A garbage sorting visual system with intelligent inorganic detection function

Abstract

The embodiment of the specification discloses a garbage sorting visual system with an inorganic intelligent detection function, which relates to the field of image processing technology, an image acquisition device for acquiring images of garbage to be sorted; a processor for acquiring images acquired by the image acquisition device, and preprocessing the images to obtain grayscale images; determining feature points of the grayscale images based on the SIFT algorithm, and calculating the amplitude of the feature points; calculating the final offset of the feature points based on the initial offset and the amplitude; offsetting the feature points based on the final offset to obtain key points; acquiring feature vectors of the key points, inputting the feature vectors into a neural network, obtaining the probability of characterizing that the garbage to be sorted is metal garbage, and generating garbage sorting instructions based on the probability; a garbage sorting device for acquiring garbage sorting instructions, and sorting metal garbage in the garbage to be sorted according to the instructions. The present invention realizes accurate garbage classification function by acquiring accurate key points.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a garbage sorting visual system with an inorganic intelligent detection function. Background Art

[0002] When using machine vision to classify conveyor belt garbage, different types of garbage have different recognition effects. Inorganic garbage (also known as inorganic garbage) generally includes metals, oil, minerals, water, etc. Common inorganic garbage includes metals, glass, plastics, etc. Visually, inorganic garbage has some more obvious characteristics compared to other garbage. For example: in the conveyor belt, metal garbage has extremely high reflectivity and clearer contour texture; glass garbage has transparent and regular characteristics, all of which can be detected and identified using machine vision technology. The conveyor belt is running in real time, and the SIFT operator is used to identify and extract features of this type of moving garbage, analyze the feature descriptors output by the operator, classify the descriptors that meet this type of feature, separate the inorganic garbage, and achieve the effect of real-time garbage sorting.

[0003] The metal waste in the inorganic waste on the conveyor belt has high reflectivity and good feature identifiability. The relevant technology can be used to extract features from its image, which can achieve the effect of metal waste identification and detection. However, in the feature extraction process, due to the reflectivity of metal waste, the precise position coordinates of the key points cannot be obtained during the key point precise positioning stage, which causes deviations in the feature point positioning, resulting in certain errors in the final feature point output results, reducing the success rate of metal waste identification during the waste sorting process, and further reducing the accuracy of metal waste sorting. Summary of the invention

[0004] The embodiment of this specification provides a garbage sorting visual system with an inorganic intelligent detection function, including:

[0005] An image acquisition device, used to acquire images of the garbage to be sorted including inorganic garbage, wherein the inorganic garbage includes metal garbage;

[0006] A processor is connected to the image acquisition device, and the processor is used to:

[0007] Acquire the image acquired by the image acquisition device, and preprocess the image to obtain a grayscale image;

[0008] Determine the feature points of the grayscale image based on the SIFT algorithm, and calculate the amplitude of the feature points, where the amplitude is used to characterize the grayscale change of the feature points;

[0009] Calculating an initial offset of the feature point based on a SIFT algorithm, and calculating a final offset of the feature point based on the initial offset and the amplitude;

[0010] Offsetting the feature points based on the final offset to obtain key points;

[0011] Acquire a feature vector of the key point, input the feature vector into a trained neural network, obtain a probability that the garbage to be sorted is the metal garbage, and generate a garbage sorting instruction based on the probability;

[0012] The garbage sorting device is connected to the processor and is used to obtain the garbage sorting instruction and sort the metal garbage in the garbage to be sorted based on the garbage sorting instruction.

[0013] Optionally, calculating the amplitude of the feature point includes:

[0014] determining a local difference region of the grayscale image;

[0015] Obtaining Gaussian filter image scale parameters of the feature points;

[0016] The amplitude of the feature point is calculated based on the Gaussian filter image scale parameter and the grayscale value of the local difference area.

[0017] Optionally, determining the local difference area of ​​the grayscale image includes:

[0018] Calculating a neighborhood reference image of the grayscale image;

[0019] The first grayscale value of each pixel in the grayscale image is subtracted from the second grayscale value of the pixel at the same position as the pixel in the neighborhood reference image, and when the absolute value of the difference obtained by subtraction is less than a threshold, the grayscale value of the pixel is set to 0; when the absolute value of the difference obtained by subtraction is greater than or equal to the threshold, the grayscale value of the pixel is set to 255, thereby obtaining a noisy local difference area.

[0020] Optionally, the threshold is 100.

[0021] Optionally, calculating the neighborhood reference image of the grayscale image includes:

[0022] For each target pixel point in the non-edge area of ​​the grayscale image, obtain the grayscale value of each pixel point in the 8-neighborhood range around the target pixel point, and take the minimum value of the grayscale values ​​as the grayscale value of the target pixel point;

[0023] For each pixel point of the grayscale image located in the edge area, the grayscale value of each pixel point is set to 0 to obtain a neighborhood reference image of the grayscale image.

[0024] Optionally, after obtaining the noisy local difference region, determining the local difference region of the grayscale image further includes:

[0025] Calculating a skeleton of the grayscale image;

[0026] Connecting adjacent pixel points with a grayscale value of 255 in the noisy local difference area along the skeleton and fitting them into multiple curves;

[0027] Performing Taylor expansion on each of the curves, and determining the overlapping parts of the curves as connecting lines based on the expanded quadratic terms;

[0028] The broken parts in the connected domain of the local difference region are connected based on the connecting lines to obtain a denoised local difference region.

[0029] Optionally, the calculating the amplitude of the feature point based on the Gaussian filter image scale parameter and the grayscale value of the local difference area includes:

[0030] The amplitude of the feature point is calculated based on the following formula:

[0031] ;

[0032] in, represents the amplitude, is the grayscale value of the pixel in the local difference area, represents the Gaussian filter image scale parameter, and i represents the sequence number of the feature point.

[0033] Optionally, the calculating a final offset of the feature point based on the initial offset and the amplitude includes:

[0034] The final offset of the feature point is calculated based on the following formula:

[0035] ;

[0036] Wherein, Mov(i) represents the final offset, represents the amplitude, i represents the sequence number of the feature point, represents the initial offset of the feature point calculated based on the SIFT algorithm, is a sine function.

[0037] Optionally, the offsetting the feature points based on the final offset to obtain key points includes:

[0038] When the final offset of the feature point is less than 0.25, keeping the position of the feature point unchanged;

[0039] When the final offset of the feature point is greater than or equal to 0.25, the feature point is offset based on the SIFT algorithm to obtain a key point.

[0040] Optionally, obtaining the feature vector of the key point includes:

[0041] The positioning information of the key point is obtained, and the positioning information is processed using a SIFT operator to obtain a feature vector.

[0042] The garbage sorting visual system with inorganic intelligent detection function provided in the embodiments of this specification may bring at least the following beneficial effects: (1) by reconstructing the distribution weights and adding the amplitude index to the weight distribution, the key points obtained in the high-reflective area will not be affected by the key points themselves due to the constraints of the reconstruction weights, making the key point positioning more accurate, thereby improving the success rate of metal garbage recognition in inorganic garbage and the accuracy of metal garbage sorting; (2) for the obtained noisy local difference area, by connecting the broken parts in the original connected domain, the adverse effects of the break caused by noise are eliminated, making the calculation of the local difference area more accurate, and further improving the success rate of metal garbage recognition; (3) comprehensively considering the local difference area and the Gaussian filter scale, so that the calculated amplitude can more accurately reflect the grayscale changes of the image, thereby making the offset based on the amplitude calculation more accurate, which is conducive to improving the success rate of metal garbage recognition.

[0043] Additional features will be described in part in the following description. For those skilled in the art, it will become apparent by reviewing the following and accompanying drawings, or it may be understood by the production or operation of the examples. The features of this specification may be implemented and obtained by practicing or using various aspects of the methods, tools, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0045] Figure 1 is a schematic diagram of an exemplary application scenario of a garbage sorting visual system with an inorganic intelligent detection function according to some embodiments of this specification;

[0046] Figure 2 is an exemplary module diagram of an image detection system with an inorganic intelligent detection function according to some embodiments of this specification;

[0047] Figure 3is a block diagram of a garbage sorting visual system with intelligent inorganic matter detection function according to some embodiments of this specification;

[0048] Figure 4 is a flow chart of steps executed by a processor in a garbage sorting visual system with an inorganic intelligent detection function according to some embodiments of this specification;

[0049] Figure 5 It is a flow chart of the steps executed by the image acquisition device and the processor in the garbage sorting visual system with inorganic intelligent detection function shown in some embodiments of this specification. DETAILED DESCRIPTION

[0050] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0051] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0052] As shown in this specification, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0053] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0054] For the metal garbage in the inorganic garbage on the conveyor belt, it has high reflectivity and good feature identifiability. Using SIFT operator for feature extraction can achieve the effect of metal garbage identification and detection. However, in the feature extraction process, due to the reflectivity of metal garbage, the precise position coordinates of the key points cannot be obtained based on the distance information alone during the key point precision positioning stage, resulting in deviations in the feature point positioning, leading to errors in the final feature point output results. The feature vector finally obtained fails to fully cover the inorganic garbage, resulting in deviations in the sorting results. To solve this deviation, it is necessary to give corresponding weights to eliminate the influence of the key points obtained by a single distance on the efficiency and accuracy of garbage sorting.

[0055] Based on the above technical problems, this specification provides a garbage sorting visual system with intelligent inorganic matter detection function.

[0056] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 It is a schematic diagram of an exemplary application scenario of a garbage sorting visual system with an intelligent inorganic matter detection function according to some embodiments of this specification.

[0058] Reference Figure 1 In some embodiments, the application scenario 100 of the garbage sorting visual system with the intelligent detection function of inorganic matter may include an image acquisition device 110, a storage device 120, a processor 130, a garbage sorting device 140 and a network 150. The various components in the application scenario 100 can be connected in a variety of ways. For example, the image acquisition device 110 can be connected to the storage device 120 and / or the processor 130 through the network 150, or directly connected to the storage device 120 and / or the processor 130. For another example, the storage device 120 can be directly connected to the processor 130 or connected through the network 150. For another example, the garbage sorting device 140 can be connected to the storage device 120 and / or the processor 130 through the network 150, or directly connected to the storage device 120 and / or the processor 130.

[0059] The image acquisition device 110 can be used to acquire images of garbage to be sorted including inorganic garbage, and the image may include multiple types of garbage to be sorted, such as organic garbage and inorganic garbage, and the inorganic garbage includes metal garbage. In some embodiments, the image acquisition device 110 can acquire images of garbage in the garbage conveyor belt according to a set image acquisition frequency (for example, once every 30 seconds). In some embodiments, in order to achieve all-round monitoring and shooting, the image acquisition device 110 can be set on the pan-tilt assembly. In some embodiments, the image acquisition device 110 can have an independent power supply, which can send the acquired images to other components (for example, storage device 120, processor 130) in the application scenario 100 by wired or wireless (for example, Bluetooth, WiFi, etc.). In some embodiments, the application scenario 100 may include multiple (for example, two or more) image acquisition devices 110, and the multiple image acquisition devices 110 can acquire images for the same or different garbage conveyor belts.

[0060] In some embodiments, the image acquisition device 110 may send the image it acquires to the storage device 120, the processor 130, etc. through the network 150. In some embodiments, the image acquired by the image acquisition device 110 may be processed by the processor 130. For example, the processor 130 may pre-process the acquired image to obtain a grayscale image.

[0061] The network 150 may facilitate the exchange of information and / or data. The network 150 may include any suitable network that can facilitate the exchange of information and / or data of the application scenario 100. In some embodiments, at least one component of the application scenario 100 (e.g., the image acquisition device 110, the storage device 120, the processor 130, and the garbage sorting device 140) may exchange information and / or data with at least one other component in the application scenario 100 through the network 150. For example, the processor 130 may obtain images collected for garbage on a garbage conveyor belt from the image acquisition device 110 and / or the storage device 120 through the network 150. It should be noted that the images of the storage device 120 are obtained from the image acquisition device 110, so the processor 130 obtaining images from the storage device 120 may be regarded as indirectly obtaining images from the image acquisition device 110. For another example, the processor 130 may send a garbage sorting instruction to the garbage sorting device 140 through the network 150. Exemplary garbage sorting instructions may include, but are not limited to, sorting metal garbage at a specified location from the garbage to be sorted.

[0062] In some embodiments, the network 150 may be any form of wired or wireless network, or any combination thereof. As an example only, the network 150 may include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, etc., or any combination thereof. In some embodiments, the network 150 may include at least one network access point, and at least one component of the application scenario 100 may be connected to the network 150 through the access point to exchange data and / or information.

[0063] The storage device 120 may store data, instructions, and / or any other information. In some embodiments, the storage device 120 may store data obtained from the image acquisition device 110 and / or the processor 130. For example, the storage device 120 may store images acquired by the image acquisition device 110; for another example, the storage device 120 may store garbage sorting instructions generated by the processor 130. In some embodiments, the storage device 120 may store data and / or instructions used by the processor 130 to execute or use to complete the exemplary methods described in this specification. In some embodiments, the storage device 120 may include a mass storage, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary mass storage may include a magnetic disk, an optical disk, a solid-state disk, etc. In some embodiments, the storage device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

[0064] In some embodiments, the storage device 120 may be connected to the network 150 to communicate with at least one other component in the application scenario 100 (e.g., the image acquisition device 110, the processor 130, the garbage sorting device 140). At least one component in the application scenario 100 may access data, instructions, or other information stored in the storage device 120 through the network 150. In some embodiments, the storage device 120 may be directly connected or communicate with one or more components in the application scenario 100 (e.g., the image acquisition device 110, the processor 130). In some embodiments, the storage device 120 may be a part of the image acquisition device 110 and / or the processor 130.

[0065] The processor 130 may process data and / or information obtained from the image acquisition device 110, the storage device 120, and other components of the application scenario 100. In some embodiments, the processor 130 may obtain an image from any one or more of the image acquisition device 110 and the storage device 120, and pre-process the image to obtain a grayscale image. In some embodiments, the processor 130 may obtain a pre-stored computer instruction from the storage device 120, and execute the computer instruction to implement the relevant steps performed by the processor 130 described in this specification.

[0066] In some embodiments, the processor 130 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 130 may be local or remote. For example, the processor 130 may access information and / or data from the image acquisition device 110 or the storage device 120 via the network 150. For another example, the processor 130 may be directly connected to the image acquisition device 110, the storage device 120 to access information and / or data. In some embodiments, the processor 130 may be implemented on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cloud-to-cloud cloud, a multi-cloud, etc., or any combination thereof.

[0067] The garbage sorting device 140 can receive data and perform garbage sorting operations. The received data may include garbage sorting instructions generated by the processor 130, and use its own sorting components (such as a mechanical arm, an electromagnet) to sort out the identified metal garbage. In some embodiments, the processor 130 may be a part of the garbage sorting device 140.

[0068] It should be noted that the above description of the application scenario 100 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the application scenario 100 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. For example, the image acquisition device 110 may include more or fewer functional components.

[0069] Figure 2 is a schematic diagram of a module of an image detection system with an inorganic intelligent detection function according to some embodiments of this specification. In some embodiments, Figure 2 The image detection system 200 with inorganic intelligent detection function can be applied to Figure 1 The application scenario 100 shown, for example, can be configured in the form of software to the processor 130 to process the image captured by the image acquisition device 110 and identify metal waste in the image.

[0070] Reference Figure 2 In some embodiments, the image detection system 200 with the intelligent detection function of inorganic matter may include an acquisition module 210, an amplitude calculation module 220, an offset calculation module 230, an offset module 240, and an identification module 250.

[0071] The acquisition module 210 can be used to acquire the image acquired by the image acquisition device and preprocess the image to obtain a grayscale image.

[0072] The amplitude calculation module 220 may be used to determine the feature points of the grayscale image based on the SIFT algorithm, and calculate the amplitude of the feature points, where the amplitude is used to characterize the grayscale variation of the feature points.

[0073] The offset calculation module 230 may be configured to calculate an initial offset of the feature point based on a SIFT algorithm, and calculate a final offset of the feature point based on the initial offset and the amplitude.

[0074] The offset module 240 may be used to offset the feature points based on the final offset to obtain key points.

[0075] The recognition module 250 can be used to obtain the feature vector of the key point, input the feature vector into a trained neural network, obtain the probability that the garbage to be sorted is the metal garbage, and generate a garbage sorting instruction based on the probability.

[0076] For more details about the above modules, please refer to other places in this manual (for example Figure 3~Figure 5 part and its related description) will not be repeated here.

[0077] It should be understood that Figure 2The image detection system 200 and its modules shown with the inorganic intelligent detection function can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).

[0078] It should be noted that the above description of the image detection system 200 with inorganic intelligent detection function is provided for illustrative purposes only and is not intended to limit the scope of this specification. It is understandable that for those skilled in the art, according to the description of this specification, without departing from this principle, the various modules can be arbitrarily combined, or a subsystem can be formed to connect with other modules. For example, Figure 2 The acquisition module 210, amplitude calculation module 220, offset calculation module 230, offset module 240, and identification module 250 described in the embodiment may be different modules in a system, or one module may realize the functions of two or more of the above modules. In some embodiments, the above modules may be part of the processor 130.

[0079] Figure 3 is a block diagram of a garbage sorting visual system with an inorganic intelligent detection function according to some embodiments of this specification, such as Figure 3 As shown, the system 300 includes:

[0080] An image acquisition device 110 is used to acquire images of the garbage to be sorted including inorganic garbage, wherein the inorganic garbage includes metal garbage;

[0081] The processor 130 is connected to the image acquisition device 110. Figure 4 , the processor 130 is used for:

[0082] S410, acquiring the image acquired by the image acquisition device, and preprocessing the image to obtain a grayscale image;

[0083] S420, determining feature points of the grayscale image based on a SIFT algorithm, and calculating amplitudes of the feature points, where the amplitudes are used to characterize grayscale changes of the feature points;

[0084] S430, calculating an initial offset of the feature point based on a SIFT algorithm, and calculating a final offset of the feature point based on the initial offset and the amplitude;

[0085] S440, offsetting the feature points based on the final offset to obtain key points;

[0086] S450, obtaining a feature vector of the key point, inputting the feature vector into a trained neural network, obtaining a probability that the garbage to be sorted is the metal garbage, and generating a garbage sorting instruction based on the probability;

[0087] The garbage sorting device 140 is connected to the processor 130, and is used to obtain the garbage sorting instruction and sort the metal garbage in the garbage to be sorted based on the garbage sorting instruction.

[0088] Specifically, the image acquisition device 110 may be a high-definition industrial camera, and the images it acquires may be in color. For details, please refer to the description of the image acquisition device 110 in the above application scenario 100.

[0089] The processor 130 is used to execute steps S410 to S450 to obtain the image captured by the image acquisition device 110, complete the identification of metal waste in the image, and finally generate a waste sorting instruction to send to the waste sorting device 140. For other aspects of the processor 130, refer to the description of the processor 130 in the application scenario 100.

[0090] Specifically, in step S410, the image acquisition device 110 may send the acquired image to the processor 130 via a wired or wireless method to obtain the image acquired by the image acquisition device 110, and perform grayscale preprocessing on the image to obtain a grayscale image, so that the image data becomes two-dimensional, which is conducive to subsequent analysis. The process of grayscale preprocessing of the image can adopt the existing technology, and the present invention does not make any specific limitation on this.

[0091] In step S420, in the process of processing the grayscale image based on the SIFT (Scale-invariant feature transform) algorithm or the SIFT operator, different interpolation centers can be fitted by Taylor expansion to obtain feature points, and then key points can be obtained by fitting and interpolating the feature points. In the process of obtaining the key points, due to the influence of high reflectivity in the metal garbage area, the key points obtained by the feature points around it are not accurately positioned. In this scenario, in order to achieve the purpose of eliminating the irreversible influence of high brightness areas on key point acquisition by changing the key point acquisition method, the amplitude of the feature point is specified here, and by adding the reference index of the amplitude to the acquisition of the key point, the small amplitude feature points that originally need to be changed in positioning are kept in their original positioning because of the existence of the reference index of the amplitude. Here, the small amplitude feature points refer to the feature points that have to change their positioning after being affected by the high reflectivity area of ​​the metal garbage in this scenario. The amplitude is obtained through the difference of local areas in the original image and the different scales. The former exists to quantify the change law of the grayscale value of the local area in the original image. The more obvious the change, the closer it is to the high reflective area of ​​metal garbage under the magnetic field. The logic of this step is obtained through prior knowledge. The greater the difference in local areas, the greater the amplitude is specified. The latter exists to distinguish feature points at different scales, because Gaussian filters of different scales retain different features of the original image. The larger the scale, the more it is smoothed and the less features are retained. Therefore, the amplitude is specified here to be inversely proportional to the scale. The amplitude is defined by the above two quantities and linked to the original image.

[0092] Optionally, in step S420, calculating the amplitude of the feature point includes:

[0093] S421, determining a local difference area of ​​the grayscale image;

[0094] S422, obtaining Gaussian filter image scale parameters of the feature points;

[0095] S423: Calculate the amplitude of the feature point based on the Gaussian filter image scale parameter and the grayscale value of the local difference area.

[0096] In step S421, the purpose of determining the local difference area is to find the local area with a large change in grayscale value in the spatial domain. The purpose is to prepare data for the next step of obtaining the amplitude. Because according to the above amplitude definition, the larger the local difference, the larger the corresponding feature point amplitude, that is, proportional.

[0097] Optionally, in step S421, determining the local difference area of ​​the grayscale image includes:

[0098] S4211, calculating a neighborhood reference image of the grayscale image;

[0099] S4212, subtract the second grayscale value of the pixel at the same position as the pixel in the neighborhood reference image from the first grayscale value of each pixel in the grayscale image, and when the absolute value of the difference obtained by subtraction is less than a threshold, set the grayscale value of the pixel to 0, and when the absolute value of the difference obtained by subtraction is greater than or equal to the threshold, set the grayscale value of the pixel to 255, to obtain a noisy local difference area.

[0100] Optionally, in step S4211, calculating a neighborhood reference image of the grayscale image includes:

[0101] For each target pixel point in the non-edge area of ​​the grayscale image, obtain the grayscale value of each pixel point in the 8-neighborhood range around the target pixel point, and take the minimum value of the grayscale values ​​as the grayscale value of the target pixel point;

[0102] For each pixel point of the grayscale image located in the edge area, the grayscale value of each pixel point is set to 0 to obtain a neighborhood reference image of the grayscale image.

[0103] Specifically, for each target pixel in the non-edge area of ​​the grayscale image, that is, a pixel that is not located at the edge of the image, the grayscale value in the 8-neighborhood range around the target pixel is detected, and the output is taken as the minimum grayscale value in the 8-neighborhood range. By analogy, the overall grayscale value average of the original image will become lower, and the edge of the originally highly reflective area will become darker. Zero padding is performed on the edge of the image, and the grayscale value of each pixel in the edge area is set to 0. The resulting image is called a neighborhood reference image.

[0104] After step S4211 is completed, step S4212 is executed, the neighborhood reference image is subtracted from the original image, a gray value threshold is specified, and the portion exceeding the threshold after subtraction is changed to 255. This portion is the edge of the local difference area. Because the area obtained by this logic is a high-change area, the edge of the highlight area is found through the above logic. In some embodiments, the threshold is 100.

[0105] In some embodiments, the local difference area of ​​the image may be determined based on the following formula:

[0106] ;

[0107] ;

[0108] in, Indicates the image segmented from the local difference area. The area with a gray value of 255 is the edge of the local difference area, which may be composed of multiple adjacent pixels or a few scattered pixels. and Respectively represent the original grayscale image and the neighborhood reference image, and subtraction refers to the subtraction of grayscale values. Among them, Indicates the threshold value. In this scenario, .

[0109] The local difference area obtained by the above logic exists in multiple local areas. Only the part of the connected domain of the change area that forms a closed shape is taken as the local difference area to be found, because small noise can also form similar high-change areas mentioned above, which may lead to misjudgment in the process of finding local difference areas. For the high-change area formed in this scenario, a certain degree of breakage may occur due to noise influence, etc. By analyzing the directionality of the connected domain, the relevant curvature is obtained, and the broken area that conforms to the curvature trend is fitted to eliminate the breakage caused by noise influence.

[0110] Optionally, after obtaining the noisy local difference region in step 4212, step S421 determines the local difference region of the grayscale image, and further includes:

[0111] S4213, calculating the skeleton of the grayscale image;

[0112] S4214, connecting adjacent pixel points with a grayscale value of 255 in the noisy local difference area along the skeleton, and fitting them into multiple curves;

[0113] S4215, performing Taylor expansion on each of the curves, and determining the overlapping parts of the curves as connecting lines based on the expanded quadratic terms;

[0114] S4216: Connect the broken parts in the connected domain of the local difference region based on the connecting line to obtain a denoised local difference region.

[0115] Specifically, in step S4213 to step S4216, the skeleton of the grayscale image can be extracted based on the prior art, and the adjacent points with a grayscale value of 255 in the noisy local difference area of ​​the above-mentioned image are connected along the skeleton of the connected domain, and fitted into multiple curves. , each curve Calculate its Taylor expansion and take its quadratic term. Take the overlapping parts after the two Taylor expansions as connecting lines to connect the broken parts in the original connected domain, eliminate the adverse effects of the break caused by noise, and obtain the denoised local difference area.

[0116] After step S421 is executed, step S422 is executed. In step S422, the Gaussian filter image scale parameter of the feature point is obtained. The Gaussian filter image scale parameter is also called the scale space factor or the fuzzy coefficient. It is a parameter in the Gaussian filter and determines the filter scale of the filter, that is, the degree of smoothness.

[0117] After step S422 is completed, step S423 is executed. In step S423, the amplitude of the feature point is calculated based on the Gaussian filter image scale parameter and the gray value of the local difference area, which may include:

[0118] The amplitude of the feature point is calculated based on the following formula:

[0119] ;

[0120] in, represents the amplitude, is the grayscale value of the pixel in the local difference area, represents the Gaussian filter image scale parameter, and i represents the sequence number of the feature point.

[0121] exist In the region, the spatial grayscale difference is considered to be non-existent, so the amplitude proportional to the grayscale difference becomes 0; In the region of , the spatial grayscale difference is considered to exist, so the amplitude proportional to the grayscale difference becomes the scale inversely proportional to the amplitude. Using power to represent the change delays its inverse effect, making the amplitude have good robustness when the scale difference changes greatly.

[0122] The concept of amplitude here is composed of the local difference and Gaussian scale in the spatial domain, and its relationship with the amplitude is as follows: the local difference is proportional to the amplitude, the larger the local difference, the larger the value corresponding to the feature point amplitude, and the proportional reason is that the larger the local difference in the spatial domain, the larger the grayscale change, and the larger the grayscale value change, the stronger the response of the feature point obtained after processing by the SIFT operator DoG algorithm, because the DoG algorithm is the difference at different scales, the greater the grayscale change, the larger the value obtained after the difference; the scale is inversely proportional to the amplitude, the larger the scale, the smaller the value corresponding to the feature point amplitude, and the inverse reason is that the effect of Gaussian filtering on smoothing the original image is proportional to the scale, and the larger the Gaussian filtering scale, the better the filtering effect, that is, the more serious the loss of details in the original image and the more obvious the contour, then the feature point corresponding to the Gaussian difference is smoothed more violently, and the corresponding value corresponding to the established amplitude concept is smaller.

[0123] In step S430, the initial offset is first calculated according to the SIFT algorithm. Specifically, the scale space DoG (Difference of Gaussian) function needs to be curve fitted according to the SIFT algorithm. In order to achieve the fitting, the Taylor expansion of the DoG function in the scale space is derived and the derivative equation is set equal to 0, so that the initial offset of the feature point can be obtained. Then, the initial offset is corrected by using the amplitude to obtain the final offset of the feature point.

[0124] Specifically, after determining the amplitude, this reference factor is introduced into the logic of obtaining the offset, that is, the offset is no longer obtained only by the distance in any dimension after fitting, but is determined by the distance and amplitude. Then the effect that the offset ultimately needs to achieve is: feature points with large amplitudes correspond to the high reflectivity characteristics of metal garbage in the scene, and their positioning is relatively fixed. When converted into more precise key points, because their distribution can be understood as attracting feature points with small amplitudes around them, then at this time, the offset of feature points with small amplitudes should be more sensitive to distance, reducing their influence by large amplitudes, so that they are not easily attracted by feature points with large amplitudes. In summary, the offset is inversely proportional to the amplitude and directly proportional to the distance. In this specification, the distance, that is, the initial offset, can be understood as: the offset based on pixel distance in the differential image obtained by derivation after Taylor expansion fitting in the original SIFT operator.

[0125] Optionally, in step S430, calculating the final offset of the feature point based on the initial offset and the amplitude includes:

[0126] The final offset of the feature point is calculated based on the following formula:

[0127] ;

[0128] Wherein, Mov(i) represents the final offset, represents the amplitude, i represents the sequence number of the feature point, represents the initial offset of the feature point calculated based on the SIFT algorithm, is a sine function. ,Will Mapping , then at this time within this range, the sine function is used to represent the inverse relationship, and the logic is: the closer the amplitude is to 1, the more severe the inverse ratio is, that is, the smaller the offset is, because the closer the amplitude is to 1, the smaller the scale of the Gaussian filter is. The purpose is to reflect this relationship, rather than simply inverse proportion.

[0129] The final offset is obtained by fitting the original feature points through Taylor expansion to obtain the key points. The purpose of fitting amplitude and distance together is to make a more reasonable division of the positioning of small-amplitude feature points affected by large amplitude (corresponding to the highly reflective area of ​​metal garbage in the image), and eliminate the misjudgment caused by the process of small-amplitude feature points becoming key points, which results in the output feature vector being unable to fully represent the metal garbage area.

[0130] In step S440, the feature points are offset based on the final offset to obtain key points, including:

[0131] When the final offset of the feature point is less than 0.25, keeping the position of the feature point unchanged;

[0132] When the final offset of the feature point is greater than or equal to 0.25, the feature point is offset based on the SIFT algorithm to obtain a key point.

[0133] Specifically, whether to offset the feature points is determined based on the following formula:

[0134] ;

[0135] Mov(i) represents the final offset. The above final offset is obtained, and the feature point is offset according to the relationship between the final offset and the threshold value 0.25. The original feature point is offset according to the logic of the formula to obtain the key point. In the original SIFT operator, only the distance is used to determine whether the offset is or not, and 0.5 is taken. The present invention combines the amplitude, and the amplitude range is , it is stipulated that when the amplitude exceeds 0.5, an offset is performed, and the final offset is half of the amplitude, so the final threshold for determining whether an offset occurs is 0.25.

[0136] By analyzing the preprocessed image, the local difference area is obtained. The determination of this area is helpful for the subsequent acquisition of the amplitude at different scales. By obtaining the amplitude of the difference area at different scales, the offset based on the amplitude is obtained. This step is helpful to eliminate the problem of inaccurate feature point positioning caused by the influence of high reflective areas. By obtaining the offset, the key point with accurate final positioning is obtained.

[0137] In step S450, obtaining the feature vector of the key point may include:

[0138] The positioning information of the key point is obtained, and the positioning information is processed using a SIFT operator to obtain a feature vector.

[0139] Specifically, the positioning information of the key points obtained in the above steps is obtained, and the SIFT operator is further processed to obtain the final feature vector. After the pre-processing of the optimization algorithm, this feature vector is not affected by the high reflective characteristics of inorganic metals. This feature vector is input into the neural network, which is a pre-trained neural network with the function of identifying metal garbage feature vectors, and the neural network input is: the garbage area feature vector obtained after the optimized SIFT operator feature extraction, which contains inorganic garbage and other garbage; the output is: the feature vector is the probability size of metal garbage in inorganic garbage, wherein the loss function during neural network training adopts the cross entropy function. The garbage classification result is determined according to the probability of the network output, which has a better effect on the sorting of inorganic garbage.

[0140] After obtaining the output probability, when the probability is greater than a preset probability threshold, a garbage sorting instruction can be generated, which may include the location information of the sorted garbage, and the garbage sorting instruction can be sent to the garbage sorting device 140, so that the processor 130 completes the identification of metal garbage in the grayscale image. In addition, the related steps performed by the image acquisition device 110 and the processor 130 can also be referred to Figure 5 The process will not be described here.

[0141] In some embodiments, steps S410 to S450 performed by the processor 130 may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on the processor 130 to perform hardware simulation), etc., or any combination thereof. Steps S410 to S450 may be stored in the storage device 120 in the form of instructions and called and / or executed by the processor 130.

[0142] The garbage sorting device 140 can obtain the garbage sorting instruction generated by the processor 130 in a wired or wireless manner, execute the garbage sorting instruction, and sort the metal garbage in the garbage to be sorted, so as to separate the metal garbage from the garbage to be sorted. For other situations of the garbage sorting device 140, please refer to the description of the garbage sorting device 140 in the above application scenario 100.

[0143] In summary, the beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) In view of the problem that in the key point positioning process of the original SIFT operator, the feature points in the high-reflective area are relatively concentrated, which affects the accurate allocation of the feature points around it, the allocation weights are reconstructed and the amplitude index is added to the weight allocation, so that the key points obtained in the high-reflective area will not be affected by itself due to the constraints of the reconstruction weights, making the key point positioning more accurate, thereby improving the success rate of metal waste recognition in inorganic waste and the accuracy of metal waste sorting; (2) For the obtained noisy local difference area, by connecting the broken parts in the original connected domain, the adverse effects of the break caused by noise are eliminated, making the calculation of the local difference area more accurate, and further improving the success rate of metal waste recognition; (3) Comprehensively considering the local difference area and the Gaussian filter scale, the calculated amplitude can more accurately reflect the grayscale changes of the image, thereby making the offset based on the amplitude calculation more accurate, which is conducive to improving the success rate of metal waste recognition.

[0144] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0145] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0146] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0147] In addition, it will be understood by those skilled in the art that various aspects of this specification may be illustrated and described by several types or situations of the prior art, including any new and useful process, machine, product or combination of substances, or any new and useful improvement thereof. Accordingly, various aspects of this specification may be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of this specification may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0148] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0149] The computer program codes required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and partly on a remote computer, or run entirely on a remote computer or processor. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

Claims

1. A garbage sorting visual system with intelligent inorganic detection function, characterized in that: include: An image acquisition device, used to acquire images of the garbage to be sorted including inorganic garbage, wherein the inorganic garbage includes metal garbage; A processor is connected to the image acquisition device, and the processor is used to: Acquire the image acquired by the image acquisition device, and preprocess the image to obtain a grayscale image; Determine the feature points of the grayscale image based on the SIFT algorithm, and calculate the amplitude of the feature points, where the amplitude is used to characterize the grayscale change of the feature points; Calculating an initial offset of the feature point based on a SIFT algorithm, and calculating a final offset of the feature point based on the initial offset and the amplitude; Offsetting the feature points based on the final offset to obtain key points; Acquire a feature vector of the key point, input the feature vector into a trained neural network, obtain a probability that the garbage to be sorted is the metal garbage, and generate a garbage sorting instruction based on the probability; A garbage sorting device, connected to the processor, for obtaining the garbage sorting instruction, and sorting the metal garbage in the garbage to be sorted based on the garbage sorting instruction; The calculating the amplitude of the feature point comprises: determining a local difference region of the grayscale image; Obtaining Gaussian filter image scale parameters of the feature points; Calculate the amplitude of the feature point based on the Gaussian filter image scale parameter and the grayscale value of the local difference area; The determining of the local difference area of ​​the grayscale image comprises: Calculating a neighborhood reference image of the grayscale image; The second grayscale value of the pixel at the same position as the pixel in the neighborhood reference image is subtracted from the first grayscale value of each pixel in the grayscale image, and when the absolute value of the difference obtained by subtraction is less than a threshold, the grayscale value of the pixel is set to 0, and when the absolute value of the difference obtained by subtraction is greater than or equal to the threshold, the grayscale value of the pixel is set to 255, to obtain a noisy local difference area; The calculating of the neighborhood reference graph of the grayscale image comprises: For each target pixel point in the non-edge area of ​​the grayscale image, obtain the grayscale value of each pixel point in the 8-neighborhood range around the target pixel point, and take the minimum value of the grayscale values ​​as the grayscale value of the target pixel point; For each pixel point of the grayscale image located in the edge area, the grayscale value of each pixel point is set to 0, so as to obtain a neighborhood reference image of the grayscale image; The calculating the amplitude of the feature point based on the Gaussian filter image scale parameter and the grayscale value of the local difference area includes: The amplitude of the feature point is calculated based on the following formula: in, represents the amplitude, is the grayscale value of the pixel in the local difference area, represents the Gaussian filter image scale parameter, i represents the sequence number of the feature point; The calculating the final offset of the feature point based on the initial offset and the amplitude includes: The final offset of the feature point is calculated based on the following formula: Wherein, Mov(i) represents the final offset, represents the amplitude, i represents the sequence number of the feature point, represents the initial offset of the feature point calculated based on the SIFT algorithm, is a sine function; The step of offsetting the feature points based on the final offset to obtain key points includes: When the final offset of the feature point is less than 0.25, keeping the position of the feature point unchanged; When the final offset of the feature point is greater than or equal to 0.25, the feature point is offset based on the SIFT algorithm to obtain a key point; The step of obtaining the feature vector of the key point includes: The positioning information of the key point is obtained, and the positioning information is processed using a SIFT operator to obtain a feature vector.

2. A garbage sorting visual system with inorganic intelligent detection function as claimed in claim 1, characterized in that: The threshold is 100.

3. A garbage sorting visual system with inorganic intelligent detection function as claimed in claim 1, characterized in that: After obtaining the noisy local difference region, determining the local difference region of the grayscale image further includes: Calculating a skeleton of the grayscale image; Connecting adjacent pixel points with a grayscale value of 255 in the noisy local difference area along the skeleton and fitting them into multiple curves; Performing Taylor expansion on each of the curves, and determining the overlapping parts of the curves as connecting lines based on the expanded quadratic terms; The broken parts in the connected domain of the local difference region are connected based on the connecting lines to obtain a denoised local difference region.

Images