Foreign matter picture generation method and device applied to color sorter and storage medium

Abstract

The invention relates to a foreign matter picture generation method and device applied to a color selector and a storage medium, and the method comprises the steps: carrying out the image segmentation extraction of a plurality of pictures containing foreign matters and / or products, and obtaining an original foreign matter material picture and an original product material picture; smoothing and gradually changing the original foreign matter material picture and the original product material picture to obtain a first foreign matter material picture and a first product material picture; importing a background picture, selecting one or more first product material pictures, and adding the selected one or more first product material pictures to random positions of the background picture to obtain a first picture; and randomly selecting one or more first foreign matter material pictures, adding the first foreign matter material pictures to the first picture in a manner that the first foreign matter material pictures are not overlapped with the product, cover the product and are partially covered by the product to obtain a training set picture containing foreign matters, and generating the position of each foreign matter material in the training set picture. Compared with the prior art, the method has the advantages that a large amount of high-quality labeled training set data for the color sorter can be generated, and the like.

Description

Technical Field

[0001] The present invention relates to the field of color sorters, and in particular to a method, a device and a storage medium for generating a foreign matter image applied to a color sorter. Background Art

[0002] Color sorters are important equipment in the food industry. In recent years, the requirements for color sorters have gradually increased. Color sorters are also required to distinguish foreign matter. Therefore, color sorters are required to be able to identify foreign matter in products.

[0003] Therefore, some technical personnel in this field gradually introduced machine learning technology into color sorters to identify foreign objects. However, due to the insufficiency of training sets, the actual effect of this function is limited. Although some existing technologies have disclosed various technical solutions, due to the lack of high-quality and sufficient training sets, although there are some related studies, they have not yet been applied in actual products.

[0004] In this regard, technicians in this field are committed to building sufficient training sets to improve the foreign matter detection capability of color sorters.

[0005] Traditional dataset construction methods usually rely on manual collection and labeling, which requires a lot of manpower and time. Especially in certain specific foreign body sorting scenarios, diverse and specifically labeled images are required, which further increases the complexity and cost of dataset production.

[0006] Although there are some image synthesis counts in some fields that are not very close to food color selection, such as advertising and public relations, by generating images, the dependence on real-world image data can be greatly reduced, and large-scale and diverse image sets can be quickly generated. Most of these technologies are mainly used in the creation of news illustrations and illustrations for self-media articles, but they perform poorly in generating high-quality and realistic images and cannot fully meet the requirements of training deep learning models. Summary of the invention

[0007] The purpose of the present invention is to provide a method, device and storage medium for generating foreign matter images for color sorting machines, which can automatically label images through a small amount of data on the object to be tested, a foreign matter database, and a large amount of synthetic data, thereby reducing the time cost of manual labeling.

[0008] The purpose of the present invention can be achieved by the following technical solutions: A method for generating a foreign body image applied to a color sorter, comprising: Performing image segmentation and extraction on multiple pictures containing foreign matter and / or products to obtain original foreign matter material pictures and original product material pictures, respectively, wherein the original foreign matter material pictures contain foreign matter information, and the original product material pictures contain product information; Performing smoothing and gradient processing on the original foreign object material picture and the original product material picture to obtain a first foreign object material picture and a first product material picture, and using the foreign object information of the original foreign object material picture as the foreign object information of the first foreign object material picture, and using the product information of the original product material picture as the product information of the first product material picture; Import the background image, and randomly select one or more first product material images, and add them to the random position of the background image to obtain the first image; One or more first foreign object material images are randomly selected and added to the first image in a manner of not overlapping with the product, covering the product, or being partially covered by the product to obtain training set images containing foreign objects, and the position of each foreign object material in the training set image is used as the foreign object position, and annotation information of the training set images is generated based on the product information of the selected first product material image, the foreign object information and the foreign object position of the first foreign object material image.

[0009] The process of smoothing and gradient processing the original foreign material image includes: A first mask image is created based on the imported original foreign object material image, wherein the pixel value of the foreground portion is set to 255, and the pixel value of the background portion is set to 0; The first mask image created is eroded using a first convolution kernel, wherein the value of each element in the first convolution kernel is equal to 1 / A, where A is the number of elements in the convolution kernel; Performing Gaussian blur processing on the first mask image after the corrosion processing; The first mask image after Gaussian blur processing is assigned as a transparent channel to the original foreign object material image to obtain the first foreign object material image in RBGA format, and the foreign object information of the original foreign object material image is used as the foreign object information of the first foreign object material image.

[0010] The first convolution kernel is a 3×3 convolution kernel.

[0011] The process of smoothing and gradient processing the original product material image is the same as that of the original foreign material image.

[0012] The process of adding the foreign matter to the first image in a non-overlapping manner to obtain a training set image containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign object image and the product image overlap. If so, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position.

[0013] The process of adding the foreign matter to the first image in a manner of covering the product to obtain a training set image containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign body image and the product image overlap. If not, return to the random position selection step. Otherwise, add the first foreign body material image to the intended adding position and place it above the layer where the product image is located.

[0014] The process of adding the foreign matter to the first image in a manner of being partially covered by the product to obtain a training set of images containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign object image and the product image overlap. If not, return to the random position selection step. Otherwise, determine whether the foreign object image is completely covered by the product image. If yes, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position and place it below the layer where the product image is located.

[0015] The image segmentation and extraction of multiple pictures containing foreign matter and / or products is achieved through a first model, and the first model is a SOLO model.

[0016] A foreign body image generating device applied to a color sorter comprises a memory, a processor, and a program stored in the memory. When the processor executes the program, the above method is implemented.

[0017] A storage medium stores a program, which implements the above method when executed.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. By collecting a small amount of data on the object to be tested and a foreign body database, a large amount of synthetic data can be automatically labeled during the synthesis process, thereby reducing the time cost of manual labeling.

[0019] 2. By generating images in three ways: without overlapping with the product, covering the product, and partially covered by the product, combined with operations such as smoothing and gradient processing, the quality and fineness of the synthesized images can be improved to meet the training requirements of the machine learning model.

[0020] 3. Synthetic images can expand the diversity of the data set, especially when the original data is limited. By synthesizing multiple samples, the model can be better generalized and the accuracy of recognition or classification can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flow chart of the main steps of the method of the present invention; Figure 2 is a schematic diagram of obtaining a first foreign matter material image from a first mask image; Figure 3 Schematic diagram of the image generated in a way that foreign matter and products do not overlap; Figure 4 A schematic diagram of a picture generated for the method of generating a foreign body covering a product; Figure 5 Schematic diagram of the image generated for how the foreign object is partially covered by the product. DETAILED DESCRIPTION

[0022] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0023] A method for generating foreign body images for color sorting machines can automatically label a large amount of synthetic data through a small amount of test object data and a foreign body database, thereby reducing the time cost of manual labeling. Figure 1 As shown, including: (1) performing image segmentation and extraction on multiple images containing foreign matter and / or products to obtain original foreign matter material images and original product material images, respectively. Specifically, in this embodiment, the product is peanuts; The image segmentation and extraction of multiple pictures containing foreign matter and / or products is realized through the first model, which is the SOLO model. A large number of pictures are collected and all objects in the pictures are segmented and labeled. These labeled data contain the boundary information of all objects in the pictures. This step provides a solid data foundation for the subsequent deep learning model training, ensuring that the model has good recognition performance in various environments. The labeled pictures are then used to train the SOLO (Segmenting Objects by Locations) model, so that the model has the ability to segment all objects in the picture.

[0024] The first model adopts the backbone+FPN (feature pyramid network) structure. The backbone is used to extract features, and the FPN is used to fuse features of different scales, so as to identify instances of different sizes in branches at different levels. Two branches are constructed at each layer of FPN: Category Branch and Mask Branch. Category Branch is used to predict the category of the instance, which is distinguished according to the center position of the instance. The output compressed image is S*S*C, where S*S is the size of the compressed image and C is the number of channels, which is the number of instance categories. Mask Branch is used to predict the mask of the instance and predict the corresponding mask for each positive sample area. By setting a certain area around the center point of the instance, such as the scale factor ε=0.2 times, the positive sample is selected and the output H*W*S is output. 2 The feature map of , H and W decibels represent the height and width of the input image, where each channel is responsible for the mask prediction of the corresponding grid. The channel number of the grid is related to its coordinates (i, j), determined by k=i*S+j, where S represents that the input image is divided into S*S grids, C represents the number of instance categories, H and W represent the height and width of the input image, and k is used to represent the mapping relationship between the grid and the channel number.

[0025] By inputting the material image with only foreign objects into the trained SOLO model, foreign objects such as plastic, stone, metal fragments and other impurities in the image can be segmented out and saved in the material library, which is used for subsequent image synthesis.

[0026] When there is a need to detect foreign matter in a specific area, for example, when we need to detect foreign matter in peanuts, this invention can be used to synthesize pictures.

[0027] Among them, the obtained original foreign body material picture contains foreign body information, and the obtained original product material picture contains product information.

[0028] (2) performing smoothing and gradient processing on the original foreign object material image and the original product material image to obtain a first foreign object material image and a first product material image, and using the foreign object information of the original foreign object material image as the foreign object information of the first foreign object material image, and using the product information of the original product material image as the product information of the first product material image; In some embodiments, the process of ensuring that the edge of the material can be integrated with the background part and performing smoothing and gradient processing on the original foreign material image includes: A first mask image is created based on the imported original foreign object material image, wherein the pixel value of the foreground portion is set to 255, and the pixel value of the background portion is set to 0; The first mask image is eroded by using a first convolution kernel to smooth edges and reduce noise, wherein the value of each element in the first convolution kernel is equal to 1 / A, and A is the number of elements in the convolution kernel. In this embodiment, the first convolution kernel is a 3×3 convolution kernel; Performing Gaussian blur processing on the first mask image after the corrosion processing; The first mask image after Gaussian blur processing is assigned as a transparent channel to the original foreign material image to obtain the first foreign material image in RBGA format, and the foreign material information of the original foreign material image is used as the foreign material information of the first foreign material image, so that there is no sense of edge separation during fusion. Figure 2 As shown, the three-channel image is fused with the first mask image to obtain a four-channel image.

[0029] In this way, by generating three ways of no overlap with the product, covering the product, and being partially covered by the product, and combining the smoothing and gradient processing operations to assign the first mask image after Gaussian blur processing as a transparent channel to the original foreign body material image, the quality and fineness of the synthesized image can be improved to meet the training requirements of the machine learning model.

[0030] Furthermore, in most embodiments, the process of smoothing and gradient processing the original product material image is the same as that of the original foreign object material image.

[0031] In addition, in this embodiment, in order to ensure as many forms as possible, data enhancement is performed on the material image, including random rotation, zooming in and out, left-right inversion, upside-down inversion, and HSV color conversion.

[0032] (3) importing a background image, and randomly selecting one or more first product material images, and adding them to random positions of the background image to obtain a first image; Since the attributes of the first product material image already include the product information when it is constructed, the product information can be obtained when it is imported.

[0033] (iv) randomly selecting one or more first foreign material images, and adding them to the first image in a manner such that they do not overlap with the product, cover the product, or are partially covered by the product, to obtain a training set of images containing foreign objects, and generating a position of each foreign material in the training set of images.

[0034] Similarly, since the first foreign body material image already contains foreign body information in its own attributes when it is constructed, the foreign body information can be obtained when it is selected for import.

[0035] Specifically, in this embodiment, when peanuts and foreign objects do not overlap, the process of adding peanuts to the first image in a non-overlapping manner with the product to obtain a training set image containing foreign objects includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum bounding rectangle of the foreign object image of the selected first foreign object material picture, and recording the foreign object position, specifically taking the position of the minimum bounding rectangle as the position of the foreign object; Determine whether the foreign object image and the product image overlap. If so, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position, and generate annotation information corresponding to the generated image based on the product information, foreign object information and the position of the foreign object, which is equivalent to annotating the composite image, making it convenient to use the composite image for deep learning training later.

[0036] like Figure 3 As shown in the figure, we finally get sample images with no overlap between foreign objects and products and their annotation results. Figure 3 The circle in the figure represents the product, and the triangle represents foreign matter.

[0037] In addition, in this embodiment, the process of adding the training set images containing foreign objects to the first image in a manner of covering the product to obtain the training set images includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign body image and the product image overlap. If not, return to the random position selection step. Otherwise, add the first foreign body material image to the intended adding position and place it above the layer where the product image is located.

[0038] Similarly, generating annotation information corresponding to the generated image based on product information, foreign object information and the location of the foreign object is equivalent to annotating the synthetic image, making it convenient to use the synthetic image for deep learning training later.

[0039] like Figure 4 As shown in the figure, we finally get the sample image of the product covered by foreign matter and its annotation results. Figure 4 The circle in the figure represents the product, and the triangle represents foreign matter.

[0040] And, in this embodiment, the process of adding the foreign matter to the first picture in a manner of being partially covered by the product to obtain a training set of pictures containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign object image and the product image overlap. If not, return to the random position selection step. Otherwise, determine whether the foreign object image is completely covered by the product image. If yes, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position and place it below the layer where the product image is located.

[0041] Similarly, generating annotation information corresponding to the generated image based on product information, foreign object information and the location of the foreign object is equivalent to annotating the synthetic image, making it convenient to use the synthetic image for deep learning training later.

[0042] like Figure 5 As shown in the figure, we finally get the sample image of the foreign body partially covered by the product and its annotation results. Figure 5 The circle in the figure represents the product, and the triangle represents foreign matter.

[0043] Since there is no stacking in most real scenes, during synthesis, the probability of the first situation occurring is set to 80%, and the probability of the other two situations occurring is set to 10%.

[0044] This application can reduce the time and labor cost of manual labeling and improve efficiency by collecting a small amount of data on the object to be tested, as well as a foreign object database and a large amount of synthetic data. Synthetic images can expand the diversity of the data set, especially when the original data is limited. By synthesizing multiple samples, the model can be better generalized and the accuracy of recognition or classification can be improved.

[0045] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

Claims

1. A method for generating a foreign body image for a color sorter, characterized in that: include: Performing image segmentation and extraction on multiple pictures containing foreign matter and / or products to obtain original foreign matter material pictures and original product material pictures, respectively, wherein the original foreign matter material pictures contain foreign matter information, and the original product material pictures contain product information; Performing smoothing and gradient processing on the original foreign object material picture and the original product material picture to obtain a first foreign object material picture and a first product material picture, and using the foreign object information of the original foreign object material picture as the foreign object information of the first foreign object material picture, and using the product information of the original product material picture as the product information of the first product material picture; Import the background image, and randomly select one or more first product material images, and add them to the random position of the background image to obtain the first image; One or more first foreign object material images are randomly selected and added to the first image in a manner of not overlapping with the product, covering the product, or being partially covered by the product to obtain training set images containing foreign objects, and the position of each foreign object material in the training set image is used as the foreign object position, and annotation information of the training set images is generated based on the product information of the selected first product material image, the foreign object information and the foreign object position of the first foreign object material image.

2. The method for generating a foreign matter image for a color sorter according to claim 1, characterized in that: The process of smoothing and gradient processing the original foreign material image includes: A first mask image is created based on the imported original foreign object material image, wherein the pixel value of the foreground portion is set to 255, and the pixel value of the background portion is set to 0; The first mask image created is eroded using a first convolution kernel, wherein the value of each element in the first convolution kernel is equal to 1 / A, where A is the number of elements in the convolution kernel; Performing Gaussian blur processing on the first mask image after the corrosion processing; The first mask image after Gaussian blur processing is assigned as a transparent channel to the original foreign object material image to obtain the first foreign object material image in RBGA format, and the foreign object information of the original foreign object material image is used as the foreign object information of the first foreign object material image.

3. The method for generating a foreign matter image for a color sorter according to claim 2, characterized in that: The first convolution kernel is a 3×3 convolution kernel.

4. A method for generating a foreign matter image for a color sorter according to claim 2 or 3, characterized in that: The process of smoothing and gradient processing the original product material image is the same as that of the original foreign material image.

5. The method for generating a foreign matter image for a color sorter according to claim 1, characterized in that: The process of adding the foreign matter to the first image in a non-overlapping manner to obtain a training set image containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign object image and the product image overlap. If so, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position.

6. The method for generating a foreign matter image for a color sorter according to claim 1, characterized in that: The process of adding the foreign matter to the first image in a manner of covering the product to obtain a training set image containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign body image and the product image overlap. If not, return to the random position selection step. Otherwise, add the first foreign body material image to the intended adding position and place it above the layer where the product image is located.

7. The method for generating a foreign matter image for a color sorter according to claim 1, characterized in that: The process of adding the foreign matter to the first image in a manner of being partially covered by the product to obtain a training set of images containing foreign matter includes: Random position selection step: randomly generating a proposed adding position in the first picture according to the size of the minimum circumscribed rectangle of the foreign object image of the selected first foreign object material picture; Determine whether the foreign object image and the product image overlap. If not, return to the random position selection step. Otherwise, determine whether the foreign object image is completely covered by the product image. If yes, return to the random position selection step. Otherwise, add the first foreign object material image to the intended adding position and place it below the layer where the product image is located.

8. The method for generating a foreign matter image for a color sorter according to claim 1, characterized in that: The image segmentation and extraction of multiple pictures containing foreign matter and / or products is achieved through a first model, and the first model is a SOLO model.

9. A foreign body image generating device for a color sorter, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images