Foreign matter picture generation method and device applied to color sorter and storage medium

Abstract

The present application relates to a kind of foreign matter picture generation method, device and storage medium applied to color sorter, wherein the method comprises: the segmentation extraction of image to multiple pictures containing foreign matter and / or product, obtain original foreign matter material picture and original product material picture;Original foreign matter material picture and original product material picture are carried out smoothing and gradient processing, obtain first foreign matter material picture and first product material picture;Import background picture, and select one or more first product material picture, add to the random position of background picture to obtain first picture;Randomly select one or more first foreign matter material picture, to be overlapped with product, cover on product and be covered by product Partially, add to first picture to obtain training set picture containing foreign matter, and generate the position of each foreign matter material in training set picture.Compared with prior art, the present application has the advantages that a large number of high-quality labeled training set data for color sorter can be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of color sorting machines, in particular to a foreign matter picture generation method and device applied to a color sorting machine and a storage medium. BACKGROUND

[0002] Color sorting machines are important equipment in the food industry. In recent years, the requirements for color sorting machines have gradually increased, and color sorting machines need to be able to distinguish foreign matter. Therefore, color sorting machines need to be able to identify foreign matter in products.

[0003] Therefore, some people in the field gradually introduce machine learning technology into color sorting machines to identify foreign matter. However, due to the insufficiency of the training set, the actual effect of this function is limited. Although some prior art discloses various technical solutions, due to the lack of a high-quality sufficient training set, although there are some related researches, there is no application in actual products at present.

[0004] In this regard, people in the field are committed to building a sufficient training set to improve the foreign matter detection capability of color sorting machines.

[0005] Traditional data set construction methods usually rely on manual collection and labeling, which requires a lot of manpower and time, especially in some specific foreign matter sorting scenarios, which require pictures with diversity and specific labeling, further increasing the complexity and cost of data set production.

[0006] Although there are some image synthesis counts in some fields that are not very similar to food color sorting, such as advertising, public relations, etc. By generating images, the dependence on real-world image data can be greatly reduced, and large-scale, diverse image sets can be quickly generated. Most of such technologies are mainly applied to news picture creation and self-media article picture arrangement, etc., and they perform poorly in generating high-quality, realistic images, and cannot fully meet the requirements of training deep learning models. SUMMARY

[0007] The purpose of the present application is to provide a foreign matter picture generation method and device applied to a color sorting machine and a storage medium, which can reduce the time cost of manual labeling by synthesizing a large amount of data from a small amount of test data and a foreign matter database.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] A foreign matter picture generation method applied to a color sorting machine, comprising:

[0010] Segmentation extraction of images of multiple pictures containing foreign matter and / or products, to obtain original foreign matter material pictures and original product material pictures respectively, wherein the original foreign matter material pictures contain foreign matter information, and the original product material pictures contain product information;

[0011] Smooth and gradient processing of the original foreign matter material pictures and the original product material pictures to obtain first foreign matter material pictures and first product material pictures, and taking the foreign matter information of the original foreign matter material pictures as the foreign matter information of the first foreign matter material pictures, and taking the product information of the original product material pictures as the product information of the first product material pictures;

[0012] Importing a background picture and randomly selecting one or more first product material pictures to add to random positions of the background picture to obtain a first picture;

[0013] Randomly selecting one or more first foreign matter material pictures to add to the first picture in a manner of no overlap with the product, covering the product, and being partially covered by the product to obtain a training set picture containing foreign matter, and taking the positions of the foreign matter materials in the training set picture as foreign matter positions, and generating labeling information of the training set picture based on the product information of the selected first product material picture, the foreign matter information of the first foreign matter material picture, and the foreign matter positions.

[0014] The process of smooth and gradient processing of the original foreign matter material pictures includes:

[0015] Creating a first mask picture based on the imported original foreign matter material picture, wherein the pixel value of the foreground part is set to 255, and the pixel value of the background part is set to 0;

[0016] Performing erosion processing on the created first mask picture using a first convolution kernel, wherein the value of each element in the first convolution kernel is equal, and is 1 / A, A is the number of elements in the convolution kernel;

[0017] Performing Gaussian blur processing on the first mask picture after erosion processing;

[0018] Assigning the first mask picture after Gaussian blur processing as a transparent channel to the original foreign matter material picture to obtain a first foreign matter material picture in RBGA format, and taking the foreign matter information of the original foreign matter material picture as the foreign matter information of the first foreign matter material picture.

[0019] The first convolution kernel is a 3x3 convolution kernel.

[0020] The process of smooth and gradient processing of the original product material picture is the same as that of the original foreign matter material picture.

[0021] The process of adding to the first picture in a manner of no overlap with the product to obtain a training set picture containing foreign matter includes:

[0022] a random position selection step: generating a position to be added to the first picture randomly according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign matter material picture;

[0023] determining whether the foreign matter image and the product image overlap, if yes, returning to the random position selection step, otherwise adding the first foreign matter material picture to the position to be added.

[0024] the process of adding the first picture in a manner of covering the product to obtain the training set picture containing the foreign matter, comprising:

[0025] a random position selection step: generating a position to be added to the first picture randomly according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign matter material picture;

[0026] determining whether the foreign matter image and the product image overlap, if no, returning to the random position selection step, otherwise adding the first foreign matter material picture to the position to be added and placing it above the layer where the product image is located.

[0027] the process of adding the first picture in a manner of being partially covered by the product to obtain the training set picture containing the foreign matter, comprising:

[0028] a random position selection step: generating a position to be added to the first picture randomly according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign matter material picture;

[0029] determining whether the foreign matter image and the product image overlap, if no, returning to the random position selection step, otherwise determining whether the foreign matter image is completely covered by the product image, if yes, returning to the random position selection step, otherwise adding the first foreign matter material picture to the position to be added and placing it below the layer where the product image is located.

[0030] the segmentation and extraction of the images of the plurality of pictures containing the foreign matter and / or the product are realized by a first model, and the first model is a SOLO model.

[0031] An foreign matter picture generation device applied to a color sorter, comprising a memory, a processor, and a program stored in the memory, and the processor realizes the method as described above when executing the program.

[0032] A storage medium having a program stored thereon, and the program realizes the method as described above when executed.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. The method can reduce the time cost of manual annotation by a small amount of data of the object to be tested, a large amount of synthesized data of the foreign matter database, and automatic annotation during the synthesis process.

[0035] 2. The quality and delicacy of the synthesized picture can be improved by generating in three ways of no overlap with the product, covering the product, and being partially covered by the product, and combining with smoothing and gradient processing, to meet the training needs of the machine learning model.

[0036] 3. The synthesized picture can expand the diversity of the data set, especially when the original data is limited, by synthesizing multiple samples, the model can better generalize, and the recognition or classification accuracy can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the main steps of the method of the present application;

[0038] Figure 2 is a schematic diagram of the first foreign matter material picture obtained from the first mask picture;

[0039] Figure 3 is a schematic diagram of the picture generated by the generation method of no overlap between the foreign matter and the product;

[0040] Figure 4 is a schematic diagram of the picture generated by the generation method of the foreign matter covering the product;

[0041] Figure 5 is a schematic diagram of the picture generated by the generation method of the foreign matter being partially covered by the product. DETAILED DESCRIPTION

[0042] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and detailed implementation and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0043] A foreign matter picture generation method applied to a color sorter can reduce the time cost of manual annotation by a small amount of data of the object to be tested, a large amount of synthesized data of the foreign matter database, and automatic annotation during the synthesis process, as shown in Figure 1 , comprising:

[0044] (1) image segmentation and extraction are performed on multiple pictures containing foreign matter and / or products to obtain original foreign matter material pictures and original product material pictures, respectively. Specifically, in the present embodiment, the product is peanuts;

[0045] The segmentation and extraction of images of multiple pictures containing foreign matters and / or products are implemented by a first model, which is a SOLO model. A large number of pictures are collected, and all objects on the pictures are segmented and labeled. The labeled data provides a solid data foundation for the training of the subsequent deep learning model, and ensures that the model has good recognition performance in various environments. Then, the SOLO (Segmenting Objects by Locations) model is trained using the labeled pictures, so that the model has the ability to segment all objects on the pictures.

[0046] The first model adopts a backbone+FPN (Feature Pyramid Network) structure. The backbone is used to extract features, and the FPN is used to fuse features of different scales, so as to identify instances of different sizes in different hierarchical branches. Two branches are constructed at each layer of the FPN: a Category Branch and a Mask Branch. The Category Branch is used to predict the category of the instance, and is distinguished according to the center position of the instance. The output compressed picture is S*S*C, where S*S is the size of the compressed picture, C is the number of channels, and the number of instance categories is taken. The Mask Branch is used to predict the mask of the instance, and predicts the corresponding mask for each positive sample region. By setting a certain region around the center point of the instance, for example, a scale factor ε=0.2 times is selected as the positive sample, and a feature map of H*W*S 2 is outputted, where H and W represent the height and width of the input image, and each channel is responsible for the mask prediction of the corresponding grid. The channel number of the grid is related to its coordinates (i, j), which is determined by k=i*S+j, where S represents that the input image is divided into S*S grids, C represents the number of instance categories, H and W represent the height and width of the input image, and k is used to represent the mapping relationship between the grid and the channel number.

[0047] The material picture containing only foreign matters is inputted into the trained SOLO model, and the foreign matters such as plastic, stone, metal fragments and other impurities in the picture can be segmented out and saved to the material library, which is used for subsequent image synthesis.

[0048] When it is necessary to detect foreign matters in a specific field, for example, when it is necessary to detect foreign matters in peanuts, the invention can be used to synthesize pictures.

[0049] Among them, the obtained original foreign matter material picture contains foreign matter information, and the obtained original product material picture contains product information.

[0050] (ii) smoothing and gradient processing is performed on the original foreign matter material picture and the original product material picture to obtain a first foreign matter material picture and a first product material picture, and the foreign matter information of the original foreign matter material picture is taken as the foreign matter information of the first foreign matter material picture, and the product information of the original product material picture is taken as the product information of the first product material picture;

[0051] In some embodiments, the edge of the material is guaranteed to be integrated with the background part, and the process of smoothing and gradient processing on the original foreign matter material picture includes:

[0052] A first mask picture is created based on the imported original foreign matter material picture, wherein the pixel value of the foreground part is set to 255, and the pixel value of the background part is set to 0;

[0053] The created first mask picture is eroded using a first convolution kernel to smooth the edge and reduce noise, wherein the value of each element in the first convolution kernel is equal, and is 1 / A, A is the number of elements in the convolution kernel, wherein in this embodiment, the first convolution kernel is a 3x3 convolution kernel;

[0054] Gaussian blur processing is performed on the first mask picture after the erosion processing;

[0055] The first mask picture after the Gaussian blur processing is taken as a transparent channel and assigned to the original foreign matter material picture to obtain a first foreign matter material picture in RBGA format, and the foreign matter information of the original foreign matter material picture is taken as the foreign matter information of the first foreign matter material picture, so that there is no edge feeling of fragmentation during integration. As shown in Figure 2 , a four-channel picture is obtained by fusing the first mask picture of the three-channel picture.

[0056] In this way, by generating in three ways of no overlap with the product, covering on the product, and being partially covered by the product, and combining the way of taking the first mask picture after the Gaussian blur processing as a transparent channel and assigning it to the original foreign matter material picture in the smoothing and gradient processing operation, the quality and delicacy of the synthesized picture can be improved, and the training needs of the machine learning model can be met.

[0057] In addition, in most embodiments, the process of smoothing and gradient processing on the original product material picture is the same as that on the original foreign matter material picture.

[0058] In addition, in this embodiment, in order to ensure as many forms as possible, data augmentation will also be performed on the material picture, including random rotation, enlargement and reduction, left-right reversal, upside-down, HSV color transformation.

[0059] (iii) importing a background picture, and randomly selecting one or more first product material pictures and adding them to random positions of the background picture to obtain a first picture;

[0060] Since the first product material picture contains product information in its attribute when it is built, the product information can be obtained when it is imported.

[0061] (Four) one or more first foreign material pictures are randomly selected to be added to the first picture in a manner of no overlap with the product, covering the product, and being partially covered by the product to obtain the training set picture containing foreign matter, and the position of each foreign material in the training set picture is generated.

[0062] Similarly, since the first foreign material picture contains foreign matter information in its attribute when it is built, the foreign matter information can be obtained when it is imported.

[0063] Specifically, in the embodiment, when the peanuts and the foreign matter do not overlap, the process of adding the first foreign material picture to the first picture in a manner of no overlap with the product to obtain the training set picture containing foreign matter includes:

[0064] Random position selection step: according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign material picture, a tentative adding position is randomly generated in the first picture, and the position of the foreign matter is recorded, specifically the position of the minimum circumscribed rectangle is taken as the position of the foreign matter;

[0065] determine whether the foreign matter image and the product image overlap, if yes, return to the random position selection step, otherwise add the first foreign material picture to the tentative adding position, and generate the label information corresponding to the generated picture according to the product information, the foreign matter information and the position of the foreign matter, which is equivalent to labeling the synthesized picture, facilitating subsequent deep learning training using the synthesized picture.

[0066] As shown in Figure 3 , the sample picture with no overlap between the foreign matter and the product and the label result thereof are finally obtained, Figure 3 where the circle represents the product and the triangle represents the foreign matter.

[0067] In addition, in the embodiment, the process of adding the first foreign material picture to the first picture in a manner of covering the product to obtain the training set picture containing foreign matter includes:

[0068] Random position selection step: according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign material picture, a tentative adding position is randomly generated in the first picture;

[0069] determine whether the foreign matter image and the product image overlap, if no, return to the random position selection step, otherwise add the first foreign material picture to the tentative adding position and place it above the layer where the product image is located.

[0070] Similarly, according to the product information, the foreign matter information and the position of the foreign matter, the annotation information corresponding to the generated picture is generated, which is equivalent to annotating the synthesized picture, facilitating subsequent deep learning training using the synthesized picture.

[0071] As shown in Figure 4 , the sample picture with the foreign matter covering the product and the annotation result thereof are finally obtained, wherein the circle represents the product and the triangle represents the foreign matter. Figure 4

[0072] In this embodiment, the process of adding the first picture to obtain the training set picture containing the foreign matter in the form of being partially covered by the product includes:

[0073] Random position selection step: according to the size of the minimum circumscribed rectangle of the foreign matter image of the selected first foreign matter material picture, a position to be added is randomly generated in the first picture;

[0074] If the foreign matter image and the product image do not overlap, the random position selection step is returned, otherwise, it is judged whether the foreign matter image is completely covered by the product image, if yes, the random position selection step is returned, otherwise, the first foreign matter material picture is added to the position to be added and placed below the layer where the product image is located.

[0075] Similarly, according to the product information, the foreign matter information and the position of the foreign matter, the annotation information corresponding to the generated picture is generated, which is equivalent to annotating the synthesized picture, facilitating subsequent deep learning training using the synthesized picture.

[0076] As shown in Figure 5 , the sample picture with the foreign matter covering the product and the annotation result thereof are finally obtained, wherein the circle represents the product and the triangle represents the foreign matter. Figure 5

[0077] Since the stacking situation is relatively common in real scenes, when synthesizing, the probability of the first situation occurring is set to 80%, and the probabilities of the other two situations occurring are set to 10%.

[0078] The present application can reduce the time and labor cost of manual annotation, improve efficiency, and expand the diversity of the data set by synthesizing a large amount of data from a small amount of test data and a foreign matter database, especially when the original data is limited. By synthesizing multiple samples, the model can be better generalized, and the recognition or classification accuracy can be improved.

[0079] ​​If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for generating images of foreign objects for use in color sorters, characterized in that, include: Multiple images containing foreign objects and / or products are segmented and extracted to obtain original foreign object material images and original product material images, wherein the original foreign object material images contain foreign object information and the original product material images contain product information, and the product is peanut; The original foreign object material image and the original product material image are smoothed and gradient processed to obtain the first foreign object material image and the first product material image. The foreign object information of the original foreign object material image is used as the foreign object information of the first foreign object material image, and the product information of the original product material image is used as the product information of the first product material image. Import the background image, and randomly select one or more first product material images, and add them to random positions on the background image to obtain the first image; One or more first foreign object material images are randomly selected and added to the first image in a manner that does not overlap with the product, covers the product, or is partially covered by the product to obtain a training set image containing foreign objects. The position of each foreign object material in the training set image is used as the foreign object position. Based on the product information of the selected first product material image, the foreign object information of the first foreign object material image, and the foreign object position, the annotation information of the training set image is generated. The process of smoothing and grading raw foreign object images includes: A first mask image is created based on the imported original foreign object material image, where the pixel value of the foreground part is set to 255 and the pixel value of the background part is set to 0. The first mask image created is eroded using the first convolution kernel, wherein each element in the first convolution kernel has the same value, which is 1 / A, and A is the number of elements in the convolution kernel; Gaussian blur is applied to the first mask image after erosion. The first mask image after Gaussian blur processing is assigned as the transparency channel to the original foreign object material image to obtain the first foreign object material image in RBGA format, and the foreign object information of the original foreign object material image is used as the foreign object information of the first foreign object material image. The process of adding images containing foreign objects to the first image in a manner that does not overlap with the product image to obtain a training set of images includes: Random location selection step: Based on the size of the minimum bounding rectangle of the foreign object image in the selected first foreign object material image, randomly generate the proposed addition location in the first image; Determine if there is overlap between the foreign object image and the product image. If yes, return to the random location selection step; otherwise, add the first foreign object image to the intended location. The process of adding training set images containing foreign objects to the first image by overlaying them onto the product includes: Random location selection step: Based on the size of the minimum bounding rectangle of the foreign object image in the selected first foreign object material image, randomly generate the proposed addition location in the first image; Determine if the foreign object image and the product image overlap. If not, return to the random location selection step; otherwise, add the first foreign object image to the intended location and place it above the product image layer. The process of adding training set images containing foreign objects to the first image by partially covering them with the product image includes: Random location selection step: Based on the size of the minimum bounding rectangle of the foreign object image in the selected first foreign object material image, randomly generate the proposed addition location in the first image; Determine if the foreign object image and the product image overlap. If not, return to the random location selection step. Otherwise, determine if the foreign object image is completely covered by the product image. If yes, return to the random location selection step. Otherwise, add the first foreign object image to the desired location and place it below the layer containing the product image.

2. The method for generating foreign object images for use in a color sorter according to claim 1, characterized in that, The first convolution kernel is a 3×3 convolution kernel.

3. A method for generating foreign object images for use in a color sorter according to claim 1 or 2, characterized in that, The process of smoothing and gradient processing the original product material images is the same as that of processing the original foreign object material images.

4. The method for generating foreign object images for use in a color sorter according to claim 1, characterized in that, The segmentation and extraction of multiple images containing foreign objects and / or products is achieved through a first model, which is the SOLO model.

5. A foreign object image generation device for use in a color sorter, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-4.

6. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-4.

Images