A data generation method and system for zero-shot object detection and a storage medium

Abstract

The application relates to the technical field of target detection, and discloses a data generation method and system for zero-sample target detection and a storage medium, wherein the method acquires a real-shooting conveyor belt image of a target newly-established production line as a background set; acquires a part mask image and a part texture image of a relevant production line similar to the target newly-established production line; determines a target image according to the part mask, the part texture image and a target background image; and mixes the target image with a real image of the relevant production line to obtain a training data set. In this way, only one empty background real-shooting image of the newly-established production line and the collected foreground part mask and texture image of the relevant production line are used to generate a foreground part image with texture distributed on a real background and a corresponding target detection training label of each image, a large amount of data of a new scene can be obtained without consuming time to collect a large amount of real data of the new scene, subsequent steps are facilitated, the precision of a detection result is facilitated to improve, and the efficiency of an obtained detection model is improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a data generation method, system and storage medium for zero-sample target detection. Background Technology

[0002] With the continuous advancement of science and technology, machine vision technology is increasingly widely used in industrial manufacturing. Deep learning-based object detection technology, which is widely applied, often requires learning useful features from large amounts of data to achieve object detection. Existing methods mostly employ few-shot object detection methods and domain transfer techniques, using a few or thousands of real-world scene images to train the relevant network. Alternatively, GA networks can be used to generate high-quality scene data. However, few-shot object detection methods still require a significant amount of real-world data from new scenes; if such data is scarce, the accuracy and recall of few-shot object detection will not reach ideal levels. Domain transfer techniques require real-world data from new domains to train the model and achieve good transfer results, which contradicts the reality of a lack of real-world data from new scenes. GA networks require a large amount of real-world data from new scenes to train the model and generate data that is difficult to distinguish between real and fake data, making them unsuitable in the early stages of a project when real-world data from new scenes is scarce.

[0003] It is evident that existing target detection models for new production lines lack real training data for this scenario, resulting in low detection accuracy and slow detection efficiency. Summary of the Invention

[0004] This invention provides a data generation method, system, and storage medium for zero-sample target detection to solve the problems of low accuracy of detection results and slow efficiency in obtaining detection models.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a data generation method for zero-sample target detection, comprising:

[0007] Acquire real-shot images of the conveyor belt of the target newly established production line as the background set;

[0008] Obtain part mask images and part texture images of related production lines similar to the target newly established production line;

[0009] The target image is determined based on the part mask, the part texture map, and the target background image, wherein the target background image is any background image in the background set;

[0010] The target image is mixed with real images of the relevant production line to obtain a training dataset.

[0011] Optionally, the method further includes:

[0012] The mask image of the target image is determined based on the pure black background texture part image;

[0013] Determine the minimum bounding rectangle of the part mask in the mask image of the target image, and determine the target information of the minimum bounding rectangle as a label. The target information is the coordinates and width and height information of the minimum bounding rectangle distributed on the mask image of the target image.

[0014] A pre-built YOLOx network is trained based on the labels and the training dataset to obtain a target detection model for the newly established production line, and target detection is performed based on the target detection model.

[0015] Optionally, determining the target image based on the part mask, the part texture map, and the target background image includes:

[0016] Perform an AND operation between the part mask image and the part texture image to obtain a textured part image;

[0017] The textured part image is set on a pre-constructed pure black background image to obtain a pure black background textured part image.

[0018] The target image is obtained by setting the texture parts in the pure black background texture part image to the corresponding positions in the target background image based on the mask image of the pre-constructed target image.

[0019] Optionally, the step of setting the textured parts in the pure black background textured part image to the corresponding positions in the target background image based on the mask image of the pre-constructed target image to obtain the target image includes:

[0020] Determine the first position of the textured part in the pure black background textured part drawing;

[0021] Determine the second position in the target background image that corresponds to the first position;

[0022] The target image is obtained by setting the texture part in the pure black background texture part image to the second position of the target background image based on the mask map of the pre-constructed target image.

[0023] Optionally, training a pre-built YOLOx network based on the labels and the training dataset to obtain a target detection model for the newly established production line includes:

[0024] The training dataset is input into a pre-built YOLOX network for training, and the output information of the pre-built YOLOX network is obtained.

[0025] The output information is compared with the label. If the overlap between the output information and the label meets a preset threshold, the training is considered complete, and the target detection model for the new production line is obtained.

[0026] Secondly, this application provides a data generation system for zero-sample target detection, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0027] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0028] Beneficial effects:

[0029] The zero-shot target detection data generation method provided by this invention first acquires a real-shot image of the conveyor belt of a newly established production line as a background set; then acquires part masks and part texture maps of related production lines similar to the newly established production line; based on the part masks, part texture maps, and target background images, a target image is determined, where the target background image is any background image from the background set; finally, the target image is mixed with real images from related production lines to obtain a training dataset. In this way, using only a single real-shot image of an empty conveyor belt background from the newly established production line, along with collected foreground part masks and texture maps from related production lines, a texture synthesis method is used to generate foreground part images with textures distributed on a real background, along with corresponding target detection training labels for each image. This method obtains data for a new scene without requiring a large amount of real data from the new scene, facilitating subsequent steps, improving the accuracy of detection results, and increasing the efficiency of obtaining the detection model. Attached Figure Description

[0030] Figure 1 This is a flowchart of a zero-sample target detection data generation method according to a preferred embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the texture synthesis method of a preferred embodiment of the present invention.

[0032] Figure 3 This is one of the data generated by the newly established production line in a preferred embodiment of the present invention.

[0033] Figure 4 Data for the background sorting production line of the relevant production line in a preferred embodiment of the present invention.

[0034] Figure 5 This is the second set of real data for the newly established production line in a preferred embodiment of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0037] Please see Figure 1 This application provides a data generation method for zero-shot target detection, comprising:

[0038] Acquire real-shot images of the conveyor belt of the target newly established production line as the background set;

[0039] Obtain part mask images and part texture images of related production lines similar to the target newly established production line;

[0040] The target image is determined based on the part mask, part texture map, and target background image, where the target background image is any background image from the background set.

[0041] The training dataset is obtained by mixing the target image with real images from the relevant production line.

[0042] It should be noted that when obtaining the part mask and part texture map of a related production line similar to the target new production line, there are several part texture maps and part masks. That is, what is obtained is a part mask map dataset and a part texture map dataset. Each time, several maps are randomly selected from the dataset as part mask maps and part texture maps to perform the steps in this application.

[0043] In this embodiment, acquiring real-shot images of the conveyor belt of the target newly established production line as the background set can be done through a camera device. Related production lines similar to the target newly established production line refer to production lines where the background of the conveyor belt changes, and the types and textures of the generated parts differ from, but not significantly from, the types and textures of the parts required by the newly established production line. It should be noted that when acquiring the part mask and part texture images of related production lines similar to the target newly established production line, these images are collected from the data of the related production lines and are not directly produced by those lines.

[0044] It should be noted that in this embodiment, the training dataset is obtained by mixing the target image with the real images of the related production line. This allows for the rapid acquisition of a new production line detection model with high detection accuracy even when there is zero real data for the new scene. The reason for this is that, in practical applications, it has been found that the detection accuracy of data generated by texture synthesis (the determined target image) directly used for detecting new production lines is only slightly over 67%, which is not ideal. This is because the foreground texture of the image generated by the texture synthesis method is not realistic enough. However, when the background is completely different, the detection model trained with similar related production line data—the grid background sorting production line—can achieve an accuracy of 83% when directly used to detect new production lines. This indicates that the two scenes have a high degree of similarity in the foreground. However, tests also revealed that the detection model trained directly with related production line data—the grid background sorting production line—often misdetects the background of the new production line as the foreground. This is because the model trained with related production line data—the grid background sorting production line—has not seen the background of the new production line. Therefore, considering the advantages of relevant production line data—grid background sorting production line data—in foreground recognition and the fact that the background of the generated data (target image) is the real background of the new production line, this application adopts a method of training a detection model by mixing the two types of data. As can be seen, even with zero real data in the new scene, we have already obtained a new production line detection model with a detection accuracy of 89%.

[0045] In a specific example, the model trained on texture synthesis-generated data achieved an accuracy (AP) of 67.959% and an AR of 79.108% for zero-shot object detection. The model trained on real data from the relevant production line achieved an accuracy of 83.545% and an AR of 88.288% for zero-shot object detection. The detection model trained on the mixed data achieved an accuracy of 89.165% and an AR of 91.336% for zero-shot object detection. A schematic diagram illustrating the data generated by the newly established production line is shown below. Figure 3-5 As shown in Table 1, the specific data comparison is as follows.

[0046] Table 1 shows the specific data.

[0047]

[0048]

[0049] The aforementioned zero-sample target detection data generation method uses only a real-world image of an empty background conveyor belt from a new production line, along with collected foreground part masks and texture maps from related production lines. It employs texture synthesis to generate a textured foreground part map with distributed textures on a real background. This method can obtain data for a new scene without requiring a large amount of real-world data from the new scene, facilitating subsequent steps, improving the accuracy of detection results, and increasing the efficiency of obtaining the detection model.

[0050] Optionally, the above method further includes:

[0051] The mask image of the target image is determined based on the pure black background texture part image;

[0052] Determine the minimum bounding rectangle of the part mask in the mask image of the target image, and determine the target information of the minimum bounding rectangle as a label. The target information is the coordinates and width and height information of the minimum bounding rectangle distributed on the mask image of the target image.

[0053] A pre-built YOLOx network is trained based on the labels and the training dataset to obtain a target detection model for the newly established production line, and target detection is performed based on the target detection model.

[0054] The step of training a pre-constructed YOLOx network based on the labels and the training dataset to obtain the target detection model for the newly established production line includes:

[0055] The training dataset is input into a pre-built YOLOX network for training, and the output information of the pre-built YOLOX network is obtained.

[0056] The output information is compared with the label. If the overlap between the output information and the label meets a preset threshold, the training is considered complete, and the target detection model for the new production line is obtained.

[0057] In this optional implementation, the specific steps for determining the label are as follows:

[0058] Using functions in OpenCV, a set of contour points can be obtained from the mask image. Based on this set, another OpenCV function can be used to obtain the minimum positive bounding moment of the part, returning the values ​​of the minimum positive bounding moment: x, y, w, and h. x and y are the coordinates of the top-left point of the matrix, and w and h are the width and height of the matrix. Saving the x, y, w, and h values ​​of the minimum positive bounding moment for each part gives the target detection label.

[0059] When training a pre-constructed YOLOx network based on the labels and the training dataset, the obtained training dataset can be divided into a training set and a validation set in a 7:3 ratio to train and validate the newly established production line. In the specific training process, the training dataset is input into the pre-constructed YOLOx network for training, and the output information of the pre-constructed YOLOx network is obtained. The output information is then compared with the labels. For example, if the output information is a first positive bounding moment, and the overlap between the first positive bounding moment and the label meets a preset threshold (i.e., the first positive bounding moment overlaps with or is nearly overlaps with the label), the first bounding moment is considered to have met the requirements, training is complete, and the target detection model for the newly established production line is obtained.

[0060] By saving the object detection labels for each image, we greatly facilitate the next step of data training. We know that labeling object detection datasets is an extremely labor-intensive and time-consuming process, while our data generation method directly generates object detection labels for each image that can be used directly for training, making it a highly efficient method.

[0061] Furthermore, the strategy of using a mixture of real data from similar scenarios and newly generated data for training can significantly improve the performance of object detection models.

[0062] Optionally, the target image is determined based on the part mask, the part texture map, and the target background image, including:

[0063] Perform an AND operation between the part mask image and the part texture image to obtain a textured part image;

[0064] A textured part image is obtained by setting a textured part image onto a pre-constructed pure black background image.

[0065] The target image is obtained by setting the texture parts in the pure black background texture part image to the corresponding positions in the target background image based on the mask image of the pre-constructed target image.

[0066] In this embodiment, "set" specifically refers to embedding or replacing; in other words, it means replacing the corresponding image pixels in the pre-constructed pure black background image with a textured part image. The pre-constructed pure black background image is the same size as the target background image.

[0067] Optionally, the target image is obtained by setting the texture parts in the pure black background texture part image to the corresponding positions in the target background image according to the mask map of the pre-constructed target image, including:

[0068] Determine the first position of the textured part in the pure black background textured part drawing;

[0069] Determine the second position in the target background image that corresponds to the first position;

[0070] The target image is obtained by setting the texture parts in the pure black background texture part image to the second position of the target background image based on the mask image of the pre-constructed target image.

[0071] It should be noted that performing a bitwise AND operation between any number N and the number 0 will result in the number 0. Performing a bitwise AND operation between any number N and the number 255 will result in the number N itself.

[0072] In this embodiment, as Figure 2 As shown, in the part mask image, the part is represented by a white area, and the background of the part is represented by a black area. The texture part in the part texture image belongs to the area between white and black. Therefore, in the part mask image, the grayscale value of each pixel in the white area is 255, and the grayscale value of each pixel in the black area is 0. The pixel values ​​of the texture in the part texture image are distributed between 0 and 255.

[0073] The part mask and the part texture are ANDed. The black positions in the part texture and the part mask will be set to 0, that is, they will remain black. The texture will be ANDed with the white positions in the part mask to preserve the texture. This AND operation can produce a part image with texture.

[0074] It is worth emphasizing that the target image is obtained by setting the texture parts in the pure black background texture part image to the corresponding positions in the target background image based on the mask image of the pre-constructed target image. The mask image of the target image is composed of the pure black background texture parts. Figure 2 It is obtained through binarization. Binarization means that all pixels in areas where the pixel value is greater than the threshold are set to 255, and all pixels in areas where the pixel value is less than the threshold are set to 0.

[0075] It's worth emphasizing that when setting the textured parts from the pure black background textured part image to their corresponding positions in the target background image based on the pre-constructed target image mask, there are actually three images of the same size: the pre-constructed pure black background textured part image, the target image mask, and the target background image. For these three images of the same size, the areas to be replaced in the pure black background textured part image and the target background image are determined by judging whether the pixels in the pre-constructed target image mask are greater than 0. Specifically, the white areas (areas with pixels greater than 0) in the target image mask are extracted and used to replace the corresponding background in the pure black background textured part image, thus obtaining the final texture synthesis image. The so-called replacement is actually using the pixel values ​​of the corresponding positions in the pure black background textured part image to apply to the pixel values ​​of the corresponding positions in the target background image. In this way, the entire target image with the parts already distributed can be obtained quickly.

[0076] The mask of the target image is determined by the pure black background texture part image. For example, the pure black background texture part image is binarized to obtain the mask of the target image. In other words, the mask of the target image is obtained before the target image.

[0077] In this optional implementation, the pure black background textured part image is obtained by replacing areas in the pure black background image with textured part images. Specifically, the replaced areas follow these rules: tiles are laid out without overlap, parts are spaced 10 pixels apart, the number of parts can be set with upper and lower limits, the type of parts is random, and the part size is scaled from 0.9 to 1.1 times. That is, the size of a part in the pure black background textured part image will not differ from the original part size by more than 0.1 times. This allows for fluctuations in part size within a certain range.

[0078] In summary, the data generation method for object detection described above can quickly generate a large batch of data for a new production line by changing a real-world image of an empty conveyor belt background. If the new production line produces new parts, we can also promptly obtain the part's mask to generate a batch of image data containing the new part. The number of foreground parts in each image can be set with upper and lower limits, generating a random number of foreground parts within that range. The type of parts in each image is random; it can be all of one type or many types. Parts can be arranged close together (with a 10-pixel interval) or scattered throughout. The texture synthesis process not only generates images with foreground parts but also saves the object detection label corresponding to each image, greatly facilitating subsequent data training. We know that labeling object detection datasets is an extremely labor-intensive and time-consuming process, while the data generation method in this embodiment directly generates object detection labels for each image, which can be directly used for training, making it a highly efficient method.

[0079] It should be understood that the data generation method for texture synthesis can not only generate data for object detection, but also generate edge maps and mask maps corresponding to each image, which can be used for template matching.

[0080] This application also provides a zero-shot target detection data generation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method. This data generation system for target detection can implement various embodiments of the above-described method and achieve the same beneficial effects; further details are omitted here.

[0081] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method steps described above. This readable storage medium can implement various embodiments of the above-described method and achieve the same beneficial effects; further details are omitted here.

[0082] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A data generation method for zero-shot target detection, characterized in that, include: Acquire real-shot images of the conveyor belt of the target newly established production line as the background set; Obtain part mask images and part texture images of related production lines similar to the target newly established production line; A textured part image is obtained by performing a bitwise AND operation between the part mask image and the part texture image; the textured part image is then set on a pre-constructed pure black background image to obtain a pure black background textured part image; the textured parts in the pure black background textured part image are set to the corresponding positions in the target background image according to the mask image of the pre-constructed target image to obtain the target image, wherein the target background image is any background image in the background set. The target image is mixed with real images of the relevant production line to obtain a training dataset.

2. The data generation method for zero-sample target detection according to claim 1, characterized in that, The method further includes: The pure black background textured part image is binarized to obtain a mask image of the target image; The minimum bounding rectangle of the part mask in the mask image of the target image is determined, and the target information of the minimum bounding rectangle is determined as a label. The target information includes the center point coordinates of the minimum bounding rectangle and the relative position information of the width and height of the minimum bounding rectangle with the width and height of the target image. A pre-built YOLOx network is trained based on the labels and the training dataset to obtain a target detection model for the newly established production line, and target detection is performed based on the target detection model.

3. The data generation method for zero-sample target detection according to claim 1, characterized in that, The step of setting the texture parts in the pure black background texture part image to the corresponding positions in the target background image based on the mask image of the pre-constructed target image includes: Determine the first position of the textured part in the pure black background textured part drawing; Determine the second position in the target background image that corresponds to the first position; The target image is obtained by setting the texture part in the pure black background texture part image to the second position of the target background image based on the mask map of the pre-constructed target image.

4. The data generation method for zero-sample target detection according to claim 2, characterized in that, The step of training a pre-constructed YOLOx network based on the labels and the training dataset to obtain the target detection model for the newly established production line includes: The training dataset is input into a pre-built YOLOX network for training, and the output information of the pre-built YOLOX network is obtained. The output information is compared with the label. If the overlap between the output information and the label meets a preset threshold, the training is considered complete, and the target detection model for the new production line is obtained.

5. A data generation system for zero-shot target detection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-4.

Images