Image data processing method, device, apparatus, and storage medium

Abstract

The application discloses a processing method, device and equipment of image data and a storage medium. The processing method comprises the following steps: acquiring a product defect image corresponding to a target category defect and a product background image of a target product; acquiring a pre-constructed semantic model, wherein the semantic model has the ability to determine the occurrence area of various category defects in the product background image; determining a target area corresponding to the target category defect in the product background image by using the semantic model; and fusing the product defect image to the target area to obtain a defect image sample corresponding to the target category defect of the target product. Through the technical scheme provided by the application, the automatic generation of the defect image sample in the industrial scene can be realized, and the rapid start of the industrial detection project is facilitated.

Description

Technical Field

[0001] This application belongs to the field of industrial inspection technology, and in particular relates to an image data processing device, equipment and storage medium. Background Technology

[0002] During the production process of industrial products, a large number of defective products are randomly generated. In order to ensure product quality and save production costs, defective products need to be removed immediately after they are generated through industrial inspection methods. With the increasing maturity of artificial intelligence technology, deep learning technology is widely used in industrial inspection to detect, identify, and segment defects in product images. This technology requires a sufficient number of defective image samples to support it.

[0003] However, in industrial settings, there are many scenarios where it is difficult to collect samples. These scenarios are as follows: 1. The core indicators of manufacturing are production capacity and yield. Companies reduce scrap rates and thus continue to make profits by maintaining high yield rates. However, excessively high yield rates can lead to the inability to collect enough samples. 2. When producing products in small batches with multiple varieties, there are situations where defect samples have not been collected before the product has been changed. 3. Defects are diverse in form and have a long tail effect, making it impossible to cover all types of defects.

[0004] Therefore, how to increase the number of defect image samples of products in industrial scenarios is an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of this application provide an image data processing method, apparatus, device, and storage medium, which can realize the automatic generation of defect image samples in industrial scenarios, facilitating the rapid start-up of industrial inspection projects.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to a first aspect of the embodiments of this application, a method for processing image data is provided, including:

[0008] Obtain the product defect image corresponding to the target category defect, as well as the product background image of the target product;

[0009] Obtain a pre-built semantic model, wherein the semantic model has the ability to determine the occurrence areas of various categories of defects in the product background image;

[0010] The semantic model is used to determine the target region corresponding to the target category defect in the product background image;

[0011] The product defect image is fused into the target region to obtain a defect image sample of the target product corresponding to the target category defect.

[0012] In some embodiments of this application, based on the foregoing scheme, the semantic model is constructed through the following steps:

[0013] Acquire training image samples and determine the boundary information and category information of the target objects in the training image samples;

[0014] Based on the boundary information, the circumscribed region of the target object is determined in the image sample to be trained.

[0015] Obtain a first image containing the external graphics region;

[0016] Set the RGB values ​​of each pixel in the outer graphic region of the first image to a first preset value to obtain the second image;

[0017] Based on the category information and the second image, the initial model is trained to obtain the semantic model.

[0018] In some embodiments of this application, based on the foregoing scheme, obtaining the first image containing the circumscribed graphics region includes:

[0019] The circumscribed region of the image sample to be trained is dilated to obtain the dilated region;

[0020] The first image is obtained by extracting the image from the dilated region in the image sample to be trained.

[0021] In some embodiments of this application, based on the aforementioned scheme, the area of ​​the expansion region is smaller than a second preset value.

[0022] In some embodiments of this application, based on the aforementioned scheme, the number of training image samples acquired is positively correlated with the complexity of the target semantic relationship, wherein the target semantic relationship is the semantic relationship between the third image and the fourth image, the third image is the image corresponding to the target object in the training image samples, and the fourth image is the image in the training image samples other than the third image.

[0023] In some embodiments of this application, based on the foregoing scheme, determining the target region corresponding to the target category defect in the product background image using the semantic model includes:

[0024] Multiple candidate regions and an expansion region corresponding to each candidate region are determined in the product background image, wherein the RGB value of each pixel in the candidate region is set to the first preset value;

[0025] The semantic model is used to score the dilated region corresponding to each candidate region to obtain a first score;

[0026] Based on the first score of the expansion region corresponding to each candidate region, the target region corresponding to the target category defect is determined from the multiple candidate regions.

[0027] In some embodiments of this application, based on the foregoing scheme, there are multiple inflated regions corresponding to each candidate region. The first score is used to characterize the probability of the target category defect occurring in the corresponding candidate region. The step of determining the target region corresponding to the target category defect from multiple candidate regions based on the first score of the inflated region corresponding to each candidate region includes:

[0028] The average of the first scores of the multiple dilated regions corresponding to each candidate region is taken as the second score corresponding to each candidate region;

[0029] The candidate region with the highest second score is determined as the target region corresponding to the defect of the target category.

[0030] In some embodiments of this application, based on the foregoing scheme, there are multiple inflated regions corresponding to each candidate region, and multiple first scores corresponding to each inflated region. Each first score is used to characterize the probability of a corresponding candidate category defect occurring in the corresponding candidate region. The step of determining the target region corresponding to the target category defect from the multiple candidate regions based on the first scores of the inflated regions corresponding to each candidate region includes:

[0031] The average of the first scores of the multiple inflated regions corresponding to each candidate region on each candidate category defect is used as the second score of each candidate region for each candidate category defect.

[0032] Based on the second score of each candidate category defect occurring in each candidate region, the candidate category defect with the highest second score is determined as the target category defect corresponding to each candidate region.

[0033] The candidate region corresponding to the defect of the target category is determined as the target region corresponding to the defect of the target category.

[0034] In some embodiments of this application, based on the foregoing scheme, obtaining the product defect image corresponding to the target category defect includes:

[0035] Obtain the original product image containing the target category defect, and the mask image containing the target category defect;

[0036] The original product image is fused with the mask image to obtain the product defect image.

[0037] In some embodiments of this application, based on the foregoing scheme, obtaining the product defect image corresponding to the target category defect includes:

[0038] Obtain the original image of the product containing the defects of the target category;

[0039] The image corresponding to the target category defect is extracted from the original product image to obtain the product defect image.

[0040] In some embodiments of this application, based on the foregoing scheme, fusing the product defect image into the target region to obtain a defect image sample of the target product corresponding to the target category defect includes:

[0041] Determine the first gradient field of the product defect image and the second gradient field of the product background image;

[0042] Replace the gradient field corresponding to the target region in the second gradient field with the first gradient field to obtain the third gradient field of the initial fused image;

[0043] The transformation matrix coefficients are determined based on the third gradient field;

[0044] Based on the transformation matrix coefficients, construct defect image samples of the target product corresponding to the target category defects.

[0045] In some embodiments of this application, based on the foregoing scheme, the image data processing method further includes:

[0046] The product defect image is subjected to multiple deformation processes to obtain multiple fifth images;

[0047] Multiple fifth images are respectively fused into the target region to obtain multiple defect image samples of the target product corresponding to the target category defects.

[0048] In some embodiments of this application, based on the foregoing scheme, the image data processing method further includes:

[0049] Obtain the original image of the product containing the target category defect selected by the user based on the first control;

[0050] In response to the user's operation on the second control, a mask image containing the target category defect is generated based on the original product image;

[0051] The original product image is fused with the mask image to obtain the product defect image;

[0052] In response to the user's operation on the third control, the step of performing multiple deformation processing on the product defect image is executed.

[0053] In some embodiments of this application, based on the foregoing scheme, the step of performing multiple deformation processing on the product defect image in response to the user's operation on the third control includes:

[0054] In response to the user's operation on the third control, a batch deformation interface is displayed;

[0055] Based on the deformation intensity and deformation quantity input by the user in the batch deformation interface, the step of performing multiple deformation processing on the product defect image is executed.

[0056] In some embodiments of this application, based on the foregoing scheme, the image data processing method further includes:

[0057] If the color of the product defect image is inconsistent with the color of the product background image, the color of the product defect image shall be modified according to the color of the product background image.

[0058] In some embodiments of this application, based on the foregoing scheme, the defect image sample is used for defect detection of the target product.

[0059] According to a second aspect of the embodiments of this application, an image data processing apparatus is provided, comprising:

[0060] The image acquisition unit is used to acquire product defect images corresponding to the target category defects, as well as product background images of the target products;

[0061] A model acquisition unit is used to acquire a pre-built semantic model, wherein the semantic model has the ability to determine the occurrence areas of various types of defects in the product background image;

[0062] The target region determination unit is used to determine the target region corresponding to the target category defect in the product background image using the semantic model.

[0063] An image fusion unit is used to fuse the product defect image into the target area to obtain a defect image sample of the target product corresponding to the target category defect.

[0064] According to a third aspect of the present application, an image data processing apparatus is provided, including a processor and a memory, wherein the memory stores computer program instructions executable by the processor, and when the processor executes the computer program instructions, it implements the steps of the method described in any of the first aspects above.

[0065] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein computer program instructions are stored therein, and when executed by a processor, the computer program instructions cause the processor to perform the steps of the method as described in any of the first aspects above.

[0066] In this application, a product defect image corresponding to a target category defect and a product background image of the target product are acquired; a pre-constructed semantic model is acquired, wherein the semantic model has the ability to determine the occurrence area of ​​various categories of defects in the product background image; the target region corresponding to the target category defect is determined in the product background image using the semantic model; and the product defect image is fused to the target region to obtain a defect image sample of the target product corresponding to the target category defect. The technical solution provided in this application enables the automatic generation of defect image samples in industrial scenarios, facilitating the rapid initiation of industrial inspection projects.

[0067] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0069] Figure 1 A flowchart illustrating an image data processing method in one embodiment is shown;

[0070] Figure 2 It shows Figure 1 A diagram illustrating the generation process of defective image samples;

[0071] Figure 3 It shows Figure 1 Another illustration of the process of generating defective image samples;

[0072] Figure 4 It shows the use of Figure 1 A diagram comparing the effects of image samples generated by a semantic model and randomly generated image samples;

[0073] Figure 5 It shows Figure 1 A schematic diagram of the inflated region during the construction process of the semantic model;

[0074] Figure 6It shows Figure 1 A schematic diagram illustrating an application scenario of semantic models in Chinese;

[0075] Figure 7 A flowchart illustrating an image data processing method in another embodiment is shown;

[0076] Figure 8 It shows the method for generating Figure 7 A schematic diagram of an operation interface for multiple defective image samples;

[0077] Figure 9 It shows Figure 8 A schematic diagram of an operation interface for medium-batch deformation;

[0078] Figure 10 It shows Figure 8 A schematic diagram of an operation interface for medium-batch fusion;

[0079] Figure 11 It shows the method for generating Figure 1 A schematic diagram of an operation interface for a defective image sample;

[0080] Figure 12 It shows the method for generating Figure 1 A schematic diagram of another user interface for a defective image sample;

[0081] Figure 13 It shows the method for generating Figure 1 A schematic diagram of another user interface for defective image samples;

[0082] Figure 14 It shows the method for generating Figure 1 A schematic diagram of another user interface for defective image samples;

[0083] Figure 15 A block diagram of an image data processing apparatus in one embodiment is shown;

[0084] Figure 16 A schematic diagram of the structure of an image data processing device in one embodiment is shown. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0087] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0088] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0089] It should also be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such uses of these terms can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described.

[0090] To enable those skilled in the art to better understand this application, the application scenarios involved in this application will be briefly described first.

[0091] With the increasing maturity of artificial intelligence technology, industrial inspection demands higher standards for defect image samples in industrial scenarios. However, the rapidly changing market and frequent product iterations make it difficult to collect defect image samples in industrial settings. To address this, this application proposes an image data processing method that can automatically generate defect image samples in industrial scenarios. Furthermore, it can rapidly prepare large batches of defect image samples from a very small number of samples, enabling industrial inspection projects to start quickly even when the number of defect image samples is small or nonexistent, thus accelerating the product quality convergence cycle. This image data processing method is particularly beneficial for scenarios with a relatively large number of good products, relatively limited defect information, and where industrial inspection projects require rapid start-up and short delivery cycles.

[0092] Figure 1A flowchart illustrating an image data processing method in one embodiment is shown, such as... Figure 1 As shown, the image data processing method may include the following steps 101 to 104.

[0093] In step 101, the product defect image corresponding to the target category defect and the product background image of the target product are obtained.

[0094] Understandably, different products often have different types of appearance defects, and the corresponding product defect images are also different for different types of appearance defects. Taking display panels as an example, they may have defects such as fragments, bubbles, scratches, missing corners, and cracks, and the corresponding product defect images for these defects will also be different.

[0095] During the implementation process, product defect diagrams corresponding to the target category of defects can be obtained in various ways.

[0096] In some embodiments, an original product image containing a target category defect can be obtained; an image corresponding to the target category defect can be extracted from the original product image to obtain a product defect image.

[0097] Refer to together Figure 2 , Figure 2 It shows Figure 1 A diagram illustrating the generation process of defective image samples. (Example) Figure 2 As shown in the figure, from left to right, there are the original product image, the product defect image, and the defect image sample. By extracting the product defect image corresponding to the target category defect from the original product image and then augmenting it onto the product background image, a more realistic defect image sample can be obtained.

[0098] In other embodiments, an original product image containing the target category defect and a mask image containing the target category defect can be obtained, and the original product image and the mask image can be fused to obtain a product defect image.

[0099] Refer to together Figure 3 , Figure 3 It shows Figure 1 Another illustration of the process of generating defective image samples. (e.g.) Figure 3 As shown in the figure, from left to right, there are the original product image, the mask image, the product background image, and the defect image sample. By fusing the original product image with the mask image, a product defect image is obtained. Then, by augmenting it onto the product background image, a more realistic defect image sample can be obtained.

[0100] In some embodiments, product defect images corresponding to various categories of defects can be saved to a defect database in advance, and then product defect images corresponding to the target category of defects can be obtained from the defect database.

[0101] It is understood that the product background image is an image containing product information, which may be a product image or other images, and this application embodiment does not limit this.

[0102] In the implementation process, multiple background images of the target product that needs to be industrially inspected can be saved to the memory in advance, and then the background images of the target product can be retrieved from the memory.

[0103] In step 102, a pre-built semantic model is obtained, wherein the semantic model has the ability to determine the occurrence areas of various types of defects in the product background image.

[0104] It is understandable that the "occurrence area" refers to the region in the product background image where various types of defects occur. Taking the target product mentioned above as an example, assuming that Type I defects typically occur in regions A and B of the product background image, then the occurrence area of ​​Type I defects in the product background image could be regions A and B.

[0105] The inventors of this application have discovered that typical product appearance defects usually have a spatial relationship with their background. When using such defects to augment defect image samples, the spatial relationship between the product appearance defect and the background can be established first. Embodiments of this application can model the semantic relationship between the target object and its surrounding environment to obtain a semantic model. Then, based on the semantic model, the relative position of the product defect image corresponding to the target category defect in the product background image is determined, ultimately yielding a defect image sample that meets the requirements.

[0106] Refer to together Figure 4 , Figure 4 The diagram shows the results of generating image samples using two different methods. Figure 4 As can be seen, the method of randomly generating image samples involves randomly pasting the target object into the background image. Obviously, this method is not suitable for scenarios where the background image is complex and the target object appears in a specific location within the background image. In such scenarios, randomly generated image samples, when applied to industrial inspection based on deep learning networks, will not only fail to improve the deep learning capabilities but may even have the opposite effect, i.e., reduce the deep learning capabilities of the network. In contrast, the method of generating image samples using semantic models can place the target object in an appropriate position within the background image based on the semantic relationship between the target object and the background image.

[0107] In some embodiments, a semantic model can be constructed through the following steps: acquiring training image samples and determining the boundary information and category information of the target object in the training image samples; determining the circumscribed region corresponding to the target object in the training image samples based on the boundary information; acquiring a first image containing the circumscribed region; setting the RGB values ​​of each pixel in the circumscribed region of the first image to a first preset value to obtain a second image; and training the initial model based on the category information and the second image to obtain a semantic model.

[0108] In the implementation process, the training image samples can be obtained by processing images of products with various types of defects produced on the production line, or by processing product defect images that are similar to the defects that may occur in the target product and selected from the defect database, or even by processing images that do not contain defects or products.

[0109] When processing images of products with various types of defects produced on the production line, it is necessary to manually segment and label the target objects in the images (in this scenario, the target objects are various types of defects) to obtain product defect images. Then, the product defect images are augmented into other images to obtain training image samples.

[0110] When processing product defect images that are similar to the defects that may occur in the target product and selected from the defect database, the selected product defect images can be directly augmented into other images to obtain training image samples.

[0111] When processing images that do not contain defects or products, it is necessary to manually segment and label the target objects in the images to obtain target object images, and then augment the target object images into other images to obtain training image samples.

[0112] Annotation refers to marking the outer contour of a target object using a border of a freeform shape, polygon, spline curve, or other common geometric shape. The boundary information of the target object in the training image sample can be the border used to annotate the target object.

[0113] The circumscribed region can be a circumscribed rectangle, a circumscribed circle, etc. To achieve better training results, the circumscribed region can be the smallest circumscribed rectangle.

[0114] In the implementation process, the first preset value can be 0, 255 or other values. That is, the RGB values ​​of each pixel in the outer graphic region of the first image can be set to black, white or other colors to obtain the second image. Then, the category information and the second image are input into the initial model for training. When the model converges, the semantic model is obtained.

[0115] It should be noted that the above initial model can be a context-based convolutional neural network.

[0116] In some embodiments, the number of acquired training image samples can be positively correlated with the complexity of the target semantic relationship, where the target semantic relationship is the semantic relationship between the third image and the fourth image, the third image is the image corresponding to the target object in the training image samples, and the fourth image is the image other than the third image in the training image samples.

[0117] It can be understood that for images with a simple target semantic relationship, the number of acquired training image samples can be relatively small. For example, about 100 training image samples can be acquired. For images with a complex target semantic relationship, in order to obtain sufficient image information, the number of acquired training image samples needs to be larger. For example, about 1000 training image samples can be acquired.

[0118] In some embodiments, the circumscribed graphic region in the training image samples can be dilated to obtain a dilated region; an image is extracted from the dilated region in the training image samples to obtain a first image.

[0119] It can be understood that the dilated region is a region larger than the circumscribed graphic region. By extracting an image from the dilated region in the training image samples, more environmental information about the target object can be obtained from the training image samples.

[0120] In the implementation process, the circumscribed graphic region can be randomly dilated, or the length and width of the circumscribed graphic region can be enlarged proportionally, or the circumscribed graphic region can be dilated based on other algorithms. The embodiments of the present application do not limit the dilation method.

[0121] Refer to Figure 5 , Figure 5 shows Figure 1 a schematic diagram of the dilated region in the construction process of the semantic model in Figure 5 As shown, when the circumscribed graphic region is the minimum circumscribed rectangle region, the dilated region is the region defined by the dashed line box. The image defined by the dilated region can actually be a second image similar to a "hui" character. In some embodiments, the area of the dilated region is less than a second preset value to avoid excessive environmental information caused by too large an area of the dilated region, thereby affecting the accuracy of the semantic model.

[0122] Based on the semantic model constructed in the above steps, the occurrence regions of different categories of target objects in different background images can be determined, and thus it also has the ability to determine the occurrence regions of various types of defects in the product background images.

[0123] In step 103, the target region corresponding to the target category defect is determined in the product background image using a semantic model.

[0124] In the implementation process, the product background image can be processed based on processing steps similar to those used in the construction of the semantic model. The processed image can then be input into the semantic model for semantic relationship reasoning to determine the target area corresponding to the target category defect.

[0125] In some embodiments, multiple candidate regions and an expansion region corresponding to each candidate region can be determined in the product background image, wherein the RGB values ​​of each pixel in the candidate region are set to a first preset value; the expansion region corresponding to each candidate region is scored by a semantic model to obtain a first score; and the target region corresponding to the target category defect is determined from the multiple candidate regions based on the first score of the expansion region corresponding to each candidate region.

[0126] In the implementation process, multiple candidate boxes can be generated in the product background image. The area defined by the candidate box is used as the candidate region. Correspondingly, the expansion region is the area defined by the outer box corresponding to the candidate box. The number of candidate regions can be 100, 200 or other values. One candidate region corresponds to one expansion region, or multiple expansion regions can be used.

[0127] It should be noted that during the training of the semantic model, the RGB values ​​of each pixel in the outer graphic region of the first image are set to the first preset value. When using the semantic model for inference, the RGB values ​​of each pixel in the candidate region also need to be set to the first preset value.

[0128] Understandably, after processing the product background image as described above, it can be input into the semantic model to obtain the first score of the dilated region corresponding to each candidate region output by the semantic model.

[0129] In some embodiments, there may be multiple inflated regions corresponding to each candidate region. The first score is used to characterize the probability of a target category defect occurring in the corresponding candidate region. Based on the first score of the inflated region corresponding to each candidate region, the target region corresponding to the target category defect is determined from multiple candidate regions. This may include: taking the average of the first scores of the multiple inflated regions corresponding to each candidate region as the second score corresponding to each candidate region; and determining the candidate region with the highest second score as the target region corresponding to the target category defect.

[0130] Taking a candidate region as an example, with three regions A, B, and C, and each candidate region corresponding to three inflated regions, if the semantic model outputs the first scores of the inflated regions corresponding to region A as 0.4, 0.5, and 0.6, the first scores of the inflated regions corresponding to region B as 0.45, 0.55, and 0.68, and the first scores of the inflated regions corresponding to region A as 0.60, 0.51, and 0.60, then the average of the first scores of the three inflated regions corresponding to region A is 0.50, meaning the second score for region A is 0.50. The average of the first scores of the three inflated regions corresponding to region B is 0.56, meaning the second score for region B is 0.56. The average of the first scores of the three inflated regions corresponding to region C is 0.57, meaning the second score for region C is 0.57. Since region C has the highest second score, region C is selected as the target region corresponding to the target category defect.

[0131] In other embodiments, there may be multiple inflated regions corresponding to each candidate region, and multiple first scores corresponding to each inflated region. Each first score is used to characterize the probability of a corresponding candidate category defect occurring in the corresponding candidate region. Based on the first scores of the inflated regions corresponding to each candidate region, the target region corresponding to the target category defect is determined from the multiple candidate regions, including: taking the average of the first scores of the multiple inflated regions corresponding to each candidate region on each candidate category defect as the second score of each candidate region for the occurrence of the corresponding candidate category defect; determining the candidate category defect with the highest second score corresponding to each candidate region; and if the candidate category defect with the highest second score is the target category defect, determining the candidate region corresponding to the candidate category defect with the highest second score as the target region corresponding to the target category defect.

[0132] It is understandable that if each inflated region corresponds to a first score for each candidate category defect, then the n inflated regions corresponding to the same candidate region will correspond to n first scores for that candidate category defect. At this time, it is necessary to calculate the average of these n first scores and use the calculated average as the second score for the candidate region to have that candidate category defect.

[0133] Taking the candidate regions as candidate region A and candidate region B, with the number of dilated regions corresponding to each candidate region being 2, and taking the target category defect as scratch as an example, if the first score of dilated region A1 corresponding to candidate region A output by the semantic model on the candidate category defect of scratch is 0.4, and the first score on the candidate category defect of crack is 0.5, and the first score of dilated region A2 corresponding to candidate region A output by the semantic model on the candidate category defect of scratch is 0.3, and the first score on the candidate category defect of crack is 0.6, then calculate the average value of the first score of dilated region A1 on the candidate category defect of scratch, obtaining 0.35, and use it as the second score for candidate region A to be classified as scratch for the candidate category defect; calculate the average value of the first score of dilated region A1 on the candidate category defect of crack, obtaining 0.56, and use it as the second score for candidate region A to be classified as crack for the candidate category defect.

[0134] Since the second score for candidate region A to be classified as crack for the candidate category defect is greater than the second score for candidate region A to be classified as scratch for the candidate category defect, the candidate category defect with the highest second score corresponding to candidate region A is determined to be crack.

[0135] Determine the candidate category defect with the highest second score corresponding to candidate region B in the same way. If the candidate category defect with the highest second score corresponding to candidate region B is scratch, then determine candidate region B as the target region.

[0136] Refer to Figure 6 , Figure 6 shows Figure 1 a schematic diagram of an application scenario of the semantic model in Figure 5 As shown, first, annotate the target object in the original image sample to obtain the bounding box of the target object, obtain the training image sample, and determine the category information of the target object; secondly, according to the bounding box, determine the minimum bounding rectangle region corresponding to the target object in the training image sample, perform random dilation processing on the minimum bounding rectangle region to obtain the dilated region, extract the image from the dilated region in the training image sample to obtain the first image containing the minimum bounding rectangle region, and set the RGB values of each pixel of the minimum bounding rectangle region in the first image to 0 to obtain the second image in the shape of a "hui" character; thirdly, based on the category information and the second image, train the initial model to obtain the semantic model.

[0137] After obtaining the semantic model, multiple candidate regions can be determined in the background image, and the semantic model is used to determine the target regions corresponding to different categories of target objects from the multiple candidate regions, and the images corresponding to different categories of target objects are fused into the target regions corresponding to the target objects to obtain the image sample for use in the later industrial detection scenario.

[0138] In step 104, the product defect image is fused to the target area to obtain a defect image sample of the target product corresponding to the target category defect.

[0139] During the implementation process, images can be fused in various ways to make the final defective image sample smoother.

[0140] In some embodiments, the idea of ​​Poisson fusion can be adopted to determine the first gradient field of the product defect image and the second gradient field of the product background image; the gradient field corresponding to the target region in the second gradient field is replaced with the first gradient field to obtain the third gradient field of the initial fused image; the transformation matrix coefficients are determined according to the third gradient field; and defect image samples of the target product corresponding to the target category defect are constructed according to the transformation matrix coefficients.

[0141] In the implementation process, the median difference method can be used to calculate the horizontal and vertical gradient fields of the product defect image. Then, the sum of these two gradient fields is used to convolve the RGB values ​​of the pixels in the product defect image to obtain the first gradient field of the product defect image. The second gradient field of the product background image can be calculated using the same method.

[0142] By replacing the gradient field, the third gradient field of the initial fused image P' can be obtained. Taking the partial derivative of the third gradient field, the divergence Dori of the initial fused image P' can be obtained. The equation M×Y=Dori is constructed, where Y is the transformation matrix coefficient to be solved, M can be a matrix with a center of -4, diagonals of 1, and a size the same as the image size, and Dori is the divergence of the initial fused image P'. The transformation matrix coefficients Y can be obtained through the above equation. Then, by multiplying the transformation matrix coefficients Y by the RGB values ​​of each pixel in the initial fused image P', the RGB values ​​of the pixels in the target fused image P (i.e., the defective image sample) are obtained, thus yielding the defective image sample.

[0143] In the above image data processing method, by using a semantic model to determine the target region of the product defect image corresponding to the target category defect in the product background image of the target product, not only is the automatic generation of defect image samples in industrial scenarios realized, but the generated defect image samples can also meet the requirements for later use. When used for neural network learning, it can improve the deep learning capability of the network.

[0144] Figure 7 A flowchart illustrating an image data processing method in another embodiment is shown. Figure 7 As shown, the image data processing method may further include the following steps:

[0145] Step 701: Perform multiple deformation processing on the product defect image to obtain multiple fifth images;

[0146] Step 702: Fuse multiple fifth images into the target region to obtain multiple defect image samples of the target product corresponding to the target category defects.

[0147] Understandably, to increase the diversity of defect image samples, defect image samples can be prepared in batches by deforming the product defect images. Deformation refers to geometric transformations such as rotation, stretching, and scaling.

[0148] In some embodiments, the number of product background images of the target product is 1. The product defect image is subjected to N1 deformation processes. Then, the N1 fifth images obtained after deformation are fused with 1 product background image to obtain N1 defect image samples, where N1 is a positive integer greater than 1.

[0149] In some embodiments, the image data processing method may further include the following steps: acquiring an original product image containing a target category defect selected by the user based on a first control; generating a mask image containing a target category defect based on the original product image in response to the user's operation on a second control; fusing the original product image and the mask image to obtain a product defect image; and performing a step of performing multiple deformation processing on the product defect image in response to the user's operation on a third control.

[0150] Figure 8 It shows the method for generating Figure 7 A schematic diagram of an operation interface for multiple defective image samples. (Example) Figure 8 As shown, users can click the first control (i.e., open image control) to select the original product image containing the target category defect, then click the second control (i.e. generate MASK control) to generate a mask image containing the target category defect, and then click the third control (i.e. batch deformation control) to achieve N deformation processing of the product defect image corresponding to the target category defect.

[0151] In some embodiments, in response to user operation of a third control, a batch deformation interface can also be displayed; based on the deformation intensity and deformation quantity input by the user in the batch deformation interface, the product defect image is subjected to multiple deformation processes.

[0152] Figure 9 It shows Figure 8 A schematic diagram of an operation interface for medium-batch deformation. For example... Figure 9As shown, users can select the save path, deformation intensity, and deformation quantity. The deformation quantity determines the final number of defect image samples generated. After selection, clicking the batch deformation control allows users to view the progress of batch deformation in real time via a progress bar. After batch deformation is complete, the software can create an "image" directory to store the deformed product defect images. Clicking the result filtering control opens the "image" folder, where users can view the deformation effect and delete images with unsatisfactory deformation results. In other embodiments, after batch deformation is complete, the software can also create two directories, "image" and "mask," to store the product image containing the deformed product defect image and the corresponding mask image, respectively. Clicking the result filtering control opens the "image" folder, where users can view the deformation effect and delete images with unsatisfactory deformation results. Subsequently, based on the filtered "image" folder, the corresponding "mask" folder is cleaned to ensure a one-to-one correspondence between the images in the "image" and "mask" folders.

[0153] Figure 10 It shows Figure 8 This is a schematic diagram of an interface for batch fusion. Users can... Figure 8 Click the batch fitting control in the operation interface shown, and in the pop-up window... Figure 10 In the interface shown, select the good product path, the result path, and / or the fusion location. The good product path is the storage path for the product background image, the result path is the storage path for the defective image samples, and the fusion location can be automatically determined based on the semantic model or selected by the user. The number of defective image samples generated by fusion is equal to the product of the number of deformations and the number of product background images.

[0154] In some embodiments, the number of product background images of the target product is M1. The product defect images are subjected to N2 deformation processes. Then, the N2 fifth images obtained after deformation are fused with the M1 product background images respectively to obtain N2×M1 defect image samples, where M1 and N2 are both positive integers greater than 1.

[0155] In some embodiments, the number of product defect images corresponding to the target category defect is N3, and the number of product background images of the target product is M2. The N3 product defect images are all subjected to K deformation processes. Then, the N3×K fifth images obtained after deformation are fused with the M2 product background images respectively to obtain N3×K×M2 defect image samples, where N3, K and M2 are all positive integers greater than 1.

[0156] During the implementation process, users can prepare defect image samples and mask images corresponding to product defect images in the defect image samples in advance, and store them in the image folder and mask folder respectively, with one-to-one correspondence between defect image samples and mask images.

[0157] Users can base their decisions on the following: Figure 9 Using the same or similar operation interface, you can select the save path, deformation intensity, and deformation quantity to obtain the product defect image after deformation.

[0158] Users can also base their decisions on... Figure 10 Users can select the defect path, good product path, result path, and / or fusion location from similar or identical user interfaces. The fusion location can be automatically determined based on a semantic model or selected by the user. The number of defect image samples generated by fusion is equal to the product of the number of defect image samples, the number of deformations, and the number of product background images. Users can also view, filter, and organize the fused defect image samples.

[0159] By preparing defective image samples in batches, the diversity of defective image samples is increased, and the preparation efficiency of defective image samples is improved.

[0160] In some embodiments, the image data processing method may further include the step of: modifying the color of the product defect image according to the color of the product background image when the color of the product defect image is inconsistent with the color of the product background image.

[0161] It's understandable that the colors of the product defect image and the product background image may not be the same. When the product defect image is grayscale, the product background image may be grayscale or color, and vice versa. To better integrate the product defect image into the target area of ​​the product background image, the colors of the product defect image can be modified.

[0162] Figure 11 It shows the method for generating Figure 1 A schematic diagram of an operation interface for a defective image sample, such as... Figure 11 As shown, when both the background image and the defective product image are grayscale, the color of the defective product image does not need to be modified.

[0163] Figure 12 It shows the method for generating Figure 1 Another schematic diagram of the user interface for a defective image sample, such as... Figure 12 As shown, when the product defect image is grayscale while the product background image is color, the color of the product defect image can be changed to color.

[0164] Figure 13 It shows the method for generating Figure 1 Another schematic diagram of the user interface for a defective image sample, such as... Figure 13 As shown, when the product defect image is in color and the product background image is in grayscale, the color of the product defect image can be changed to grayscale.

[0165] Figure 14 It shows the method for generating Figure 1 Another user interface diagram for defective image samples, such as... Figure 14 As shown, when both the product background image and the product defect image are in color, the color of the product defect image does not need to be modified.

[0166] By adaptively modifying the color of the product defect image based on the color of the product background image, the product defect image can be better integrated into the target area of ​​the product background image.

[0167] In some embodiments, defect image samples can be used for defect detection of a target product.

[0168] Understandably, in the industrial inspection sector, there are various methods for defect detection of target products, with deep learning networks being a common approach. After obtaining defect image samples, these samples can be applied to a deep learning network. Because the various types of defects in the image samples are located in appropriate positions, and the image samples meet the requirements of deep learning networks for training data, these defect image samples can improve the deep learning capabilities of the network.

[0169] It should be noted that the current problem of difficulty in collecting defect image samples in industrial scenarios can be solved by the image data processing method proposed in this application. This application can automatically generate defect image samples corresponding to various types of defects in the target product, and can quickly and in large quantities prepare defect image samples. Thus, industrial inspection of the target product can be initiated based on the generated defect image samples, achieving the goal of accelerating the deployment of industrial inspection projects.

[0170] The following describes an apparatus embodiment of this application, which can be used to execute the image data processing method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the image data processing method described above in this application.

[0171] See Figure 15 The diagram shows a block diagram of an image data processing apparatus according to an embodiment of this application.

[0172] like Figure 15As shown, the image data processing apparatus of this application embodiment includes: an image acquisition unit 1501, a model acquisition unit 1502, a target region determination unit 1503, and an image fusion unit 1504. The image acquisition unit 1501 is used to acquire a product defect image corresponding to a target category defect, and a product background image of the target product. The model acquisition unit 1502 is used to acquire a pre-constructed semantic model, wherein the semantic model has the ability to determine the occurrence area of ​​various categories of defects in the product background image. The target region determination unit 1503 is used to determine the target region corresponding to the target category defect in the product background image using the semantic model. The image fusion unit 1504 is used to fuse the product defect image into the target region to obtain a defect image sample of the target product corresponding to the target category defect.

[0173] In some embodiments, the model acquisition unit 1502 is further configured to acquire training image samples and determine the boundary information and category information of the target object in the training image samples; determine the circumscribed graphic region corresponding to the target object in the training image samples based on the boundary information; acquire a first image containing the circumscribed graphic region; set the RGB values ​​of each pixel of the circumscribed graphic region in the first image to a first preset value to obtain a second image; and train the initial model based on the category information and the second image to obtain a semantic model.

[0174] In some embodiments, the model acquisition unit 1502 is further configured to acquire a first image containing a circumscribed graphic region, including: performing dilation processing on the circumscribed graphic region in the training image sample to obtain a dilated region; and extracting an image from the dilated region in the training image sample to obtain the first image.

[0175] In some embodiments, the area of ​​the expansion region is smaller than a second preset value.

[0176] In some embodiments, the number of training image samples acquired is positively correlated with the complexity of the target semantic relationship, wherein the target semantic relationship is the semantic relationship between the third image and the fourth image, the third image is the image corresponding to the target object in the training image samples, and the fourth image is the image other than the third image in the training image samples.

[0177] In some embodiments, the target region determination unit 1503 is further configured to determine multiple candidate regions and an expansion region corresponding to each candidate region in the product background image, wherein the RGB value of each pixel of the candidate region is set to a first preset value; score the expansion region corresponding to each candidate region by a semantic model to obtain a first score; and determine the target region corresponding to the target category defect from the multiple candidate regions based on the first score of the expansion region corresponding to each candidate region.

[0178] In some embodiments, there are multiple inflated regions corresponding to each candidate region. The first score is used to characterize the probability of a target category defect occurring in the corresponding candidate region. The target region determination unit 1503 is also used to take the average of the first scores of the multiple inflated regions corresponding to each candidate region as the second score corresponding to each candidate region; and to determine the candidate region with the highest second score as the target region corresponding to the target category defect.

[0179] In some embodiments, each candidate region has multiple inflated regions, and each inflated region has multiple first scores. Each first score is used to characterize the probability of a corresponding candidate category defect occurring in the corresponding candidate region. The target region determination unit 1503 is further used to take the average of the first scores of the multiple inflated regions corresponding to each candidate region on each candidate category defect as the second score of each candidate region for each candidate category defect; based on the second score of each candidate region for each candidate category defect, the candidate category defect with the highest second score is determined as the target category defect corresponding to each candidate region; and the candidate region corresponding to the target category defect is determined as the target region corresponding to the target category defect.

[0180] In some embodiments, the image acquisition unit 1501 is further configured to acquire an original product image containing a target category defect and a mask image containing a target category defect; and fuse the original product image and the mask image to obtain a product defect image.

[0181] In some embodiments, the image acquisition unit 1501 is further configured to acquire an original product image containing a target category defect; and extract an image corresponding to the target category defect from the original product image to obtain a product defect image.

[0182] In some embodiments, the image fusion unit 1504 is further configured to determine a first gradient field of the product defect image and a second gradient field of the product background image; replace the gradient field corresponding to the target region in the second gradient field with the first gradient field to obtain a third gradient field of the initial fused image; determine the transformation matrix coefficients based on the third gradient field; and construct defect image samples of the target product corresponding to the target category defect based on the transformation matrix coefficients.

[0183] In some embodiments, the image data processing apparatus further includes a batch preparation unit (not shown) for performing multiple deformation processing on the product defect image to obtain multiple fifth images; and fusing the multiple fifth images into the target region to obtain multiple defect image samples of the target product corresponding to the target category of defects.

[0184] In some embodiments, the batch preparation unit is further configured to acquire an original product image containing a target category defect selected by the user based on a first control; generate a mask image containing a target category defect based on the original product image in response to the user's operation on a second control; fuse the original product image and the mask image to obtain a product defect image; and perform a step of performing multiple deformation processing on the product defect image in response to the user's operation on a third control.

[0185] In some embodiments, the batch preparation unit is further configured to display a batch deformation interface in response to a user's operation on a third control; and to perform a step of performing multiple deformation processing on the product defect image based on the deformation intensity and deformation quantity input by the user in the batch deformation interface.

[0186] In some embodiments, the image data processing apparatus further includes an image fitting unit (not shown) for modifying the color of the product defect image according to the color of the product background image when the color of the product defect image is inconsistent with the color of the product background image.

[0187] In some embodiments, defect image samples are used for defect detection of a target product.

[0188] Based on the same inventive concept, embodiments of this application also provide an image data processing device, see reference. Figure 16 The diagram shows a schematic of the structure of an image data processing device according to an embodiment of this application. The image data processing device includes one or more memories 1604, one or more processors 1602, and at least one computer program (computer program instructions) stored in the memory 1604 and executable on the processor 1602. When the processor 1602 executes the computer program, it implements the method described above.

[0189] Among them, Figure 16 In this document, a bus architecture (represented by bus 1600) is used. Bus 1600 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1602 and memory represented by memory 1604. Bus 1600 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1605 provides an interface between bus 1600 and receiver 1601 and transmitter 1603. Receiver 1601 and transmitter 1603 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 1602 is responsible for managing bus 1600 and general processing, while memory 1604 can be used to store data used by processor 1602 during operation.

[0190] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method described above.

[0191] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0193] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program instructions, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0195] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for processing image data, characterized in that, include: Obtain the product defect image corresponding to the target category defect, as well as the product background image of the target product; Obtain a pre-built semantic model, wherein the semantic model has the ability to determine the occurrence areas of various categories of defects in the product background image; The semantic model is used to determine the target region corresponding to the target category defect in the product background image; The product defect image is fused to the target region to obtain a defect image sample of the target product corresponding to the target category defect; wherein, the semantic model is constructed through the following steps: Acquire training image samples and determine the boundary information and category information of the target objects in the training image samples; Based on the boundary information, the circumscribed region of the target object is determined in the image sample to be trained. Obtain a first image containing the external graphics region; Set the RGB values ​​of each pixel in the outer graphic region of the first image to a first preset value to obtain the second image; Based on the category information and the second image, the initial model is trained to obtain the semantic model; The step of using the semantic model to determine the target region corresponding to the target category defect in the product background image includes: Multiple candidate regions and an expansion region corresponding to each candidate region are determined in the product background image, wherein the RGB value of each pixel in the candidate region is set to the first preset value; The semantic model is used to score the dilated region corresponding to each candidate region to obtain a first score; Based on the first score of the expansion region corresponding to each candidate region, the target region corresponding to the target category defect is determined from the multiple candidate regions.

2. The image data processing method according to claim 1, characterized in that, The step of acquiring the first image containing the outer graphic region includes: The circumscribed region of the image sample to be trained is dilated to obtain the dilated region; The first image is obtained by extracting the image from the dilated region in the image sample to be trained.

3. The image data processing method according to claim 2, characterized in that, The area of ​​the expansion region is smaller than the second preset value.

4. The image data processing method according to claim 1, characterized in that, The number of training image samples acquired is positively correlated with the complexity of the target semantic relationship. The target semantic relationship is the semantic relationship between the third image and the fourth image. The third image is the image corresponding to the target object in the training image samples, and the fourth image is the image in the training image samples other than the third image.

5. The image data processing method according to claim 1, characterized in that, There are multiple inflated regions corresponding to each candidate region. The first score is used to characterize the probability that the target category defect occurs in the corresponding candidate region. The step of determining the target region corresponding to the target category defect from multiple candidate regions based on the first score of the inflated region corresponding to each candidate region includes: The average of the first scores of the multiple dilated regions corresponding to each candidate region is taken as the second score corresponding to each candidate region; The candidate region with the highest second score is determined as the target region corresponding to the defect of the target category.

6. The image data processing method according to claim 1, characterized in that, There are multiple expansion regions corresponding to each candidate region, and multiple first scores corresponding to each expansion region. Each first score is used to characterize the probability of a corresponding candidate category defect occurring in the corresponding candidate region. The step of determining the target region corresponding to the target category defect from the multiple candidate regions based on the first scores of the expansion regions corresponding to each candidate region includes: The average of the first scores of the multiple inflated regions corresponding to each candidate region on each candidate category defect is taken as the second score of each candidate region for the corresponding candidate category defect. Determine the candidate category defect with the highest second score for each of the candidate regions; If the candidate category defect with the highest second score is the target category defect, the candidate region corresponding to the candidate category defect with the highest second score is determined as the target region corresponding to the target category defect.

7. The image data processing method according to claim 1, characterized in that, The step of obtaining the product defect image corresponding to the target category defect includes: Obtain the original product image containing the target category defect, and the mask image containing the target category defect; The original product image is fused with the mask image to obtain the product defect image.

8. The image data processing method according to claim 1, characterized in that, The step of obtaining the product defect image corresponding to the target category defect includes: Obtain the original image of the product containing the defects of the target category; The image corresponding to the target category defect is extracted from the original product image to obtain the product defect image.

9. The image data processing method according to claim 7 or 8, characterized in that, The step of fusing the product defect image into the target region to obtain a defect image sample of the target product corresponding to the target category defect includes: Determine the first gradient field of the product defect image and the second gradient field of the product background image; Replace the gradient field corresponding to the target region in the second gradient field with the first gradient field to obtain the third gradient field of the initial fused image; The transformation matrix coefficients are determined based on the third gradient field; Based on the transformation matrix coefficients, construct defect image samples of the target product corresponding to the target category defects.

10. The image data processing method according to claim 1, characterized in that, Also includes: The product defect image is subjected to multiple deformation processes to obtain multiple fifth images; Multiple fifth images are respectively fused into the target region to obtain multiple defect image samples of the target product corresponding to the target category defects.

11. The image data processing method according to claim 10, characterized in that, Also includes: Obtain the original image of the product containing the target category defect selected by the user based on the first control; In response to the user's operation on the second control, a mask image containing the target category defect is generated based on the original product image; The original product image is fused with the mask image to obtain the product defect image; In response to the user's operation on the third control, the step of performing multiple deformation processing on the product defect image is executed.

12. The image data processing method according to claim 11, characterized in that, The step of performing multiple deformation processing on the product defect image in response to the user's operation on the third control includes: In response to the user's operation on the third control, a batch deformation interface is displayed; Based on the deformation intensity and deformation quantity input by the user in the batch deformation interface, the step of performing multiple deformation processing on the product defect image is executed.

13. The image data processing method according to claim 1, characterized in that, Also includes: If the color of the product defect image is inconsistent with the color of the product background image, the color of the product defect image shall be modified according to the color of the product background image.

14. The image data processing method according to claim 1, characterized in that, The defect image samples are used for defect detection of the target product.

15. An image data processing apparatus, characterized in that, include: The image acquisition unit is used to acquire product defect images corresponding to the target category defects, as well as product background images of the target products; A model acquisition unit is used to acquire a pre-built semantic model, wherein the semantic model has the ability to determine the occurrence areas of various types of defects in the product background image; The target region determination unit is used to determine the target region corresponding to the target category defect in the product background image using the semantic model. An image fusion unit is used to fuse the product defect image into the target region to obtain a defect image sample of the target product corresponding to the target category of defect; wherein... The model acquisition unit is also used to acquire training image samples and determine the boundary information and category information of the target object in the training image samples; determine the circumscribed graphic region corresponding to the target object in the training image samples based on the boundary information; acquire a first image containing the circumscribed graphic region; set the RGB values ​​of each pixel of the circumscribed graphic region in the first image to a first preset value to obtain a second image; and train the initial model based on the category information and the second image to obtain a semantic model. The target region determination unit is further configured to determine multiple candidate regions and an expansion region corresponding to each candidate region in the product background image, wherein the RGB values ​​of each pixel of the candidate region are set to the first preset value; score the expansion region corresponding to each candidate region using the semantic model to obtain a first score; and determine the target region corresponding to the target category defect from the multiple candidate regions based on the first score of the expansion region corresponding to each candidate region.

16. An image data processing device, comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the steps of the method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 14.

Images