Data processing method, device, electronic device, medium and program product

By automatically processing the industrial product images, using the initial defect detection model to determine candidate areas and train the object detection model, the problems of low manual detection efficiency and poor accuracy in the prior art are solved, and efficient and accurate defect detection is achieved.

CN117036227BActive Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211154727.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-21
Publication Date
2024-12-06
Estimated Expiration
2042-09-21

AI Technical Summary

Technical Problem

The quality inspection of existing industrial defects mainly relies on manual visual inspection, which is inefficient and has low accuracy, and cannot meet the needs of automation and intelligence.

Method used

By acquiring the sample product image, calling the initial defect detection model to determine the candidate region, sampling based on the foreground loss characteristics, training the target defect detection model to realize automated defect detection.

Benefits of technology

It improves the efficiency and accuracy of defect detection, reduces manual intervention, and meets the automation and intelligence needs of industrial defect detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117036227B_ABST
    Figure CN117036227B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose data processing methods, devices, electronic devices, media and program products, which can be applied to the field of data processing technology. The method includes: calling the initial defect detection model to determine multiple candidate areas in the sample product image and the foreground probability of each candidate area; determining multiple first sampling candidate areas according to the foreground loss characteristics of each positive candidate area; sampling multiple second sampling candidate areas from multiple first sampling candidate areas, and calling the initial defect detection model to determine the classification probability sets and position offset information corresponding to the multiple second sampling candidate areas, so as to train the initial defect detection model, and use the trained target defect detection model to perform defect detection. The use of the embodiments of the present application helps to improve the efficiency and accuracy of defect detection of products. The embodiments of the present application can also be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, assisted driving, and smart home appliances.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to data processing methods, devices, electronic devices, media and program products. Background Art

[0002] Industrial defect quality inspection refers to the quality inspection of industrial products in the manufacturing process, that is, the detection of defects in products. At present, the method of detecting defects in products is generally manual visual inspection by relevant personnel. This method requires a lot of human resources, is inefficient, and has low accuracy. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device, electronic device, medium and program product, which help to improve the efficiency and accuracy of product defect detection.

[0004] On the one hand, an embodiment of the present application discloses a data processing method, the method comprising:

[0005] Acquire a sample product image, and call an initial defect detection model to determine a plurality of candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0006] Determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0007] Sampling the plurality of first sampling candidate regions to obtain a plurality of second sampling candidate regions, and calling the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions; the classification probability set includes the probability that the second sampling candidate region belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate region and the sample defect region;

[0008] The initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and the target defect detection model is used to perform defect detection on the product image.

[0009] In one aspect, an embodiment of the present application discloses a data processing device, the device comprising:

[0010] An acquisition unit, used for acquiring a sample product image, and calling an initial defect detection model to determine a plurality of candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0011] A processing unit, configured to determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine a plurality of first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0012] The processing unit is further used to sample from the multiple first sampling candidate areas to obtain multiple second sampling candidate areas, and call the initial defect detection model to determine the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas; the classification probability set includes the probability that the second sampling candidate area belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate area and the sample defect area;

[0013] The processing unit is further used to train the initial defect detection model according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and use the target defect detection model to perform defect detection on the product image.

[0014] On the one hand, an embodiment of the present application provides an electronic device, the electronic device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to perform the following steps:

[0015] Acquire a sample product image, and call an initial defect detection model to determine a plurality of candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0016] Determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0017] Sampling the plurality of first sampling candidate regions to obtain a plurality of second sampling candidate regions, and calling the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions; the classification probability set includes the probability that the second sampling candidate region belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate region and the sample defect region;

[0018] The initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and the target defect detection model is used to perform defect detection on the product image.

[0019] On the one hand, an embodiment of the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed by a processor, the computer program instructions are used to perform the following steps:

[0020] Acquire a sample product image, and call an initial defect detection model to determine a plurality of candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0021] Determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0022] Sampling the plurality of first sampling candidate regions to obtain a plurality of second sampling candidate regions, and calling the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions; the classification probability set includes the probability that the second sampling candidate region belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate region and the sample defect region;

[0023] The initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and the target defect detection model is used to perform defect detection on the product image.

[0024] In one aspect, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions. When the computer instructions are executed by a processor, the method provided in the above aspect can be implemented.

[0025] By using the embodiments of the present application, a sample product image can be obtained, and an initial defect detection model can be called to determine multiple candidate areas in the sample product image, thereby determining multiple first sampling candidate areas from multiple forward candidate areas based on the foreground loss characteristics of the multiple candidate areas, and then sampling multiple second sampling candidate areas from the multiple sampling candidate areas, so as to train the initial defect detection model based on the multiple second sampling candidate areas. For example, the initial defect detection model can be called to determine the classification probability set and position offset information corresponding to each second sampling candidate area, and the initial defect detection model can be trained based on the classification probability set and position offset information corresponding to each second sampling candidate area, and then defect detection can be performed on the product image based on the trained target defect detection model. On the one hand, the present application can train a target defect detection model for automatic defect detection of products, which meets the needs of automation and intelligence of product defect detection and improves the efficiency of product defect detection; on the other hand, when training the defect detection product model, the generated multiple candidate areas can be sampled twice, that is, multiple first sampling candidate areas can be determined based on the foreground loss feature to accelerate the convergence of the model, and multiple second sampling candidate areas can be sampled again from the multiple first sampling candidate areas to reduce the impact of incorrectly labeled sample defect areas in the sample product image on the model training, thereby improving the accuracy of the trained model and helping to improve the accuracy of product defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0028] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0029] Figure 3 This is a schematic diagram of the effect of a sample product image provided by an embodiment of the present application;

[0030] Figure 4 This is a schematic diagram of the effect of a sample product image provided by an embodiment of the present application;

[0031] Figure 5 is a schematic diagram of a training process of an initial defect detection model provided in an embodiment of the present application;

[0032] Figure 6 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0033] Figure 7 is a schematic diagram of a defect detection process provided by an embodiment of the present application;

[0034] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0035] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0037] The embodiment of the present application proposes a data processing scheme, which can obtain a sample product image and call an initial defect detection model to determine multiple candidate areas in the sample product image, thereby determining multiple first sampling candidate areas from multiple forward candidate areas based on the foreground loss characteristics of the multiple candidate areas, and then sampling multiple second sampling candidate areas from the multiple sampling candidate areas, so as to facilitate training the initial defect detection model based on the multiple second sampling candidate areas. For example, the initial defect detection model can be called to determine the classification probability set and position offset information corresponding to each second sampling candidate area, and the initial defect detection model can be trained based on the classification probability set and position offset information corresponding to each second sampling candidate area, and then defect detection can be performed on the product image based on the trained target defect detection model. On the one hand, the present application can train a target defect detection model for automatic defect detection of products, which meets the needs of automation and intelligence of product defect detection and improves the efficiency of product defect detection; on the other hand, when training the defect detection product model, the generated multiple candidate areas can be sampled twice, that is, multiple first sampling candidate areas can be determined based on the foreground loss feature to accelerate the convergence of the model, and multiple second sampling candidate areas can be sampled again from the multiple first sampling candidate areas to reduce the impact of incorrectly labeled sample defect areas in the sample product image on the model training, thereby improving the accuracy of the trained model and helping to improve the accuracy of product defect detection.

[0038] In a possible implementation, the embodiment of the present application can be applied to a data processing system. Figure 1 , Figure 1: is a structural diagram of a data processing system provided by an embodiment of the present application, and the data processing system may include a client and a data processing background. The client can be used to select a product image that needs to be defect-detected, and then the client uploads the selected product image to the data processing background for defect detection. The product image for defect detection may be a photographed image of the product that needs to be defect-detected, and the product image may be stored in a storage area after shooting, and then the user may select a product image from the storage area and upload it to the data processing background for defect detection; the product image may also be an image taken on site, that is, the user may take a photo of the product on site and upload the photographed image to the data processing background for defect detection. In one embodiment, the product that needs to be defect-detected may be an industrial product, such as an industrial intermediate product such as a valve, a pipe fitting, an instrument, a tool, or a final industrial product such as an electrical appliance, equipment, etc., which is not limited here. The data processing background may be the device where the client is located, or may not be the device where the client is located (such as a server or other user device), which is not limited here.

[0039] The data processing background can execute the above data processing scheme to obtain the target defect detection model, and then after receiving the product image sent by the client, use the target defect detection model to perform defect detection on the product image to obtain the defect detection result. Optionally, the defect detection result can indicate the area in the product image where there are defects and the defect category of the defects in the area, and then the defect detection result can be sent to the client. After receiving the defect detection result, the client can display the product image marked with defect areas and defect categories for the user to view the defects detected in the product image. It can be understood that the use of this data processing system for defect detection meets the automation and intelligent requirements of product defect detection and improves the accuracy of product defect detection; and, because when training the defect detection product model, the generated multiple candidate areas can be sampled twice, so as to reduce the impact of the sample defect area with incorrect labeling in the sample product image on the model training, improve the accuracy of the trained model, and help improve the accuracy of product defect detection.

[0040] In one possible implementation, the embodiments of the present application can also be applied to the field of artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0043] The technical solution of the present application can be used in electronic devices, such as the above-mentioned data processing devices. The electronic device can be a terminal, a server, or other devices for data processing, which is not limited in the present application. Optional. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, smart speakers, smart home appliances, etc.

[0044] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] Based on the above description, the present application embodiment proposes a data processing method. Figure 2 , Figure 2 1 is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by the above-mentioned electronic device. The data processing method can include the following steps.

[0046] S201, obtaining a sample product image, and calling an initial defect detection model to determine multiple candidate regions in the sample product image and a foreground probability of each candidate region.

[0047] The sample product image may be a product image used to train the initial defect detection model, and the sample product image is annotated with a sample defect area, and the sample defect area may be an area annotated on the product image that best represents the location of the defect. The product image may be an image taken of a part of the product that needs to be defect-detected, such as an image of the surface of the product, and the corresponding sample defect area may be annotated on the photographed product image, so that a sample product image annotated with the sample defect area may be obtained.

[0048] For example, see Figure 3 , Figure 3 is a schematic diagram of the effect of a sample product image provided by an embodiment of the present application. It can be seen that there is a marked box (such as Figure 3 ), the area indicated by the marked box is the sample defect area.

[0049] In a possible implementation, the sample product image may also be annotated with a sample defect category corresponding to the sample defect area. The sample defect category is used to indicate the category to which the defect in the sample defect area belongs. The sample defect category belongs to a defect category set, and the defect category set may include multiple defect categories. For example, the defect category may be a scratch, roller mark, pit, roughness, corrugation, burr, etc., which is not limited here.

[0050] The initial defect detection model may be a model that needs to be trained for defect detection, the initial defect detection model may be an untrained basic model or a trained model that does not meet the test conditions, etc. The initial defect detection model may be a deep neural network, such as an object detection model.

[0051] The candidate region may be a region in the product image that is more likely to belong to the foreground, also known as a candidate frame. The foreground probability may be the probability that a region belongs to the foreground, that is, the foreground probability is used to characterize the possibility that the candidate region belongs to the foreground. That is, the foreground probability corresponding to the candidate region needs to be greater than or equal to a certain threshold (i.e., the foreground probability threshold), and the foreground probability threshold is used to indicate the minimum value of the foreground probability required for a region to be determined as a candidate region. Foreground may be a concept relative to background, indicating that there is content that needs to be detected in a region. For example, there is a dirty foreign object (not actually a product defect) in the sample product image, that is, the dirty foreign object is the foreground content relative to the product surface, and there is a scratch defect in the sample product, and the scratch marks are also the foreground content relative to the product surface. When the initial defect detection model is used for detection, it is very likely that the area where the dirty foreign object or the scratch defect is located is determined to be a region that is more likely to belong to the foreground (i.e., the candidate area), and the corresponding foreground probability is greater than or equal to the foreground probability threshold.

[0052] In one embodiment, any candidate region may have corresponding position information. Optionally, the position information of any candidate region may be represented by corresponding region coordinate information (also referred to as region position information). For example, the region coordinate information of any candidate region may be represented as the coordinates (x, y) of the upper left corner of the candidate region and the width w and height h, that is, it may be represented as [x, y, w, h].

[0053] In a possible implementation, calling the initial defect detection model to determine multiple candidate regions in the sample product image and the foreground probability of each candidate region may specifically include the following steps:

[0054] ① Call the initial defect detection model to perform feature extraction processing on the sample product image to obtain the sample image feature corresponding to the sample product image. The sample image feature can be used to characterize the image information of the sample product image, and the sample image feature can be represented as a corresponding feature map. In one embodiment, the initial defect detection model may include a feature extraction network, and the feature extraction network may perform feature extraction processing on the product image input to the defect detection model to obtain the corresponding image feature. The feature extraction network may be a deep neural network, such as a CNN network (a neural network), so as to perform feature extraction on the sample product image based on the CNN network to obtain the corresponding feature map (i.e., the sample image feature). For example, the method of generating a feature map can be represented by the following formula (Formula 1).

[0055] F=f(x;θ) Formula 1

[0056] Among them, f( ) represents a feature extraction network, such as a CNN network, θ represents the weight of the CNN network, x represents the input product image, and F represents the feature map (i.e., image feature) of the input product image x. Therefore, the input product image can be processed based on the feature extraction network in the initial defect detection model to obtain the corresponding image features.

[0057] ② Based on the image features, multiple candidate regions in the sample product image and the foreground probability of each candidate region are determined, and the foreground probability of each candidate region is greater than or equal to the foreground probability threshold. In one embodiment, multiple candidate regions in the sample product image are determined based on the image features, and can be determined based on the candidate region generation network in the initial defect detection model. The candidate region generation network can be an RPN network (Region Proposal Network, a neural network for generating candidate boxes). For example, the generation method of the candidate region can be represented by the following formula (Formula 2).

[0058] Proposal N =RPN(F;θ rpn )

[0059] Proposal i ∈[x,y,w,h,conf] Formula 2

[0060] Among them, Proposal N Represents the N candidate regions generated based on the candidate region generation network (such as the RPN network). RPN() represents the RPN candidate region generation network, θ rpn represents the network parameters of the RPN candidate region generation network, and F represents the image features of the product image input to the initial defect detection model. It can be understood that for any candidate region Proposali It can be represented as a five-dimensional vector [x, y, w, h, conf], where the first four dimensions represent the location information of the candidate region, namely the coordinates of the upper left corner of the candidate region (x, y), as well as the width w and height h, and the last conf represents the foreground probability of the candidate region.

[0061] S202, determining a foreground loss feature of each positive candidate region according to the foreground probabilities of multiple positive candidate regions among the multiple candidate regions, and determining multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss feature of each positive candidate region.

[0062] The overlap between the positive candidate region and the sample defect region is greater than an overlap threshold. The overlap threshold may be a minimum overlap required to determine the candidate region as a positive candidate region. The overlap is used to indicate the degree of overlap between the candidate region and the sample defect region. The more overlap between the candidate region and the sample defect region, the higher the corresponding overlap. The smaller the overlap between the candidate region and the sample defect region, the lower the corresponding overlap.

[0063] In a possible implementation, the degree of overlap can be expressed as the area intersection-and-union ratio between the candidate area and the sample defect area, and the method for determining the positive candidate area can include the following steps: ① Determine the area intersection-and-union ratios between multiple candidate areas and the sample defect area respectively. The area intersection-and-union ratio (IOU, also referred to as intersection-and-union ratio) can be used to indicate the proportion of the area where any candidate area overlaps with the sample defect area (i.e., the intersection area between the candidate area and the sample defect area) to the union area of ​​the candidate area and the sample defect area, thereby indicating the degree of overlap between the candidate area and the sample defect area. The larger the area intersection-and-union ratio between the candidate area and the sample defect area, the greater the corresponding degree of overlap, and the smaller the area intersection-and-union ratio between the candidate area and the sample defect area, the smaller the corresponding degree of overlap.

[0064] ② Determine the candidate regions whose IoU ratio is greater than the IoU ratio threshold among the multiple candidate regions as positive candidate regions. The IoU ratio threshold may be the minimum IoU ratio required to determine the candidate region as a positive candidate region. For example, if the IoU ratio threshold is 0.4, if the IoU ratio between the candidate region a and the sample defect region is 0.6, the candidate region a may be determined as a positive candidate region. If the IoU ratio between the candidate region a and the sample defect region is 0.3, the candidate region a may not be determined as a positive candidate region.

[0065] In one embodiment, the overlap degree may also be obtained by measuring the ratio of the overlapping area between the candidate area and the sample defect area (ie, the intersection area between the candidate area and the sample defect area) to the total area of ​​the candidate area and the sample defect area, which is not limited here.

[0066] The foreground loss feature can be generated according to the foreground probability of the candidate region, and the foreground loss feature can be expressed as a loss value. The higher the foreground probability, the smaller the foreground loss feature, and the lower the foreground probability, the higher the foreground loss feature.

[0067] For example, the foreground loss feature of any candidate region can be expressed by the following formula (Formula 3).

[0068] L cls =-log(conf) Formula 3

[0069] Among them, L cls represents the foreground loss feature, conf represents the foreground probability of the candidate region, and thus the foreground loss feature of any positive candidate region can be obtained according to the foreground probability of any positive candidate region.

[0070] The first sampling candidate area may be a partial candidate area screened from the positive candidate area. The first sampling candidate area may be a partial positive candidate area with a larger foreground loss feature. It is understandable that, generally speaking, for a neural network model, in the process of training the neural network model, a candidate area with a larger classification loss (such as the foreground loss feature of the present application) is often more beneficial for model learning, because it can generate a larger gradient for parameter updating, thereby speeding up the convergence of the model and improving the efficiency of model training. In this way, multiple first sampling candidate areas with larger foreground loss features can be obtained by sampling, thereby speeding up the convergence of the initial defect detection model and improving the training efficiency of the initial defect detection model during the training of the initial defect detection model.

[0071] In a possible implementation, the first sampling candidate region may be a positive candidate region whose foreground loss feature is ranked in the top K, where K is a positive integer. Alternatively, the first sampling candidate region may also be a positive candidate region whose foreground loss feature is greater than a certain foreground loss threshold. Alternatively, the first sampling candidate region may be a positive candidate region whose foreground loss feature is ranked in the top K and greater than a certain foreground loss threshold.

[0072] In one embodiment, as described above, the first sampling candidate region may be a positive candidate region ranked in the top K in terms of foreground loss feature, then multiple first sampling candidate regions are determined from multiple positive candidate regions according to the foreground loss feature of each positive candidate region, and the following steps may be specifically included: ① Sort multiple positive candidate regions in order from large to small according to the foreground loss feature. ② Determine the positive candidate regions ranked in the top K as the first sampling candidate regions to obtain multiple first sampling candidate regions. Thus, K first sampling candidate regions with larger foreground loss features among multiple positive candidate regions can be obtained. Among them, the value of K may be a preset fixed value, such as, if K is 10, then 10 first sampling candidate regions can be determined from multiple positive candidate regions; the value of K may also be a value obtained according to a preset ratio and the number of positive candidate regions, for example, if K is 50%, then when the number of positive candidate regions is 16, K=16*50%=8, and then 8 first sampling candidate regions can be determined from multiple positive candidate regions.

[0073] In one embodiment, as described above, the first sampling candidate region can be a positive candidate region whose foreground loss feature is ranked in the top K and is greater than a certain foreground loss threshold. Then, multiple first sampling candidate regions are determined from multiple positive candidate regions according to the foreground loss feature of each positive candidate region. Specifically, the following steps may be included: ① Sort multiple positive candidate regions in order from large to small according to the foreground loss feature. ② Determine the positive candidate region that is ranked in the top K and whose foreground loss feature is greater than the foreground loss threshold as the first sampling candidate region, and obtain multiple first sampling candidate regions. Among them, K is a positive integer. The method for determining K can refer to the above description and will not be repeated here. Thus, the positive candidate region whose foreground loss feature is ranked in the top K can be obtained as the first sampling candidate region to improve the convergence speed of the model training process.

[0074] S203 , sampling from the multiple first sampling candidate regions to obtain multiple second sampling candidate regions, and calling the initial defect detection model to determine the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate regions.

[0075] The second sampling candidate area can be a partial candidate area screened from multiple first sampling candidate areas. The initial defect detection model can be trained based on the multiple second sampling candidate areas determined, which can reduce the impact of incorrectly labeled sample defect areas on the initial defect detection model training and improve the anti-interference ability of the trained defect detection model to noise labeling information.

[0076] In a possible implementation, sampling from a plurality of first sampling candidate regions to obtain a plurality of second sampling candidate regions may be performed by randomly selecting some first sampling candidate regions from the plurality of first sampling candidate regions to obtain the plurality of second sampling candidate regions.

[0077] It is understandable that the training of neural network models can usually be based on labeled sample data (also called training data). For example, the training of the initial defect detection model for the neural network model can be based on sample product images labeled with sample defect areas and corresponding sample defect categories. However, training the model based on labeled data usually requires relying on a large amount of accurately labeled training data. If the sample data is incorrectly labeled, it is easy to cause the prediction accuracy of the trained model to be low, that is, the model's ability to resist interference from noise labels is weak. Some current models for defect detection train the model based on the above-mentioned multiple positive candidate areas. If there are some sample defect areas in the sample product image that are not defects but are incorrectly labeled as defects, the generated positive candidate areas will move closer to the incorrectly labeled sample defect areas as the training progresses, resulting in the trained model having low defect detection accuracy. The present application can reduce the chance of model training based on incorrectly labeled sample defect areas by randomly sampling multiple first sampling candidate areas, thereby improving the model's ability to resist interference from noise labels, enhancing the robustness of the trained defect detection model, and improving the detection accuracy of the trained defect detection model for defects.

[0078] For example, in industrial scenarios, due to environmental problems in production lines and workshops, some dust, flocculent dirt, etc. will be generated on the surface of products (such as parts). In this case, in the product surface image captured by the camera, the real defects are often very similar to the characteristics of these dirt and foreign objects. This makes it difficult for labelers to accurately label, and inevitably produces many incorrectly labeled sample defect areas. Please refer to Figure 4 , Figure 4 is a schematic diagram of the effect of a sample product image provided in an embodiment of the present application, such as Figure 4 As shown, only Figure 4 The area shown in 401 is the area where the real defect is located, but the annotator also annotated other locations with similar shapes and defects, which resulted in Figure 4Therefore, after determining multiple positive candidate areas during the training process, multiple second samples can be obtained by random sampling and then the initial defect detection model can be trained by area, thereby reducing the chance of model training based on positive candidate areas with high overlap of incorrectly labeled sample defect areas, thereby improving the model's anti-interference ability to noise labels and improving the defect detection accuracy of the trained defect detection model.

[0079] The classification probability set includes the probability that the second sampling candidate area belongs to each defect category. The position offset information is used to indicate the position deviation between the second sampling candidate area determined by the initial defect detection model and the sample defect area. Optionally, the position deviation information of any second sampling candidate area can be represented by referring to the area coordinate information of the above-mentioned candidate area, such as the position difference, width difference and height difference between the upper left corner coordinates of the second sampling candidate area and the sample defect area. It can be understood that there is a corresponding classification probability and a corresponding position offset information for each second sampling candidate area. For example, the above-mentioned defect category set may include defect 1, defect 2, defect 3, and defect 4. Then, for any second sampling candidate area, the classification probability that any second sampling candidate area belongs to defect 1, defect 2, defect 3, and defect 4 respectively can be obtained, and the position offset information of any second sampling candidate area and the sample defect area can be obtained.

[0080] In a possible implementation, the initial defect detection model may include a defect recognition subnetwork, and then the classification probability sets and position offset information corresponding to the plurality of second sampling candidate regions may be determined based on the defect recognition subnetwork. The defect recognition subnetwork may be a neural network, such as an RCNN network (a neural network).

[0081] In one embodiment, when determining the classification probability sets and position offset information corresponding to multiple second sampling candidate regions based on the defect recognition subnetwork, each second sampling candidate region can be mapped back to the sample image features to obtain the regional features corresponding to each second sampling candidate region, and then the classification probability sets and position offset information corresponding to each second sampling candidate region are determined based on the regional features corresponding to each second sampling candidate frame. In one embodiment, each second sampling candidate region is mapped back to the sample image features to obtain the regional features corresponding to each second sampling candidate region, and region of interest pooling (RoI Pooling) can be used for processing, which is not described in detail here.

[0082] S204: Train the initial defect detection model according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions to obtain a target defect detection model, and use the target defect detection model to perform defect detection on the product image.

[0083] Among them, in the process of training the initial defect detection model, the position offset information of the second sampling candidate area can be gradually reduced, that is, the position difference between the second sampling candidate area and the sample defect area gradually decreases, and the defect category corresponding to the largest classification probability in the classification probability set of the second sampling candidate area can be made the same as the sample defect category with greater confidence.

[0084] In a possible implementation, the initial defect detection model may be trained by the sample defect category corresponding to the sample defect area, the classification probability set and the position offset information corresponding to the plurality of second sampling candidate areas. For example, the classification loss value may be determined according to the classification probability set and the sample defect category corresponding to the plurality of second sampling candidate areas, and the regression loss value may be determined according to the position offset information of the plurality of second sampling candidate areas, and then the model parameters of the initial defect detection model may be optimized based on the classification loss value and the regression loss value to obtain the target defect detection model.

[0085] In a possible implementation, the initial defect detection model is trained, and the initial defect detection model can also be trained using the above-mentioned multiple candidate areas, the foreground probability of each candidate area, the sample defect area, the sample defect category corresponding to the sample defect area, the classification probability sets corresponding to the multiple second sampling candidate areas, and the position offset information.

[0086] In one embodiment, as described above, the sample product image is annotated with the sample defect category corresponding to the sample defect area. Then, the initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate areas to obtain the target defect detection model, which may specifically include the following steps:

[0087] ① Determine the first classification loss feature according to the foreground probability of multiple candidate areas, and determine the first regression loss feature according to the position deviation between the multiple candidate areas and the sample defect area. Among them, the first classification loss feature, which can also be called the first classification loss value, can be determined according to the foreground loss feature determined based on the foreground probability of each candidate area, such as the first classification loss value can be determined according to the average value of the foreground loss feature of each candidate area. The determination method of the foreground loss feature corresponding to each candidate area can refer to the above description and will not be repeated here. The first regression loss feature, which can also be called the first regression loss value, can be determined according to the position deviation between the multiple candidate areas and the sample defect area, such as the average value of the position deviation between the multiple candidate areas and the sample defect area. It can be understood that if the average value of the position deviation between the multiple candidate areas and the sample defect area is larger, the first regression loss feature is larger, and if the average value of the position deviation between the multiple candidate areas and the sample defect area is smaller, the first regression loss feature is smaller. The first classification loss feature and the first regression loss feature can be used to train the initial defect detection model, specifically, to train the candidate region generation model of the initial defect detection model. As the model training proceeds, the first classification loss feature and the second regression loss feature gradually decrease.

[0088] In a possible implementation, the position deviation between any candidate region and the sample defect region may also be referred to as a candidate region regression loss feature. For example, the candidate region regression loss feature of any candidate region may be represented by the following formula (Formula 4).

[0089] L reg =|box-GT| Formula 4

[0090] Among them, L cls represents the regression loss feature of the candidate region, box represents the regional coordinate information of the candidate region, and GT represents the regional coordinate information of the sample defect region. Therefore, the candidate region regression loss feature of any candidate region can be obtained according to the deviation between the regional coordinate information of any candidate region and the regional coordinate information of the sample defect region.

[0091] ② Determine the second classification loss feature based on the classification probability sets and sample defect categories corresponding to the multiple second sampling candidate areas, and determine the second regression loss feature based on the position offset information of the multiple second sampling candidate areas. The second classification loss feature, which can also be called the second classification loss value, can characterize the difference between the classification probability of each defect category in the classification probability set of the multiple second sampling candidate areas and the labeled probability for each defect category. The labeled probability for each defect category can be determined based on the sample defect category. For example, if the probability corresponding to the sample defect category is 1 and the probability corresponding to the remaining categories is 0, then as the model training proceeds, the second classification loss feature gradually decreases, so that the classification probability sets corresponding to the respective second sampling candidate areas gradually approach the labeled probability for each defect category.

[0092] The second regression loss feature, which may also be referred to as the second regression loss value, can characterize the position deviation between multiple second sampling candidate regions and the sample defect region, that is, the position offset information. As the model training proceeds, the second regression loss feature gradually decreases, so that the deviation between the position offset information of each second sampling candidate region and the sample defect region gradually decreases.

[0093] ③ The initial defect detection model is trained according to the first classification loss feature, the first regression loss feature, the second classification loss feature and the second regression loss feature to obtain the target defect detection model. It can be understood that the initial defect detection model is trained as a whole according to the first classification loss feature, the first regression loss feature, the second classification loss feature and the second regression loss feature obtained above, that is, the model parameters of the feature extraction network, the candidate region generation network and the defect recognition sub-network in the defect detection model can be optimized, so that the initial defect detection model can gradually have the ability to more accurately detect the defect area of ​​the input product image, and can more accurately determine the defect category corresponding to the defect area, so as to obtain the target defect detection model through training for defect detection.

[0094] In one possible implementation, after the initial defect detection model is trained to obtain the target defect detection model, the target defect detection model can be tested. When the test passes, the target defect detection model that passes the test is applied to defect detection on product images; when the test fails, the target defect detection model obtained through this training is trained again as the initial defect detection model to obtain a target defect detection model that can pass the test.

[0095] In one embodiment, the target defect detection model is used to perform defect detection on the product image. The target defect detection model can be called to process the product image to be detected to obtain a defect detection result. The defect detection result is used to indicate the defect of the detected product image. Optionally, the defect detection result can be used to indicate the defect area in the product image and the defect category corresponding to the defect. Optionally, if the target defect detection model does not detect the defect area, the defect detection result can be empty, indicating that no defect is detected in the product image.

[0096] Here we explain the training process of the entire initial defect detection model with diagrams, see Figure 5 , Figure 5 Schematic diagram of a training process of an initial defect detection model provided in an embodiment of the present application. First, a sample product image (such as Figure 5 Then, the sample image features of the sample product image are obtained (as shown in 501 in FIG. Figure 5 ), which can be obtained based on a feature extraction network such as a CNN network based on the initial defect detection model; then, based on the sample image features, multiple candidate regions and a foreground probability of each candidate region (such as Figure 5 As shown in 503 in FIG. 5 , it can be determined by the RPN network; and then multiple first sampling candidate regions (such as Figure 5 Specifically, the foreground loss feature can be determined according to the foreground probability of the positive candidate region, and then the positive candidate regions with the foreground loss feature ranked in the top K are determined as the first sampling candidate regions; and then multiple second sampling candidate regions are obtained by random sampling from the multiple first sampling candidate regions (such as Figure 5 Then, the position offset information and classification probability set of each second sampling candidate region can be determined (as shown in 505 in FIG. Figure 5 ), such as can be determined by the RCNN network. Then, the first classification loss feature (such as Figure 5 507 in FIG), determining a first regression loss feature (as shown in FIG5 ) based on the position deviation between the plurality of candidate regions and the sample defect region. Figure 5 ), and determining a second classification loss feature (such as Figure 5 ), determining a second regression loss feature (such as Figure 5 ), thereby training the initial defect detection model based on the first classification loss feature, the first regression loss feature, the second classification loss feature and the second regression loss feature to obtain the target defect detection model (as shown in 510 in FIG. 1 ). Figure 5 ), so as to facilitate subsequent defect detection using the target defect detection model.

[0097] By using the embodiments of the present application, a sample product image can be obtained, and an initial defect detection model can be called to determine multiple candidate areas in the sample product image, thereby determining multiple first sampling candidate areas from multiple forward candidate areas based on the foreground loss characteristics of the multiple candidate areas, and then sampling multiple second sampling candidate areas from the multiple sampling candidate areas, so as to train the initial defect detection model based on the multiple second sampling candidate areas. For example, the initial defect detection model can be called to determine the classification probability set and position offset information corresponding to each second sampling candidate area, and the initial defect detection model can be trained based on the classification probability set and position offset information corresponding to each second sampling candidate area, and then defect detection can be performed on the product image based on the trained target defect detection model. On the one hand, the present application can train a target defect detection model for automatic defect detection of products, which meets the needs of automation and intelligence of product defect detection and improves the efficiency of product defect detection; on the other hand, when training the defect detection product model, the generated multiple candidate areas can be sampled twice, that is, multiple first sampling candidate areas can be determined based on the foreground loss feature to accelerate the convergence of the model, and multiple second sampling candidate areas can be sampled again from the multiple first sampling candidate areas to reduce the impact of incorrectly labeled sample defect areas in the sample product image on the model training, thereby improving the accuracy of the trained model and helping to improve the accuracy of product defect detection.

[0098] See also Figure 6 , Figure 6 1 is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by the above-mentioned electronic device. The data processing method can include the following steps.

[0099] S601, obtaining a sample product image, and calling an initial defect detection model to determine multiple candidate regions in the sample product image and a foreground probability of each candidate region.

[0100] S602, determining a foreground loss feature of each positive candidate region according to the foreground probabilities of multiple positive candidate regions among the multiple candidate regions, and determining multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss feature of each positive candidate region.

[0101] Steps S601-S602 may refer to the relevant description of steps S201-S202, which will not be repeated here.

[0102] S603 , sampling from the multiple first sampling candidate regions to obtain multiple second sampling candidate regions, and calling the initial defect detection model to determine the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate regions.

[0103] In a possible implementation, sampling from multiple first sampling candidate areas to obtain multiple second sampling candidate areas may include the following steps: ① Obtaining the sampling ratio for the multiple first sampling candidate areas. The sampling ratio is used to indicate the number of second sampling candidate areas that need to be sampled. ② Sampling from the multiple first sampling candidate areas to obtain multiple second sampling candidate areas according to the sampling ratio. In one embodiment, the adopted ratio may be the ratio of the number of candidate areas that need to be sampled to the number of first candidate areas. For example, if the sampling ratio is 1 / 2, and the number of first sampling candidate areas is 20, then according to the sampling ratio and the number of first sampling candidate areas, it can be obtained that the number of candidate areas that need to be sampled is 20*1 / 2=10, and then 10 second sampling candidate areas can be randomly sampled from the 20 first sampling candidate areas.

[0104] In one possible implementation, the present application may determine the sampling ratio according to the actual application scenario. For example, for some application scenarios where labeling errors are prone to occur, areas that are not defects are often easily labeled as defective areas. In this case, a slightly smaller sampling ratio may be determined to sample a smaller proportion of second sampling candidate areas, thereby reducing the chances of performing model training based on sample defective areas that are incorrectly labeled as defects, so as to improve the prediction accuracy of the trained model.

[0105] S604 , training the initial defect detection model according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions to obtain a target defect detection model.

[0106] S605: Acquire a target product image of the product to be inspected.

[0107] The product to be inspected may be a product that needs to be inspected for defects. The target product image may be an image captured of a region of the product to be inspected that needs to be inspected for defects, such as the surface of the product to be inspected, to obtain the target product image.

[0108] S606: Calling the target defect detection model to determine the target image features corresponding to the target product image, and determining a plurality of target candidate regions in the target product image according to the target image features.

[0109] Among them, the target image feature can be an image feature that represents the image information of the target product image, and the target image feature can be represented as a corresponding feature map. It can be understood that the target defect detection model has the same model structure as the initial defect detection model mentioned above. In the process of training the initial detection model, by continuously adjusting the model parameters, the trained target defect detection model has the ability to accurately identify the defect area in the product image and the defect category corresponding to the defect area.

[0110] In one embodiment, the target image features can be obtained by performing feature extraction processing through a feature extraction network in a target defect detection model. It is understandable that the feature extraction network in the target defect detection model can accurately extract a feature map of the target product image suitable for defect detection, thereby facilitating subsequent accurate defect detection.

[0111] The target candidate area can be a candidate area determined by the target defect detection model based on the target product image. The relevant description of the target candidate area can refer to the relevant description of the above candidate area, which will not be repeated here. It can be understood that any target candidate area has corresponding location information, such as corresponding area coordinate information, and each target candidate area can be associated with a corresponding foreground probability, and the foreground probability is greater than or equal to the foreground probability threshold.

[0112] In one embodiment, multiple target candidate regions in the target product image are determined based on the target image features, and can be determined based on a candidate region generation network (such as an RPN network) in the target defect detection model. It is understandable that the candidate region generation network in the target defect detection model can accurately extract the target candidate region belonging to the foreground, thereby facilitating subsequent accurate defect detection.

[0113] S607 , calling the target defect detection model to determine position offset information of multiple target candidate regions and a classification probability set of each target candidate region.

[0114] Among them, the method for determining the position offset information of multiple target candidate areas and the classification probability set of each target candidate area can refer to the above-mentioned description of determining the position offset information of multiple second sampling candidate areas and the classification probability set of each target candidate area, which will not be repeated here. In one embodiment, the position offset information of multiple target candidate areas and the classification probability set of each target candidate area can be determined by the defect recognition subnetwork in the target defect detection model. It can be understood that the defect recognition subnetwork in the target defect detection model has the ability to identify the position offset information corresponding to each target candidate area and the classification probability for each defect category, thereby quickly determining the position offset information of multiple target candidate areas and the classification probability set of each target candidate area.

[0115] S608 : Determine a target defect region of the target product image and a target defect category corresponding to the target defect region according to the position offset information of the plurality of target candidate regions and a classification probability set of each target candidate region.

[0116] The target defect area may be the area corresponding to the location of the defect that best reflects the target product image determined based on the target defect detection model. The target defect category corresponding to the target defect area may be the defect category corresponding to the maximum classification probability in the classification probability set corresponding to the target defect area.

[0117] In one embodiment, multiple offset candidate regions can be determined based on the position offset information of multiple target candidate regions, each offset candidate region is associated with a corresponding classification probability set, a target defect region is determined from the multiple offset candidate regions, and the target defect category corresponding to the target defect region is determined according to the classification probability set corresponding to the target defect region. Among them, multiple offset candidate regions can be determined based on the position offset information of multiple target candidate regions, which is equivalent to fine-tuning each target candidate region based on the corresponding position offset information, so as to obtain multiple offset candidate regions that are closer to the location of the real defect. In one embodiment, the target defect region is determined from the multiple offset candidate regions, and the target defect region can be determined from the multiple offset candidate regions according to the maximum classification probability in the classification probability set of each offset candidate region (also referred to as the classification probability of the predicted defect category corresponding to each offset candidate region), for example, the target defect region is determined from the multiple offset candidate regions by non-maximum suppression (NMS), which is not repeated here. In one embodiment, the target defect category corresponding to the target defect region is determined according to the classification probability set corresponding to the target defect region, which can be the defect category corresponding to the maximum classification probability in the classification probability set based on the target defect region.

[0118] Here, the process of obtaining the target defect area in the target product image is described in combination with the image. Figure 7 , Figure 7It is a schematic diagram of a defect detection process provided by an embodiment of the present application. First, step S701 can be executed: obtain the target product image, and then input the target product image into the target defect detection model (i.e., Input Image), and then in the target defect detection model, step S702 is executed: feature extraction, specifically, feature extraction can be performed based on the feature extraction network in the target defect detection model, such as extracting features (ExtractFeatures) through the convolutional layers (CONV Layers) in the CNN network, thereby obtaining feature maps (Feature Maps), i.e., target product features. Then step S703 is executed: predict candidate areas, specifically, the candidate area generation network in the target defect detection model can be called to determine multiple target candidate areas and the foreground probability of each target candidate area based on the target product features. Then S704 is executed: region of interest pooling (RoI Pooling) to further extract features from each target candidate area. Then S705 is executed: defect classification, specifically, the defect recognition subnetwork in the defect detection model can be called to determine the classification probability set and position offset information of each target candidate area. Finally, step S706 may be executed: outputting the defect detection result, that is, determining the target defect region and the corresponding target defect category based on the classification probability set of each target candidate region and the position offset information, so as to obtain and output the defect detection result.

[0119] By using the embodiments of the present application, a sample product image can be obtained, and an initial defect detection model can be called to determine multiple candidate areas in the sample product image, thereby determining multiple first sampling candidate areas from multiple forward candidate areas based on the foreground loss characteristics of the multiple candidate areas, and then sampling multiple second sampling candidate areas from the multiple sampling candidate areas, so as to train the initial defect detection model based on the multiple second sampling candidate areas. For example, the initial defect detection model can be called to determine the classification probability set and position offset information corresponding to each second sampling candidate area, and the initial defect detection model can be trained based on the classification probability set and position offset information corresponding to each second sampling candidate area, and then defect detection can be performed on the product image based on the trained target defect detection model. On the one hand, the present application can train a target defect detection model for automatic defect detection of products, which meets the needs of automation and intelligence of product defect detection and improves the efficiency of product defect detection; on the other hand, when training the defect detection product model, the generated multiple candidate areas can be sampled twice, that is, multiple first sampling candidate areas can be determined based on the foreground loss feature to accelerate the convergence of the model, and multiple second sampling candidate areas can be sampled again from the multiple first sampling candidate areas to reduce the impact of incorrectly labeled sample defect areas in the sample product image on the model training, thereby improving the accuracy of the trained model and helping to improve the accuracy of product defect detection.

[0120] See also Figure 8 , Figure 8 is a structural diagram of a data processing device provided in an embodiment of the present application. Optionally, the data processing device may be provided in the above-mentioned electronic device. Figure 8 As shown, the data processing device described in this embodiment may include:

[0121] The acquisition unit 801 is used to acquire a sample product image and call the initial defect detection model to determine multiple candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0122] The processing unit 802 is configured to determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine a plurality of first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0123] The processing unit 802 is further configured to obtain a plurality of second sampling candidate regions by sampling from the plurality of first sampling candidate regions, and to call the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions; the classification probability set includes the probability that the second sampling candidate region belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate region and the sample defect region;

[0124] The processing unit 802 is further used to train the initial defect detection model according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and use the target defect detection model to perform defect detection on the product image.

[0125] In one implementation, the processing unit 802 is specifically configured to:

[0126] Sorting the multiple positive candidate regions in descending order according to the foreground loss features;

[0127] The forward candidate regions ranked in the top K are determined as first sampling candidate regions to obtain the multiple first sampling candidate regions, where K is a positive integer.

[0128] In one implementation, the overlap degree is represented by an area intersection-over-union ratio between the candidate area and the sample defect area; the processing unit 802 is further configured to:

[0129] respectively determining the area intersection-over-union ratios between the plurality of candidate areas and the sample defect area;

[0130] A candidate region whose region IoU ratio is greater than an IoU ratio threshold value among the multiple candidate regions is determined as the positive candidate region.

[0131] In one implementation, the processing unit 802 is specifically configured to:

[0132] Acquire a sampling ratio for the plurality of first sampling candidate regions; the sampling ratio is used to indicate the number of second sampling candidate regions that need to be sampled;

[0133] The plurality of second sampling candidate regions are obtained by sampling from the plurality of first sampling candidate regions according to the sampling ratio.

[0134] In one implementation, the processing unit 802 is specifically configured to:

[0135] Calling the initial defect detection model to perform feature extraction processing on the sample product image to obtain sample image features corresponding to the sample product image;

[0136] Based on the sample image features, multiple candidate regions in the sample product image and a foreground probability of each candidate region are determined, and the foreground probability of each candidate region is greater than or equal to a foreground probability threshold.

[0137] In one implementation, the sample product image is annotated with a sample defect category corresponding to a sample defect area, and the processing unit 802 is specifically configured to:

[0138] Determine a first classification loss feature according to the foreground probabilities of the multiple candidate regions, and determine a first regression loss feature according to the position deviations between the multiple candidate regions and the sample defect region;

[0139] Determine a second classification loss feature according to the classification probability sets respectively corresponding to the plurality of second sampling candidate regions and the sample defect category, and determine a second regression loss feature according to the position offset information of the plurality of second sampling candidate regions;

[0140] The initial defect detection model is trained according to the first classification loss feature, the first regression loss feature, the second classification loss feature, and the second regression loss feature to obtain the target defect detection model.

[0141] In one implementation, the processing unit 802 is specifically configured to:

[0142] Acquire a target product image of a product to be inspected;

[0143] Calling the target defect detection model to determine a target image feature corresponding to the target product image, and determining a plurality of target candidate regions in the target product image according to the target image feature;

[0144] Calling the target defect detection model to determine the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region;

[0145] A target defect region of the target product image and a target defect category corresponding to the target defect region are determined according to the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region.

[0146] See also Fig. 9 , Fig. 9 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device described in this embodiment includes: a processor 901 and a memory 902. Optionally, the electronic device may also include a network interface or a power supply module. The processor 901 and the memory 902 may exchange data.

[0147] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0148] The above-mentioned network interface may include an input device and / or an output device. For example, the input device may be a control panel, a microphone, a receiver, etc., and the output device may be a display screen, a transmitter, etc., which are not listed here one by one.

[0149] The memory 902 may include a read-only memory and a random access memory, and provide program instructions and data to the processor 901. A portion of the memory 902 may also include a non-volatile random access memory. When the processor 901 calls the program instructions, it is used to execute:

[0150] Acquire a sample product image, and call an initial defect detection model to determine a plurality of candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with a sample defect region;

[0151] Determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, and determine multiple first sampling candidate regions from the multiple positive candidate regions according to the foreground loss features of each positive candidate region; the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold;

[0152] Sampling the plurality of first sampling candidate regions to obtain a plurality of second sampling candidate regions, and calling the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions; the classification probability set includes the probability that the second sampling candidate region belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate region and the sample defect region;

[0153] The initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the multiple second sampling candidate areas to obtain a target defect detection model, and the target defect detection model is used to perform defect detection on the product image.

[0154] In one implementation, the processor 901 is specifically configured to:

[0155] Sorting the multiple positive candidate regions in descending order according to the foreground loss features;

[0156] The forward candidate regions ranked in the top K are determined as first sampling candidate regions to obtain the multiple first sampling candidate regions, where K is a positive integer.

[0157] In one implementation, the overlap degree is represented by an area intersection-over-union ratio between the candidate area and the sample defect area; the processor 901 is further configured to:

[0158] respectively determining the area intersection-over-union ratios between the plurality of candidate areas and the sample defect area;

[0159] A candidate region whose region IoU ratio is greater than an IoU ratio threshold value among the multiple candidate regions is determined as the positive candidate region.

[0160] In one implementation, the processor 901 is specifically configured to:

[0161] Acquire a sampling ratio for the plurality of first sampling candidate regions; the sampling ratio is used to indicate the number of second sampling candidate regions that need to be sampled;

[0162] The plurality of second sampling candidate regions are obtained by sampling from the plurality of first sampling candidate regions according to the sampling ratio.

[0163] In one implementation, the processor 901 is specifically configured to:

[0164] Calling the initial defect detection model to perform feature extraction processing on the sample product image to obtain sample image features corresponding to the sample product image;

[0165] Based on the sample image features, multiple candidate regions in the sample product image and a foreground probability of each candidate region are determined, and the foreground probability of each candidate region is greater than or equal to a foreground probability threshold.

[0166] In one implementation, the sample product image is annotated with a sample defect category corresponding to a sample defect area, and the processor 901 is specifically configured to:

[0167] Determine a first classification loss feature according to the foreground probabilities of the multiple candidate regions, and determine a first regression loss feature according to the position deviations between the multiple candidate regions and the sample defect region;

[0168] Determine a second classification loss feature according to the classification probability sets respectively corresponding to the plurality of second sampling candidate regions and the sample defect category, and determine a second regression loss feature according to the position offset information of the plurality of second sampling candidate regions;

[0169] The initial defect detection model is trained according to the first classification loss feature, the first regression loss feature, the second classification loss feature, and the second regression loss feature to obtain the target defect detection model.

[0170] In one implementation, the processor 901 is specifically configured to:

[0171] Acquire a target product image of a product to be inspected;

[0172] Calling the target defect detection model to determine a target image feature corresponding to the target product image, and determining a plurality of target candidate regions in the target product image according to the target image feature;

[0173] Calling the target defect detection model to determine the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region;

[0174] A target defect region of the target product image and a target defect category corresponding to the target defect region are determined according to the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region.

[0175] Optionally, when the program instructions are executed by the processor, other steps of the method in the above embodiment may also be implemented, which will not be described in detail here.

[0176] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above method, such as executing the method executed by the above electronic device, which is not repeated here.

[0177] Optionally, the storage medium involved in the present application, such as computer-readable storage medium, may be non-volatile or volatile.

[0178] Optionally, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of blockchain nodes, etc. Among them, the blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0179] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0180] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0181] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction, and when the computer instruction is executed by a processor, some or all of the steps in the above method can be implemented. For example, the computer instruction is stored in a computer-readable storage medium. The processor of a computer device (such as the above-mentioned electronic device) reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction so that the computer device executes the steps performed in the embodiments of the above methods. For example, the computer device can be a terminal, or it can be a server.

[0182] The data processing method, device, electronic device, medium and program product provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, It is characterized in that The method comprises: A sample product image is obtained, and an initial defect detection model is called to determine multiple candidate regions in the sample product image and a foreground probability of each candidate region; the sample product image is annotated with an erroneous sample defect region; the initial defect detection model is a model that needs to be trained for defect detection, including an untrained basic model or a trained model that does not meet the test conditions; determining a foreground loss feature of each positive candidate region according to the foreground probabilities of multiple positive candidate regions among the multiple candidate regions, sorting the multiple positive candidate regions in descending order according to the foreground loss features, determining the top K positive candidate regions in the sorting as first sampling candidate regions, and obtaining multiple first sampling candidate regions, where K is a positive integer; and the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold; Acquire a sampling ratio for the plurality of first sampling candidate areas, the sampling ratio being used to indicate the number of second sampling candidate areas to be sampled, sample the plurality of second sampling candidate areas from the plurality of first sampling candidate areas according to the sampling ratio, and call the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate areas; the classification probability set includes the probability that the second sampling candidate area belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate area and the sample defect area; The initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions to obtain a target defect detection model, and the target defect detection model is used to perform defect detection on the product image; The overlap degree is expressed as an IoU ratio between the candidate region and the sample defect region; the method further comprises: determining the IoU ratios between the multiple candidate regions and the sample defect region respectively; determining the candidate region whose IoU ratio among the multiple candidate regions is greater than an IoU ratio threshold as the positive candidate region; The calling of the initial defect detection model to determine multiple candidate areas in the sample product image and the foreground probability of each candidate area includes: calling the initial defect detection model to perform feature extraction processing on the sample product image to obtain sample image features corresponding to the sample product image; based on the sample image features, determining multiple candidate areas in the sample product image and the foreground probability of each candidate area, wherein the foreground probability of each candidate area is greater than or equal to a foreground probability threshold.

2. The method according to claim 1, It is characterized in that The sample product image is annotated with incorrect sample defect areas and sample defect categories corresponding to the sample defect areas; the initial defect detection model is trained according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate areas to obtain a target defect detection model, including: Determine a first classification loss feature according to the foreground probabilities of the multiple candidate regions, and determine a first regression loss feature according to the position deviations between the multiple candidate regions and the sample defect regions with errors and the sample defect regions; Determine a second classification loss feature according to the classification probability sets respectively corresponding to the plurality of second sampling candidate regions and the sample defect category, and determine a second regression loss feature according to the position offset information of the plurality of second sampling candidate regions; The initial defect detection model is trained according to the first classification loss feature, the first regression loss feature, the second classification loss feature, and the second regression loss feature to obtain the target defect detection model.

3. The method according to claim 1, It is characterized in that The method of performing defect detection on the product image by using the target defect detection model includes: Acquire a target product image of a product to be inspected; Calling the target defect detection model to determine a target image feature corresponding to the target product image, and determining a plurality of target candidate regions in the target product image according to the target image feature; Calling the target defect detection model to determine the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region; A target defect region of the target product image and a target defect category corresponding to the target defect region are determined according to the position offset information of the multiple target candidate regions and a classification probability set of each target candidate region.

4. A data processing device, It is characterized in that The device comprises: An acquisition unit is used to acquire a sample product image, and call an initial defect detection model to determine multiple candidate areas in the sample product image and a foreground probability of each candidate area; the sample product image is annotated with an erroneous sample defect area; the initial defect detection model is a model that needs to be trained for defect detection, including an untrained basic model or a trained model that does not meet the test conditions; a processing unit, configured to determine a foreground loss feature of each positive candidate region according to the foreground probabilities of the multiple positive candidate regions among the multiple candidate regions, sort the multiple positive candidate regions in descending order according to the foreground loss features, determine the positive candidate regions in the top K in the sorting as first sampling candidate regions, and obtain multiple first sampling candidate regions, where K is a positive integer; and the overlap between the positive candidate region and the sample defect region is greater than an overlap threshold; The processing unit is further configured to obtain a sampling ratio for the plurality of first sampling candidate areas, the sampling ratio being used to indicate the number of second sampling candidate areas to be sampled, sampling the plurality of second sampling candidate areas from the plurality of first sampling candidate areas according to the sampling ratio, and calling the initial defect detection model to determine classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate areas; the classification probability set includes the probability that the second sampling candidate area belongs to each defect category, and the position offset information is used to indicate the position deviation between the second sampling candidate area and the sample defect area; The processing unit is further configured to train the initial defect detection model according to the classification probability sets and position offset information respectively corresponding to the plurality of second sampling candidate regions to obtain a target defect detection model, and perform defect detection on the product image using the target defect detection model; The overlap degree is expressed as an area intersection-over-union ratio between the candidate area and the sample defect area; the processing unit is further used to respectively determine the area intersection-over-union ratios between the multiple candidate areas and the sample defect area; and determine the candidate area whose area intersection-over-union ratio among the multiple candidate areas is greater than the intersection-over-union ratio threshold as the positive candidate area; The processing unit is also used to call the initial defect detection model to perform feature extraction processing on the sample product image to obtain sample image features corresponding to the sample product image; based on the sample image features, multiple candidate areas in the sample product image and the foreground probability of each candidate area are determined, and the foreground probability of each candidate area is greater than or equal to a foreground probability threshold.

5. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 3.

7. A computer program product, It is characterized in that The method comprises computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 3.