A method, apparatus, device, storage medium and program product for image processing

Abstract

Embodiments of the present application disclose a kind of image processing method, device, equipment, storage medium and program product, at least involve artificial intelligence and so on technology.The method can train target image detection model with less labeled data, realize the image defect detection of to-be-labeled image.The method comprises: obtaining a plurality of to-be-labeled images;Each to-be-labeled image is carried out image value estimation processing based on Gaussian mixture prediction model, and the image value confidence of each to-be-labeled image is obtained;From the plurality of to-be-labeled images, the target to-be-labeled image is selected based on the image value confidence of each to-be-labeled image;Based on target labeled image and initial labeled image, the model parameter of initial image detection model is updated, and the target image detection model is obtained;Each to-be-labeled image is carried out image detection processing based on target image detection model, and the image defect result of each to-be-labeled image is obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an image processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Object detection is a technique for locating and recognizing objects in images, and it is one of the three fundamental tasks of computer vision. In recent years, with the development of deep learning, object detection has made continuous breakthroughs in various fields. Deep learning-based object detection algorithms can automatically mine task-related information from training data and extract more robust features.

[0003] Currently, the two-stage detector method is a highly accurate and stable object detection method. In this method, the Faster R-CNN model is typically used to detect defects in industrial images. Specifically, the region proposal network (RPN) in the Faster R-CNN model predicts multiple bounding boxes in the image, and the region convolutional neural network (R-CNN) model in the Faster R-CNN model performs fine-grained identification and coordinate adjustment of these bounding boxes to obtain the final defect detection result. However, in existing two-stage methods, the Faster R-CNN network uses ResNet with a large number of parameters as the backbone network, resulting in a large amount of data to be trained. Moreover, a huge amount of annotation cost is required to meet the model training needs, which is not only unfavorable for rapid image detection, but also cannot guarantee the over-detection rate and under-detection rate of products. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, storage medium, and program product that can train a target image detection model with less labeled data, quickly realize image defect detection of the labeled image, and reduce the over-detection rate and under-detection rate of the corresponding product.

[0005] In a first aspect, embodiments of this application provide an image processing method. The method includes: acquiring multiple images to be labeled; performing image value prediction processing on each image to be labeled based on a Gaussian mixture prediction model to obtain an image value confidence score for each image to be labeled, wherein the Gaussian mixture prediction model is a machine learning model obtained through iterative training using an initial labeled image as training data; selecting a target image to be labeled from the multiple images to be labeled based on the image value confidence score of each image to be labeled; updating the model parameters of an initial image detection model based on the target labeled image and the initial labeled image to obtain a target image detection model, wherein the target labeled image is obtained by labeling the target image to be labeled; and performing image detection processing on each image to be labeled based on the target image detection model to obtain an image defect result for each image to be labeled.

[0006] Secondly, embodiments of this application provide an image processing apparatus. This image processing apparatus includes, but is not limited to, terminal devices, servers, etc. The image processing apparatus includes an acquisition unit and a processing unit. The acquisition unit is used to acquire multiple images to be labeled. The processing unit is used to perform image value prediction processing on each image to be labeled based on a Gaussian mixture prediction model to obtain the image value confidence score of each image to be labeled. The Gaussian mixture prediction model is a machine learning model obtained by iterative training using an initial labeled image as training data. The processing unit is used to select a target image to be labeled from the multiple images to be labeled based on the image value confidence score of each image to be labeled. The processing unit is used to update the model parameters of an initial image detection model based on the target labeled image and the initial labeled image to obtain a target image detection model. The target labeled image is obtained by labeling the target image to be labeled. The processing unit is used to perform image detection processing on each image to be labeled based on the target image detection model to obtain the image defect result of each image to be labeled.

[0007] In some optional implementations, the processing unit is configured to: extract at least one first feature map of a first image to be labeled, wherein the first image to be labeled is any one of a plurality of images to be labeled; perform classification and regression processing on the second feature maps generated in each anchor box on each first feature map to obtain a first feature vector and a second feature vector for each anchor box, wherein the first feature vector is used to indicate the Gaussian mixture probability distribution of the classification result of the corresponding anchor box, and the second feature vector is the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box; calculate the class value confidence of the corresponding anchor box based on the first feature vector and the second feature vector of each anchor box, wherein the class value confidence of the anchor box is used to indicate the value degree of the first image to be labeled in the class corresponding to the anchor box; and calculate the image value confidence of the first image to be labeled based on the class value confidence of each anchor box.

[0008] In some alternative embodiments, the processing unit is configured to: extract classification mean information, classification weight information, and classification variance information from the first feature vector of the first anchor box; and extract regression mean information, regression weight information, and regression variance information from the second feature vector of the first anchor box, wherein the first anchor box is any one of the anchor boxes; calculate a first classification confidence score of the first anchor box based on the classification variance information and classification weight information; calculate a second classification confidence score of the first anchor box based on the classification mean information and classification weight information; calculate a first regression confidence score of the first anchor box based on the regression variance information and regression weight information; and calculate a class value confidence score of the first anchor box based on the first classification confidence score, the second classification confidence score, the first regression confidence score, and the second regression confidence score.

[0009] In some alternative implementations, the processing unit is configured to: weight the classification weight information with the classification variance information based on each prediction sub-model to obtain the first classification confidence value of the first anchor box in the corresponding prediction sub-model, wherein the Gaussian mixture prediction model includes prediction sub-models; and sum the first classification confidence values ​​of the first anchor boxes in each prediction sub-model to obtain the first classification confidence level of the first anchor boxes.

[0010] In some alternative implementations, the processing unit is configured to: calculate the target classification mean of the first anchor box based on classification weight information and classification mean information; calculate the distance difference between the classification mean information and the target classification mean; weight the distance difference based on the classification weight information for each prediction sub-model to obtain the second classification confidence value of the first anchor box in the corresponding prediction sub-model, wherein the Gaussian mixture prediction model includes prediction sub-models; and sum the second classification confidence values ​​of the first anchor box in each prediction sub-model to obtain the second classification confidence level of the first anchor box.

[0011] In some alternative implementations, the processing unit is configured to: weight the classification weight information with the classification variance information based on each prediction sub-model to obtain the target classification confidence value of the first anchor box in the corresponding prediction sub-model; and sum the target classification confidence values ​​of the first anchor box in each prediction sub-model to obtain the target classification mean of the first anchor box.

[0012] In some alternative implementations, the processing unit is configured to: select the maximum confidence value from the first classification confidence value, the second classification confidence value, the first regression confidence value, and the second regression confidence value as the category value confidence value of the first anchor box.

[0013] In some alternative implementations, the processing unit is configured to: sum the category value confidence scores of each anchor box in the first image to be labeled, to obtain the image value confidence score of the first image to be labeled.

[0014] In some alternative implementations, the processing unit is configured to: construct an image value confidence matrix for each image to be labeled based on a first image value confidence score for each image to be labeled; perform prediction processing on the image value confidence matrix for each image to be labeled based on a target prediction model to obtain the image value prediction probability for the corresponding image to be labeled; and select the image to be labeled that has an image value prediction probability greater than or equal to a preset threshold from multiple images to be labeled as the target image to be labeled.

[0015] In some alternative embodiments, the processing unit is further configured to: after selecting the images to be labeled from a plurality of images to be labeled, corresponding to the images whose image value prediction probability is greater than or equal to a preset threshold, as target images to be labeled, calculate the value confidence distance between each pair of target images to be labeled based on the first image value confidence of each pair of target images to be labeled; sum the value confidence distances between each pair of target images to be labeled to obtain the total value confidence distance of the target images to be labeled; and update the model parameters of the target prediction model based on the total value confidence distance.

[0016] In some alternative implementations, the processing unit is used to: process the image value confidence of every two target images to be labeled based on the KL divergence model to obtain the value confidence distance between every two target images to be labeled.

[0017] In some alternative implementations, the processing unit is configured to: input each of the images to be labeled into the region proposal network in the target image detection model to obtain a detection box corresponding to the image to be labeled; and process the detection box of each of the images to be labeled based on the region convolutional neural network in the target image detection model to obtain an image defect result corresponding to the image to be labeled.

[0018] A third aspect of this application provides an image processing apparatus, including: a memory, an input / output (I / O) interface, and a processor. The memory stores program instructions. The processor executes the program instructions in the memory to perform the image processing method corresponding to the embodiment of the first aspect described above.

[0019] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the embodiments of the first aspect described above.

[0020] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the implementation method of the first aspect.

[0021] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0022] In this embodiment, after acquiring multiple images to be labeled, a machine learning model (i.e., a Gaussian mixture prediction model) trained iteratively using the initial labeled image as training data is used to perform image value prediction processing on each image to be labeled, thereby estimating the image value confidence level of each image to be labeled. Then, after obtaining the image value confidence level of each image to be labeled, a target image to be labeled is selected from the multiple images to be labeled based on the image value confidence level of each image to be labeled. Further, the model parameters of the initial image detection model are updated according to the target labeled image and the initial labeled image, thereby training a target image detection model. The target labeled image mentioned is obtained by labeling the target image to be labeled. In this way, the target image detection model is used to perform image detection processing on each image to be labeled, thereby obtaining the image defect result of each image to be labeled. In this embodiment, the Gaussian mixture prediction model is used to process each image to be labeled. Instead of outputting the bounding box position and image category of each image as in existing two-step methods, it estimates the image value confidence level of each image. This image value confidence level reflects the current image value of each image, helping to characterize its value. Furthermore, this embodiment selects target images from a massive number of images based on the image value confidence level. This not only eliminates the significant annotation costs required for manual annotation of a large number of images but also provides training data for subsequent iterations of the initial image detection model, reducing the amount of data needed for model learning. This allows the finally trained target image detection model to detect image defects in each image. In other words, this embodiment trains the target image detection model with less labeled data, quickly achieving image detection of images to be labeled and reducing the over-detection and under-detection rates of the corresponding products. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This illustration shows an application scenario diagram of the image processing method provided in the embodiments of this application;

[0025] Figure 2 This illustration shows a schematic diagram of an image processing workflow framework provided in an embodiment of this application;

[0026] Figure 3 A flowchart of an image processing method provided in an embodiment of this application is shown;

[0027] Figure 4 A flowchart illustrating the process of estimating image value confidence provided in an embodiment of this application is shown.

[0028] Figure 5 This paper illustrates the output diagram of the Gaussian distribution feature vector provided in an embodiment of this application.

[0029] Figure 6 This illustration shows an update diagram of the target prediction model provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram of the functional modules of the image processing apparatus provided in the embodiments of this application is shown;

[0031] Figure 8 A schematic diagram of the hardware structure of the image processing device provided in an embodiment of this application is shown. Detailed Implementation

[0032] This application provides an image processing method, apparatus, device, storage medium, and program product that can train a target image detection model with less labeled data, quickly achieve image detection of the labeled image, and reduce the over-detection rate and under-detection rate of the corresponding product.

[0033] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that implementations of the application described herein can be implemented, for example, in sequences other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] With the research and advancement of artificial intelligence (AI) technology, AI technology is being researched and applied in many fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, self-driving cars, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI technology will be applied in more fields and play an increasingly important role.

[0037] Artificial intelligence-based object detection technology is a technique for locating and recognizing objects in images, and it is one of the fundamental tasks in computer vision within the field of artificial intelligence. Object detection is widely used in various image processing devices, improving the efficiency of object recognition in labeled images.

[0038] However, existing solutions primarily rely on the Faster R-CNN model to detect defects in industrial scenarios. But the Faster R-CNN model requires a large number of parameters in its backbone network, resulting in a large amount of data to be trained. Furthermore, it demands significant investment in annotation to meet the model's training requirements, which not only hinders rapid image detection but also fails to guarantee both over-detection and under-detection rates.

[0039] Therefore, to solve the technical problems described above, this application provides an image processing method. The image processing method provided in this application is based on artificial intelligence. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0040] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech recognition, natural language processing, and machine learning / deep learning.

[0041] In the embodiments of this application, the main artificial intelligence technologies involved include the aforementioned computer vision technology, machine learning, and other related fields. For example, it may involve image processing and image recognition in computer vision (CV); it may also involve deep learning in machine learning (ML), including artificial neural networks; and it may also involve active learning in teaching learning within machine learning.

[0042] The image processing method provided in this application can be applied to image processing devices with data processing capabilities, such as terminal devices, servers, and question-answering robots. Terminal devices may include, but are not limited to, smartphones, desktop computers, laptops, tablets, smart speakers, in-vehicle devices, smartwatches, wearable smart devices, smart voice interaction devices, smart home appliances, and aircraft. Servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services; this application does not impose specific limitations. Furthermore, the mentioned terminal devices and servers can be directly or indirectly connected via wired or wireless communication; this application does not impose specific limitations.

[0043] The image processing device mentioned above is capable of implementing the aforementioned computer vision technology. Computer vision technology is a science that studies how to enable machines to "see," specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition, trajectory tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition. In the embodiments of this application, the image processing device can use this computer vision technology to perform feature map extraction and other processing on the image to be labeled.

[0044] Furthermore, this image processing device can also possess machine learning capabilities. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning, deep learning, and instructional learning typically include technologies such as neural networks. The image processing method provided in this application uses an artificial intelligence model, mainly involving the application of neural networks. Through neural networks, the image value of the images to be labeled is predicted, thereby continuously updating and adjusting the model parameters for the selected images with higher image value, and then performing recognition processing on each image to be labeled.

[0045] To facilitate understanding of the technical solutions of this application, the image processing methods provided in the embodiments of this application are described below in conjunction with application scenarios. Figure 1 The illustration shows an application scenario diagram of the image processing method provided in the embodiments of this application.

[0046] As in Figure 1 In the application scenarios shown, the image processing method provided in this application embodiment can be applied to image processing devices such as portable computers. As an illustrative description, Figure 1 Taking the portable computer shown as an example, the portable computer includes 131 imaging points. Using a camera or similar device, images can be captured and processed at each of these 131 imaging points, thereby acquiring images from different imaging points.

[0047] It should be noted that, in practical applications, the image processing method provided in this application embodiment can also be applied to other 3C devices with a large number of imaging points. This application embodiment only uses the method described above. Figure 1 The portable computer shown in the image is used as an example for illustration; no specific example is specified.

[0048] Figure 2 A schematic diagram of an image processing workflow framework provided in an embodiment of this application is shown.

[0049] like Figure 2 As shown, after acquiring multiple images to be labeled, a Gaussian mixture prediction model can be used to predict the image value of each image, thereby estimating the image value confidence score of each image. Furthermore, since the image value confidence score of each image reflects its initial image value, the target images with higher image value can be selected from these multiple images based on the image value confidence score. Thus, by labeling the target images and obtaining the corresponding labeled images, the model parameters of the initial image detection model can be updated in conjunction with the initial labeled images. Finally, the updated target image detection model can detect the image defects of each image. By using the image value confidence score to characterize the image value of the corresponding image to be labeled, suitable target images can be selected, thus requiring only the selected target images to be labeled, saving labeling costs. Moreover, training the target image detection model with less labeled data enables rapid image detection of the images to be labeled.

[0050] The following describes an image processing method provided by an embodiment of this application with reference to the accompanying drawings. Figure 3 A flowchart illustrating an image processing method provided in an embodiment of this application is shown. Figure 3 As shown, the image processing method may include the following steps:

[0051] 301. Obtain multiple images to be labeled.

[0052] In this example, defects can be detected in products that are either defective or not, such as defects in the product's frame. For instance, an image can be taken of the product to be inspected at the imaging point, thus obtaining multiple images of the product to be labeled. For example, as described above... Figure 1 Taking the portable computer shown as an example, the portable computer can be photographed at 131 imaging points to obtain 131 images of the portable computer to be labeled.

[0053] It should be noted that the product to be tested mentioned above can be a 3C product, or other products that require image detection, etc., and this application embodiment does not limit the description. The 3C products described are a collective term for computer electronic products, communication electronic products, and consumer electronic products, such as laptops, tablets, etc.

[0054] 302. Based on the Gaussian mixture prediction model, image value prediction processing is performed on each image to be labeled to obtain the image value confidence of each image to be labeled. The Gaussian mixture prediction model is a machine learning model obtained by iterative training using the initial labeled images as training data.

[0055] In this example, the Gaussian mixture prediction model is a machine learning model obtained through iterative training using the initially labeled images as training data. The training objective of this Gaussian mixture prediction model is to predict the image value of unlabeled images to be labeled. By predicting the image value confidence level of the images to be labeled, the confidence level of the image value of the corresponding images to be labeled can be reflected from the image value confidence level, so as to select high-value images to be labeled in the future for adjusting the model parameters.

[0056] After acquiring multiple images to be labeled, each of these images can be used as input to a Gaussian mixture prediction model. This model then estimates the image value of each image, thereby predicting the corresponding image value confidence level. The described image value confidence level indicates the image value of the corresponding image to be labeled. It should be noted that the described Gaussian mixture prediction model may include, but is not limited to, a Faster region convolutional neural network (GMM-based Faster R-CNN) model based on a Gaussian mixture distribution; specific implementations in this application are not limited to this model.

[0057] For example, the estimation process for determining the image value confidence level for each image to be labeled can be referred to as follows. Figure 4 The method embodiments shown are for understanding purposes. Figure 4 As shown, the prediction process includes at least the following steps:

[0058] S401. Extract at least one first feature map from the first image to be labeled, wherein the first image to be labeled is any one of a plurality of images to be labeled.

[0059] In this example, after obtaining multiple images to be labeled, feature maps can be extracted from each image using network models such as convolutional neural networks to extract at least one first feature map of the corresponding image. Taking any one of the multiple images to be labeled (i.e., the first image to be labeled) as an example, the first image to be labeled is used as the input to a convolutional neural network to extract at least one first feature map of the first image to be labeled.

[0060] S402. Classify and regress the second feature maps generated in each anchor box on each first feature map to obtain the first feature vector and the second feature vector of each anchor box. The first feature vector is used to indicate the Gaussian mixture probability distribution of the classification result of the corresponding anchor box, and the second feature vector is used to indicate the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box.

[0061] In this example, for each first feature map in the first image to be labeled, a region proposal network (RPN) can be used to generate m anchor boxes in each first feature map. Here, m ≥ 1, and m is an integer. After generating each first feature map, classification and regression processing are performed on the second feature maps within each anchor box generated on each first feature map to obtain the first feature vector and second feature vector of each anchor box in each first feature map. It should be noted that the first feature vector indicates the Gaussian mixture probability distribution of the classification result of the corresponding anchor box. In other words, the first feature vector reflects the probability distribution of the anchor box's category. The second feature vector indicates the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box. That is, the second feature vector reflects the probability distribution of the detection box position of the anchor box.

[0062] As an illustrative description, Figure 5 A schematic diagram of the output of the Gaussian distribution eigenvector provided in an embodiment of this application is shown. For example... Figure 5 As shown, taking any first feature map as an example, each sliding window in the first feature map is associated with m anchor boxes. The sizes of these m anchor boxes are all different. The intermediate layers in the convolutional neural network process the second feature map within each anchor box, outputting two branch vectors: a classification branch vector (i.e., the first feature vector described above) and a regression branch vector (i.e., the second feature vector described above). It should be noted that the first feature vector is a vector of length 2m × 3k, and the second feature vector is a vector of length 4m × 3k. Here, m is the number of anchor boxes, and k is the dimension in the Gaussian mixture probability distribution.

[0063] from Figure 5It can also be seen that the first feature vector includes classification mean information. Classification weight information and classification variance information Among them, the classification mean information Classification weight information and classification variance information The second feature vector includes information about the regression mean. Regression weight information and regression variance information Among them, regression mean information Regression weight information and regression variance information It should be noted that k represents the k-th predictor sub-model in the Gaussian mixture prediction model, 1≤k≤K, where K is the total number of predictor sub-models in the Gaussian mixture prediction model, and k and K are integers. Furthermore, p represents the class, and b can specifically be represented as {x,y,w,h}, where (x,y) represents the position of the detection box, and w and h represent hyperparameters.

[0064] Furthermore, when the aforementioned classification variance information The larger the value, the less accurate the category of the anchor box. Similarly, when regression variance information... The larger the value of , the more unstable the detection box position of the anchor point. The less accurate the anchor point category and the more unstable the detection box position, the higher the degree of confusion in the image region where the anchor point is located, and therefore the more valuable the annotation.

[0065] S403. Calculate the class value confidence of the corresponding anchor box based on the first feature vector and the second feature vector of each anchor box. The class value confidence of the anchor box is used to indicate the value of the first image to be labeled in the class corresponding to the anchor box.

[0066] In this example, after calculating the first and second feature vectors for each anchor box, the class value confidence score of the corresponding anchor box can be calculated based on the first and second feature vectors. It should be noted that the class value confidence score indicates the value of the first image to be labeled within the class corresponding to the anchor box.

[0067] Regarding the process of calculating the class value confidence of anchor boxes, this application embodiment uses any one anchor box (i.e., the first anchor box) among the multiple anchor boxes in the first feature map as an example for explanation. The class value confidence of other anchor boxes can also be understood by referring to the calculation process of the class value confidence of the first anchor box, and will not be elaborated upon here. Taking the first anchor box as an example, its class value confidence can be understood in the following way:

[0068] First, the classification mean, classification weight, and classification variance information are extracted from the first feature vector of the first anchor box. Similarly, the regression mean, regression weight, and regression variance information are extracted from the second feature vector of the first anchor box. It should be noted that the classification mean, classification weight, and classification variance information described here can be understood by referring to the content shown in step S402 above, and will not be repeated here.

[0069] Then, the classification mean information is extracted. Classification weight information and classification variance information Then, based on the classification variance information and classification weight information Calculate the first classification confidence score of the first anchor box. As an example, since the Gaussian mixture prediction model includes K prediction sub-models, the classification variance information is extracted... and classification weight information Then, the classification variance information can be... and classification weight information As input to each prediction sub-model, to process the classification weight information through each prediction sub-model. Classification variance information Weighted processing is performed to calculate the first classification confidence value of the first anchor box in the corresponding prediction sub-model. For example, the first classification confidence value of the first anchor box in the k-th prediction sub-model is... Thus, after obtaining the first classification confidence value of the first anchor box in each prediction sub-model, the first classification confidence values ​​of the first anchor box in each prediction sub-model are summed to calculate the first classification confidence level u of the first anchor box. alp For example, u alp It can be used To express.

[0070] Similarly, in extracting the classification mean information Classification weight information and classification variance information Then, it can also be based on classification mean information. and classification weight information Calculate the second classification confidence score of the first anchor box. As an illustrative example, the classification weight information can be first... and classification variance information As input to each prediction sub-model, to process the classification weight information through each prediction sub-model. Classification variance information Weighted processing is performed to calculate the target classification confidence value of the first anchor box in the corresponding prediction sub-model. For example, the target classification confidence value of the first anchor box in the i-th prediction sub-model is... Where 1 ≤ i ≤ k, and i is an integer. Thus, after obtaining the target classification confidence value of the first anchor box in each prediction sub-model, the target classification confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the target classification mean of the first anchor box, i.e.

[0071] After calculating the target classification mean of the first anchor box Then, the distance difference between the classification mean and the target classification mean is further calculated, i.e. Then, the classification weight information and distance difference As input to each prediction sub-model, to process the classification weight information through each prediction sub-model. Distance difference Weighted values ​​are then applied to calculate the second classification confidence value of the first anchor box in the corresponding prediction sub-model. For example, the second classification confidence value of the first anchor box in the k-th prediction sub-model is... Thus, after obtaining the second classification confidence value of the first anchor box in each prediction sub-model, the second classification confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the second classification confidence level u of the first anchor box. epp ,Right now

[0072]

[0073] Similarly, the regression mean information is extracted from the second feature vector. Regression weight information and regression variance information Then, it can also be based on regression weight information. and regression variance information Calculate the first regression confidence score for the first anchor box. Similarly, based on the regression mean information... and regression weight information Calculate the second regression confidence level of the first anchor box.

[0074] As an illustrative description, the calculation of the first regression confidence score of the first anchor box can also be understood in the following way: after extracting the regression variance information... and regression weight information Then, the regression variance information can be... and regression weight information As input to each predictive sub-model, to process the regression variance information through each predictive sub-model. and regression weight information Weighted values ​​are then applied to calculate the first regression confidence value of the first anchor box in the corresponding prediction sub-model. For example, the first regression confidence value of the first anchor box in the k-th prediction sub-model is... Thus, after obtaining the first regression confidence value of the first anchor box in each prediction sub-model, the first regression confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the first regression confidence level u of the first anchor box. alb ,Right now

[0075] As an illustrative description, the calculation of the second regression confidence score of the first anchor box can also be understood in the following way: after extracting the regression mean information... Regression weight information and regression variance information Afterwards, it can also be based on the regression mean information. and regression weight information Calculate the second regression confidence level of the first anchor box.

[0076] For example, regression weight information can be used. and regression variance information As input to each predictive sub-model, to process the regression weight information through each predictive sub-model. and regression variance information Weighted values ​​are then applied to calculate the target regression confidence value of the first anchor box in the corresponding prediction sub-model. For example, the target regression confidence value of the first anchor box in the i prediction sub-models is... Where 1 ≤ i ≤ k, and i is an integer. Thus, after obtaining the target regression confidence value of the first anchor box in each prediction sub-model, the target regression confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the target regression mean of the first anchor box, i.e. After calculating the target regression mean of the first anchor point box, the regression mean information can be calculated. Regression mean with target The distance difference between them, i.e. Then, the regression weight information and distance difference As input to each predictive sub-model, the regression weight information π is passed through each predictive sub-model. b k Distance difference Weighted values ​​are then applied to calculate the second regression confidence value of the first anchor box in the corresponding prediction sub-model. For example, the second regression confidence value of the first anchor box in the k-th prediction sub-model is... Thus, after obtaining the second regression confidence value of the first anchor box in each prediction sub-model, the second regression confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the second regression confidence level u of the first anchor box. epb ,Right now

[0077] Finally, after the above steps, the first classification confidence level u is calculated. alp Second category confidence level u epp First regression confidence level u alb and the second regression confidence level u epb Then, based on the first classification confidence level u alp Second category confidence level u epp First regression confidence level u alb and the second regression confidence level u epb Calculate the category value confidence score of the first anchor box. For example, this can be derived from the first category confidence score u. alp Second category confidence level u epp First regression confidence level u alb and the second regression confidence level u epb In the selection process, the maximum confidence score is chosen as the category value confidence score of the first anchor box, i.e., P. ic (I,θ)=max(u epp ,u alp ,u epb ,u alb ), where P ic (I,θ) represents the category value confidence of the first anchor box. In other words, P ic (I,θ) can also be understood as the confidence level that the category of the first anchor box belongs to category c. It should be noted that category c is only used to represent one of the categories and is not specifically limited.

[0078] S404. Calculate the image value confidence of the first image to be labeled based on the category value confidence of each anchor box.

[0079] In this example, after calculating the class value confidence of each anchor box in the same way as described above, the image value confidence of the first image to be labeled can be calculated based on the class value confidence of each anchor box. For example, the class value confidence of each anchor box in the first image to be labeled can be summed to calculate the image value confidence P of the first image to be labeled. c .

[0080] 303. Select the target image to be labeled from multiple images based on the image value confidence of each image to be labeled.

[0081] In this example, based on the above Figure 4 In the processing steps S401 to S404, after calculating the image value confidence score of each image to be labeled, the target image to be labeled with high value can be selected from multiple images to be labeled based on the image value confidence score of the image to be labeled.

[0082] For example, based on the image value confidence score of each image to be labeled, an image value confidence score matrix P is constructed for each image to be labeled. c ×P c Then, the image value confidence matrix P for each image to be labeled is... c ×P c As input to the target prediction model, the image value confidence matrix P for each image to be labeled is obtained through the target prediction model. c ×P c Prediction processing is performed to predict the image value prediction probability of the corresponding image to be labeled. After calculating the image value prediction probability of each image to be labeled, a target image to be labeled can be selected from multiple images based on this probability. As an illustrative example, after calculating the image value prediction probability of each image to be labeled, this probability can be compared with a preset threshold. Images whose image value prediction probabilities are greater than or equal to the preset threshold are selected as target images to be labeled. For example, suppose the image value prediction probabilities of images to be labeled 1, 2, 3, 4, and 5 are 0.7, 0.8, 0.55, 0.45, and 0.68, respectively, and the preset threshold is 0.7. By comparison, it can be seen that only the image value prediction probabilities of images to be labeled 1 and 2 are greater than or equal to 0.7. Therefore, images to be labeled 1 and 2 can be selected as target images to be labeled.

[0083] It should be noted that the target prediction model described above may include, but is not limited to, bidirectional long short-term memory (Bi-LSTM) models, etc., and is not limited in the specific embodiments of this application.

[0084] In other examples, after selecting the images whose predicted value probability is greater than or equal to a preset threshold as target images for annotation, the model parameters of the target prediction model can be adjusted to continuously optimize its prediction performance. As an illustrative example, a Bi-LSTM model is used as the target prediction model. Figure 6 A schematic diagram illustrating the update of the target prediction model provided in an embodiment of this application is shown. For example... Figure 6As shown, this update process can be understood in the following way:

[0085] Constructing the image value confidence matrix P corresponding to the image to be labeled c ×P c Then, the image value confidence matrix P of each image to be labeled is generated. c ×P c As input to the target prediction model, the image value confidence matrix P for each image to be labeled is obtained through the target prediction model. c ×P c Prediction processing is performed to predict the image value prediction probability of the corresponding target image. Then, based on the image value confidence of each pair of target images, the value confidence distance between each pair of target images is calculated. For example, the first image value confidence of each pair of target images is processed based on the KL (Kullbank-Leibler) divergence model to obtain the value confidence distance between each pair of target images. .in,

[0086]

[0087] I N This represents two target images to be labeled when performing distance calculations.

[0088] Calculate the value confidence distance between every two target images to be labeled. The sum of the value confidence distances between any two target images is then calculated by summing the value confidence distances between them. This yields the total value confidence distance between the target images. Thus, based on the total distance d of value confidence... Ib Update the model parameters of the target prediction model.

[0089] 304. Based on the target labeled image and the initial labeled image, update the model parameters of the initial image detection model to obtain the target image detection model. The target labeled image is obtained by labeling the target image to be labeled.

[0090] In this example, because we want the output of the deep neural network to be as close as possible to the actual predicted value, we can compare the current network's predicted value with the actual target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the deep neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Therefore, training the deep neural network becomes the process of minimizing this loss as much as possible.

[0091] Therefore, after selecting the target image to be labeled, it can be labeled to obtain the labeled image. This labeled image is then added to the training data to update the parameters of the initial image detection model based on the labeled image and the initial labeled image, thereby updating the target image detection model.

[0092] As an illustrative description, image detection processing can be performed on the target labeled image and the initial labeled image based on the initial image detection model to predict the image defect prediction probability of each labeled image. The image defect prediction probability indicates the image defect category of the corresponding labeled image. Then, the difference between the image defect prediction probability of each labeled image and the label of the corresponding labeled image is calculated, and this difference is determined as the target loss value. After calculating the target loss value, the model parameters of the initial image detection model can be updated based on the target loss value to train the target image detection model. For example, the target loss value can be iteratively trained and optimized based on the gradient descent algorithm until the training result meets the condition for terminating model training. It should be noted that the described gradient descent algorithm can include stochastic gradient descent, stochastic gradient descent with moving average, adaptive gradient algorithm, adaptive moment estimation algorithm, etc., and is not specifically limited in this application embodiment. In addition, the condition for terminating model training can include setting the number of training iterations to meet a preset value, or the target loss value being less than a preset value, etc., and is not specifically limited in this application.

[0093] It should be noted that the initial image detection model mentioned may include, but is not limited to, the Faster R-CNN model in the two-stage method described above, and is not limited in the embodiments of this application.

[0094] In other examples, after selecting the target image to be labeled, it can be labeled to obtain a labeled image. This labeled image is then added to the training data to update the Gaussian mixture prediction model's parameters based on both the initial and target labeled images, resulting in an updated Gaussian mixture prediction model. This updated model can more accurately predict the image value confidence of each target image, laying the foundation for selecting high-quality target images for subsequent labeling.

[0095] 305. Based on the target image detection model, perform image detection processing on each of the images to be labeled to obtain the image defect results of each image to be labeled.

[0096] In this example, after updating the target image detection model, each image to be labeled can be used as input to the target image detection model to detect image defects in each image. For example, each image to be labeled can be input into the region proposal network in the target image detection model to obtain a detection box corresponding to the image to be labeled. Then, the detection boxes of each image to be labeled are processed based on the region convolutional neural network in the target image detection model to obtain the image defect results corresponding to the image to be labeled.

[0097] In this embodiment, each image to be labeled is processed based on a Gaussian mixture prediction model. Instead of outputting the bounding box position and image category of each image as in existing two-step methods, it estimates the image value confidence level of each image. This image value confidence level reflects the current image value of each image, helping to characterize its value. Thus, this embodiment selects target images from a massive number of images based on the image value confidence level. This not only saves the significant annotation costs required for manual annotation of a large number of images but also provides training data for subsequent iterations of training the initial image detection model, reducing the amount of data needed for model learning. This allows the finally trained target image detection model to detect image defects in each image. In other words, this embodiment trains the target image detection model with less labeled data, quickly achieving image detection of images to be labeled and reducing the over-detection and under-detection rates of the corresponding products.

[0098] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0100] The image processing apparatus in the embodiments of this application will now be described in detail. Figure 7 This is a schematic diagram of one embodiment of the image processing apparatus provided in this application. The image processing apparatus includes, but is not limited to, terminal devices, servers, etc. Figure 7 As shown, the image processing apparatus may include an acquisition unit 701 and a processing unit 702.

[0101] The acquisition unit 701 is used to acquire multiple images to be labeled. For details, please refer to the preceding description. Figure 3 The content described in step 301 is understood and will not be repeated here.

[0102] Processing unit 702 is used to perform image value prediction processing on each image to be labeled based on a Gaussian mixture prediction model to obtain the image value confidence score of each image to be labeled. The Gaussian mixture prediction model is a machine learning model obtained by iterative training using the initial labeled images as training data. For details, please refer to the aforementioned... Figure 3 The content described in step 302 is for your understanding and will not be repeated here.

[0103] Processing unit 702 is used to select a target image to be labeled from multiple images to be labeled based on the image value confidence score of each image to be labeled. For details, please refer to the foregoing. Figure 3 The content described in step 303 is for your understanding and will not be repeated here.

[0104] Processing unit 702 is used to update the model parameters of the initial image detection model based on the target labeled image and the initial labeled image to obtain the target image detection model. The target labeled image is obtained by labeling the target image to be labeled. See the foregoing for details. Figure 3 The content described in step 304 is for your understanding and will not be repeated here.

[0105] The processing unit 702 is used to perform image detection processing on each image to be labeled based on the target image detection model, and obtain the image defect results for each image to be labeled. For details, please refer to the foregoing. Figure 3 The content described in step 305 is for your understanding and will not be repeated here.

[0106] In some optional implementations, the processing unit 702 is configured to: extract at least one first feature map of a first image to be labeled, wherein the first image to be labeled is any one of a plurality of images to be labeled; perform classification and regression processing on the second feature maps generated in each anchor box on each first feature map to obtain a first feature vector and a second feature vector for each anchor box, wherein the first feature vector is used to indicate the Gaussian mixture probability distribution of the classification result of the corresponding anchor box, and the second feature vector is the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box; calculate the class value confidence of the corresponding anchor box based on the first feature vector and the second feature vector of each anchor box, wherein the class value confidence of the anchor box is used to indicate the value degree of the first image to be labeled in the category corresponding to the anchor box; and calculate the first image value confidence of the first image to be labeled based on the class value confidence of each anchor box.

[0107] In some alternative embodiments, the processing unit 702 is configured to: extract classification mean information, classification weight information, and classification variance information from the first feature vector of the first anchor box; and extract regression mean information, regression weight information, and regression variance information from the second feature vector of the first anchor box, wherein the first anchor box is any one of the anchor boxes; calculate a first classification confidence score of the first anchor box based on the classification variance information and classification weight information; calculate a second classification confidence score of the first anchor box based on the classification mean information and classification weight information; calculate a first regression confidence score of the first anchor box based on the regression variance information and regression weight information; and calculate a class value confidence score of the first anchor box based on the first classification confidence score, the second classification confidence score, the first regression confidence score, and the second regression confidence score.

[0108] In some alternative implementations, the processing unit 702 is configured to: weight the classification weight information with the classification variance information based on each prediction sub-model to obtain the first classification confidence value of the first anchor box in the corresponding prediction sub-model, wherein the Gaussian mixture prediction model includes the prediction sub-model; and sum the first classification confidence values ​​of the first anchor box in each prediction sub-model to obtain the first classification confidence degree of the first anchor box.

[0109] In some alternative embodiments, the processing unit 702 is configured to: calculate the target classification mean of the first anchor box based on classification weight information and classification mean information; calculate the distance difference between the classification mean information and the target classification mean; weight the distance difference based on the classification weight information for each prediction sub-model to obtain the second classification confidence value of the first anchor box in the corresponding prediction sub-model, wherein the Gaussian mixture prediction model includes prediction sub-models; and sum the second classification confidence values ​​of the first anchor box in each prediction sub-model to obtain the second classification confidence level of the first anchor box.

[0110] In some alternative implementations, the processing unit 702 is configured to: weight the classification weight information with the classification variance information based on each prediction sub-model to obtain the target classification confidence value of the first anchor box in the corresponding prediction sub-model; and sum the target classification confidence values ​​of the first anchor box in each prediction sub-model to obtain the target classification mean of the first anchor box.

[0111] In some alternative implementations, the processing unit 702 is configured to: select the maximum confidence value from the first classification confidence value, the second classification confidence value, the first regression confidence value, and the second regression confidence value as the category value confidence value of the first anchor box.

[0112] In some alternative embodiments, the processing unit 702 is configured to: sum the category value confidence scores of each anchor box in the first image to be labeled, to obtain the first image value confidence score of the first image to be labeled.

[0113] In some alternative embodiments, the processing unit 702 is configured to: construct an image value confidence matrix for each image to be labeled based on the image value confidence of each image to be labeled; perform prediction processing on the image value confidence matrix of each image to be labeled based on the target prediction model to obtain the image value prediction probability of the corresponding image to be labeled; and select the image to be labeled that has an image value prediction probability greater than or equal to a preset threshold from multiple images to be labeled as the target image to be labeled.

[0114] In some alternative embodiments, the processing unit 702 is further configured to: after selecting the images to be labeled from a plurality of images to be labeled as target images when the image value prediction probability is greater than or equal to a preset threshold, calculate the value confidence distance between each pair of target images to be labeled based on the image value confidence of each pair of target images to be labeled; sum the value confidence distances between each pair of target images to be labeled to obtain the total value confidence distance of the target images to be labeled; and update the model parameters of the target prediction model based on the total value confidence distance.

[0115] In some alternative implementations, the processing unit 702 is used to: process the image value confidence of every two target images to be labeled based on the KL divergence model to obtain the value confidence distance between every two target images to be labeled.

[0116] In some alternative embodiments, the processing unit 702 is configured to: input each of the images to be labeled into the region proposal network in the target image detection model to obtain a detection box corresponding to the image to be labeled; and process the detection box of each image to be labeled based on the region convolutional neural network in the target image detection model to obtain an image defect result corresponding to the image to be labeled.

[0117] The image processing apparatus in the embodiments of this application has been described above from the perspective of modular functional entities. The image processing device in the embodiments of this application will now be described below from the perspective of hardware processing. The described image processing device may include, but is not limited to, terminal devices, servers, etc. Figure 8 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. The image processing device can vary considerably due to differences in configuration or performance. The image processing device may include at least one processor 801, a communication line 807, a memory 803, and at least one communication interface 804.

[0118] The processor 801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more integrated circuits used to control the execution of the program of the present application.

[0119] Communication line 807 may include a path for transmitting information between the aforementioned components.

[0120] The communication interface 804 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0121] The memory 803 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions. The memory can exist independently and be connected to the processor via communication line 807. The memory can also be integrated with the processor.

[0122] The memory 803 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 801. The processor 801 executes the computer execution instructions stored in the memory 803, thereby implementing the image processing method provided in the above embodiments of this application.

[0123] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0124] In a specific implementation, as one example, the image processing device may include multiple processors, such as... Figure 8 Processors 801 and 802 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0125] In a specific implementation, as one embodiment, the image processing device may further include an output device 805 and an input device 806. The output device 805 communicates with the processor 801 and can display information in various ways. The input device 806 communicates with the processor 801 and can receive input from the target object in various ways. For example, the input device 806 may be a mouse, a touch screen device, or a sensing device, etc.

[0126] The aforementioned image processing device can be a general-purpose device or a dedicated device. In specific implementations, the image processing device can be a server, terminal device, or other similar device. Figure 8 A device with a similar structure. The embodiments of this application do not limit the type of image processing device.

[0127] It should be noted that Figure 8 The processor 801 in the memory can call computer execution instructions stored in the memory 803 to cause the image processing device to perform actions such as... Figures 3 to 4 The method in the corresponding method embodiment.

[0128] Specifically, Figure 7 The function / implementation process of the processing unit 702 can be achieved through... Figure 8 The processor 801 in the memory calls computer execution instructions stored in the memory 803 to implement the function. Figure 7 The function / implementation process of the acquisition unit 701 can be achieved through... Figure 8 It is implemented using the 804 communication interface.

[0129] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the image processing methods described in the above method embodiments.

[0130] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the image processing methods described in the above method embodiments.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0138] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs), etc.

[0139] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Obtain multiple images to be labeled; Image value prediction processing is performed on each of the images to be labeled based on the Gaussian mixture prediction model to obtain the image value confidence of each image to be labeled. The Gaussian mixture prediction model is a machine learning model obtained by iterative training using the initial labeled images as training data. Based on the image value confidence score of each of the images to be labeled, an image value confidence matrix corresponding to each image to be labeled is constructed. Based on the target prediction model, the image value confidence matrix of each image to be labeled is predicted to obtain the image value prediction probability of the corresponding image to be labeled. From the plurality of images to be labeled, the images corresponding to which the image value prediction probability is greater than or equal to a preset threshold are selected as target images to be labeled; Based on the image value confidence of every two target images to be labeled, calculate the value confidence distance between each pair of target images to be labeled; The total value confidence distance between any two target images to be labeled is obtained by summing the value confidence distances between them. The target prediction model is updated based on the total distance of the value confidence score. The model parameters of the initial image detection model are updated based on the target labeled image and the initial labeled image to obtain the target image detection model. The target labeled image is obtained by labeling the target image to be labeled. Based on the target image detection model, image detection processing is performed on each of the images to be labeled to obtain the image defect results of each image to be labeled.

2. The method according to claim 1, characterized in that, The image value prediction process based on the Gaussian mixture prediction model is performed on each of the images to be labeled to obtain the image value confidence score of each image to be labeled, including: Extract at least one first feature map from the first image to be labeled, wherein the first image to be labeled is any one of the plurality of images to be labeled; Classification and regression processing are performed on the second feature maps generated in each anchor box on each of the first feature maps to obtain a first feature vector and a second feature vector for each anchor box. The first feature vector is used to indicate the Gaussian mixture probability distribution of the classification result of the corresponding anchor box, and the second feature vector is used to indicate the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box. The category value confidence of each anchor box is calculated based on the first feature vector and the second feature vector of each anchor box. The category value confidence of the anchor box is used to indicate the value of the first image to be labeled in the category corresponding to the anchor box. The image value confidence of the first image to be labeled is calculated based on the category value confidence of each anchor box.

3. The method according to claim 2, characterized in that, Calculate the class value confidence score of each anchor box based on its first and second feature vectors, including: The classification mean, classification weight, and classification variance information are extracted from the first feature vector of the first anchor box, and the regression mean, regression weight, and regression variance information are extracted from the second feature vector of the first anchor box, wherein the first anchor box is any one of the anchor boxes. The first classification confidence score of the first anchor box is calculated based on the classification variance information and the classification weight information, and the second classification confidence score of the first anchor box is calculated based on the classification mean information and the classification weight information. The first regression confidence score of the first anchor box is calculated based on the regression variance information and the regression weight information, and the second regression confidence score of the first anchor box is calculated based on the regression mean information and the regression weight information. The category value confidence of the first anchor box is calculated based on the first classification confidence, the second classification confidence, the first regression confidence, and the second regression confidence.

4. The method according to claim 3, characterized in that, The step of calculating the first classification confidence score of the first anchor box based on the classification variance information and the classification weight information includes: Based on each prediction sub-model, the classification weight information is weighted and the classification variance information is weighted to obtain the first classification confidence value of the first anchor box in the corresponding prediction sub-model. The Gaussian mixture prediction model includes the prediction sub-model. The first classification confidence value of the first anchor box in each prediction sub-model is summed to obtain the first classification confidence level of the first anchor box.

5. The method according to claim 3, characterized in that, The calculation of the second classification confidence score of the first anchor box based on the classification mean information and the classification weight information includes: Based on the classification weight information and the classification mean information, calculate the target classification mean of the first anchor box; Calculate the distance difference between the classification mean information and the target classification mean; Based on each prediction sub-model, the classification weight information is used to weight the distance difference to obtain the second classification confidence value of the first anchor box in the corresponding prediction sub-model. The Gaussian mixture prediction model includes the prediction sub-model. The second classification confidence value of the first anchor box in each prediction sub-model is summed to obtain the second classification confidence level of the first anchor box.

6. The method according to claim 5, characterized in that, Based on the classification weight information and the classification mean information, the target classification mean of the first anchor box is calculated, including: Based on each prediction sub-model, the classification weight information is weighted and the classification variance information is weighted to obtain the target classification confidence value of the first anchor box in the corresponding prediction sub-model. The target classification confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the target classification mean of the first anchor box.

7. The method according to any one of claims 3 to 6, characterized in that, The category value confidence of the first anchor box is calculated based on the first classification confidence, the second classification confidence, the first regression confidence, and the second regression confidence, including: The highest confidence level among the first classification confidence level, the second classification confidence level, the first regression confidence level, and the second regression confidence level is selected as the category value confidence level of the first anchor box.

8. The method according to any one of claims 2 to 5, characterized in that, Calculating the image value confidence of the first image to be labeled based on the category value confidence of each anchor box includes: The image value confidence of the first image to be labeled is obtained by summing the category value confidence of each anchor box in the first image to be labeled.

9. The method according to claim 1, characterized in that, Based on the image value confidence of every two target images to be labeled, the value confidence distance between each pair of target images to be labeled is calculated, including: The image value confidence of each pair of target images to be labeled is processed based on the KL divergence model to obtain the value confidence distance between each pair of target images to be labeled.

10. The method according to any one of claims 1 to 5, characterized in that, The step of performing image detection processing on each of the images to be labeled based on the target image detection model to obtain the image defect results for each image to be labeled includes: Each of the images to be labeled is input into the region proposal network in the target image detection model to obtain the detection box corresponding to the image to be labeled; The detection boxes of each image to be labeled are processed based on the region convolutional neural network in the target image detection model to obtain the image defect results corresponding to the image to be labeled.

11. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire multiple images to be labeled; The processing unit is used to perform image value prediction processing on each of the images to be labeled based on a Gaussian mixture prediction model to obtain the image value confidence of each image to be labeled. The Gaussian mixture prediction model is a machine learning model obtained by iterative training using the initial labeled images as training data. The processing unit is configured to construct an image value confidence matrix corresponding to each of the images to be labeled based on the image value confidence of each image to be labeled; perform prediction processing on the image value confidence matrix of each image to be labeled based on the target prediction model to obtain the image value prediction probability of the corresponding image to be labeled; and select the image to be labeled that has an image value prediction probability greater than or equal to a preset threshold from the plurality of images to be labeled as the target image to be labeled. The processing unit is used to calculate the value confidence distance between each pair of target images to be labeled based on the image value confidence of each pair of target images to be labeled. The total value confidence distance between any two target images to be labeled is obtained by summing the value confidence distances between them. The target prediction model is updated based on the total distance of the value confidence score. The processing unit is used to update the model parameters of the initial image detection model based on the target labeled image and the initial labeled image to obtain the target image detection model. The target labeled image is obtained by labeling the target image to be labeled. The processing unit is used to perform image detection processing on each of the images to be labeled based on the target image detection model, so as to obtain the image defect results of each image to be labeled.

12. The apparatus according to claim 11, characterized in that, The processing unit is used for: Extract at least one first feature map from the first image to be labeled, wherein the first image to be labeled is any one of the plurality of images to be labeled; Classification and regression processing are performed on the second feature maps generated in each anchor box on each of the first feature maps to obtain a first feature vector and a second feature vector for each anchor box. The first feature vector is used to indicate the Gaussian mixture probability distribution of the classification result of the corresponding anchor box, and the second feature vector is used to indicate the Gaussian mixture probability distribution of the detection box result of the corresponding anchor box. The category value confidence of each anchor box is calculated based on the first feature vector and the second feature vector of each anchor box. The category value confidence of the anchor box is used to indicate the value of the first image to be labeled in the category corresponding to the anchor box. The image value confidence of the first image to be labeled is calculated based on the category value confidence of each anchor box.

13. The apparatus according to claim 12, characterized in that, The processing unit is used for: The classification mean, classification weight, and classification variance information are extracted from the first feature vector of the first anchor box, and the regression mean, regression weight, and regression variance information are extracted from the second feature vector of the first anchor box, wherein the first anchor box is any one of the anchor boxes. The first classification confidence score of the first anchor box is calculated based on the classification variance information and the classification weight information, and the second classification confidence score of the first anchor box is calculated based on the classification mean information and the classification weight information. The first regression confidence score of the first anchor box is calculated based on the regression variance information and the regression weight information, and the second regression confidence score of the first anchor box is calculated based on the regression mean information and the regression weight information. The category value confidence of the first anchor box is calculated based on the first classification confidence, the second classification confidence, the first regression confidence, and the second regression confidence.

14. The apparatus according to claim 13, characterized in that, The processing unit is used for: Based on each prediction sub-model, the classification weight information is weighted and the classification variance information is weighted to obtain the first classification confidence value of the first anchor box in the corresponding prediction sub-model. The Gaussian mixture prediction model includes the prediction sub-model. The first classification confidence value of the first anchor box in each prediction sub-model is summed to obtain the first classification confidence level of the first anchor box.

15. The apparatus according to claim 13, characterized in that, The processing unit is used for: Based on the classification weight information and the classification mean information, calculate the target classification mean of the first anchor box; Calculate the distance difference between the classification mean information and the target classification mean; Based on each prediction sub-model, the classification weight information is used to weight the distance difference to obtain the second classification confidence value of the first anchor box in the corresponding prediction sub-model. The Gaussian mixture prediction model includes the prediction sub-model. The second classification confidence value of the first anchor box in each prediction sub-model is summed to obtain the second classification confidence level of the first anchor box.

16. The apparatus according to claim 15, characterized in that, The processing unit is used for: Based on each prediction sub-model, the classification weight information is weighted and the classification variance information is weighted to obtain the target classification confidence value of the first anchor box in the corresponding prediction sub-model. The target classification confidence values ​​of the first anchor box in each prediction sub-model are summed to obtain the target classification mean of the first anchor box.

17. The apparatus according to any one of claims 13 to 16, characterized in that, The processing unit is used for: The highest confidence level among the first classification confidence level, the second classification confidence level, the first regression confidence level, and the second regression confidence level is selected as the category value confidence level of the first anchor box.

18. The apparatus according to any one of claims 12 to 15, characterized in that, The processing unit is used for: The image value confidence of the first image to be labeled is obtained by summing the category value confidence of each anchor box in the first image to be labeled.

19. The apparatus according to claim 11, characterized in that, The processing unit is used for: The image value confidence of each pair of target images to be labeled is processed based on the KL divergence model to obtain the value confidence distance between each pair of target images to be labeled.

20. The apparatus according to any one of claims 11 to 15, characterized in that, The processing unit is used for: Each of the images to be labeled is input into the region proposal network in the target image detection model to obtain the detection box corresponding to the image to be labeled; The detection boxes of each image to be labeled are processed based on the region convolutional neural network in the target image detection model to obtain the image defect results corresponding to the image to be labeled.

21. An image processing device, characterized in that, include: Input / output interface, processor, and memory, wherein the memory stores program instructions; The processor is configured to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 10.

Images