Image processing method, device, and related apparatus

Abstract

Embodiments of the present application disclose an image processing method, device and related equipment, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The image processing method comprises: obtaining a to-be-processed image; generating one or more candidate detection boxes corresponding to the to-be-processed image; calling a target detection model to process the one or more candidate detection boxes to obtain a detection result of the to-be-processed image, wherein the detection result comprises a target detection box corresponding to a target category in the one or more candidate detection boxes, and the target detection model is obtained by training a first branch network and a second branch network included in a first scoring network using a second scoring network, a sample image and a category label of the sample image. Through the embodiments of the present application, the accuracy of target detection on the image can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus and related equipment. Background Technology

[0002] In recent years, thanks to the development of deep learning, various computer vision tasks have made tremendous progress and breakthroughs. Object detection, as one of the most fundamental tasks in computer vision, is a challenging task and an indispensable foundational technology for many advanced applications (such as robot vision, face recognition, and autonomous driving).

[0003] Object detection aims to determine the category and location information of specific objects in an image, involving the classification and localization of object instances. Current weakly supervised object detection techniques are mostly based on multi-instance learning. This approach struggles to automatically learn highly accurate candidate detection boxes and is limited by candidate detection boxes corresponding to certain regions with strong decision-making power. Consequently, the trained detectors do not perform ideally, potentially leading to errors in the object detection results. Therefore, it is necessary to investigate a new method to improve the accuracy of object detection. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and related equipment, which can effectively improve the accuracy of target detection in images.

[0005] One embodiment of this application provides an image processing method, including:

[0006] Obtain the image to be processed;

[0007] Generate one or more candidate detection boxes corresponding to the image to be processed;

[0008] The object detection model is invoked to process one or more candidate detection boxes to obtain the detection result of the image to be processed. The detection result includes the object detection boxes in one or more candidate detection boxes that correspond to the object category. The object detection model is trained by using the second scoring network, sample images, and the category labels of the sample images to train the first scoring network, which includes the first branch network and the second branch network.

[0009] One embodiment of this application provides an image processing apparatus, including:

[0010] The acquisition module is used to acquire the image to be processed;

[0011] The generation module is used to generate one or more candidate detection boxes corresponding to the image to be processed;

[0012] The processing module is used to call the object detection model to process one or more candidate detection boxes to obtain the detection result of the image to be processed. The detection result includes the object detection boxes in one or more candidate detection boxes that correspond to the object category. The object detection model is trained by using the second scoring network, sample images, and the category labels of the sample images to train the first scoring network, which includes the first branch network and the second branch network.

[0013] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the image processing method of this application embodiment.

[0014] One aspect of this application provides a computer-readable storage medium storing a computer program, which includes program instructions. When executed by a processor, the program instructions perform the image processing method described in this application.

[0015] Accordingly, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium; the processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided in one aspect of the embodiments of this application.

[0016] In this embodiment, candidate detection boxes that may contain real targets are first generated for the image to be processed. Then, the candidate detection boxes are processed by calling the target detection model to obtain the detection results. Since the target detection model is trained on the first branch network and the second branch network included in the first score, the target detection model can utilize the complementarity of the dual-branch network to accurately determine the target detection box corresponding to the target category during the processing of these candidate detection boxes, thereby effectively improving the accuracy of target detection in the image. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an architecture diagram of an image processing system provided in an embodiment of this application;

[0019] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a visualization of a detection result provided in an embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating a target detection model training method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an exemplary image processing network provided in an embodiment of this application;

[0023] Figure 6 This is a flowchart illustrating another object detection model training method provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of another exemplary image processing network structure provided in the embodiments of this application;

[0025] Figure 8 This is a schematic diagram of another exemplary image processing network structure provided in the embodiments of this application;

[0026] Figure 9 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0030] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0031] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0032] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition. In one embodiment, the image processing scheme provided in this application, which generates candidate detection boxes corresponding to the image to be processed and obtains target detection boxes corresponding to the target category, specifically involves image processing and image semantic understanding technologies.

[0033] The image processing solutions provided in this application belong to the fields of computer vision (CV) and machine learning (ML) within the field of artificial intelligence. Furthermore, the solutions provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving, as illustrated in the following embodiments:

[0034] The architecture of the image processing system provided in the embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Please see Figure 1 , Figure 1 This is an architecture diagram of an image processing system provided in an embodiment of this application. For example... Figure 1 As shown in the diagram, the architecture may include a database 100 and an image processing device 101, wherein the database 100 and the image processing device 101 can communicate with each other via wired or wireless means. The database 100 may be a local database of the image processing device 101, or a cloud database accessible by the image processing device 101, etc. The image processing device 101 may be located in a computer device such as a server or terminal.

[0036] Database 100 may include an image library and a detection result library. The image library is used to store various types of image data and video data, which may include various real-time acquired pictures or videos, or videos or pictures uploaded by users through terminal devices. Based on these image data or video data, images to be processed can be obtained, such as directly using an image as the image to be processed or extracting a frame from a video as the image to be processed. The image library can also be used to store sample images, which may be training sets from professional sample libraries or custom sample images, such as sample images created using various images crawled from the Internet. Each sample image carries a real category label, which is used to indicate the category of the target object included in the sample image. Since object detection can identify the target of interest and its location in a given image, it is a foundational task for subsequent higher-order computer vision analysis tasks. Therefore, a dedicated detection result library can be allocated to store the detection results of the image to be processed. This allows other algorithms to directly call the detection results from the library when processing similar or identical images or when they need to use relevant results. The detection results can include one or more candidate detection boxes corresponding to the image to be processed and the target detection boxes corresponding to the target category. The target category can be one or more predefined categories, and each target category can correspond to multiple target detection boxes. The image within the region corresponding to the target detection box is the target object of that target category. In other words, the detection result includes the location information (target detection box) and category information (target category) of the target object.

[0037] Image processing device 101 can automatically acquire images to be processed from an image library or acquire images in real time through a camera device. Using the image processing scheme provided in this application, the images to be processed can be processed to obtain detection results. In this embodiment, candidate detection boxes corresponding to the images to be processed can first be generated. Then, a target detection model is used to detect the candidate detection boxes. This target detection model can be a trained first scoring network, including a first branch network and a second branch network, used to determine the first prediction score and the second prediction score of each candidate detection box, respectively. Subsequently, the prediction scores output by the two branch networks for processing the candidate detection boxes can be fused to obtain a target prediction score. Then, based on this target prediction score, a target detection box corresponding to the target category is determined from one or more candidate detection boxes.

[0038] Furthermore, the training of the object detection model can also be achieved using the image processing device 101. The training process can be roughly as follows: First, a series of candidate detection boxes that may contain real objects are generated for the input sample image using a candidate detection box generation method (e.g., unsupervised candidate detection box generation technology). Then, supervised learning can be performed using image-level labels (i.e., the class labels of the samples) through a multiple instance learning method. After obtaining the initial score for each candidate detection box, secondary pseudo-labels are generated for the candidate detection boxes in two different ways, including a first pseudo-label and a second pseudo-label. The two branch networks complement each other, reducing ambiguity in the model optimization process. Under the supervision of the secondary pseudo-labels, the two branches can automatically assign a class score to a candidate detection box, thereby achieving the training of the object detection model. Finally, during prediction, the two scores are fused (e.g., average voting) to obtain the final prediction score for the detection box. The final detection result can be output using methods such as non-maximum suppression. It should be noted that "score" and "scoring" in the embodiments provided in this application can have the same meaning unless their distinction is emphasized.

[0039] It can be observed that the above-mentioned object detection method is a weakly supervised object detection method. It only requires providing the object category contained in a certain image, and no longer requires providing precise object location information. This can greatly alleviate the problem of general object detectors requiring a large amount of labeled data during training, significantly reducing the labeling complexity. Furthermore, this can effectively utilize the massive amount of images obtained from the Internet for model training, thereby achieving object detection for large-scale / long-tail categories. The weakly supervised object detection method provided in this application differs from conventional methods. The weakly supervised object detection technology based on a dual-branch structure provided in this solution can not only train the object detector (i.e., the object detection model) with only image-level labels, thereby achieving efficient training, but also complement the advantages and disadvantages of the dual-branch network structure, further improving the accuracy of the model. By using a high-accuracy object detection model, i.e., the different branch networks included in the first scoring network after training, the candidate detection boxes are evaluated from different perspectives, resulting in high-accuracy scores and thus accurate detection results.

[0040] In this embodiment, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. No restrictions are placed here. The terminal can be a smartphone, tablet computer, smart wearable device, smart voice interaction device, smart home appliance, personal computer, in-vehicle terminal, etc., and no restrictions are placed here either.

[0041] It should be noted that the image processing solution of this application can be applied to various scenarios related to robot vision, and the images to be processed can differ depending on the scenario. For example, in industrial quality inspection applications, the images to be processed can be images of various industrial parts, such as PCB circuit boards, steel surfaces, solar panels, metal surfaces, etc. In autonomous driving applications, the images to be processed can be images of various roads acquired in real-time or in advance, including highways, urban roads, rural roads, etc. Furthermore, in augmented reality navigation applications, the images to be processed can be images acquired based on environments such as streets and shopping malls. Therefore, no specific application scenarios or content of the images to be processed are limited in this application.

[0042] Furthermore, the image processing solution provided in this application can be widely applied in many fields such as robot navigation, intelligent video surveillance, aerospace, and industrial inspection. For example, the image processing solution can be deployed on an industrial quality inspection platform to perform surface defect detection processing on images of various industrial parts, and output the defect location (detection box) and defect type of the image to be processed. The image processing solution can also be integrated into obstacle avoidance planning algorithms for autonomous driving, performing detection processing on real-time acquired images to identify the type and location of target obstacles, thereby achieving effective obstacle avoidance and driving route planning in autonomous driving.

[0043] The specific implementation of the image processing method proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] Please see Figure 2 , Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method can be performed by a computer device (e.g., Figure 1 The image processing device 101 shown is used for execution. The image processing method includes, but is not limited to, the following steps:

[0045] S201, Obtain the image to be processed.

[0046] In one embodiment, the image to be processed can be an image captured in real time by an image processing device, or an image pre-captured and stored in an image library. The image can be a single picture or any frame extracted from a video; the source and acquisition method of the image to be processed are not limited. The image to be processed can include different objects and their corresponding instances. For example, an image captured based on an urban road environment can include different objects such as people, traffic lights, cars, and buildings, with different vehicles being different instances of the object "car".

[0047] S202, Generate one or more candidate detection boxes corresponding to the image to be processed.

[0048] In one embodiment, a computer device (e.g., an image processing device) can automatically generate one or more candidate detection boxes corresponding to the image to be processed. These candidate detection boxes may include candidate detection boxes containing targets, where the target refers to the target object corresponding to the target category. Each candidate detection box typically defines a region in the image with a closed rectangle. Therefore, each candidate detection box corresponds to a different region in the image to be processed, and the regions may partially overlap. The images within each region may belong to the same target category or different target categories. By generating candidate detection boxes, a preliminary determination can be made of the location of the target categories included in the image to be processed. Subsequently, an object detection model can be used to filter out the detection boxes most likely to contain the real target and use them as the detection result of the model.

[0049] Optionally, the candidate detection box generation method for the image to be processed can adopt unsupervised detection algorithms, such as selective search, or classic detection algorithms, such as edge boxes, sliding window method, etc., or deep learning networks, such as using RPN (Region Proposal Network) to extract candidate detection boxes. Here, there are no restrictions on the specific implementation method used to generate candidate detection boxes. For ease of explanation, we will use selective search in the unsupervised candidate detection box extraction algorithm as an example to briefly describe the candidate detection box generation process: ① Initialize the original region using image segmentation methods, that is, divide the image to be processed into multiple small sub-regions; ② Calculate the similarity between every two adjacent sub-regions according to a greedy strategy. Here, the similarity can be a weighted sum of color, texture, size, and shape similarity; ③ Merge the two most similar sub-regions into one region; ④ Calculate the similarity between the merged region and adjacent sub-regions; ⑤ Repeat steps ③ and ④, that is, continuously perform iterative merging of regions until the entire image is merged into one region position. Here, a region position means that no new region can be merged based on similarity. The granularity of the final region is based on the region where each target object is located; ⑥ Output the outer rectangle of each merged region, that is, the candidate detection box. In this way, there are candidate detection boxes in every merging. Therefore, for a certain target category, there can be candidate detection boxes of different granularities, resulting in one or more candidate detection boxes corresponding to the image to be processed. It should be noted that, in this application, "candidate box" and "candidate detection box" can have the same meaning when their distinction is not emphasized.

[0050] The aforementioned selective search method calculates similarity between superpixels (i.e., sub-regions obtained by region segmentation of an image). It combines these superpixels in a bottom-up manner (which can be understood as merging image regions with increasing granularity), and outputs the intermediate regions generated as candidate boxes. These candidate boxes are then used as potential object-containing bounding boxes. This method utilizes a series of traditional image features for similarity evaluation. By combining diverse similarity evaluation metrics, it helps improve the probability of object detection and ensures the accuracy of candidate bounding box extraction. Furthermore, this method does not require model training, is computationally fast and efficient, and has a high recall rate, providing excellent initial detection results for object detectors. It should be noted that unsupervised candidate box generation methods can be other more advanced approaches that generate candidate boxes from images without annotations. The basic requirement is that the generated candidate boxes have the highest possible recall rate, ensuring that training and detection cover the real objects as much as possible and improving the object detection coverage of the target category.

[0051] In addition, the sliding window method can be used to exhaustively search all regions in the image to be processed by sliding a window from left to right and top to bottom. Alternatively, edge boxes can be used to determine candidate detection boxes using edge information. Furthermore, deep learning networks, such as RPN, can directly generate candidate detection boxes. It should be noted that compared to traditional detection methods, methods like RPN, due to their intelligent learning, can significantly improve the speed of candidate detection box generation.

[0052] S203, call the object detection model to process one or more candidate detection boxes to obtain the detection result of the image to be processed.

[0053] In this embodiment of the application, the detection result includes one or more candidate detection boxes corresponding to the target category. The target detection model is obtained by training the first branch network and the second branch network of the first scoring network using the second scoring network, the sample image, and the category label of the sample image.

[0054] Each target category may have one or more bounding boxes. Different bounding boxes may correspond to different instances of the same target category or instances of different target categories. The image within the region corresponding to each bounding box contains objects of the target category, which can be one or more predefined categories. For example, if the target categories are "people" and "cars," there may be two bounding boxes for the "people" category, indicating that the image contains two different people. There may also be five bounding boxes for the "cars" category, indicating that the image contains five different cars. The detection results can be stored digitally or displayed in the image to be processed, i.e., the bounding boxes and their corresponding target categories are labeled. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of a visualization of the detection result provided in this application. The detection result can be the processing result of an image processing solution applied in the field of industrial quality inspection. That is, taking an image of a single PCB circuit board as input, the output detection result is the location of each defect (target detection box), the defect category (target category), and the score of the defect category in the image, including short circuit defect (short, 1.00), open circuit defect (open, 1.00), edge defect (mousebite, 1.00), intermediate defect (pin-hole, 1.00), and redundancy (copper, 1.00).

[0055] The first scoring network is trained using a second scoring network, sample images, and the category labels of the sample images. The resulting object detection model has the ability to distinguish whether an object of the target category is present in the image to be processed. The first scoring network includes two branches trained under different supervision information. The second scoring network can also include two branches, referred to here as a classification branch network and a detection branch network, which are used to calculate different scores for candidate detection boxes corresponding to the sample images in the horizontal and vertical dimensions, respectively. The horizontal dimension refers to the probability that a candidate detection box belongs to each target category, and the vertical dimension refers to the contribution of each candidate detection box under a fixed target category. Optionally, the second scoring network can adopt the classification branch and detection branch of WSDDN (Weakly Supervised Deep Detection Networks).

[0056] During training, both the first and second scoring networks are used simultaneously. The second scoring network can be considered an auxiliary training network for the first scoring network. In practical applications, only the trained first scoring network is used; that is, the trained first scoring network is used as the object detection model. Since the first scoring network includes a first branch network and a second branch network, these two branches can be collectively referred to as a dual-branch network. In other words, the object detection model includes a dual-branch network. The dual-branch network is used to process the image to obtain the detection result, while the second scoring network is not used to process the image. For the specific process of training the object detection model, please refer to [link to relevant documentation]. Figure 4 The details of the corresponding embodiments will not be described in detail here.

[0057] In one embodiment, the second scoring network includes a classification branch network and a detection branch network. The classification branch network, the detection branch network, the first branch network, and the second branch network can all be composed of fully connected layers and normalization layers. The difference lies in the fact that the parameters of the fully connected layer (FC) and the processing parameters of the normalization layer (softmax) can be different. For example, in the classification branch network and the detection branch network, the fully connected layer (FC) outputs a vector of the same size, while the normalization layer calculates the score of the candidate detection box along different dimensions, considering the relationship between the candidate detection box and the target category from different dimensions.

[0058] Optionally, before calling the object detection model to process one or more candidate detection boxes in the image to be processed, an image feature extraction network can be used to extract the image features of the corresponding image regions in the image to be processed. Then, the object detection model is called to process these image features to obtain the detection result. This feature extraction network can be regarded as the backbone network, and any network structure can be used, as long as it can extract high-resolution features with strong semantic information (i.e., the image features corresponding to the candidate detection boxes).

[0059] In summary, the embodiments of this application have at least the following advantages:

[0060] Candidate detection boxes are automatically extracted from the image to be processed through an unsupervised candidate detection box generation method. While ensuring the accuracy of candidate box extraction, no model training is required, which can reduce the computational load caused by model training and thus reduce the amount of preparatory work for object detection. The object detection model with a dual-branch network structure processes the candidate detection boxes, which can accurately select the target detection boxes corresponding to the target category, thereby improving the accuracy of target object classification and localization.

[0061] Please see Figure 4 , Figure 4 This is a flowchart illustrating a target detection model training method provided in an embodiment of this application. This method can be performed using a computer device (e.g., Figure 1 The image processing device 101 shown is used for execution. This method includes, but is not limited to, the following steps:

[0062] S401, Obtain the training sample set.

[0063] In this embodiment, the training sample set includes multiple sample images and a category label for each sample image. The training sample set may include a large number of sample images carrying real category labels. The category labels of the sample images are image-level labels, which are category annotations of the objects included in the sample images. All categories indicated by the category labels in the training sample set can be regarded as target categories.

[0064] This scheme employs a novel weakly supervised object detection structure based on a dual-branch architecture. It does not require bounding box level supervision; model training only requires specifying the categories present in the image. This allows for effective utilization of massive internet image resources. Therefore, the sample images can be from a vast collection of images obtained from the internet, such as images initially labeled using keyword search engine results. The training set can also include different sample images for different application scenarios. For example, for industrial quality inspection, the sample images could include both images with surface defects and images with intact surfaces. Object detection models trained on different sample sets can then distinguish between different object categories.

[0065] S402, the second scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image, to obtain the initial score of each candidate detection box in each target category and the image score of each sample image in each target category.

[0066] In one embodiment, before calling the second scoring network to process one or more candidate detection boxes corresponding to each sample image, it is necessary to generate candidate detection boxes for each sample image. The specific method can be the same as or different from the method used in the previous embodiment to generate one or more candidate detection boxes corresponding to the image to be processed; no limitation is imposed here. Since the training sample set does not provide real target detection boxes, this scheme can also use an unsupervised candidate detection box generation method to generate some candidate detection boxes that may contain real targets. Then, the model can be used to select the detection boxes most likely to contain real targets for supervised learning.

[0067] In one embodiment, after generating candidate detection boxes for each sample image, the step of calling a second scoring network to process the candidate detection boxes may include: obtaining image features of the image region corresponding to each candidate detection box in one or more candidate detection boxes for each sample image, wherein the image features are obtained by using a feature extraction network to extract features from the image region corresponding to the candidate detection box; determining an initial category score and an initial region score for each candidate detection box based on the image features, wherein the initial category score represents the probability that each candidate detection box belongs to each target category, and the initial region score represents the probability that each candidate detection box contributes to each target category; fusing the initial category score and the initial region score to obtain an initial score for each candidate detection box in each target category; and determining the image score of each sample image in each target category based on the initial scores of each candidate detection box corresponding to each sample image.

[0068] The feature extraction network can employ various pre-trained networks with feature extraction capabilities, such as convolutional neural networks pre-trained on ImageNet. Specifically, it can be a VGG (Visual Geometry Group) series convolutional network model, or a convolutional neural network (CNN) with SPP (Spatial Pyramid Pooling) layers. The specific structure of the feature extraction network is not limited here. For one or more candidate detection boxes in each sample image, the feature extraction network can be used to extract the image features of the corresponding image region within the corresponding sample image.

[0069] Regarding the specific process of feature extraction, this application uses a deep convolutional neural network with an SPP (Spatial Pyramid Pooling) layer as an example. The entire sample image to be detected is first processed by a deep convolutional network (CNN) to extract depth image features. Then, for each candidate detection box, the region pooling technique of the SPP layer is used to extract the depth features corresponding to that candidate detection box. That is, the image features of the corresponding image region in the corresponding sample image of the candidate detection box are also the pooled depth image features. Here, the SPP layer is used to perform pyramid pooling on the depth image features, which can process sample images of arbitrary input size, avoiding the model being limited to a fixed input size. At the same time, by extracting spatial feature information of different sizes, the robustness of the model to spatial layout and object transformation can be improved. Furthermore, the feature extraction network can also include multiple fully connected layers. After obtaining the image features, the image features can be processed through multiple (e.g., two) fully connected layers to map the abstract information of different receptive fields to a larger space, obtaining a comprehensive representation of the image features of each candidate detection box. This can increase the model's representational ability.

[0070] Optionally, the second scoring network may include a classification branch network and a detection branch network. These two branches can give different scores based on the same image features, namely the initial class score and the initial region score. These two scores measure the degree of association between a detection box and each target class from different perspectives. In the specific processing, the image features can be mapped to different feature spaces through a fully connected layer to obtain two features. These two features can be used to predict the class of the two branches. Branch 1 (i.e., the classification branch network or the classification channel) is responsible for predicting the probability distribution of each candidate detection box belonging to each class, and Branch 2 (i.e., the detection branch network or the detection channel) is responsible for predicting the probability of the current candidate detection box contributing to each class. For the prediction of each branch, the two features can be processed in different dimensions through a normalization layer. The softmax calculation of Branch 1 can be shown in Equation (1) as follows:

[0071]

[0072] Among them, [σ class (x c )] ij Let x represent the initial category score, i.e., the probability that the j-th candidate detection box belongs to the i-th target category. c This represents the feature obtained by processing the image features through the fully connected layer of the classification branch network. This feature is a matrix of size C×|R|, where |R| represents the total number of candidate detection boxes, C represents the total number of target categories, and k represents the k-th target category among all target categories.

[0073] The softmax calculation for branch two can be shown in equation (2) below:

[0074]

[0075] Among them, [σ det (x d )] ij Let x represent the initial region score, which is the informativeness of the j-th candidate detection box corresponding to the i-th target category, or the probability of the j-th candidate detection box contributing to the i-th target category. d This represents the image features obtained after processing by the fully connected layer of the detection branch network. This feature is a matrix of size C×|R|, where |R| represents the total number of candidate detection boxes, C represents the total number of target categories, and k represents the k-th candidate detection box among all candidate detection boxes.

[0076] It can be observed that the initial category score and initial region score obtained above are calculated based on matrices of the same size, starting from the dimensions of the candidate detection box and the target category, respectively. These two scores measure the correlation between a candidate detection box and each target category from different perspectives. By fusing these two scores (e.g., multiplying them), the initial score of the candidate detection box is obtained. This initial score can comprehensively characterize the probability that the detection box belongs to a certain category.

[0077] Alternatively, element-wise multiplication (i.e., Hadamard product) can be performed on the initial class score and initial region score after normalization of all candidate detection boxes in each sample image. That is, the element values ​​at corresponding positions of the matrix of initial class score and the matrix of initial region score are multiplied together. This results in a lower data dimensionality, which is beneficial to improving computational efficiency and speed.

[0078] For each candidate detection box corresponding to each sample image, there is an initial score belonging to each target category. The initial scores of all candidate detection boxes under a certain target category can be fused, for example, by averaging the initial scores through instance-level pooling, to obtain the image score of each sample image in that target category. For the initial scores of each target category included in each sample image, the above process is repeated to obtain the image score of each sample image in each target category. This image score is a predicted image-level score, which can be used to combine with the true category label to determine the loss of the second scoring network. For example, suppose there are 10 category labels for the sample image (i.e., 10 target categories), and there are 20 candidate detection boxes for the sample image. Each candidate detection box corresponds to 10 initial scores under the 10 target categories. The image score of the sample image under that target category can be obtained by averaging the 10 initial scores under that target category.

[0079] In other words, since there are no class labels for individual candidate detection boxes, only the probability distribution of all candidate detection boxes belonging to each target class, averaging the probabilities of all candidate detection boxes under the same target class yields the probability distribution of the entire sample image. This probability distribution includes predicted probabilities of the sample image belonging to each target class. Then, supervised learning is performed on this probability distribution at the image level, i.e., calculating the loss value using the predicted values ​​of this probability distribution and the true class labels. During the inference phase, the probability distribution of each candidate detection box can be directly used as its initial predicted score, that is, the initial score for each candidate detection box belonging to each target class.

[0080] For each sample image, the aforementioned processing of candidate detection boxes using the second scoring network can be viewed as a multi-instance learning approach, utilizing image-level labels for supervised learning of the model. Specifically, all candidate detection boxes are treated as a bag containing both ground truth boxes and background boxes. The bag's label represents the true object category contained in the sample image; that is, each candidate detection box possesses a label for each object category. However, the authenticity of belonging to each object category can be measured by an initial score, which can then be used to further filter candidate detection boxes containing true object categories.

[0081] S403, the first scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image, and the first prediction score and the second prediction score of each candidate detection box in each target category are obtained.

[0082] In one embodiment, the first scoring network and the second scoring network share the same image features corresponding to the same candidate detection boxes; that is, the image features input into the second scoring network are also input into the first scoring network for processing. Optionally, the step of calling the first scoring network to process the candidate detection boxes of the sample image can be: calling the first branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score for each candidate detection box in each target category; calling the second branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a second prediction score for each candidate detection box in each target category.

[0083] In this model, both the first and second branch networks can use cascaded fully connected layers and normalized layers. The processing of the fully connected and normalized layers in the two branch networks can be similar or completely different. The first and second prediction scores obtained by the two branch networks are used to calculate the loss of various branch networks in the second scoring network. Furthermore, the supervision information used by the two branch networks differs. The first branch network undergoes supervised learning through instance-level pseudo-labels, while the second branch network undergoes supervised learning through bag-level pseudo-labels. This allows the two branch networks to evaluate the same candidate detection box from different dimensions, thus enabling the model to not only possess instance-level discriminative ability but also avoid ambiguity in model optimization through bag-level label supervision. The specific supervised learning process can be found in the following embodiments, which will not be detailed here.

[0084] S404. Using the first predicted score, the second predicted score, the initial score, the image score, and the category label, the first scoring network and the second scoring network are trained to obtain the object detection model.

[0085] In this embodiment, the loss of each branch network can be determined using its corresponding score and label. The losses of all branch networks are then fused to obtain the total loss. This total loss is used to train the first and second scoring networks, and the trained first scoring network is ultimately used as the object detection model. For the specific training process (i.e., the supervised learning process), please refer to [link to relevant documentation]. Figure 6 The contents of the corresponding embodiments will not be described in detail here.

[0086] For the model structure during the training phase, please refer to [link / reference]. Figure 5 The illustrated schematic diagram of an exemplary image processing network includes a feature extraction network, a first branch network and a second branch network of a first scoring network, a detection branch network and a classification branch network of the second scoring network, and the results obtained from each network processing. The methods for determining the losses of the first and second scoring networks can be found in [reference needed]. Figure 6 The content corresponding to the embodiments is not elaborated here, and other processing principles will not be repeated.

[0087] As shown above, this solution can output the target regions and their categories contained in a given image. Its main feature is that the model training phase does not require providing real object detection boxes; it only needs to provide all object categories contained in each image. This is significant for training object detection models with large-scale training images because it avoids a large amount of manual annotation, especially for some long-tail categories (i.e., categories with few samples). The annotation of both category labels and candidate detection boxes often requires the assistance of industry experts, which is time-consuming and laborious, and difficult to meet the large amount of training data required by deep learning models.

[0088] In summary, the embodiments of this application have at least the following advantages:

[0089] During training, there is no need to provide manually labeled candidate detection boxes. The target detection model is trained using sample images with only category labels, which greatly reduces the cost and workload of manual labeling and improves training efficiency. The two different branches of the first scoring network are used to predict the scores of the candidate detection boxes of the sample images, resulting in different predicted scores. Then, the two scoring networks are supervised learning by combining the predicted scores and other relevant training information to obtain a highly accurate trained first scoring network. This first scoring network can be used as the target detection model, which can effectively improve the accuracy of detection results in the application stage.

[0090] Please see Figure 6 , Figure 6 This is a flowchart illustrating another object detection model training method provided in this application embodiment. This method can be performed by a computer device (e.g., Figure 1 The image processing device 101 shown is used for execution. This method includes, but is not limited to, the following steps:

[0091] S601, Obtain the training sample set.

[0092] S602, the second scoring network is called to process one or more candidate detection boxes corresponding to each sample image to obtain the initial score of each candidate detection box in each target category and the image score of each sample image in each target category.

[0093] S603, the first scoring network is called to process one or more candidate detection boxes corresponding to each sample image, and the first prediction score and the second prediction score of each candidate detection box in each target category are obtained.

[0094] For details on how to implement the above steps, please refer to [link / reference]. Figure 4 The contents of steps S401 to S403 in the corresponding embodiments will not be repeated here.

[0095] S604, using the initial score of each candidate detection box in each target category to obtain the first pseudo-label corresponding to each candidate detection box, and using the initial score of each candidate detection box in each target category to obtain the second pseudo-label corresponding to the multi-instance package to which each candidate detection box belongs.

[0096] In one embodiment, an optional implementation of obtaining the first pseudo-label corresponding to each candidate detection box may be as follows: For each sample image, remove candidate detection boxes whose initial score for each target category is less than a score threshold from one or more candidate detection boxes, resulting in N candidate detection boxes, where N is a positive integer; select the candidate detection box with the largest initial score from the N candidate detection boxes, use it as a reference detection box, and determine the corresponding target category as the first pseudo-label of the reference detection box; determine the overlap between the reference detection box and the candidate detection boxes other than the reference detection box in the N candidate detection boxes, and remove candidate detection boxes with an overlap greater than or equal to a first overlap threshold; select the candidate detection box with the largest initial score from the removed candidate detection boxes, and re-execute the process of using the candidate detection box with the largest initial score as the reference detection box, until all N candidate detection boxes are traversed.

[0097] For each sample image and one or more candidate detection boxes, a score threshold is first used to eliminate candidate detection boxes whose initial score in each target category is less than or equal to the score threshold. This method filters out candidate detection boxes with low confidence that contain the real target, allowing for faster selection of candidate detection boxes with high confidence that contain the real target. Before this, each target category can be iterated over, using the target category as a constant dimension. All candidate detection boxes in that target category are sorted in order based on their initial scores, for example, from high to low. Then, candidate detection boxes with lower initial scores are eliminated. After eliminating all candidate detection boxes in that target category, N candidate detection boxes remain.

[0098] Then, the candidate detection box with the highest initial score among the N candidate detection boxes is taken as the reference detection box. Simultaneously, the corresponding target category can be determined as the first pseudo-label of the reference detection box. It should be noted that this first pseudo-label is annotated on the candidate detection box, and the object included in the candidate detection box can be considered an instance; therefore, this label is an instance-level label. It is called a pseudo-label here because its accuracy is unknown. The reference detection box, as the currently determined true target box, can be used to filter other redundant candidate detection boxes. That is, by calculating the overlap between the reference detection box and other candidate detection boxes among the N candidate detection boxes, candidate detection boxes with an overlap greater than or equal to a first overlap threshold are removed. Optionally, the overlap calculation can use IOU (Intersection Over Union). When the IOU value is between 0 and 1, it represents the overlap between two candidate detection boxes. The higher the value, the higher the overlap. Removing candidate detection boxes with high overlap can reduce the number of candidate detection boxes, thereby facilitating the extraction of other true target boxes and the annotation of instance-level labels on the candidate detection boxes.

[0099] Next, the reference detection box with the highest initial score can be selected from the remaining candidate detection boxes that have not been eliminated. This reference detection box is then used as the true target box. The steps of calculating overlap, eliminating candidate detection boxes that do not meet the conditions, and selecting reference detection boxes are repeated in a loop until all N candidate detection boxes have been traversed. That is, all candidate detection boxes of this target category are either labeled as belonging to this target category or are eliminated because they do not meet the conditions. The above process is repeated for all categories. Finally, all candidate detection boxes can be labeled with a label. This label can include the category label corresponding to the target category or the background label. The first pseudo-label here usually refers to the category label corresponding to the target category. This type of label is usually a foreground target label. The first pseudo-label can be used for secondary supervised learning of candidate detection boxes. For example, the loss can be calculated using the first pseudo-label and the first predicted score, and the loss can be backpropagated to adjust the first branch network. Optionally, candidate detection boxes that are not labeled with the target category after repeating the above process are all labeled with background labels. In addition, candidate detection boxes with an overlap higher than the first overlap threshold can also be ignored. Finally, for all candidate detection boxes of a certain target category, there are three types of candidate detection boxes: candidate detection boxes labeled with the target category, ignored candidate detection boxes, and candidate detection boxes whose initial score is less than the score threshold and have not been traversed.

[0100] The above process is also known as OICR (Online Instance Classifier Refinement) supervision. This branch of supervision is instance-level pseudo-label supervision. Since the pseudo-labels are generated, some noise is inevitably introduced. Therefore, directly using instance-level supervision can lead to ambiguity in learning. For example, some backgrounds with high initial scores may be mislabeled as foreground objects, and some objects may only have some regions labeled as positive samples, while the rest may be labeled as negative samples due to insufficient overlap. This can cause ambiguity in the model during optimization. However, this branch directly performs instance-level supervision on candidate bounding boxes. When the pseudo-labels are accurate enough, it can enable the model to achieve optimal prediction accuracy.

[0101] In one embodiment, an optional implementation of obtaining the second pseudo-label corresponding to the multi-example bag where each candidate detection box belongs may be: selecting the first candidate detection box with the largest initial score in each target category from one or more candidate detection boxes corresponding to each sample image, and determining the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in one or more candidate detection boxes; combining the first candidate detection box and candidate detection boxes with an overlap greater than or equal to a second overlap threshold into a multi-example bag, and determining the corresponding target category as the second pseudo-label of the multi-example bag; removing candidate detection boxes from one or more candidate detection boxes in the multi-example bag to obtain P candidate detection boxes, where P is a positive integer; selecting the first candidate detection box with the largest initial score from the removed candidate detection boxes, and re-determining the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in the P candidate detection boxes, until one or more candidate detection boxes are clustered into each multi-example bag.

[0102] First, iterate through each target category, using the target category as the standard dimension, compare the initial scores of all candidate detection boxes under that target category, and select the first candidate detection box with the highest initial score among all candidate detection boxes corresponding to that target category. Then, calculate the overlap between the first candidate detection box and other candidate detection boxes. The method for calculating the overlap can be the same as the method for determining the overlap mentioned above, or it can be different. Next, combine the first candidate detection box and all surrounding candidate detection boxes with an overlap higher than the second overlap threshold into a multi-instance bag. It is called a multi-instance bag because the image in the region corresponding to each candidate detection box can be regarded as an instance. The second pseudo-label of this multi-instance bag is the corresponding target category. The second overlap threshold can be the same as the first overlap threshold, or it can be different. The overlap threshold can be set based on manual experience or iteratively optimized through training. Then, identify the candidate detection boxes that are not included in any multi-instance bag from the remaining P candidate detection boxes of that target category, and repeat the above operation until all candidate detection boxes are selected into a multi-instance bag. Repeat the above process for all target categories. Then, supervised learning is performed separately using multi-instance learning for each multi-instance package. For candidate detection boxes of the background category, instance-level supervised learning can be used to directly learn how to classify the background.

[0103] It should be noted that candidate detection boxes in the background class can be categorized into a single multi-instance bag, where the label of the multi-instance bag is the background label. Optionally, these candidate detection boxes in the background class may be labeled as such due to low scores and not categorized into the same multi-instance bag. Furthermore, for the remaining P candidate detection boxes in each target category, or the aforementioned N candidate detection boxes, the specific selection process for candidate detection boxes differs for each target category; that is, P or N is variable across different target categories. The second pseudo-label of the multi-instance bag can represent the pseudo-label of all candidate detection boxes it contains, i.e., labeling a batch of candidate detection boxes. This second pseudo-label is then used to supervise multiple candidate detection boxes simultaneously, preventing the model from getting trapped in local optima.

[0104] The above-described processing is also known as PCL (Proposal Cluster Learning) supervision. This supervision method uses multi-instance learning to refine the score. Because multiple candidate detection boxes in a multi-instance bag are supervised simultaneously, the optimal detection box cannot stand out, which contradicts the requirement in the testing phase that the optimal detection box should have the highest score. This inconsistency between the training and testing phases leads to a decrease in model accuracy. However, by supervising multiple candidate detection boxes simultaneously, it overcomes the ambiguity caused by instance-level supervision in OICR, i.e., it avoids directly supervising individual instances and prevents the rigidity caused by false label noise. The image processing scheme provided in this application combines these two methods, leveraging their complementary advantages and disadvantages to maximize the accuracy of the trained object detection model, thereby improving the accuracy of the detection results.

[0105] The first and second pseudo-labels obtained above can supervise different branches of the first scoring network. That is, by calculating the loss, the first and second scoring networks are adjusted according to the total loss. For details, please refer to the description of step S605.

[0106] S605, using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label, determine the total loss corresponding to the first rating network and the second rating network.

[0107] In one embodiment, this step may be implemented as follows: determining the loss of the second scoring network using image scores and category labels; determining the loss of the first branch network of the first scoring network using the first predicted score and the first pseudo-label; determining the loss of the second branch network of the first scoring network using the second predicted score and the second pseudo-label; and determining the total loss corresponding to the first and second scoring networks using the loss of the second scoring network, the loss of the first branch network, and the loss of the second branch network.

[0108] Optionally, the loss of the second scoring network, the loss of the first branch network, and the loss of the second branch network can all be calculated using the cross-entropy loss method. The loss of the first branch network and the loss of the second branch network can be fused to obtain the loss of the first scoring network. Furthermore, the loss of the first scoring network and the loss of the second scoring network can be fused to obtain the total loss corresponding to the first scoring network and the second scoring network. The loss fusion can be performed using a weighted summation method.

[0109] For a schematic diagram of the image processing network structure during the training phase, Figure 5 Based on, combined Figure 4 The exemplary content of each scoring network in the corresponding embodiments and the related descriptions in this embodiment can be further refined; please refer to [link / reference]. Figure 7 , Figure 7 Another exemplary diagram of an image processing network structure is shown. The first and second pseudo-labels obtained from the initial score can be used in the first and second branches of the first scoring network, respectively, to calculate the losses of the first and second branches, thus obtaining the loss of the first scoring network. The loss of the second scoring network is calculated based on the image score and the category label. Fusing the losses of these two scoring networks enables supervised learning of the model.

[0110] S606 uses the total loss to train the first and second scoring networks, and uses the trained first scoring network as the object detection model.

[0111] In this embodiment, the parameters or structure of the first and second scoring networks can be adjusted using the total loss. The adjusted networks are then used to continue processing candidate detection boxes. Through iterative training, the total loss converges, and the first scoring network, which is finally trained, is used as the object detection model, including the first and second branch networks.

[0112] In one embodiment, the first scoring network includes one or more scoring units, each scoring unit including a first branch network and a second branch network, and the one or more scoring units are connected in parallel; the input of each scoring unit is the image features of the image region corresponding to each candidate detection box in the sample image, and the fusion score output by the previous scoring unit is used to determine the first pseudo-label and the second pseudo-label used by the next scoring unit. The fusion score is obtained by fusing the first prediction score output by the first branch network and the second prediction score output by the second branch network.

[0113] The above training process can be considered as the training flow when the first scoring network includes one scoring unit. Each scoring unit undergoes the same processing flow, and the number of scoring units may affect the calculation of the total loss and the final structure of the object detection model. When the first scoring network includes multiple scoring units, these units are connected in parallel. Each scoring unit includes a first branch network and a second branch network, sharing the same input data. That is, the input data processed is the image features of the corresponding image region in the sample image for each candidate detection box. Each scoring unit can output a fusion score, which is obtained by fusing the first and second prediction scores obtained by the two branch networks in the scoring unit. Specifically, this can be achieved through average voting, i.e., averaging the two prediction scores, or by weighted summation of the two prediction scores. The specific fusion method is not limited here. Based on the fusion score of the previous scoring unit, the required first and second pseudo-labels can be determined for the next scoring unit. These two pseudo-labels are used for supervised learning of the next scoring unit. By combining their respective prediction scores, the loss corresponding to each scoring unit is determined, thus obtaining the total loss. It should be noted that each scoring unit can be viewed as a refinement classifier (or refinement branch), and the second scoring network can be viewed as a base classifier. The supervision of the first refinement classifier depends on the output of the base classifier, and the supervision of the i-th refinement classifier depends on the output of the (i-1)-th refinement classifier, where i is a positive integer greater than or equal to 2. That is, each refinement branch uses the result of the previous refinement branch as input for the next iteration of optimization.

[0114] Combining the first scoring network described above, the image processing network can be further refined as follows: Figure 8 As shown, Figure 8 This is a schematic diagram of another exemplary image processing network structure provided in this application embodiment. The first scoring network includes K scoring units, where K is a positive integer. Each scoring unit includes two branch networks, each consisting of a fully connected layer and a normalization layer. The second scoring network and other parts of the network follow the same structure. Figure 7 The content shown. For Figure 8 The first and second pseudo-labels determined by the fusion score of the i-th scoring unit among the K scoring units shown will be applied to the first and second branch networks in the (i+1)-th scoring unit, respectively. The first and second pseudo-labels obtained from the initial score output by the second scoring network will be applied to the first and second branch networks of the first scoring unit, respectively. The loss corresponding to each branch network will be calculated by combining their respective predicted scores, and the loss of the scoring unit, the loss of the first scoring network, and the total loss will be obtained in sequence.

[0115] The above scheme, based on the multi-instance learning framework, improves the pseudo-label generation process for candidate detection boxes as follows: Based on the initial scores of individual candidate boxes output during the multi-instance learning phase, one or more possible true target boxes can be identified for each target category. These are then used as pseudo-labels to supervise the model's learning, enabling the model to acquire instance-level discriminative ability. Compared to previous methods that either assumed the box with the highest initial score as the true target box or treated all boxes within a certain range around the highest-scoring box as a single packet for re-multi-instance learning, this scheme avoids the former's approach of treating non-maximum-score candidate boxes as negative samples while ignoring the possibility that these negative samples might cover parts of the true target region, leading to ambiguity during model training. It also avoids the latter's approach of treating multiple candidate boxes as a single packet for supervision, where the lack of instance-level supervision results in inconsistent model performance during training and testing. Specifically, during model testing, the optimal detection box is required to have the highest score, while other boxes must have sufficiently low scores; during model training, there is no optimal detection box.

[0116] In summary, the embodiments of this application have at least the following advantages:

[0117] Compared to previous weakly supervised object detection techniques, this method utilizes a dual-branch structure in the first scoring network. The first branch network filters candidate detection boxes that may contain real targets, encouraging the best candidate detection box scores to stand out. The second branch network, through bag-level label supervision, improves the detection coverage of target categories, thus avoiding the situation where only some regions of the target category are correctly identified. Using the initial scores of candidate detection boxes, different pseudo-labels are generated based on two different methods for supervised learning of different branches of the model. By complementing the two branches—the second branch network resolves the ambiguity caused by pseudo-labels in the first branch network during optimization, and the first branch network resolves the inconsistency between training and testing results in the second branch network—the object detection model can possess instance-level discriminative ability and avoid local optima during optimization, ensuring consistency between training and testing, thereby effectively improving the accuracy of the trained object detection model.

[0118] Please see Figure 9 , Figure 9 This is a schematic flowchart of another image processing method provided in an embodiment of this application. This method can be performed by a computer device (e.g., Figure 1 The image processing device 101 shown is used for execution. This method includes, but is not limited to, the following steps:

[0119] S901, Obtain the image to be processed.

[0120] S902, Generate one or more candidate detection boxes corresponding to the image to be processed.

[0121] In one implementation, the specific implementation method of steps S901 to S902 can be found in [reference needed]. Figure 2 The steps S201 to S202 in the corresponding embodiments will not be described again here. Steps S903 to S906 in this embodiment are... Figure 2 As further explained in the corresponding embodiment of step S203, the prerequisite for performing the subsequent steps is that the target detection model includes the first branch network and the second branch network of the trained first scoring network.

[0122] S903, the first branch network is called to process one or more candidate detection boxes to obtain a third prediction score for each candidate detection box in each target category, and the second branch network is called to process one or more candidate detection boxes to obtain a fourth prediction score for each candidate detection box in each target category.

[0123] In one embodiment, after training the first scoring network as described in the previous embodiment, an object detection model is obtained. The first and second branch networks in the trained first scoring network can obtain different predicted scores for candidate detection boxes by processing them. Before calling the first branch network, each candidate detection box can pass through a feature extraction network to extract features from the image to be processed, obtaining the image features corresponding to the candidate detection box. The specific method is not elaborated here. Specifically, the first branch network processes the image features corresponding to the candidate detection boxes. Optionally, the image features can be sequentially passed through the fully connected layer and normalization layer of the first branch network to obtain a third predicted score for each candidate detection box in each target category. Similar to the first branch network, the second branch network can also sequentially pass the image features corresponding to the candidate detection boxes through a fully connected layer and a normalization layer to obtain a fourth predicted score for each candidate detection box in each target category. Unlike the predicted scores obtained during the training phase, the predicted scores obtained during the application phase are more accurate scores obtained after refinement by the various branch networks of the first scoring network.

[0124] S904, the third prediction score and the fourth prediction score are fused to obtain the target prediction score of each candidate detection box in each target category.

[0125] In one embodiment, the third and fourth prediction scores of each candidate detection box for each target category can be fused. Optionally, the fusion method can be an average voting method for candidate detection boxes, that is, averaging the third and fourth prediction scores to obtain a fused score, thereby determining the target prediction score of each candidate detection box for each target category. Alternatively, a weighted average method can be used, assigning different weight values ​​to each prediction score, summing and averaging them to obtain a fused score, thereby determining the target prediction score of each candidate detection box for each target category.

[0126] Furthermore, the fusion score can be directly used as the target prediction score, or the fusion score can be processed accordingly to obtain the target prediction score, depending on the number of scoring units included in the trained first scoring network. Optionally, the first scoring network includes one or more scoring units, each scoring unit including a first branch network and a second branch network, and the one or more scoring units are connected in parallel; the input of each scoring unit is the image features of the image region corresponding to each candidate detection box in the image to be processed, and the output of each scoring unit is the target prediction score of each candidate detection box in each target category.

[0127] If the trained first scoring network includes one scoring unit, the fused score of the corresponding third and fourth predicted scores can be directly used as the target prediction score. Conversely, if the trained first scoring network has at least two scoring units, and the score obtained by each scoring unit for processing the candidate detection box includes both a third and a fourth predicted score, the third and fourth predicted scores obtained by each scoring unit can be fused to obtain the fused score of each scoring unit for processing the candidate detection box. Furthermore, the target prediction score of the candidate detection box in each target category can be obtained by fusing the fused scores of each scoring unit. This fusion can be an average or a weighted average. For example, as shown... Figure 8 The K scoring units shown are assumed to have a fusion score α after the average voting of the candidate boxes for the i-th scoring unit. i The target prediction score is

[0128] S905, determine the target detection box corresponding to the target category from one or more candidate detection boxes based on the target prediction score, and use the target detection box corresponding to the target category as the detection result of the image to be processed.

[0129] In one embodiment, since one or more candidate detection boxes in the generated image to be processed may not contain the target category, and the detection result usually only needs to provide the target detection box for a specific target category, most of these candidate detection boxes are redundant. The number of candidate detection boxes can be reduced by using the target detection score and preset rules, thereby determining the target detection box corresponding to the target category and using it as the detection result.

[0130] Optionally, Non-Maximum Suppression (NMS) can be used to obtain the final detection result. The specific processing flow of NMS is illustrated below with an example. Assuming the target classification task has 3 classes (i.e., 3 target categories), and there are 1000 candidate detection boxes generated for the image to be processed, the final output score-related vector is 1000×3, with each column corresponding to a target category and each row representing the target prediction score of each candidate detection box. According to the NMS algorithm steps, each column of the 1000×3 matrix can be sorted in descending order. Starting with the candidate detection box with the highest target prediction score in each column, the overlap with subsequent candidate detection boxes in that column is calculated. Boxes with an overlap greater than a given threshold and a lower target prediction score are removed. The remaining candidate detection boxes indicate that multiple targets of that class may exist in the image. This process is repeated for the remaining candidate detection boxes until all candidate detection boxes in the column have been traversed. After traversing all columns of the 1000×3 matrix, the detection result is obtained.

[0131] In addition to the above, other related NMS algorithms can be used to obtain detection results and find the optimal object detection location (i.e., the target detection box), such as Soft NMS, Softer NMS, IOU-Guided NMS, etc. Compared with traditional NMS, the improvement of these algorithms lies in using different filtering strategies based on the target prediction scores of candidate detection boxes. For example, Soft NMS does not directly exclude boxes that overlap with the selected boxes by more than a certain threshold, but instead uses a certain strategy to reduce the score of the corresponding boxes until it falls below a certain threshold, thus avoiding the excessive deletion of correctly located boxes in crowded situations.

[0132] In summary, the embodiments of this application have at least the following advantages:

[0133] The first scoring network, including the first and second branch networks, is used to process the candidate detection boxes of the image to be processed. Since the accuracy of the model is maximized during training, the prediction scores of the two branch networks are relatively accurate prediction results. By fusing the prediction scores of the two branch networks, the correlation between the candidate detection boxes and the target category can be described more comprehensively and accurately. Therefore, the target detection box corresponding to the target category can be determined more accurately based on the target prediction score.

[0134] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this application. The aforementioned image processing apparatus may be a computer program (including program code) running on a computer device; for example, the image processing apparatus may be application software. The image processing apparatus may be used to execute corresponding steps in the methods provided in the embodiments of this application. Figure 10 As shown, the image processing device 1000 may include: an acquisition module 1001, a generation module 1002, and a processing module 1003, wherein:

[0135] Acquisition module 1001 is used to acquire the image to be processed;

[0136] The generation module 1002 is used to generate one or more candidate detection boxes corresponding to the image to be processed;

[0137] The processing module 1003 is used to call the object detection model to process one or more candidate detection boxes to obtain the detection result of the image to be processed. The detection result includes one or more object detection boxes in the candidate detection boxes that correspond to the object category. The object detection model is trained by using the second scoring network, the sample image, and the category label of the sample image to train the first scoring network, which includes the first branch network and the second branch network.

[0138] In one embodiment, the image processing apparatus 1000 may further include: a training module 1004, wherein:

[0139] The acquisition module 1001 is also used to acquire a training sample set, which includes multiple sample images and the category label of each sample image;

[0140] The processing module 1003 is also used to call the second scoring network to process one or more candidate detection boxes corresponding to each sample image, so as to obtain the initial score of each candidate detection box in each target category and the image score of each sample image in each target category.

[0141] The processing module 1003 is also used to call the first scoring network to process one or more candidate detection boxes corresponding to each sample image, and obtain the first prediction score and the second prediction score of each candidate detection box in each target category;

[0142] Training module 1004 is used to train the first scoring network and the second scoring network using the first predicted score, the second predicted score, the initial score, the image score, and the category label to obtain the object detection model.

[0143] In one embodiment, the training module 1004 is specifically configured to: obtain a first pseudo-label corresponding to each candidate detection box using the initial score of each candidate detection box in each target category; and obtain a second pseudo-label corresponding to the multi-example bag to which each candidate detection box belongs using the initial score of each candidate detection box in each target category; determine the total loss corresponding to the first scoring network and the second scoring network using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label; train the first scoring network and the second scoring network using the total loss; and use the trained first scoring network as the target detection model.

[0144] In one embodiment, the training module 1004 is specifically configured to: remove candidate detection boxes whose initial scores for each target category are less than a threshold from one or more candidate detection boxes corresponding to each sample image, resulting in N candidate detection boxes, where N is a positive integer; select the candidate detection box with the largest initial score from the N candidate detection boxes, use the candidate detection box with the largest initial score as the reference detection box, and determine the corresponding target category as the first pseudo-label of the reference detection box; determine the overlap between the reference detection box and the candidate detection boxes other than the reference detection box in the N candidate detection boxes, and remove candidate detection boxes whose overlap is greater than or equal to a first overlap threshold; select the candidate detection box with the largest initial score from the removed candidate detection boxes, and re-execute the process of using the candidate detection box with the largest initial score as the reference detection box, until all N candidate detection boxes are traversed.

[0145] In one embodiment, the training module 1004 is specifically configured to: select the first candidate detection box with the largest initial score in each target category from one or more candidate detection boxes corresponding to each sample image, and determine the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in one or more candidate detection boxes; combine the first candidate detection box and candidate detection boxes with an overlap greater than or equal to a second overlap threshold into a multi-example package, and determine the corresponding target category as the second pseudo-label of the multi-example package; remove the candidate detection boxes in the multi-example package from one or more candidate detection boxes to obtain P candidate detection boxes, where P is a positive integer; select the first candidate detection box with the largest initial score from the removed candidate detection boxes, and re-execute the determination of the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in the P candidate detection boxes, until one or more candidate detection boxes are clustered into each multi-example package.

[0146] In one embodiment, the training module 1004 is specifically configured to: determine the loss of the second rating network using image scores and category labels; determine the loss of the first branch network of the first rating network using a first predicted score and a first pseudo-label; and determine the loss of the second branch network of the first rating network using a second predicted score and a second pseudo-label; and determine the total loss corresponding to the first rating network and the second rating network using the loss of the second rating network, the loss of the first branch network, and the loss of the second branch network.

[0147] In one embodiment, the processing module 1003 is specifically configured to: call the first branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image, and obtain a first prediction score for each candidate detection box in each target category; call the second branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image, and obtain a second prediction score for each candidate detection box in each target category.

[0148] In one embodiment, the processing module 1003 is specifically configured to: acquire image features of the image region corresponding to each candidate detection box in each sample image within one or more candidate detection boxes corresponding to each sample image, wherein the image features are obtained by extracting features from the image region corresponding to the candidate detection box using a feature extraction network; determine an initial category score and an initial region score for each candidate detection box based on the image features, wherein the initial category score represents the probability that each candidate detection box belongs to each target category, and the initial region score represents the probability that each candidate detection box contributes to each target category; fuse the initial category score and the initial region score to obtain an initial score for each candidate detection box in each target category; and determine the image score of each sample image in each target category based on the initial scores of each candidate detection box corresponding to each sample image.

[0149] In one embodiment, the first scoring network includes one or more scoring units, each scoring unit including a first branch network and a second branch network, and the one or more scoring units are connected in parallel; the input of each scoring unit is the image features of the image region corresponding to each candidate detection box in the sample image, and the fusion score output by the previous scoring unit is used to determine the first pseudo-label and the second pseudo-label used by the next scoring unit. The fusion score is obtained by fusing the first prediction score output by the first branch network and the second prediction score output by the second branch network.

[0150] In one embodiment, the object detection model includes a first branch network and a second branch network of a trained first scoring network. The processing module 1003 is further configured to: call the first branch network to process one or more candidate detection boxes to obtain a third prediction score for each candidate detection box in each object category; call the second branch network to process one or more candidate detection boxes to obtain a fourth prediction score for each candidate detection box in each object category; fuse the third prediction score and the fourth prediction score to obtain a target prediction score for each candidate detection box in each object category; determine the target detection box corresponding to the target category from one or more candidate detection boxes based on the target prediction score, and use the target detection box corresponding to the target category as the detection result of the image to be processed.

[0151] It is understood that the functions of each functional module of the image processing apparatus described in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here either.

[0152] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 1100 may include one or more independent devices (such as servers, nodes, terminals, etc.), or it may include components within independent devices (such as chips, software modules, or hardware modules). The computer device 1100 may include at least one processor 1101 and a communication interface 1102. Further optionally, the computer device 1100 may also include at least one memory 1103 and a bus 1104. The processor 1101, the communication interface 1102, and the memory 1103 are connected via the bus 1104.

[0153] The processor 1101 is a module that performs arithmetic and / or logical operations. Specifically, it can be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a microcontroller unit (MCU).

[0154] The communication interface 1102 can be used to provide information input or output to the at least one processor. And / or, the communication interface 1102 can be used to receive data sent from the outside and / or send data to the outside, and can be a wired link interface including such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicle short-range communication technology and other short-range wireless communication technologies, etc.).

[0155] The memory 1103 provides storage space, in which data such as the operating system and computer programs can be stored. The memory 1103 can be one or a combination of several of the following: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0156] At least one processor 1101 in the computer device 1100 is used to invoke a computer program stored in at least one memory 1103 to execute the aforementioned image processing method, such as the one described above. Figure 2 , Figure 4 , Figure 6 , Figure 9 The image processing method described in the illustrated embodiment.

[0157] In one possible implementation, the processor 1101 in the computer device 1100 is used to call a computer program stored in at least one memory 1103 to perform the following operations: acquire an image to be processed through a communication interface 1102; generate one or more candidate detection boxes corresponding to the image to be processed; call an object detection model to process the one or more candidate detection boxes to obtain a detection result of the image to be processed, the detection result including object detection boxes in one or more candidate detection boxes corresponding to the object category, wherein the object detection model is trained by using a second scoring network, sample images, and category labels of the sample images to train a first branch network and a second branch network of the first scoring network.

[0158] In one embodiment, the processor 1101 is further configured to: acquire a training sample set, the training sample set including multiple sample images and category labels for each sample image; invoke a second scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain an initial score for each candidate detection box in each target category and an image score for each sample image in each target category; invoke a first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score and a second prediction score for each candidate detection box in each target category; and train the first scoring network and the second scoring network using the first prediction score, the second prediction score, the initial score, the image score, and the category label to obtain a target detection model.

[0159] In one embodiment, when the processor 1101 trains the first scoring network and the second scoring network using the first predicted score, the second predicted score, the initial score, the image score, and the category label to obtain the object detection model, it specifically performs the following steps: obtaining the first pseudo-label corresponding to each candidate detection box using the initial score of each candidate detection box in each target category, and obtaining the second pseudo-label corresponding to the multi-example bag where each candidate detection box is located using the initial score of each candidate detection box in each target category; determining the total loss corresponding to the first scoring network and the second scoring network using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label; training the first scoring network and the second scoring network using the total loss, and using the trained first scoring network as the object detection model.

[0160] In one embodiment, when the processor 1101 obtains the first pseudo-label corresponding to each candidate detection box using the initial score of each candidate detection box for each target category, it specifically performs the following steps: First, it removes candidate detection boxes whose initial scores for each target category are less than a score threshold from one or more candidate detection boxes corresponding to each sample image, obtaining N candidate detection boxes, where N is a positive integer. Second, it selects the candidate detection box with the largest initial score from the N candidate detection boxes, uses the candidate detection box with the largest initial score as a reference detection box, and determines the corresponding target category as the first pseudo-label of the reference detection box. Third, it determines the overlap between the reference detection box and the candidate detection boxes other than the reference detection box in the N candidate detection boxes, and removes candidate detection boxes whose overlap is greater than or equal to a first overlap threshold. Fourth, it selects the candidate detection box with the largest initial score from the removed candidate detection boxes, and re-executes the process of using the candidate detection box with the largest initial score as a reference detection box, until all N candidate detection boxes are traversed.

[0161] In one embodiment, when the processor 1101 obtains the second pseudo-label corresponding to the multi-example bag to which each candidate detection box belongs using the initial score of each candidate detection box in each target category, it is specifically used to: select the first candidate detection box with the largest initial score in each target category from one or more candidate detection boxes corresponding to each sample image, and determine the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in one or more candidate detection boxes; combine the first candidate detection box and the candidate detection boxes with an overlap greater than or equal to a second overlap threshold into a multi-example bag, and determine the corresponding target category as the second pseudo-label of the multi-example bag; remove the candidate detection boxes in the multi-example bag from one or more candidate detection boxes to obtain P candidate detection boxes, where P is a positive integer; select the first candidate detection box with the largest initial score from the removed candidate detection boxes, and re-execute the determination of the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in the P candidate detection boxes, until one or more candidate detection boxes are clustered into each multi-example bag.

[0162] In one embodiment, when the processor 1101 determines the total loss corresponding to the first scoring network and the second scoring network using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label, it is specifically used to: determine the loss of the second scoring network using the image score and the category label; determine the loss of the first branch network of the first scoring network using the first predicted score and the first pseudo-label; and determine the loss of the second branch network of the first scoring network using the second predicted score and the second pseudo-label; and determine the total loss corresponding to the first scoring network and the second scoring network using the loss of the second scoring network, the loss of the first branch network, and the loss of the second branch network.

[0163] In one embodiment, when the processor 1101 calls the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score and a second prediction score for each candidate detection box in each target category, it specifically performs the following steps: calling the first branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score for each candidate detection box in each target category; and calling the second branch network of the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a second prediction score for each candidate detection box in each target category.

[0164] In one embodiment, when the processor 1101 calls the second scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain the initial score of each candidate detection box in each target category and the image score of each sample image in each target category, the processor specifically performs the following steps: It acquires the image features of the image region corresponding to each candidate detection box in each sample image within one or more candidate detection boxes corresponding to each sample image. The image features are obtained by using a feature extraction network to extract features from the image region corresponding to the candidate detection box. Based on the image features, it determines the initial category score and initial region score of each candidate detection box. The initial category score represents the probability that each candidate detection box belongs to each target category, and the initial region score represents the probability that each candidate detection box contributes to each target category. It then fuses the initial category score and the initial region score to obtain the initial score of each candidate detection box in each target category. Finally, it determines the image score of each sample image in each target category based on the initial scores of each candidate detection box corresponding to each sample image.

[0165] In one embodiment, the first scoring network includes one or more scoring units, each scoring unit including a first branch network and a second branch network, and the one or more scoring units are connected in parallel; the input of each scoring unit is the image features of the image region corresponding to each candidate detection box in the sample image, and the fusion score output by the previous scoring unit is used to determine the first pseudo-label and the second pseudo-label used by the next scoring unit. The fusion score is obtained by fusing the first prediction score output by the first branch network and the second prediction score output by the second branch network.

[0166] In one embodiment, the processor 1101 object detection model includes a first branch network and a second branch network of a trained first scoring network. When the object detection model is invoked to process one or more candidate detection boxes to obtain the detection result of the image to be processed, it is specifically used for: invoking the first branch network to process one or more candidate detection boxes to obtain a third prediction score for each candidate detection box in each target category; invoking the second branch network to process one or more candidate detection boxes to obtain a fourth prediction score for each candidate detection box in each target category; fusing the third prediction score and the fourth prediction score to obtain a target prediction score for each candidate detection box in each target category; determining the target detection box corresponding to the target category from one or more candidate detection boxes based on the target prediction score, and using the target detection box corresponding to the target category as the detection result of the image to be processed.

[0167] It should be understood that the computer device 1100 described in the embodiments of this application can perform the image processing method described in the corresponding embodiments above, and can also perform the methods described above. Figure 10 The description of the image processing apparatus 1000 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0168] Furthermore, it should be noted that an exemplary embodiment of this application also provides a storage medium storing a computer program for the aforementioned image processing method. This computer program includes program instructions that, when loaded and executed by one or more processors, can implement the image processing method described in the embodiments, which will not be repeated here. The beneficial effects of employing the same method will also not be repeated here. It is understood that the program instructions can be deployed on one or more computer devices capable of communicating with each other for execution.

[0169] The aforementioned computer-readable storage medium can be an internal storage unit of the image processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0170] One aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of this application.

[0171] In one aspect of this application, another computer program product is provided, which includes a computer program or computer instructions that, when executed by a processor, implement the steps of the image processing method provided in the embodiments of this application.

[0172] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed; Generate one or more candidate detection boxes corresponding to the image to be processed; The object detection model is invoked to process the one or more candidate detection boxes to obtain the detection result of the image to be processed. The detection result includes the object detection boxes in the one or more candidate detection boxes that correspond to the object category. The training process of the target detection model includes: Obtain a training sample set, which includes multiple sample images and a category label for each sample image; The second scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image to obtain an initial score for each candidate detection box in each target category and an image score for each sample image in each target category; wherein, the initial score is obtained by fusing the initial category score and the initial region score of each candidate detection box, the initial category score is used to represent the probability that each candidate detection box belongs to each target category, and the initial region score is used to represent the probability that each candidate detection box contributes to each target category; The first scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score and a second prediction score for each candidate detection box in each target category; The first scoring network and the second scoring network are trained using the first predicted score, the second predicted score, the initial score, the image score, and the category label to obtain an object detection model.

2. The method as described in claim 1, characterized in that, The step of training the first scoring network and the second scoring network using the first predicted score, the second predicted score, the initial score, the image score, and the category label to obtain the object detection model includes: The first pseudo-label corresponding to each candidate detection box is obtained using the initial score of each candidate detection box in each target category, and the second pseudo-label corresponding to the multi-example package to which each candidate detection box belongs is obtained using the initial score of each candidate detection box in each target category. Using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label, the total loss corresponding to the first scoring network and the second scoring network is determined; The first scoring network and the second scoring network are trained using the total loss, and the trained first scoring network is used as the object detection model.

3. The method as described in claim 2, characterized in that, The step of obtaining the first pseudo-label corresponding to each candidate detection box using the initial score of each candidate detection box in each target category includes: For each sample image, candidate detection boxes with initial scores less than a score threshold in each target category are removed from one or more candidate detection boxes to obtain N candidate detection boxes, where N is a positive integer; Select the candidate detection box with the largest initial score from the N candidate detection boxes, use the candidate detection box with the largest initial score as the reference detection box, and determine the corresponding target category as the first pseudo label of the reference detection box; Determine the overlap between the reference detection box and the candidate detection boxes in the N candidate detection boxes other than the reference detection box, and remove the candidate detection boxes whose overlap is greater than or equal to a first overlap threshold. Select the candidate detection box with the largest initial score from the eliminated candidate detection boxes, and re-execute the process of using the candidate detection box with the largest initial score as the reference detection box until all N candidate detection boxes have been traversed.

4. The method as described in claim 2, characterized in that, The step of obtaining the second pseudo-label corresponding to the multi-example bag containing each candidate detection box using the initial score of each candidate detection box in each target category includes: From one or more candidate detection boxes corresponding to each sample image, select the first candidate detection box with the largest initial score in each target category, and determine the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in the one or more candidate detection boxes; The first candidate detection box and the candidate detection boxes with an overlap greater than or equal to the second overlap threshold are combined into a multi-example package, and the corresponding target category is determined as the second pseudo-label of the multi-example package; The candidate detection boxes in the multi-example package are removed from the one or more candidate detection boxes to obtain P candidate detection boxes, where P is a positive integer; Select the first candidate detection box with the highest initial score from the removed candidate detection boxes, and re-execute the process of determining the overlap between the first candidate detection box and the candidate detection boxes other than the first candidate detection box in the P candidate detection boxes, until all one or more candidate detection boxes are clustered into each multi-instance package.

5. The method as described in claim 2, characterized in that, The step of determining the total loss corresponding to the first scoring network and the second scoring network using the first predicted score, the second predicted score, the first pseudo-label, the second pseudo-label, the image score, and the category label includes: The loss of the second scoring network is determined using the image score and the category label; The loss of the first branch network of the first scoring network is determined using the first predicted score and the first pseudo-label, and the loss of the second branch network of the first scoring network is determined using the second predicted score and the second pseudo-label. The total loss corresponding to the first scoring network and the second scoring network is determined by using the loss of the second scoring network, the loss of the first branch network, and the loss of the second branch network.

6. The method according to any one of claims 1 to 5, characterized in that, The step of calling the first scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score and a second prediction score for each candidate detection box in each target category includes: The first branch network of the first scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image to obtain the first prediction score of each candidate detection box in each target category; The second branch network of the first scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image, so as to obtain a second prediction score for each candidate detection box in each target category.

7. The method according to any one of claims 1 to 5, characterized in that, The step of calling the second scoring network to process one or more candidate detection boxes corresponding to each sample image to obtain an initial score for each candidate detection box in each target category and an image score for each sample image in each target category includes: One or more candidate detection boxes are obtained for each sample image. The image features of the image region corresponding to each candidate detection box in each sample image are obtained by using a feature extraction network to extract features from the image region corresponding to the candidate detection box. Based on the image features, an initial category score and an initial region score are determined for each candidate detection box. The initial category score represents the probability that each candidate detection box belongs to each target category, and the initial region score represents the probability that each candidate detection box contributes to each target category. The initial category score and the initial region score are fused to obtain the initial score of each candidate detection box for each target category; The image score for each sample image in each target category is determined based on the initial score of each candidate detection box corresponding to each sample image.

8. The method according to any one of claims 1 to 5, characterized in that, The first scoring network includes one or more scoring units, each scoring unit including a first branch network and a second branch network, and the one or more scoring units are connected in parallel; the input of each scoring unit is the image features of the image region corresponding to each candidate detection box in the sample image, and the fusion score output by the previous scoring unit is used to determine the first pseudo-label and the second pseudo-label used by the next scoring unit. The fusion score is obtained by fusing the first prediction score output by the first branch network and the second prediction score output by the second branch network.

9. The method as described in claim 1, characterized in that, The object detection model includes a first branch network and a second branch network of a trained first scoring network. The step of calling the object detection model to process the one or more candidate detection boxes to obtain the detection result of the image to be processed includes: The first branch network is invoked to process the one or more candidate detection boxes to obtain a third prediction score for each candidate detection box in each target category; The second branch network is invoked to process the one or more candidate detection boxes to obtain a fourth prediction score for each candidate detection box in each target category; The third prediction score and the fourth prediction score are fused to obtain the target prediction score of each candidate detection box in each target category; Based on the target prediction score, a target detection box corresponding to the target category is determined from one or more candidate detection boxes, and the target detection box corresponding to the target category is used as the detection result of the image to be processed.

10. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed; The generation module is used to generate one or more candidate detection boxes corresponding to the image to be processed; The processing module is used to call the object detection model to process the one or more candidate detection boxes to obtain the detection result of the image to be processed. The detection result includes the object detection boxes in the one or more candidate detection boxes that correspond to the object category. The training process of the target detection model includes: Obtain a training sample set, which includes multiple sample images and a category label for each sample image; The second scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image to obtain an initial score for each candidate detection box in each target category and an image score for each sample image in each target category; wherein, the initial score is obtained by fusing the initial category score and the initial region score of each candidate detection box, the initial category score is used to represent the probability that each candidate detection box belongs to each target category, and the initial region score is used to represent the probability that each candidate detection box contributes to each target category; The first scoring network is invoked to process one or more candidate detection boxes corresponding to each sample image to obtain a first prediction score and a second prediction score for each candidate detection box in each target category; The first scoring network and the second scoring network are trained using the first predicted score, the second predicted score, the initial score, the image score, and the category label to obtain an object detection model.

11. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the image processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 9.

Images