Training method of active learning model, image processing method and device

Abstract

The application relates to an image processing technology in the field of computer vision in the field of artificial intelligence, and provides a training method and an image processing method and device of an active learning model, which comprise the following steps: using a labeled image data set C i training an active learning model M i , obtaining an active learning model M i1 , using an unlabeled image data set D i training an active learning model M i1 , obtaining an active learning model M i2 ; using the labeled image data set C i and the unlabeled image data set D i training an active learning model M i2 , obtaining a divergence weight of each unlabeled image, selecting N unlabeled images according to the divergence weight of each unlabeled image for manual labeling, and obtaining a labeled image data set C i+1 ; and determining a target active learning model according to the labeled image data set C i+1 . According to the embodiment of the application, the model precision can be ensured and human resources can be saved under the condition that as few images as possible need to be manually labeled.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an active learning model training method, an image processing method, and an apparatus. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems in application fields such as manufacturing, inspection, document analysis, and medical diagnosis. It's the science of how to use cameras / video cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras / video cameras) and a brain (algorithms) to replace human eyes in identifying, tracking, and measuring targets, thus enabling computers to perceive their environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as the science of studying how to enable artificial systems to "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs in acquiring input information, and then the computer replaces the brain in processing and interpreting this input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, possessing the ability to autonomously adapt to their environment.

[0003] Image classification (IC), object detection (OD), and image segmentation (IS) are crucial problems in high-level visual semantic understanding tasks. With the rapid development of artificial intelligence technology, these three fundamental tasks are increasingly widely applied in the field of computer vision. In existing technologies, the process of selecting manually labeled data from unlabeled datasets during the training of neural network models performing these three tasks lacks regularity or exhibits poor regularity, and easily leads to the selection of redundant samples. This results in long training times and slow convergence speeds for the neural network.

[0004] Therefore, how to select the most representative unlabeled samples from the unlabeled dataset for manual labeling has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a training method, image processing method, and apparatus for an active learning model. It can manually label the most representative unlabeled samples from an unlabeled dataset, thereby improving the prediction accuracy of the model while selecting as little unlabeled data as possible.

[0006] Firstly, this application provides a method for training an active learning model, the method comprising: S1: using a labeled image dataset C iTraining active learning model M i The active learning model M is obtained. i1 Active learning model M i Includes a backbone network and multiple parallel classifiers, with i being a positive integer; S2: Utilizes an unlabeled image dataset D i Training active learning model M i1 The active learning model M is obtained. i2 Active learning model M i2 The parameters of multiple parallel classifiers and the active learning model M i1 Multiple parallel classifiers in the model have the same parameters, and the active learning model M... i2 Parameters of the backbone network and the active learning model M i1 The parameters of the backbone network are different; S3: using the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of multiple parallel classifiers in the active learning model M are different. i+1 Parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same, and the divergence weights are used to characterize the deviation between multiple predictions for each unlabeled image. Multiple predictions for each unlabeled image are output by multiple parallel classifiers; S4: Based on the unlabeled image dataset D... i The divergence weight of each unlabeled image in the unlabeled image dataset D i N unlabeled images are selected from the dataset and manually labeled. These N manually labeled images are then added to the labeled image dataset C. i The labeled image dataset C is obtained. i+1 and unlabeled dataset D i+1 N is a positive integer; S5: Based on the labeled image dataset C i+1 Define the target active learning model.

[0007] As can be seen, in this embodiment, an active learning model is first trained using a labeled image dataset, and the parameters of the backbone network and multiple parallel classifiers in the active learning model are adjusted to obtain the active learning model M. i1 Then fix the active learning model M. i1 The classifier parameters are adjusted, and the active learning model M is updated using a large number of unlabeled images from the unlabeled image dataset. i1The parameters of the backbone network are adjusted to ensure that the prediction results of multiple parallel classifiers for the same unlabeled image are as consistent as possible, resulting in the active learning model M. i2 At this point, the parameters in the backbone network can be considered optimized to their best; therefore, the active learning model M can be fixed. i2 The parameters of the backbone network are used to train the active learning model M using images from both the unlabeled and labeled image datasets. i2 The parameters of the classifier are updated to obtain the divergence weight for each unlabeled image. Since the divergence weight represents the degree of deviation of the classifier's prediction result for the same image, it can quantitatively represent the difficulty of the machine to recognize the unlabeled image. Therefore, the most representative unlabeled images that need to be manually labeled can be selected through the divergence weight. Thus, a high-precision active learning model is obtained with as few images as possible that need to be manually labeled, saving the cost of manual labeling. In addition, in step S3, the model is trained simultaneously using the unlabeled image dataset and the labeled image dataset. This can ensure that when the prediction values ​​of multiple classifiers for the same image are similar, the prediction values ​​of the multiple classifiers are also as close as possible to the true value (i.e., the labeled value) of the image.

[0008] In one feasible implementation, the above is based on the labeled image dataset C i+1 Determine the target active learning model, including: when the labeled image dataset C i+1 Number of images and unlabeled dataset D i+1 The ratio of the number of images in the middle is greater than or equal to a preset ratio value, or by using an active learning model M. i+1 For the labeled image dataset C i+1 When the accuracy of prediction for the image in the dataset is greater than or equal to the preset accuracy, the active learning model M will be activated. i+1 If the target is an active learning model, then let i = i + 1 and repeat steps S1-S5.

[0009] As can be seen, in this embodiment of the application, by setting the termination condition of the active learning model training to one of the two methods in the above embodiments, it can be ensured that the model stops training when the preset condition is met, and the target active learning model is obtained, thereby ensuring that the model meets the requirements while saving model training time as much as possible.

[0010] In one feasible implementation, the above utilizes an unlabeled image dataset D i Training active learning model M i1 The active learning model M is obtained. i2 This includes: unlabeled image dataset D i Input active learning model M i1 The unlabeled image dataset D is obtained. iFor each unlabeled image, multiple first predicted values ​​are generated, and each of these first predicted values ​​corresponds one-to-one with multiple parallel classifiers. Based on these first predicted values ​​and the first loss function for each unlabeled image, the active learning model M is updated. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function consists of multiple first predicted values ​​for each unlabeled image.

[0011] It should be understood that the aforementioned backbone network can be a convolutional neural network or other neural networks with similar functions, and its role is to extract features from the images in the input model.

[0012] As can be seen in the embodiments of this application, by utilizing a large number of unlabeled images to train the active learning model M i1 The parameters in the backbone network are updated to make them as optimal as possible, thereby ensuring that the influence of the backbone network on the differences in classifier prediction results is eliminated when updating the parameters of multiple classifiers in step S3.

[0013] In one feasible implementation, the above utilizes a labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image in the dataset include: Q1: the weights for each unlabeled image in the dataset C. i and unlabeled image dataset D i The j-th image P j Input active learning model M j-1 Image P is obtained. j Multiple second predicted values, image P j The multiple second predicted values ​​correspond one-to-one with multiple parallel classifiers, where j is a positive integer; Q2: Based on image P j Multiple second-predicted values ​​update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j When image P j For the unlabeled image dataset D i When looking at the image in the image, according to image P j The multiple second predictions determine the divergence weights of the image; Q3: When j is less than the sum of E and F, let j = j + 1, and repeat steps Q1-Q2; when j equals the sum of E and F, actively learn model M. j As an active learning model M i+1 When j=1, the active learning model M j-1For active learning model M i2 E and F represent the labeled image dataset C, respectively. i and unlabeled image dataset D i The number of images in the dataset, where E and F are positive integers.

[0014] It should be understood that, in the embodiments of this application, labeled and unlabeled images are used to train the active learning model M. i2 In this process, the input image data has no sequential order; the active learning model M can be trained first using labeled images. i2 Then, the active learning model M is trained using unlabeled images. i2 Alternatively, an active learning model M can be trained using unlabeled images first. i2 Then, the labeled images are used to train the active learning model M. i2 Alternatively, labeled images can be used to train the active learning model M alternately. i2 Alternatively, other input orders can be used to input the unlabeled and labeled images to train the model.

[0015] As can be seen, in this embodiment, both unlabeled and labeled images are used to update the active learning model M. i2 The parameters of the classifier are designed to ensure that the predicted values ​​of multiple classifiers are as close as possible to or the same as the true values, thus avoiding a decrease in model prediction performance when only unlabeled images are used for training.

[0016] In one feasible implementation, the above is based on image P j Multiple second-predicted values ​​update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j Including: when image P j For the unlabeled image dataset D i When looking at the image in the image, according to image P j Multiple second predicted values ​​and a second loss function update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the second loss function is the image P. j Multiple second predicted values, the function value of the second loss function, and the image P j The divergence weight is proportional to the image P; j For the labeled image dataset C i When looking at the image in the image, according to image P j The active learning model M is updated using multiple second-predicted values ​​and a third loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. jThe input data for the third loss function is the image P. j Multiple second predicted values ​​and the image P j The true value.

[0017] As can be seen in the embodiments of this application, when unlabeled images are used to train the active learning model M j-1 When dealing with the parameters of multiple classifiers, the divergence weights, which characterize the prediction bias of the classifiers, are added to the second loss function. By using this second loss function to adjust the parameters of the classifiers, the parameters of the classifiers can be adjusted in a targeted manner for images with different divergence weights, thereby accelerating the convergence speed of the model and saving model training time.

[0018] In one feasible implementation, the above is based on the unlabeled image dataset D. i The divergence weight of each unlabeled image in the unlabeled image dataset D i N unlabeled images were selected from the dataset and manually labeled, including: [the process of manually labeling] the unlabeled image dataset D. i Each unlabeled image is sorted from largest to smallest according to its divergence weight, and the top N unlabeled images are selected from the sorted results for manual labeling.

[0019] It should be understood that a larger divergence weight indicates a greater difference in the multiple predictions of multiple classifiers for the same image, and conversely, a smaller divergence weight indicates that the multiple predictions of multiple classifiers for the same image are closer.

[0020] As can be seen in the embodiments of this application, in each iteration, the unlabeled images are sorted from largest to smallest according to their divergence weights. The top N images selected from the sorting results are the most representative images, i.e. images that are difficult for machines to recognize. The selected N images are manually labeled, and the labeled image dataset is updated using the labeled N images. Therefore, when the labeled images are used for model training in the future, the convergence speed of the model can be accelerated, thereby saving the cost of manual labeling.

[0021] Secondly, this application provides an image processing method, comprising: acquiring an image to be processed; performing image processing on the image to be processed using a target active learning model to obtain a predicted value of the image to be processed; the target active learning model includes a backbone network and multiple parallel classifiers, wherein the multiple parallel classifiers are connected to the backbone network. Specifically, the target active learning model is obtained through K training iterations, wherein in the i-th training iteration of the above K training iterations: using a labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 Active learning model M iThe labeled image dataset C is obtained from the (i-1)th training iteration in K training iterations. i This is the result of updating after the (i-1)th training iteration; obtained from the unlabeled image dataset D. i Training active learning model M i1 The active learning model M is obtained. i2 Unlabeled image dataset D i The active learning model M is obtained after the (i-1)th training iteration. i2 The parameters of multiple parallel classifiers and the active learning model M i1 Multiple parallel classifiers in the model have the same parameters, and the active learning model M... i2 Parameters of the backbone network and the active learning model M i1 The parameters of the backbone network are different; through the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of multiple parallel classifiers in the active learning model M are different. i+1 Parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same; the labeled image dataset C is updated according to the divergence weights of each unlabeled image. i and unlabeled image dataset D i The labeled dataset C is obtained. i+1 and unlabeled image dataset D i+1 , where i and K are positive integers.

[0022] As can be seen, in the embodiments of this application, the target active learning model trained by the method in the first aspect has a high prediction accuracy while minimizing human resource costs. Therefore, when using the target active learning model for image processing, the image processing effect can be improved.

[0023] In one feasible implementation, based on the labeled image dataset C i+1 Number of images and unlabeled dataset D i+1 The ratio of the number of images in the middle, or through the active learning model M i+1 For the labeled image dataset C i+1 The accuracy of predictions based on the images in the dataset determines the number of training iterations, K.

[0024] It should be understood that the specific training process of the target active learning model is the same as the corresponding steps in the first aspect, and will not be repeated here. It can be seen that in this embodiment, by determining the number of training iterations K using one of the two methods described in the above embodiments, it can be ensured that the model stops training when it reaches the preset conditions, thus obtaining the target active learning model, thereby ensuring that the model meets the requirements while saving model training time as much as possible.

[0025] In one feasible implementation, the above image processing includes at least one of image classification, object detection, and image segmentation.

[0026] As can be seen, the method in this application embodiment can be used in any of the tasks of image classification, object detection and image segmentation. By applying the image processing method in this application embodiment to the above three tasks, the image processing effect can be improved.

[0027] Thirdly, this application provides an apparatus for training an active learning model, the apparatus including modules for performing the method in the first aspect.

[0028] Fourthly, this application provides an image processing apparatus that includes a module for performing the method in the second aspect.

[0029] Fifthly, this application provides an apparatus for training an active learning model, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor executes the method in the first aspect.

[0030] In a sixth aspect, this application provides an image processing apparatus, which includes a processor and a memory for storing program instructions, and the processor for invoking the program instructions to execute the method in the second aspect.

[0031] In a seventh aspect, this application provides a computer-readable storage medium storing program code for execution by a device, the program code including methods for performing the first aspect and / or the second aspect.

[0032] Eighthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the first and / or second aspects above.

[0033] Ninthly, this application provides a chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the methods of the first aspect and / or the second aspect.

[0034] Alternatively, as one implementation, the chip may also include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the methods in the first aspect and / or the second aspect.

[0035] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0036] In a tenth aspect, this application provides an electronic device that includes the means described in any one of the third to sixth aspects above. Attached Figure Description

[0037] The accompanying drawings used in the embodiments of this application are described below.

[0038] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0039] Figure 2 This application provides a schematic diagram of a backbone network structure;

[0040] Figure 3 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of another system architecture provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of an active learning model provided in an embodiment of this application;

[0043] Figure 6 This is a schematic flowchart illustrating a method for training an active learning model provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram illustrating an example of sample selection based on the training method of this application, provided in an embodiment of this application.

[0045] Figure 8 This is a schematic diagram illustrating the effect of image classification based on a target active learning model, as provided in an embodiment of this application.

[0046] Figure 9 This is a schematic diagram illustrating the effect of another image classification based on a target active learning model provided in this application embodiment;

[0047] Figure 10This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0048] Figure 11 This is a schematic block diagram of a training device for an active learning model provided in an embodiment of this application;

[0049] Figure 12 This is a schematic block diagram of an image processing apparatus provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of the hardware structure of a training device for an active learning model provided in an embodiment of this application;

[0051] Figure 14 This is a schematic diagram of the hardware structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings.

[0053] The embodiments of this application can be applied to image detection, album management, video recording, smart city projects, human-computer interaction, and other scenarios that require image processing, such as image classification, image segmentation, and object detection.

[0054] It should be understood that the images in the embodiments of this application can be static images (or static pictures) or moving images (or moving pictures). For example, the images in this application can be videos or animated pictures, or they can be static pictures or photographs. For ease of description, in the following embodiments, static images or moving images will be uniformly referred to as images.

[0055] The method described in this application can be specifically applied to album management and target detection scenarios. The following is a detailed description of these two scenarios.

[0056] Photo album management:

[0057] A user's terminal device, such as a mobile phone, may store a large number of images in its photo album, for example, images acquired through taking photos, screenshots, or downloading from the internet. When a user needs to find the image they need from a large amount of image data, the method in this application embodiment can be used to classify the large number of images in the album. Different types of images are stored in different directories, such as animal, landscape, and people categories. The animal category can also be further subdivided into different subcategories, for example, identifying the animal in the image based on its specific animal category and assigning it to the corresponding subcategory.

[0058] As can be seen, the method described in this application can quickly help users locate the category of the image they are looking for, thereby saving users time and improving user experience.

[0059] Object detection:

[0060] Object detection involves identifying objects of interest within an image and determining their location and size. For example, if a user wants to find images containing cats in their device's photo album, the method described in this application can be used to identify all cat-containing images on the user's device for selection.

[0061] As can be seen, the method in this application embodiment can accurately detect targets in images, thereby filtering images containing objects of interest to the user and improving the user experience.

[0062] It should be understood that the album management and object detection described above are only two specific scenarios in which the methods of this application embodiment are applied. The methods of this application embodiment are not limited to the above two scenarios. The methods of this application embodiment can be applied to any scenario that requires image processing, such as image segmentation. Alternatively, the methods of this application embodiment can also be similarly applied to other fields, such as speech recognition and natural language processing, etc., and this application embodiment does not limit this.

[0063] The methods and apparatus provided in this application can also be used to expand the training database, such as... Figure 1 The I / O interface 112 of the execution device 110 can output the unlabeled images that need to be manually labeled after being processed by the execution device to the client device 140. The client device 140 manually labels the unlabeled images and sends the labeled images as training data pairs to the database 130, so that the training data maintained by the database 130 is richer, thereby providing richer training data for the training work of the training device 120.

[0064] The method provided in this application is described below from the perspectives of model training and model application:

[0065] The active learning model training method provided in this application relates to computer vision processing and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. It involves symbolic and formal intelligent information modeling, extraction, preprocessing, and training of training data (such as the image to be processed in this application) to ultimately obtain a trained target active learning model. Furthermore, the image processing method provided in this application can utilize the trained target active learning model, inputting input data (such as the image to be processed in this application) into the trained target active learning model to obtain output data (such as the predicted value of the image to be processed in this application). It should be noted that the active learning model training method and the image processing method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0066] This application involves a large number of applications related to neural networks. In order to better understand the solutions of this application, the relevant terms and concepts in the fields of neural networks and computer vision that may be involved in this application will be introduced below.

[0067] (1) Image classification

[0068] Determine the category of targets contained in the image or video to be processed.

[0069] (2) Target Detection

[0070] Identifying all targets (objects) of interest from a given image and determining their category and location is a crucial task in computer vision. Due to the diverse appearances, shapes, and poses of various objects, coupled with interference from factors such as imaging lighting and occlusion, object detection remains one of the core and most challenging problems in computer vision.

[0071] (3) Image segmentation

[0072] Image segmentation is divided into instance segmentation and scene segmentation. Image segmentation is mainly used to determine which target or object each pixel in the image to be processed belongs to.

[0073] (4) Neural Network

[0074] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0075]

[0076] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For xs The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0077] (5) Deep Neural Networks

[0078] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0079] (6) Convolutional Neural Networks

[0080] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units on the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0081] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0082] (7) Loss Function

[0083] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0084] (8) Backpropagation algorithm

[0085] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0086] (9) Pixel value

[0087] A pixel value in an image can be a red-green-blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value of 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. Within each color component, the smaller the value, the lower the brightness; the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

[0088] (10) Entropy: It can represent the certainty of things. The higher the certainty, the lower the entropy, and vice versa. For classification tasks, if the confidence of the classification result of an image is closer to 0 or 1, its entropy is lower. The closer the classification result is to 0.5, the higher the entropy, which means that the classification result is uncertain.

[0089] The system architecture provided in the embodiments of this application is described below.

[0090] See appendix Figure 1 , Figure 1 This application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to acquire training data. In this application embodiment, the training data includes labeled images and unlabeled images. The label of a labeled image may be the category corresponding to the image, the category corresponding to a target in the image, or the category corresponding to each pixel of the image. The mathematical representation of the above categories is a multi-dimensional vector.

[0091] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 (that is, the target active learning model in this application embodiment) based on the training data maintained in the database 130.

[0092] The following describes in more detail, using Embodiment 1, how the training device 120 obtains the target model / rule 101 based on training data. This target model / rule 101 can be used to implement the image processing method provided in this application embodiment. That is, by inputting the image to be processed into the target model / rule 101 after relevant preprocessing, the predicted value of the image to be processed can be obtained. Specifically, the target model / rule 101 in this application embodiment can be a target active learning model. In the embodiments provided in this application, this target active learning model is obtained by training an adversarial classifier and a backbone network composed of convolutional neural networks. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as a limitation on the embodiments of this application.

[0093] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1 The execution device 110 shown can be a terminal, such as a mobile phone, tablet, laptop, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, etc., or it can be a server or cloud device. (See attached...) Figure 1In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include various image or video data.

[0094] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0095] Finally, I / O interface 112 returns the processing results, such as the predicted value of the image to be processed obtained above (i.e., the category label of the image to be processed, or the target identified from the image to be processed, or the result of segmenting the image to be processed), to client device 140, thereby providing it to the user.

[0096] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0097] In the appendix Figure 1 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0098] It is worth noting that, attached Figure 1 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0099] like Figure 1 As shown, the target model / rule 101 is trained using training device 120. In this embodiment, the target model / rule 101 can be a target active learning model trained using the active learning model training method of this embodiment. Specifically, the target active learning model provided in this embodiment may include an adversarial classifier and a backbone network. In the backbone network provided in this embodiment, the backbone network may be a convolutional neural network, and the adversarial classifier may be a convolutional neural network, a fully connected neural network, or other similar neural networks. This solution does not specifically limit this.

[0100] As introduced in the basic concepts above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning (DL) architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network, where each neuron can respond to the input image.

[0101] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230.

[0102] Convolutional / pooling layers 220:

[0103] Convolutional layers:

[0104] like Figure 2 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0105] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0106] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0107] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0108] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0109] Pooling layer:

[0110] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0111] Neural network layer 230:

[0112] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize neural network layers 230 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 230 can include multiple hidden layers (such as...). Figure 2 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multiple hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0113] After the multiple hidden layers in neural network layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 2 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0114] It should be noted that, as Figure 2The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0115] The following describes a chip hardware structure provided by an embodiment of this application.

[0116] Figure 3 A chip hardware structure provided in this embodiment of the invention includes a neural network processor 50. This chip can be configured as follows: Figure 1 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 2 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 3 This is achieved in the chip shown.

[0117] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the host CPU, which allocates tasks to it. The core of the NPU is the arithmetic circuit 503, which is controlled by the controller 504 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0118] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0119] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0120] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0121] In some implementations, vector computation unit 507 can store the processed output vector into unified buffer 506. For example, vector computation unit 507 can apply a nonlinear function to the output of arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0122] The unified memory 506 is used to store input data and output data.

[0123] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0124] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0125] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0126] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0127] Generally, the unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0128] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0129] The above-mentioned Figure 1 The training device 120 in the embodiment of this application is capable of performing the various steps of the method for training an active learning model. Figure 1 The execution device 110 is capable of performing various steps of the image processing method (e.g., image classification, image segmentation, and object detection) according to the embodiments of this application. Figure 2 The neural network model shown and Figure 3 The chip shown can also be used to perform various steps of the image processing method according to the embodiments of this application. Figure 3 The chip shown can also be used to perform the various steps of the method for training an active learning model in the embodiments of this application.

[0130] like Figure 4 As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 210, and a data storage system 250, wherein the local devices 301 and 302 are connected to the execution device 210 through a communication network.

[0131] The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the neural network training method or image processing method (e.g., image super-resolution method, image denoising method, image de-mosaic method, and image deblurring method) of the embodiments of this application.

[0132] Specifically, the execution device 210 can perform the following processes:

[0133] Obtain the input feature map of the training image; use the feature extraction kernel of the neural network to perform feature extraction processing on the input feature map to obtain the first candidate feature map. The feature extraction processing makes each element in the first candidate feature map the L1 regular distance between the feature extraction kernel and the corresponding data in the input feature map; add the first candidate feature map and the second candidate feature map to obtain the output feature map. The second candidate feature map is the feature map obtained by expanding the input feature map by N times, where N is greater than 0; determine the image processing result of the training image based on the output feature map; adjust the parameters of the neural network based on the image processing result.

[0134] The above process execution device 210 can train a neural network, which can be used for image processing, speech processing and natural language processing, etc. For example, the neural network can be used to implement the image super-resolution method, image denoising method, image de-mosaic method and image deblurring method in the embodiments of this application.

[0135] Alternatively, the above-described process execution device 210 can be configured into an image processing apparatus, which can be used for image processing (e.g., it can be used to implement the image super-resolution method, image denoising method, image de-mosaic method, and image deblurring method in the embodiments of this application).

[0136] Users can interact with execution device 210 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0137] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0138] In one implementation, local devices 301 and 302 obtain relevant parameters of the neural network from execution device 210, deploy the neural network on local devices 301 and 302, and use the neural network to perform image processing on the image to be processed to obtain the processing result of the image to be processed.

[0139] In another implementation, a neural network can be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local devices 301 and 302, and uses the neural network to perform image processing on the image to obtain the processing result of the image to be processed.

[0140] In one implementation, local devices 301 and 302 obtain relevant parameters of the image processing device from execution device 210, deploy the image processing device on local devices 301 and 302, and use the image processing device to perform image processing on the image to be processed to obtain the processing result of the image to be processed.

[0141] In another implementation, an image processing device can be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local device 301 and the local device 302, and uses the image processing device to perform image processing on the image to be processed to obtain the processing result of the image to be processed.

[0142] In other words, the execution device 210 can also be a cloud device, in which case the execution device 210 can be deployed in the cloud; or, the execution device 210 can also be a terminal device, in which case the execution device 210 can be deployed on the user terminal side. This application embodiment does not limit this.

[0143] The following description, in conjunction with the accompanying drawings, details the method for training an active learning model and the image processing method (e.g., the image processing method may include image classification, object detection, and image segmentation) according to embodiments of this application.

[0144] Please see Figure 5 , Figure 5 This is a schematic diagram of the active learning model in this application. Figure 5 As shown in step (a), the active learning model includes a backbone network g and two adversarial classifiers f1 and f2. The backbone network g can be a convolutional neural network or other neural networks. This application does not make any specific limitations on this. The backbone network g extracts features from the input image to be processed to obtain the feature vector of the image to be processed. The adversarial classifiers are used to obtain the predicted value of the image to be processed based on the feature vector of the input image to be processed. The predicted value can be the category of the image, the category of the target object in the image, or the category of each pixel in the image. The mathematical representation of the above three categories is a multi-dimensional vector.

[0145] Please see Figure 6 , Figure 6 This is a schematic flowchart illustrating a method 600 for training an active learning model, as provided in an embodiment of this application. Figure 6 As shown, method 600 includes steps S1, S2, S3, S4 and S5.

[0146] In some examples, method 600 can be... Figure 1 The execution device 120 in Figure 3 The chip shown and Figure 4 The execution device 210 and other devices in the process are executed.

[0147] Step S1: Utilize the labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 Active learning model M i It includes a backbone network and multiple parallel classifiers, with the parallel classifiers connected to the backbone network, where i is a positive integer.

[0148] For specific details, please refer to the corresponding references. Figure 5 Step (a) shown in the figure involves the labeled image dataset C. i Each labeled image x in l Input active learning model M i In the process, the labeled image x is obtained. l Two predicted values and Then, based on the backpropagation algorithm, the binary cross-entropy loss function shown in formula (1-2), and the two predicted values ​​mentioned above, the active learning model M is adjusted. i The parameters of the backbone network and two parallel classifiers are used to obtain the active learning model M. i The active learning model M i The two classifiers have the same parameters.

[0149] Among them, the active learning model M i The adjustment direction is to minimize the value of the binary cross-entropy loss function in formula (1-2), that is, to make the predicted values ​​of the two adversarial classifiers as close as possible to the labeled image x. l The true value of . When i equals zero, the above active learning model M i For the initial active learning model, the parameters of the backbone network and the two parallel classifiers in the initial active learning model are randomly initialized and determined.

[0150]

[0151] Among them, Figure 5 In step (a), x l Let f represent the labeled input image, g represent the feature extractor, and f1 and f2 represent two classifiers, respectively. and These represent the labels of the input image. and These represent the predicted values ​​of the two classifiers, each a multi-dimensional vector, where C represents the dimension of that multi-dimensional vector. and Let represent the values ​​of the c-th dimension in the two multidimensional vectors output by the classifier. and These represent the values ​​of the c-th dimension in the multidimensional vector corresponding to the true values ​​(i.e., labeled values) of the labeled image. It should be understood that... and Equal, L l argmin represents the binary cross-entropy loss function, and argmin indicates that the model parameters are adjusted in a way that minimizes the loss value of the loss function.

[0152] The following will describe in detail the use of the labeled image dataset C i The labeled images in the training active learning model M i The active learning model M is obtained. i1 The process:

[0153] (1) The labeled image dataset C i The labeled image Xr is input into the active learning model M r-1 In the process, the backbone network extracts features from the labeled image Xr to obtain a feature vector, and inputs the feature vector into two adversarial classifiers. The two adversarial classifiers output two predicted values ​​for the labeled image Xr.

[0154] (2) Based on the backpropagation algorithm, the binary cross-entropy loss function shown in formula (1-2), and the two predicted values ​​of the image Xr, update the active learning model M. r-1 The parameters of the backbone network and two parallel classifiers are used to obtain the active learning model M. r ;

[0155] (3) Let r = r + 1, and repeat steps (1) and (2) above until r equals E, where E is the labeled image dataset C. i The number of labeled images in the dataset, where r and E are positive integers;

[0156] Where, when r equals 1, the active learning model M r-1 For the initial active learning model, when r equals E, the active learning model M... r This is the active learning model M. i1 .

[0157] Step S2: Utilize the unlabeled image dataset D i Training active learning model M i1 The active learning model M is obtained. i2 Active learning model M i2 The parameters of multiple parallel classifiers and the active learning model M i1 Multiple parallel classifiers in the model have the same parameters, and the active learning model M... i2 Parameters of the backbone network and the active learning model M i1 The parameters of the backbone network are different.

[0158] In one feasible implementation, please refer to the corresponding documentation. Figure 5 In step (b), the unlabeled image dataset D is... i Input active learning model M i1 The unlabeled image dataset D is obtained. i For each unlabeled image, obtain multiple first predicted values; based on these first predicted values ​​and a first loss function, update the active learning model M. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function consists of multiple first predicted values ​​for each unlabeled image.

[0159] Specifically, the unlabeled image dataset D i Each unlabeled image x in u Input active learning model M i1 In the process, the unlabeled image x is obtained. u Two first prediction values and Then, based on the backpropagation algorithm, the first loss function shown in formulas (1-3) and (1-4), and the two first predicted values ​​mentioned above, the active learning model M is adjusted. i1 The parameters of the backbone network and the parameters of the adversarial classifier are kept constant to obtain the active learning model M. i2 .

[0160] Among them, the active learning model M i1 The direction of adjustment is to minimize the value of the first loss function represented by formula (1-3), that is, to make the first prediction values ​​of the two adversarial classifiers as similar as possible.

[0161]

[0162]

[0163] Among them, L a For the first loss function, and For the unlabeled image x u The two first predicted values ​​are C, where C is the number of unlabeled images input into the active learning model each time.

[0164] The following will describe in detail the use of the unlabeled image dataset D. i Training the active learning model M using unlabeled images i1 The active learning model M is obtained. i2 The process:

[0165] (1) The unlabeled image dataset D i The unlabeled image Xa is input into the active learning model Ma-1 In the process, the backbone network extracts features from the unlabeled image Xa to obtain a feature vector, and inputs the feature vector into two adversarial classifiers. The two adversarial classifiers output two first predicted values ​​for the labeled image Xa.

[0166] (2) Based on the backpropagation algorithm, the first loss function shown in formula (1-3), and the two first predicted values ​​of the unlabeled image Xa, update the active learning model M. a-1 The parameters of the backbone network and the two adversarial classifiers remain unchanged, resulting in the active learning model M. a The average entropy E(u) of the prediction results of the two adversarial classifiers is calculated based on the two first predicted values ​​of the unlabeled image Xa and formula (1-5). The alignment weight Wa of the unlabeled image Xa is calculated based on the average entropy E(u) and formula (1-6). The average entropy E(u) and the alignment weight Wa are used to characterize the deviation of the classifier prediction results, that is, the certainty of the prediction results: the greater the difference between the two first predicted values, the greater the average entropy and the smaller the alignment weight; the closer the two first predicted values ​​are, the smaller the average entropy and the greater the alignment weight.

[0167]

[0168]

[0169] Where n represents the number of images input to the model each time, τ is a hyperparameter that can be set to 0.1, and δ(x) is the activation function.

[0170] Optionally, the activation function shown can be a non-linear activation function such as the Sigmoid function, tanh function, or ReLU function, or it can be an activation function with other functions. This application embodiment does not limit this.

[0171] (3) Let a = a + 1, and repeat steps (1) and (2) above until a equals F, where F is the unlabeled image dataset D. i The number of unlabeled images in the dataset, where a and F are positive integers;

[0172] Where, when a equals 1, the active learning model M a-1 For active learning model M i1 When a equals F, the active learning model M a This is the active learning model M. i2 .

[0173] Step S3: Utilize the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1and unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i3 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of multiple parallel classifiers in the active learning model M are different. i3 Parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same, and the divergence weight is used to characterize the deviation between multiple predictions for each unlabeled image. The multiple predictions for each unlabeled image are output by multiple parallel classifiers.

[0174] Please refer to the corresponding references. Figure 5 In step (c), the unlabeled image dataset D is... i Each unlabeled image x in u and labeled image dataset C i Each labeled image x in l Input active learning model M i2 The active learning model M is updated based on the predicted values ​​output by the classifier. i2 The parameters of the adversarial classifier, the active learning model M i2 With the parameters of the backbone network remaining unchanged, we obtain the active learning model M. i+1 The active learning model M i+1 The two classifiers have the same parameters.

[0175] The above utilizes the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image are determined, specifically including steps Q1, Q2, and Q3:

[0176] Step Q1: Transfer the labeled image dataset C i and unlabeled image dataset D i The j-th image P j Input active learning model M j-1 Image P is obtained. j The multiple second predicted values, where j is a positive integer; where the multiple parallel classifiers refer to... Figure 5 Two adversarial classifiers in the model;

[0177] Step Q2: Based on image P j Multiple second-predicted values ​​update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j When image Pj For the unlabeled image dataset D i When looking at the image in the image, according to image P j Multiple second-predicted values ​​determine the divergence weights of the image.

[0178] Among them, the divergence weight Wd is based on the image P j The divergence weight Wd of the unlabeled image is calculated using multiple second predicted values ​​and formula (1-7) to characterize the deviation between the two second predicted values ​​output by the two adversarial classifiers. As can be easily understood from formula (1-7), when the difference between the two second predicted values ​​is large, the average entropy is greater, and the divergence weight Wd is larger; when the two second predicted values ​​are relatively close, the average entropy is smaller, and the divergence weight Wd is smaller.

[0179]

[0180] In one feasible implementation, the above is based on image P j Multiple second-predicted values ​​update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j Specifically, this includes two situations:

[0181] When image P j For the unlabeled image dataset D i When looking at the image in the image, according to image P j Multiple second predicted values ​​and a second loss function update the active learning model M j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the second loss function is the image P. j The second loss function has multiple predicted values, and the function value of the second loss function is proportional to the divergence weight; the mathematical expression of the second loss function is shown in formulas (1-8) and (1-9), and the active learning model M... j-1 The update direction of the classifier parameters is to minimize the value of the second loss function.

[0182]

[0183]

[0184] When image P j For the labeled image dataset C i When looking at the image in the image, according to image P j The active learning model M is updated using multiple second-predicted values ​​and a third loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the third loss function is the image P. jMultiple second predicted values ​​and image P j The true value of the third loss function is expressed mathematically as the cross-entropy loss function shown in formula (1-2).

[0185] Step Q3: When j is less than the sum of E and F, let j = j + 1, and repeat steps Q1-Q2; when j is equal to the sum of E and F, activate the active learning model M. i3 As an active learning model M i+1 E and F represent the labeled image dataset C, respectively. i and unlabeled image dataset D i The number of images in the dataset, where E and F are positive integers.

[0186] It should be understood that, in the embodiments of this application, labeled and unlabeled images are used to train the active learning model M. i2 In this process, the input image data has no sequential order; the active learning model M can be trained first using labeled images. i2 Then, the active learning model M is trained using unlabeled images. i2 Alternatively, an active learning model M can be trained using unlabeled images first. i2 Then, the labeled images are used to train the active learning model M. i2 Alternatively, labeled images can be used to train the active learning model M alternately. i2 Alternatively, other input orders can be used to input the unlabeled and labeled images to train the active learning model M. i2 This application does not impose specific limitations on this.

[0187] Step S4: Based on the unlabeled image dataset D i The divergence weight of each unlabeled image in the unlabeled image dataset D i N unlabeled images are selected from the dataset and manually labeled. These N manually labeled images are then added to the labeled image dataset C. i The labeled image dataset C is obtained. i+1 and unlabeled dataset D i+1 N is a positive integer.

[0188] Specifically, the unlabeled image dataset D i After manually labeling the N unlabeled images selected from the dataset, they are added to the labeled image dataset C. i And never labeled the image dataset D i Delete the selected N unlabeled images.

[0189] In one feasible implementation, the above is based on the unlabeled image dataset D. i The divergence weight of each unlabeled image in the unlabeled image dataset Di N unlabeled images were selected from the dataset and manually labeled, including: [the process of manually labeling] the unlabeled image dataset D. i Each unlabeled image is sorted from largest to smallest according to its divergence weight, and the top N unlabeled images are selected from the sorted results for manual labeling.

[0190] As described in step Q2 of step S3 above, the larger the divergence weight, the more difficult it is for the adversarial classifier to reach a consensus on the prediction results of the input image to be processed. Such samples with large divergence weights are complex samples that are difficult for machines to recognize. Conversely, samples with small divergence weights are simple samples that are easy for machines to recognize.

[0191] Step S5: Based on the labeled image dataset C i+1 Define the target active learning model.

[0192] In one feasible implementation, when the labeled image dataset C i+1 Number of images and unlabeled dataset D i+1 The ratio of the number of images in the middle is greater than or equal to a preset ratio value, or by using an active learning model M. i+1 For the labeled image dataset C i+1 When the accuracy of prediction for the image in the dataset is greater than or equal to the preset accuracy, the active learning model M will be activated. i+1 As a target-oriented active learning model, it should be understood that the parameters of multiple classifiers in the trained target-oriented active learning model are the same; wherein, the aforementioned preset ratio value and preset accuracy can be set according to the actual scenario. In addition, other conditions can also be used as conditions for terminating the training of the active learning model, and this application does not make specific limitations on this.

[0193] When the labeled image dataset C i+1 Number of images and unlabeled dataset D i+1 The ratio of the number of images in the middle is less than a preset ratio, or an active learning model M is used. i+1 For the labeled image dataset C i+1 If the accuracy of the prediction of the image is less than the preset accuracy, let i = i + 1, and repeat the above steps S1-S5.

[0194] The above utilizes the active learning model M i+1 For the labeled image dataset C i+1 The process of improving the accuracy of image prediction is specifically as follows: using the active learning model M i+1 For the labeled image dataset C i+1For each labeled image in the dataset, a prediction is made, resulting in a predicted value for each image. The predicted value and the true value of each image are then compared. The number of labeled images whose predicted and true values ​​match is then counted, and this number is correlated with the number of labeled image datasets C. i+1 The ratio of the number of images in the middle is the prediction accuracy.

[0195] As can be seen, in this embodiment, an active learning model is first trained using a labeled image dataset, and the parameters of the backbone network and multiple parallel classifiers in the active learning model are adjusted to obtain the active learning model M. i1 Then fix the active learning model M. i1 The classifier parameters are adjusted, and the active learning model M is updated using a large number of unlabeled images from the unlabeled image dataset. i1 The parameters of the backbone network are adjusted to ensure that the prediction results of multiple parallel classifiers for the same unlabeled image are as consistent as possible, resulting in the active learning model M. i2 At this point, the parameters in the backbone network can be considered optimized to their best; therefore, the active learning model M can be fixed. i2 The parameters of the backbone network are used to train the active learning model M using images from both the unlabeled and labeled image datasets. i2 The parameters of the classifier are updated to obtain the divergence weight for each unlabeled image. Since the divergence weight represents the degree of deviation of the classifier's prediction results for the same image, that is, quantitatively represents the difficulty of the machine to distinguish unlabeled images, the most representative unlabeled images that need to be manually labeled can be selected through the divergence weight. Thus, a high-precision active learning model is obtained with as few images as possible that need to be manually labeled, saving the cost of manual labeling. In addition, in step S3, the model is trained simultaneously using the unlabeled image dataset and the labeled image dataset. This can ensure that when the prediction values ​​of multiple classifiers for the same image are similar, the prediction values ​​of the multiple classifiers are also as close as possible to the true value of the image.

[0196] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an example of image classification using the method described in this application, whereby sample selection and training of an active learning model are performed according to the method of this application. Figure 7 The "agreement-disagreement-selection" (ADS) method is the method used in this application to train the active learning model. Figure 7 As shown, the purpose of training the active learning model is to enable it to identify whether an image being processed belongs to a cat, dog, or cow. The specific training process involves inputting labeled and unlabeled images into the active learning model. For labeled images, the model's predicted values ​​are compared with the actual values ​​of the images. Figure 6 The model parameters are adjusted using the cross-entropy loss function described in [the document]. For unlabeled images, after training with ADS, predicted values ​​for the unlabeled images are obtained. These predicted values ​​are multi-dimensional vectors, where each element corresponds to an animal category. The value of each element represents the probability that the unlabeled image belongs to the animal category corresponding to that element. Figure 7 As shown, the predicted value is a three-dimensional vector, where the three elements represent the probability values ​​of the unlabeled image being a dog, a cat, and a cow, respectively. Figure 7 As shown, when the two 3D vectors obtained by the two classifiers are relatively close, and each 3D vector has an element, such as the element corresponding to a cat, whose value is close to 1 or much higher than the values ​​of the other two elements, then the prediction results of the two predictors are consistent, and this unlabeled image is regarded as a simple image, that is, an image that is easy for the machine to recognize. When the two 3D vectors obtained by the two classifiers are significantly different or the values ​​of the three elements in any 3D vector are relatively close, then the unlabeled image is regarded as a sample that is difficult to recognize. Then, the divergence weight of each image in the unlabeled image is calculated to quantify the degree to which each unlabeled image is easy or difficult to recognize. The images that need to be manually labeled are selected by the divergence weight and manually labeled. The labeled images are then added to the labeled dataset.

[0197] The following section uses an image classification task as an example to verify the performance of the target active learning model obtained after training using the above method 600.

[0198] like Figure 8 As shown, on the CIFAR-10 dataset, the active learning model was trained using both the ADS method and the learning loss for active learning (LL4AL) method from the embodiments of this application, respectively, and the results are as follows. Figure 8 The diagram shows the prediction accuracy of different target active learning models obtained through two different methods, under conditions of different labeled sample ratios. Figure 8 As can be seen, ADS significantly outperforms the existing technology LL4AL, especially in the early iterations. When using 2%, 4%, and 6% of labeled samples, ADS's prediction accuracy is 6.00%, 4.43%, and 2.09% higher than LL4AL, respectively. This improvement validates ADS's ability to align the distribution of unlabeled and labeled samples (i.e., predicting the same or similar results for unlabeled and labeled images of the same type) and to manually label representative unlabeled samples using a small dataset of labeled images. In the final iteration using 20% ​​of the samples, ADS achieves an accuracy of 90.63%, very close to the full training set.

[0199] like Figure 9 As shown, on the CIFAR-10 dataset, the active learning models were trained using the ADS method, LL4AL method, state-relabeling adversarial active learning (SRAAL), variational adversarial active learning (VAAL), and active learning for convolutional neural networks: a core-set method, respectively, resulting in the following: Figure 9 The diagram shows the prediction accuracy of different target active learning models obtained through the four methods described above, under different proportions of labeled samples. Figure 9 As can be seen, ADS significantly outperforms the other three methods in terms of prediction accuracy when the standard deviation is small. In particular, when using 10%, 15%, and 20% of the labeled samples, ADS's prediction accuracy is 2.51%, 4.4%, and 2.25% higher than that of the SRAAL method, respectively.

[0200] In summary, the target active learning model trained by the ADS method in this application embodiment can achieve the same prediction accuracy as the target active learning model trained by other methods while selecting as few unlabeled samples as possible for manual annotation; or, that is, when the active learning model is trained with the same proportion of labeled samples, the prediction accuracy of the target active learning model obtained by the ADS method is higher than that of the target active learning model obtained by other methods.

[0201] Figure 10 This is a schematic flowchart of the image processing method of this application. Figure 10 Method 1000 includes steps 1010 and 1020.

[0202] In some examples, method 1000 can be... Figure 1 Execution device 110 in Figure 3 The chip shown and Figure 4 The execution device 210 and other devices in the process are executed.

[0203] Step S1010: Obtain the input feature map of the image to be processed.

[0204] Step S1020: The target active learning model is used to process the image to be processed, obtaining the predicted value of the image. The target active learning model includes a backbone network and multiple parallel classifiers, which are connected to the backbone network.

[0205] This objective-based active learning model is obtained through K training iterations. In the i-th training iteration of the above K training iterations: using the labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 Active learning model M i The labeled image dataset C is obtained from the (i-1)th training iteration in K training iterations. i This is the result of updating after the (i-1)th training iteration; obtained from the unlabeled image dataset D. i Training active learning model M i1 The active learning model M is obtained. i2 Unlabeled image dataset D i The active learning model M is obtained after the (i-1)th training iteration. i2 The parameters of multiple parallel classifiers and the active learning model M i1 Multiple parallel classifiers in the model have the same parameters, and the active learning model M... i2 Parameters of the backbone network and the active learning model M i1 The parameters of the backbone network are different; through the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of multiple parallel classifiers in the active learning model M are different. i+1 Parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same; the labeled image dataset C is updated according to the divergence weights of each unlabeled image. i and unlabeled image dataset D i The labeled dataset C is obtained. i+1 and unlabeled image dataset D i+1 , where i and K are positive integers.

[0206] According to the labeled image dataset C i+1 The number of images in the dataset and the unlabeled dataset D i+1 The ratio of the number of images in the middle, or through the active learning model Mi+1 For the labeled image dataset C i+1 The accuracy of predictions based on the images in the dataset determines the number of training iterations K.

[0207] The specific training process of the above-mentioned active learning model can be referred to Figure 6 The specific description of method 600 is omitted here.

[0208] Optionally, the above method 500 can be processed by a CPU, or by a combination of a CPU and a GPU, or it can be processed without a GPU, using other processors suitable for neural network computing. This application does not impose any restrictions.

[0209] The image processing described above may include image classification, image segmentation, object detection, or other related image processing, which are not specifically limited in this application. The application of method 1000 in the fields of image classification, image segmentation, and object detection will be described in detail below.

[0210] Image classification: The image to be processed is input into the target active learning model. The backbone network in the model extracts the features of the image to be processed, and obtains the feature vector of the image to be processed. The feature vector is then input into the classifier to obtain the predicted value of the image to be processed. The predicted value is a multi-dimensional vector. Each element in the multi-dimensional vector corresponds to an image category. Each element in the multi-dimensional vector is used to represent the probability value of the image to be processed belonging to the image category corresponding to each element.

[0211] Image segmentation: The image to be processed is input into the target active learning model. The backbone network of the model extracts features from the image to obtain a feature vector. This feature vector is then input into a classifier to obtain multiple predicted values ​​for the image. These predicted values ​​correspond one-to-one with multiple pixels in the image. Each predicted value is a multi-dimensional vector, and each element in any multi-dimensional vector corresponds to an image category. Each element in any multi-dimensional vector represents the probability that the pixel corresponding to that element belongs to that category.

[0212] Object detection: The image to be processed is input into the active learning model for objects. The backbone network in the model extracts the features of the image to be processed, and obtains the feature vector of the image to be processed. This feature vector is then input into the classifier. The classifier first identifies and segments the target objects in the image to be processed, and obtains the target regions corresponding to the target objects. Finally, it outputs a multi-dimensional vector corresponding to the target regions. The meaning of this multi-dimensional vector is the same as that in the image classification above, and will not be repeated here.

[0213] Understandable Figure 6 The described embodiment is the training phase of this active learning model (e.g.) Figure 1 The training equipment 120 shown is used in the following phases: The specific training is conducted using... Figure 6 The embodiments shown or any possible implementations based on these embodiments provide a backbone network and multiple adversarial classifiers; while Figure 10 The described embodiments can be understood as the application stage of this active learning model (e.g.) Figure 1 The stage executed by the execution device 110 shown can be specifically manifested as the use of a process by... Figure 6 The active learning model trained in the illustrated embodiment obtains a predicted value for the image to be processed based on the image input by the user.

[0214] Figure 11 The apparatus 1100 for training an active learning model provided in this application embodiment includes: a training unit 1101 for executing steps S1-S3, wherein S1: using a labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 Active learning model M i Includes a backbone network and multiple parallel classifiers, with i being a positive integer; S2: Utilizes an unlabeled image dataset D i Training active learning model M i1 The active learning model M is obtained. i2 Active learning model M i2 The parameters of multiple parallel classifiers and the active learning model M i1 Multiple parallel classifiers in the model have the same parameters, and the active learning model M... i2 Parameters of the backbone network and the active learning model M i1 The parameters of the backbone network are different; S3: using the labeled image dataset C i and unlabeled image dataset D i Training active learning model M i2 The active learning model M is obtained. i+1 and unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of multiple parallel classifiers in the active learning model M are different. i+1 Parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same, and the divergence weight is used to characterize the deviation between multiple prediction results for each unlabeled image. The multiple prediction results for each unlabeled image correspond one-to-one with multiple parallel classifiers.

[0215] Filtering unit 1102 is used to perform step S4: based on the unlabeled image dataset D i The divergence weight of each unlabeled image in the unlabeled image dataset D i N unlabeled images are selected from the dataset and manually labeled. These N manually labeled images are then added to the labeled image dataset C. i The labeled image dataset C is obtained. i+1 and unlabeled dataset D i+1 N is a positive integer;

[0216] Determining unit 1103 is used to perform step S5: based on the labeled image dataset C i+1 Define the target active learning model.

[0217] In one feasible implementation, the aforementioned determining unit is specifically used to: when the labeled image dataset C i+1 Number of images and unlabeled dataset D i+1 The ratio of the number of images in the middle is greater than or equal to a preset ratio value, or by using an active learning model M. i+1 For the labeled image dataset C i+1 When the accuracy of prediction for the image in the dataset is greater than or equal to the preset accuracy, the active learning model M will be activated. i+1 If the target is an active learning model, then let i = i + 1 and repeat steps S1-S5.

[0218] It should be understood that the training unit 1101, the screening unit 1102, and the determination unit 1103 in the device 1100 can also perform... Figure 6 The methods shown in the embodiments will not be described again here.

[0219] Figure 12 The image processing apparatus 1200 provided in this application embodiment includes:

[0220] Acquisition unit 1201: Used to acquire the image to be processed.

[0221] Processing unit 1202: Used to perform image processing on the image to be processed using a target active learning model to obtain the predicted value of the image to be processed.

[0222] Specifically, the image processing device 1200 can be used to process... Figure 10 The corresponding steps of the image processing method 1000 described herein will not be repeated here.

[0223] Figure 13 This is a schematic diagram of the hardware structure of a training device for an active learning model provided in an embodiment of this application. Figure 13The training device 1300 for the active learning model shown (specifically, the device 1300 can be a computer device) includes a memory 1301, a processor 1302, a communication interface 1303, and a bus 1304. The memory 1301, processor 1302, and communication interface 1303 are interconnected via the bus 1304.

[0224] The memory 1301 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1301 may store a program. When the program stored in the memory 1301 is executed by the processor 1302, the processor 1302 and the communication interface 1303 are used to execute the various steps of the training method of the active learning model of the present application embodiment.

[0225] The processor 1302 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the training device of the active learning model in the embodiments of this application, or to execute the training method of the active learning model in the method embodiments of this application.

[0226] The processor 1302 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the training method for the active learning model of this application can be completed by the integrated logic circuits in the hardware of the processor 1302 or by software instructions. The processor 1302 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1301. The processor 1302 reads the information in the memory 1301 and, in conjunction with its hardware, performs the functions required by the units included in the training device of the active learning model of this application embodiment, or executes the training method of the active learning model of this application embodiment.

[0227] The communication interface 1303 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 1300 and other devices or communication networks. For example, training data (such as the labeled image dataset and the unlabeled image dataset described in the embodiments of this application) can be obtained through the communication interface 1303.

[0228] Bus 1304 may include a pathway for transmitting information between various components of device 1300 (e.g., memory 1301, processor 1302, communication interface 1303).

[0229] Figure 14 This is a schematic diagram of the hardware structure of the image processing device provided in the embodiments of this application. Figure 14 The image processing device 1400 shown (which may specifically be a computer device) includes a memory 1401, a processor 1402, a communication interface 1403, and a bus 1404. The memory 1401, the processor 1402, and the communication interface 1403 are interconnected via the bus 1404.

[0230] The memory 1401 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1401 may store a program. When the program stored in the memory 1401 is executed by the processor 1402, the processor 1402 and the communication interface 1403 are used to execute the various steps of the image processing method of the embodiments of this application.

[0231] The processor 1402 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image processing apparatus of this application embodiment, or to execute the image processing method of this application method embodiment.

[0232] The processor 1402 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image processing method of this application can be completed by the integrated logic circuits in the hardware of the processor 1402 or by instructions in software form. The processor 1402 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1401. The processor 1402 reads the information in the memory 1401 and, in conjunction with its hardware, performs the functions required by the units included in the image processing apparatus of this application embodiment, or executes the image processing method of the method embodiment of this application.

[0233] Communication interface 1403 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between device 1400 and other devices or communication networks. For example, training data (such as that described in this application) can be acquired through communication interface 1403. Figure 10 (The image to be processed in the described embodiments).

[0234] Bus 1404 may include a pathway for transmitting information between various components of device 1400 (e.g., memory 1401, processor 1402, communication interface 1403).

[0235] It should be noted that, although Figure 13 and Figure 14 The illustrated devices 1300 and 1400 only show the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 1300 and 1400 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 1300 and 1400 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 1300 and 1400 may also include only the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 13 or Figure 14 All the devices shown.

[0236] It can be understood that the aforementioned device 1300 is equivalent to Figure 1 The training equipment 120 and device 1400 are equivalent to Figure 1 The execution device 110 is described herein. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0237] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0239] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0241] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an active learning model, characterized in that, The method includes: S1: Using the labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 The active learning model M i The dataset includes a backbone network and multiple parallel classifiers connected to the backbone network, where i is a positive integer, and the labeled image dataset C... i Includes tagged images; S2: Utilizing the unlabeled image dataset D i Training the active learning model M i1 The active learning model M is obtained. i2 The active learning model M i2 The parameters of multiple parallel classifiers and the active learning model M i1 The parameters of multiple parallel classifiers are the same, and the active learning model M i2 The parameters of the backbone network and the active learning model M i1 The parameters of the backbone network differ, and the unlabeled image dataset D... i Including unlabeled images, the use of the unlabeled image dataset D i Training the active learning model M i1 The active learning model M is obtained. i2 This includes: transferring the unlabeled image dataset D i Input the active learning model M i1 The unlabeled image dataset D is obtained. i The active learning model M is updated based on multiple first predicted values ​​for each unlabeled image and a first loss function. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function is a plurality of first predicted values ​​for each unlabeled image; S3: Utilizing the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of the multiple parallel classifiers in the active learning model M are different. i+1 The parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same, and the divergence weight is used to characterize the deviation between multiple predicted values ​​of each unlabeled image. The multiple predicted values ​​of each unlabeled image are output by the multiple parallel classifiers respectively. S4: Based on the unlabeled image dataset D i The divergence weights for each unlabeled image in the unlabeled image dataset D are derived from this dataset. i N unlabeled images are selected from the dataset and manually labeled. These N manually labeled images are then added to the labeled image dataset C. i The labeled image dataset C is obtained. i+1 and unlabeled dataset D i+1 N is a positive integer; S5: Based on the labeled image dataset C i+1 Define the target active learning model.

2. The method according to claim 1, characterized in that, The data is based on the labeled image dataset C. i+1 Define the target active learning model, including: When the labeled image dataset C i+1 The number of images in the dataset and the unlabeled dataset D i+1 The ratio of the number of images in the middle is greater than or equal to a preset ratio value, or by using an active learning model M. i+1 For the labeled image dataset C i+1 When the accuracy of prediction of the image in the model is greater than or equal to a preset accuracy, the active learning model M is activated. i+1 As the target active learning model; Otherwise, let i = i + 1, and repeat steps S1-S5.

3. The method according to claim 1 or 2, characterized in that, The use of the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i The divergence weights for each unlabeled image include: Q1: The labeled image dataset C... i and the unlabeled image dataset D i The j-th image P j Input active learning model M j-1 The image P is obtained. j Multiple second predicted values, where j is a positive integer; Q2: Based on the image P j The active learning model M is updated by multiple second predicted values. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j When the image P j For the unlabeled image dataset D i When the image is in the image, according to the image P j The multiple second predicted values ​​determine the divergence weights of the image; Q3: When j is less than the sum of E and F, let j = j + 1, and repeat steps Q1-Q2; when j is equal to the sum of E and F, set the active learning model M... j As an active learning model M i+1 When j=1, the active learning model M j-1 The active learning model M i2 E and F are respectively the labeled image dataset C i and the unlabeled image dataset D i The number of images in the dataset, where E and F are positive integers.

4. The method according to claim 3, characterized in that, According to the image P j The active learning model M is updated by multiple second predicted values. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j ,include: When the image P j For the unlabeled image dataset D i When the image is in the image, according to the image P j The active learning model M is updated using multiple second predicted values ​​and a second loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the second loss function is the image P. j Multiple second predicted values, and the function value of the second loss function is related to the image P. j The weight of the disagreement is directly proportional to its weight; When the image P j For the labeled image dataset C i When the image is in the image, according to the image P j The active learning model M is updated using multiple second predicted values ​​and a third loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the third loss function is the image P. j Multiple second predicted values ​​and the image P j The true value.

5. The method according to any one of claims 1-4, characterized in that, The data based on the unlabeled image dataset D i The divergence weights for each unlabeled image in the unlabeled image dataset D are derived from this dataset. i N unlabeled images were selected and manually labeled, including: For the unlabeled image dataset D i Each unlabeled image is sorted from largest to smallest according to its divergence weight to obtain a sorting result. The top N unlabeled images are selected from the sorting result for manual labeling.

6. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The target active learning model is used to process the image to be processed to obtain the predicted value of the image to be processed. The target active learning model includes a backbone network and multiple parallel classifiers, and the multiple parallel classifiers are connected to the backbone network. The target active learning model is obtained through K training iterations. In the i-th training iteration of the K training iterations: using the labeled image dataset C... i Training active learning model M i The active learning model M is obtained. i1 The active learning model M i The labeled image dataset C is obtained from the (i-1)th training iteration of the K training iterations. i This is the result updated after the (i-1)th training iteration; obtained from the unlabeled image dataset D. i Training the active learning model M i1 The active learning model M is obtained. i2 The unlabeled image dataset D i The active learning model M is obtained by updating it after the (i-1)th training iteration. i2 The parameters of multiple parallel classifiers and the active learning model M i1 The parameters of multiple parallel classifiers are the same, and the active learning model M i2 The parameters of the backbone network and the active learning model M i1 The parameters of the backbone network differ, and the unlabeled image dataset D is used. i Training the active learning model M i1 The active learning model M is obtained. i2 This includes: transferring the unlabeled image dataset D i Input the active learning model M i1 The unlabeled image dataset D is obtained. i The active learning model M is updated based on multiple first predicted values ​​for each unlabeled image and a first loss function. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function is multiple first predicted values ​​for each unlabeled image; through the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of the multiple parallel classifiers in the active learning model M are different. i+1 The parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same; the labeled image dataset C is updated according to the divergence weights of each unlabeled image. i and the unlabeled image dataset D i The labeled dataset C is obtained. i+1 and unlabeled image dataset D i+1 The i and the K are positive integers.

7. The method according to claim 6, characterized in that, According to the labeled image dataset C i+1 The number of images in the dataset and the unlabeled dataset D i+1 The ratio of the number of images in the middle, or through the active learning model M i+1 For the labeled image dataset C i+1 The accuracy of predictions based on the images in the dataset determines the number of training iterations, K.

8. The method according to claim 6 or 7, characterized in that, The image processing includes at least one of image classification, object detection, and image segmentation.

9. A training device for an active learning model, characterized in that, The device includes: Training unit, used to execute steps S1-S3; A filtering unit is used to perform step S4; The unit is designated to execute step S5; Step S1: Utilizing the labeled image dataset C i Training active learning model M i The active learning model M is obtained. i1 The active learning model M i The dataset includes a backbone network and multiple parallel classifiers connected to the backbone network, where i is a positive integer, and the labeled image dataset C... i Includes tagged images; Step S2: Utilizing the unlabeled image dataset D i Training the active learning model M i1 The active learning model M is obtained. i2 The active learning model M i2 The parameters of multiple parallel classifiers and the active learning model M i1 The parameters of multiple parallel classifiers are the same, and the active learning model M i2 The parameters of the backbone network and the active learning model M i1 The parameters of the backbone network differ, and the unlabeled image dataset D... i Including unlabeled images, the use of the unlabeled image dataset D i Training the active learning model M i1 The active learning model M is obtained. i2 This includes: transferring the unlabeled image dataset D i Input the active learning model M i1 The unlabeled image dataset D is obtained. i The active learning model M is updated based on multiple first predicted values ​​for each unlabeled image and a first loss function. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function is a plurality of first predicted values ​​for each unlabeled image; Step S3: Utilizing the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of the multiple parallel classifiers in the active learning model M are different. i+1 The parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same, and the divergence weight is used to characterize the deviation between multiple predicted values ​​of each unlabeled image. The multiple predicted values ​​of each unlabeled image are output by the multiple parallel classifiers respectively. Step S4: Based on the unlabeled image dataset D i The divergence weights for each unlabeled image in the unlabeled image dataset D are derived from this dataset. i N unlabeled images are selected from the dataset and manually labeled. These N manually labeled images are then added to the labeled image dataset C. i The labeled image dataset C is obtained. i+1 and unlabeled dataset D i+1 N is a positive integer; Step S5: Based on the labeled image dataset C i+1 Define the target active learning model.

10. The apparatus according to claim 9, characterized in that, The determining unit is specifically used for: When the labeled image dataset C i+1 The number of images in the dataset and the unlabeled dataset D i+1 The ratio of the number of images in the middle is greater than or equal to a preset ratio value, or by using an active learning model M. i+1 For the labeled image dataset C i+1 When the accuracy of prediction for the image in the dataset is greater than or equal to the preset accuracy, the active learning model M will be activated. i+1 As the target active learning model; Otherwise, let i = i + 1, and use the training unit to execute steps S1-S3, the filtering unit to execute step S4, and the determining unit to execute step S5.

11. The apparatus according to claim 9 or 10, characterized in that, In the use of the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i Regarding the aspect of the divergence weights for each unlabeled image, the training unit is specifically used for: Q1: The labeled image dataset C... i and the unlabeled image dataset D i The j-th image P j Input active learning model M j-1 The image P is obtained. j Multiple second predicted values, where j is a positive integer; Q2: Based on the image P j The active learning model M is updated by multiple second predicted values. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j When the image P j For the unlabeled image dataset D i When the image is in the image, according to the image P j The multiple second predicted values ​​determine the divergence weights of the image; Q3: When j is less than the sum of E and F, let j = j + 1, and repeat steps Q1-Q2; when j is equal to the sum of E and F, set the active learning model M... j As an active learning model M i+1 When j=1, the active learning model M j-1 The active learning model M i2 E and F are respectively the labeled image dataset C i and the unlabeled image dataset D i The number of images in the dataset, where E and F are positive integers.

12. The apparatus according to claim 11, characterized in that, According to the image P j The active learning model M is updated by multiple second predicted values. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j In this respect, the training unit is specifically used for: When the image P j For the unlabeled image dataset D i When the image is in the image, according to the image P j The active learning model M is updated using multiple second predicted values ​​and a second loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the second loss function is the image P. j Multiple second predicted values, and the function value of the second loss function is related to the image P. j The weight of the disagreement is directly proportional to its weight; When the image P j For the labeled image dataset C i When the image is in the image, according to the image P j The active learning model M is updated using multiple second predicted values ​​and a third loss function. j-1 The parameters of multiple classifiers are used to obtain the active learning model M. j The input data for the third loss function is the image P. j Multiple second predicted values ​​and the image P j The true value.

13. The apparatus according to any one of claims 9-12, characterized in that, The filtering unit is specifically used for: For the unlabeled image dataset D i Each unlabeled image is sorted from largest to smallest according to its divergence weight to obtain a sorting result. The top N unlabeled images are selected from the sorting result for manual labeling.

14. An image processing apparatus, characterized in that, The device includes: Acquisition Unit: Acquires the image to be processed; Processing unit: Uses a target active learning model to perform image processing on the image to be processed to obtain a predicted value of the image to be processed; the target active learning model includes a backbone network and multiple parallel classifiers, and the multiple parallel classifiers are connected to the backbone network; The target active learning model is obtained through K training iterations. In the i-th training iteration of the K training iterations: using the labeled image dataset C... i Training active learning model M i The active learning model M is obtained. i1 The active learning model M i The labeled image dataset C is obtained from the (i-1)th training iteration of the K training iterations. i This is the result updated after the (i-1)th training iteration; obtained from the unlabeled image dataset D. i Training the active learning model M i1 The active learning model M is obtained. i2 The unlabeled image dataset D i The active learning model M is obtained by updating it after the (i-1)th training iteration. i2 The parameters of multiple parallel classifiers and the active learning model M i1 The parameters of multiple parallel classifiers are the same, and the active learning model M i2 The parameters of the backbone network and the active learning model M i1 The parameters of the backbone network differ, and the unlabeled image dataset D is used. i Training the active learning model M i1 The active learning model M is obtained. i2 This includes: transferring the unlabeled image dataset D i Input the active learning model M i1 The unlabeled image dataset D is obtained. i The active learning model M is updated based on multiple first predicted values ​​for each unlabeled image and a first loss function. i1 The parameters of the backbone network are used to obtain the active learning model M. i2 The input data for the first loss function is multiple first predicted values ​​for each unlabeled image; through the labeled image dataset C i and the unlabeled image dataset D i Training the active learning model M i2 The active learning model M is obtained. i+1 and the unlabeled image dataset D i The divergence weights for each unlabeled image in the active learning model M i+1 The parameters of multiple parallel classifiers and the active learning model M i2 The parameters of the multiple parallel classifiers in the active learning model M are different. i+1 The parameters of the backbone network and the active learning model M i2 The parameters of the backbone network are the same; the labeled image dataset C is updated according to the divergence weights of each unlabeled image. i and the unlabeled image dataset D i The labeled dataset C is obtained. i+1 and unlabeled image dataset D i+1 The i and the K are positive integers.

15. The apparatus according to claim 14, characterized in that, According to the labeled image dataset C i+1 The number of images in the dataset and the unlabeled dataset D i+1 The ratio of the number of images in the middle, or through the active learning model M i+1 For the labeled image dataset C i+1 The accuracy of predictions based on the images in the dataset determines the number of training iterations, K.

16. The apparatus according to claim 14 or 15, characterized in that, The image processing includes at least one of image classification, object detection, and image segmentation.

17. An apparatus for training an active learning model, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 5.

18. An image processing apparatus, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 6 to 8.

19. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 8.

20. A chip, characterized in that, The chip includes a processor and a data interface, wherein the processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 8.

Images