Image processing model processing method, device, equipment and storage medium

By identifying noise-free sample images in the image processing model, updating the positive and negative sample thresholds, and combining multi-label prediction loss and quantization loss, the problems of time-consuming and labor-intensive multi-label annotation and noise impact are solved, achieving accurate quantization of image semantics and effective retrieval.

CN116450870BActive Publication Date: 2024-11-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210017658.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-07
Publication Date
2024-11-05
Estimated Expiration
2042-01-07

AI Technical Summary

Technical Problem

Existing image processing models struggle to effectively recall semantically similar images due to the time-consuming and labor-intensive nature of multi-label annotation and the inaccurate annotation results of pre-trained models, which in turn affect the accuracy of quantization features caused by noisy sample images.

Method used

By acquiring a set of sample images, the model updated in the previous iteration is used to extract image features, perform quantization and multi-label prediction, identify noise-free sample images and correct the annotation information, update the positive and negative sample thresholds, update the model by combining multi-label prediction loss and quantization loss, remove the influence of noisy samples, and dynamically adjust the thresholds to improve training effectiveness.

Benefits of technology

The accuracy of the image processing model is improved in a large number of noisy sample images, ensuring that the quantized features can accurately express the semantics and content of the image, thereby improving the accuracy and efficiency of image retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116450870B_ABST
    Figure CN116450870B_ABST
Patent Text Reader

Abstract

The present application provides a processing method, apparatus, device and storage medium for an image processing model, which can realize effective training of the model in a large number of noisy sample images, including: extracting the quantitative features of the sample image of the current iteration through the image processing model updated in the previous iteration to obtain the multi-label prediction probability of the sample image; determining the noise-free sample image of each label based on the positive and negative sample thresholds of each label updated in the previous iteration and the multi-label prediction probability of the sample image, and determining the multi-label prediction loss of the current iteration based on the annotation information of the corrected noise-free sample image and the multi-label prediction probability; determining the quantitative loss of the current iteration based on the quantitative features of the sample image, similar sample images and non-similar sample images, and the quantitative features of the sample image and the corresponding quantitative targets; iteratively updating the image processing model updated in the previous iteration based on the multi-label prediction loss and the quantitative loss.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a processing method, apparatus, computer equipment, storage medium and computer program product for an image processing model. Background Art

[0002] Image retrieval is the process of searching for the semantics and content of an input image and extracting images from an image database that have similar semantics and content to the input image. The usual processing flow is: quantize the input image to obtain the quantitative features of the input image, and compare the quantitative features with the quantitative features of each image in the image database to obtain the recalled image.

[0003] From this, we can see that in order to accurately recall images with similar image semantics and image content to the input image, the image processing model needs to have the ability to accurately extract quantitative features that measure image semantics and image content.

[0004] However, learning image semantics requires multi-label annotation of sample images. Annotating semantic labels for large-scale images is time-consuming and labor-intensive. Although using existing or pre-trained multi-label annotation models to perform multi-label annotation on sample images can reduce the label annotation cost, the annotation results of such models are usually not accurate enough, resulting in image processing models being based on sample images with a lot of noise, making it difficult to accurately learn the quantitative features of the image. Summary of the invention

[0005] Based on this, it is necessary to provide a processing method, device, computer equipment, storage medium and computer program product for an image processing model that can improve the accuracy of the image processing model based on a large number of noisy sample images in response to the above technical problems.

[0006] A method for processing an image processing model, the method comprising:

[0007] Get the sample image set of the current iteration;

[0008] Extracting image features of sample images in the sample image set through the image processing model updated in the previous iteration, quantizing the image features to obtain quantized features, and performing multi-label prediction based on the quantized features to obtain multi-label prediction probabilities of the sample images;

[0009] Based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image, determine the noise-free sample image corresponding to each label from the sample image set and correct the label information of the noise-free sample image, and determine the multi-label prediction loss of the current iteration according to the label information of the noise-free sample image and the multi-label prediction probability;

[0010] According to the multi-label prediction probability of the noise-free sample image, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration;

[0011] Determine similar sample images and non-similar sample images of the sample image from the sample image set, and determine the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization targets;

[0012] Based on the multi-label prediction loss and the quantization loss, the image processing model updated in the previous iteration is updated to obtain the image processing model updated in the current iteration.

[0013] A processing device for an image processing model, the device comprising:

[0014] A sample image set acquisition module is used to acquire the sample image set of the current iteration;

[0015] The forward processing module of the current iteration is used to extract image features of the sample images in the sample image set through the image processing model updated in the previous iteration, quantize the image features to obtain quantized features, and perform multi-label prediction based on the quantized features to obtain the multi-label prediction probability of the sample images;

[0016] A multi-label prediction loss determination module is used to determine the noise-free sample images corresponding to each label from the sample image set and correct the labeling information of the noise-free sample images based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image, and determine the multi-label prediction loss of the current iteration according to the labeling information of the noise-free sample images and the multi-label prediction probability;

[0017] A positive and negative sample threshold updating module is used to update the positive and negative sample thresholds corresponding to each label updated in the previous iteration according to the multi-label prediction probability of the noise-free sample image, so as to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration;

[0018] a quantization loss determination module, used to determine similar sample images and non-similar sample images of the sample image from the sample image set, and determine the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization targets;

[0019] The backward processing module of the current iteration is used to update the image processing model updated in the previous iteration based on the multi-label prediction loss and the quantization loss to obtain the image processing model updated in the current iteration.

[0020] In one embodiment, the multi-label prediction loss determination module is also used to take each label in the multi-label as a target label in turn; obtain the positive sample threshold and the negative sample threshold corresponding to the target label updated in the previous iteration; when the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is greater than or equal to the positive sample threshold of the target label, then determine that the sample image is the noise-free positive sample image corresponding to the target label in the current iteration, and correct the labeling information of the sample image corresponding to the target label to the first value; when the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is less than or equal to the negative sample threshold of the target label, then determine that the sample image is the noise-free negative sample image corresponding to the target label in the current iteration, and correct the labeling information of the sample image corresponding to the target label to the second value.

[0021] In one embodiment, the positive and negative sample threshold updating module is further used to take each label in the multiple labels as the target label in turn; according to the noise-free positive sample image of the target label in the current iteration, the mean value of the predicted probability corresponding to the target label is used to update the positive sample threshold corresponding to the target label updated in the previous iteration, so as to obtain the positive sample threshold corresponding to the target label updated in the current iteration; according to the noise-free negative sample image of the target label in the current iteration, the mean value of the predicted probability corresponding to the target label is used to update the negative sample threshold corresponding to the target label updated in the previous iteration, so as to obtain the negative sample threshold corresponding to the target label updated in the current iteration.

[0022] In one embodiment, the positive and negative sample threshold updating module is further used to perform weighted summation of the positive sample threshold corresponding to the target label updated in the previous iteration and the mean of the predicted probability of the target label corresponding to the noise-free positive sample image of the target label in the current iteration to obtain the positive sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

[0023] In one embodiment, the positive and negative sample threshold updating module is further used to perform weighted summation of the negative sample threshold corresponding to the target label updated in the previous iteration and the mean of the predicted probability of the target label corresponding to the noise-free negative sample image of the target label in the current iteration to obtain the negative sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

[0024] In one embodiment, the multi-label prediction loss determination module is also used to take each label in the multi-label as the target label in turn; for each noise-free sample image corresponding to the target label, the prediction loss of the target label is calculated according to the corrected annotation information of the noise-free sample image and the prediction probability corresponding to the target label in the multi-label prediction probability of the noise-free sample image; the prediction losses corresponding to each label are summed to obtain the multi-label prediction loss of the iteration.

[0025] In one embodiment, the sample image set includes a similar sample image pair, and the two sample images included in the similar sample image pair are similar;

[0026] The quantization loss determination module is also used to generate a triplet corresponding to each similar sample image pair of the sample image set; the triplet includes a target sample image, a similar sample image of the target sample image and a non-similar sample image of the target sample image; for each triplet, the triplet quantization loss is calculated according to the difference in quantization features between the target sample image and the similar sample images, and the difference between the quantization features of the target sample image and the non-similar sample images; the image quantization loss is calculated according to the difference between the quantization features of each sample image of the sample image set and the corresponding quantization target; and the triplet quantization loss and the image quantization loss are weightedly summed to obtain the quantization loss of the current iteration.

[0027] In one embodiment, the quantization loss determination module is further used to use the two sample images in the similar sample image pair as the target sample image and the similar sample images of the target sample image; based on the image features extracted this time, determine at least one sample image that is not similar to the target sample image from the sample image set as the non-similar sample image of the target sample image; and form at least one triplet with the target sample image, the similar sample image of the target sample image, and the non-similar sample image of the target sample image.

[0028] In one embodiment, the device further comprises: a similar sample image pair acquisition module, which is used to acquire the storyboards of the video; using the images of the same storyboard as similar sample images to obtain a plurality of similar sample image pairs corresponding to the same storyboard;

[0029] The quantization loss determination module is also used to obtain the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time; the candidate sample images are sample images of other similar sample image pairs in the sample image set except the similar sample image pair to which the target sample image belongs; based on the distances between the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time, the candidate sample images are sorted in ascending order of distance; the candidate sample images ranked after a set number of digits are determined, and sample images that are dissimilar to the target sample image are determined from the candidate sample images ranked after the set number of digits as non-similar sample images of the target sample image.

[0030] In one embodiment, the quantization loss determination module is further used to input the quantization features of the sample image into a preset sign function to obtain the sign of the quantization features of the sample image as the quantization target of the sample image; and calculate the image quantization loss of the sample image based on the difference between the quantization features of the sample image and the corresponding sign.

[0031] In one embodiment, the image processing model includes a quantization network and a multi-label prediction network;

[0032] The backward processing module of the current iteration is used to perform weighted summation of the multi-label prediction loss and the quantization loss to obtain the total loss of the current iteration; based on the total loss of the current iteration, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in sequence to obtain the image processing model updated in the current iteration.

[0033] In one embodiment, each iteration is any iteration in a full training round, and the sample image set used in each iteration is a subset of the full sample image set;

[0034] The device also includes a positive and negative sample threshold determination module, which is used to obtain the original annotation information of each sample image in the full sample image set and the multi-label prediction probability of each sample image after completing the previous full training round when the current iteration is the first iteration in the full training round in which the noise-free sample image is determined based on the positive and negative sample thresholds of each label; take each label in the multi-label as the target label in turn; for each target label, determine the positive sample image and the negative sample image of the target label according to the original annotation information; initialize the positive sample threshold of the target label required for the current iteration according to the mean value of the prediction probability corresponding to the target label in the multi-label prediction probability of the positive sample image of the target label after completing the previous full training round; initialize the negative sample threshold of the target label required for the current iteration according to the mean value of the prediction probability corresponding to the target label in the multi-label prediction probability of the negative sample image of the target label after completing the previous full training round.

[0035] In one embodiment, the device also includes an image retrieval module, which is used to obtain a pre-trained image processing model when a training stop condition is met; obtain an input image for image retrieval; extract image features of the input image through the pre-trained image processing model, and quantify the image features of the input image to obtain quantitative features of the input image; compare the quantitative features of the input image with the quantitative features of each stock image in an image library; and determine a retrieval image that matches the input image based on the comparison result.

[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program and the processor executes the processing method of the above-mentioned image processing model.

[0037] A computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the processing method of the above-mentioned image processing model.

[0038] A computer program product stores a computer program, wherein the computer program is executed by a processor to implement the processing method of the above-mentioned image processing model.

[0039] The processing method, apparatus, computer equipment, storage medium and computer program product of the above-mentioned image processing model determine the noise-free sample images of each label from the sample image set through the positive and negative sample thresholds corresponding to each label updated in the previous iteration, and the multi-label prediction probability of the sample image obtained based on the image processing model updated in the previous iteration, so as to realize the judgment of the noise-free sample images corresponding to each label in the current iteration, so as to remove the noise sample images in the calculation of the multi-label prediction loss of the current iteration, so as to avoid the overfitting of the parameters used for image quantization processing in the image processing model to the noise sample images, and realize the effective training of the image processing model in a large number of noise sample images; and, through the multi-label prediction probability of the noise-free sample images of the current iteration, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated, and the obtained noise-free sample images of the current iteration are obtained. The updated positive and negative sample thresholds corresponding to each label are used as the thresholds for judging the noise-free sample images corresponding to each label in the next iteration, so as to realize the dynamic update of the positive and negative sample thresholds of each label used in different iterations, and further improve the effectiveness of training the image processing model in a large number of noisy sample images; in addition, the quantization loss of the current iteration is obtained according to the quantization features of the sample image, similar sample images and non-similar sample images, as well as the quantization features of the sample image and the corresponding quantization targets, and the image processing model updated in the previous iteration is updated through the multi-label prediction loss and the quantization loss, so that in the current iteration, the parameters in the image processing model are adjusted in the direction of minimizing the multi-label loss and the quantization loss, so that the quantization features of the image obtained based on the image processing model after the training are completed can accurately express the image semantics and image content of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An application environment diagram of a processing method of an image processing model in one embodiment;

[0041] Figure 2 is a flowchart of a processing method of an image processing model in one embodiment;

[0042] Figure 3 is a schematic diagram of a first mirror in an embodiment;

[0043] Figure 4 is a schematic diagram of a second mirror in an embodiment;

[0044] Figure 5 A schematic diagram of a process for determining the positive and negative sample thresholds used in the current iteration in one embodiment;

[0045] Figure 6 A flowchart of a processing method of an image processing model in another implementation;

[0046] Figure 7 A schematic diagram of the training process of an image processing model in implementation;

[0047] Figure 8 is a schematic diagram of the architecture of a processing method of an image processing model in one embodiment;

[0048] Fig. 9 is a structural block diagram of a processing device of an image processing model in one embodiment;

[0049] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] Mentioning "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments. It should be noted that the "first", "second", etc. involved in the introduction of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] The processing method of the image processing model provided in this application belongs to the field of artificial intelligence technology. The following is an introduction to the terms involved in this application:

[0053] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0056] Image recognition technology refers to recognition at the category level. In the image recognition process, instead of considering specific instances of objects, we can consider the category of the object (such as people, dogs, cats, birds, etc.) and give the category to which the object belongs. For example, in the image recognition task based on the general object recognition open source dataset imagenet, the category of the target object in the image to be predicted is identified from the categories included in the general object recognition open source dataset imagenet.

[0057] Image retrieval includes image semantic retrieval, which refers to the retrieval process of finding images semantically related to the input image from the image library. Semantic relevance can be a class of images with the same category (such as belonging to the dog category) or the same attributes (there can be multiple attributes, such as yellow, happy, natural environment, etc.).

[0058] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0059] The training of the image processing model can be a process of iteratively updating the parameters of the image processing model using the full set of sample images to minimize the loss. Iteratively updating the model parameters using the full set of sample images can be regarded as a full training round, that is, an epoch. In each epoch, the full set of sample images is divided into multiple batches, and each batch of sample images is input into the image processing model in turn. After each batch of sample images is input into the image processing model, the image processing model predicts the batch of sample images, and based on the predicted values ​​of the batch of sample images and the true values ​​of the batch of sample images (the true values ​​can be the annotation information of the sample images), the loss corresponding to the batch of sample images is obtained to update the parameters of the image processing model; wherein, the process of the image processing model predicting the batch of sample images is called the forward processing process, and the process of updating the parameters of the image processing model based on the loss of the batch of sample images is called the backward processing process. A forward processing process and the backward processing process corresponding to the forward processing process are regarded as an iteration. The batch corresponding to an epoch is consistent with the number of iterations included in the epoch. For example, the full sample image set is divided into 100 batches, the epoch corresponds to 100 batches, and the number of iterations included in the epoch is also 100.

[0060] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0061] Figure 1The application environment diagram of the processing method of the image processing model provided in this application. Among them, the terminal 102 can interact with the server 104 through a communication network; the terminal 102 can be, but not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster or cloud server composed of multiple servers.

[0062] In one embodiment, when the training stop condition is met, the server 104 can obtain a pre-trained image processing model; obtain an input image for image retrieval; extract image features of the input image through the pre-trained image processing model, and quantify the image features of the input image to obtain quantitative features of the input image; compare the quantitative features of the input image with the quantitative features of each stock image in the image library; and determine a retrieval image that matches the input image based on the comparison result.

[0063] The training stop condition is set for the image processing model and can be: when the loss of a full training round no longer decreases compared to the loss of the previous full training round, it is determined that the training stop condition is reached; wherein the loss of each full training round is determined based on the average of the losses of each iteration of the full training round.

[0064] The pre-trained image processing model includes a feature extraction network and a quantization network; the server 104 can obtain image features through the feature extraction network, and input the image features into the quantization network to obtain the quantization features of the image. The feature extraction network can be a convolutional neural network (CNN); when the feature extraction network is a convolutional neural network, the convolutional neural network can adopt a structure such as resnet101, resnet18CNN, etc.; correspondingly, the image features obtained through the convolutional neural network can be called the deep features of the image. The quantization network can be cascaded using multiple fully connected layers (FC).

[0065] The input image is an image to be retrieved, and can be sent from the terminal 102 to the server 104 ; the server 104 retrieves an image matching the input image from the stock images, and the server 104 feeds back the image matching the input image as a retrieved image to the terminal 102 .

[0066] The image library is formed based on the stock images and the quantitative features of the stock images themselves. Specifically, the server 104 can use a pre-trained image processing model to extract the image features of each stock image, and use the image features of the stock images of the pre-trained image processing model to quantify, to obtain the quantitative features of each stock image, use the quantitative features of each stock image as the query index of each stock image, and store each stock image and its own quantitative features in the data storage system to form an image library. The data storage system can be integrated in the server 104, or can be set separately from the server 104.

[0067] The server 104 may also cluster the stock images according to the set number of clusters based on the quantitative features of the stock images to obtain multiple clusters, use the quantitative features of the stock images at the center of the cluster as the query index of the cluster, and establish an association relationship between the quantitative features of the stock images at the center of the cluster and the cluster. The clustering method may be kmeans clustering.

[0068] The terminal 102 may send the input image to the server 104 , and the server 104 may extract the image features of the input image using a pre-trained image processing model, and quantize the image features of the input image using the pre-trained image processing model to obtain the quantized features of the input image.

[0069] When determining the retrieval image that matches the input image, the server 104 can compare the quantitative features of the input image with the quantitative features of the stock images in the image library one by one. The comparison results include the distance between the quantitative features of the input image and the quantitative features of each stock image. The server 104 can use the stock image with a distance less than or equal to a preset distance as the retrieval image that matches the input image; the server 104 can sort the distance from small to large and feed back each retrieval image to the terminal 102 in turn.

[0070] When determining the retrieval image that matches the input image, the server 104 may also compare the quantitative features of the input image with the quantitative features that serve as query indexes for each cluster, and the comparison results obtained include the distance between the quantitative features of the input image and the quantitative features that serve as query indexes for each cluster; the server 104 determines the quantitative features that are closest to the quantitative features of the input image, or the quantitative features whose distance to the quantitative features of the input image is less than or equal to a preset distance, among the quantitative features that serve as query indexes for each cluster, and uses the cluster associated with the quantitative features as the target cluster; uses the stock image of the target cluster as the retrieval image that matches the input image; the server 104 obtains the distances between the quantitative features of the input image and each retrieval image, respectively, and determines the retrieval image that ranks before the preset number of digits in ascending order of distance, and feeds back the retrieval images that rank before the preset number of digits in ascending order of distance.

[0071] In the above embodiment, the quantitative features of the input image are obtained by a pre-trained image processing model, and the quantitative features of the input image are compared with the quantitative features of the stock image to determine the retrieval image that matches the input image, thereby improving the image retrieval efficiency; and, since the quantitative features obtained by the pre-trained image processing model have the ability to measure image semantics and image similarity, it can ensure that the input image and the retrieval image are relatively consistent in image semantics and relatively similar.

[0072] In addition, with the user's authorization, the server 104 can also recommend multiple images of interest to the user based on the quantitative features of the images obtained by the pre-trained image processing model, where the images are semantically consistent and have similar image performances.

[0073] In one embodiment, a method for processing an image processing model is provided to implement training of the image processing model; the method can be executed by the terminal 102 or the server 104, or can be executed collaboratively by the terminal 102 and the server 104. In the embodiment of the present application, the method is illustrated by taking the execution of the server 104 as an example. Figure 2 Schematic diagram of a processing method of an image processing model, the method comprising the following steps:

[0074] Step S202, obtaining a sample image set of the current iteration.

[0075] When training an image processing model, multiple similar sample image pairs can be used as a full sample image set; when using the full sample image set to perform a full training round, the server 104 can divide the multiple similar sample image pairs into multiple batches, and each batch of similar sample image pairs is used for one iteration in the full training round, and each batch includes bs similar sample image pairs.

[0076] The current iteration is an iteration of determining the noise-free sample images of each label based on the positive and negative sample thresholds of each label; the current iteration may belong to a certain iteration in the set full training round; the sample images used for the multi-label prediction loss of each iteration of each set full training round are the noise-free sample images of each label of the iteration. Among them, the full training round with a round greater than the set round can be used as the set full training round. For example, if the set round is 5, then the full training round after the 5th full training round is the set full training round, and in each iteration of the full training round after the 5th full training round, the multi-label prediction loss used in the iteration is determined based on the noise-free sample images of each label of each iteration; and in the full training round after the 5th full training round, the first set full training round is the 6th epoch, and the non-first set full training round is the 7th epoch and the full training rounds after the 7th epoch.

[0077] The sample images used for the multi-label prediction loss of each iteration of the non-set full training round are: positive and negative sample images of each label determined based on an existing or pre-trained multi-label annotation model. The non-set full training round is performed before the set full training round.

[0078] The server 104 may set one batch of similar sample image pairs among multiple batches of similar sample image pairs of the full training round as the sample image set of the current iteration.

[0079] Step S204, extracting image features of sample images in the sample image set through the image processing model updated in the previous iteration, quantizing the image features to obtain quantized features, and performing multi-label prediction based on the quantized features to obtain multi-label prediction probabilities of the sample images.

[0080] When training an image processing model, the image processing model may include a feature extraction network, a quantization network, and a multi-label prediction network; the multi-label prediction network obtains the prediction probability of the image under each label based on the quantization features of the image output by the quantization network.

[0081] When the feature extraction network is a convolutional neural network, the image feature can be called a deep feature of the image. The image feature can be a D-dimensional feature vector. The quantized feature can be a feature obtained by performing vector normalization on the image feature, and the value range of the feature after vector normalization can be a floating point number between -1 and 1.

[0082] The multi-label prediction probability of a sample image is the prediction probability of the sample image under each label. The multi-label prediction probability of a sample image is, for example, P[image] = {P 1 [image],P 2 [image],…,P m [image], …}, where P 1 [image] indicates the predicted probability that the image corresponds to label label_1, P m [image] indicates the predicted probability of the image corresponding to the label label_m. The multi-label prediction probability of the sample image in each iteration is obtained by the forward processing of the iteration, that is, the parameter output of the multi-label prediction network based on the image processing model updated in the previous iteration.

[0083] After the previous iteration is completed, the parameters of each network in the feature extraction network, quantization network and multi-label prediction network of the image processing model are updated. Therefore, after the previous iteration is completed, the image processing model can be called the image processing model updated by the previous iteration.

[0084] When performing the current iteration, for each sample image of the current iteration, the server 104 uses the feature extraction network updated in the previous iteration to obtain the image features of each sample image, quantizes the image features of each sample image to obtain the quantized features of each image, and uses the multi-label prediction network to obtain the multi-label prediction probability of each sample image based on the quantized features of each sample image.

[0085] Step S206, based on the positive and negative sample thresholds corresponding to each label and the multi-label prediction probability of the sample image updated in the previous iteration, determine the noise-free sample image corresponding to each label from the sample image set and correct the labeling information of the noise-free sample image, and determine the multi-label prediction loss of the current iteration according to the labeling information of the noise-free sample image and the multi-label prediction probability.

[0086] When the current iteration is a non-first iteration of the first set full training round or any iteration other than the first set full training round, the previous iteration of the current iteration has the corresponding updated positive and negative sample thresholds of each label.

[0087] The positive and negative sample thresholds of each label updated in the previous iteration are used to judge the noise-free positive and negative samples in the current iteration to determine the noise-free sample images of each label in the current iteration, and the prediction loss of each label in the current iteration is obtained based on the noise-free sample images of each label in the current iteration. The multi-label prediction loss includes the prediction loss of each label.

[0088] The annotation information of the sample image includes the annotation information of the sample image corresponding to each label; the annotation information corresponding to a certain label indicates whether the sample image is a positive sample image of the label or a negative sample image of the label. In this iteration, the annotation information of the sample image before correction is determined by the previous full training round; the annotation information of the sample image after correction is considered to be determined by the current full training round.

[0089] A noise-free sample image of a certain label means that the confidence that the sample image is a positive or negative sample image of the label is high; for example, if the sample image is a positive sample image of the label, then the confidence that the sample image is a positive sample image of the label is high; if the sample image is a negative sample image of the label, then the confidence that the sample image is a negative sample image of the label is high.

[0090] If the sample image image_11 used in the current iteration, in the forward processing of the current iteration, the image processing model updated by the previous iteration can obtain the multi-label prediction probability of the sample image image_11 as {P 1 [image_11],P 2 [image_11],P 3 [image_11],…,P m[image_11], …}; where P 1 [image_11], P 2 [image_11], P 3 [image_11], P m [image_11] is the predicted probability of the sample image image_11 corresponding to the labels label_1, label_2, label_3, and label_m. For label_1, the positive sample threshold of label_1 updated in the previous iteration is thr_11 - , the negative sample threshold of the label label_1 updated in the previous iteration is thr_10 - .

[0091] Server 104 can 1 [image_11] respectively with thr_11 - and thr_10 - For comparison, if P 1 [image_11] greater than or equal to thr_11 - , or P 1 [image_11] is less than or equal to thr_10 - , then it is determined that in the current iteration, the sample image P 1 [image_11] is a noise-free sample image with label label_1.

[0092] Specifically, if P 1 [image_11] greater than or equal to thr_11 - , then it is determined that in the current iteration, the sample image image_11 is a noise-free positive sample image of the label label_1. If the label value corresponding to the label label_1 in the label information of the sample image image_11 indicates that the sample image image_11 is a negative sample image of the label label_1, then the label information of the sample image image_11 corresponding to the label label_1 is corrected so that the corrected label information indicates that the sample image image_11 is a positive sample image of the label label_1; since in the current iteration, the sample image image_11 is a positive sample image of the label label_1, it is the positive sample threshold thr_11 updated by the previous iteration. - The judgment is made with a high confidence level, so the sample image image_11 can be called a noise-free positive sample image of the label label_1.

[0093] If the sample image image_11 actually belongs to the positive sample image of the label label_1, but the label information of the sample image image_11 corresponding to the label label_1 before correction cannot represent that the sample image image_11 is the positive sample image of the label label_1, then the sample image image_11 can be called an unlabeled positive sample (positive unlabel data, PU data); if the label information of the sample image image_11 corresponding to the label label_1 after correction can represent that the sample image image_11 is the positive sample image of the label label_1, then the sample image image_11 can be called a labeled positive sample (labeled positive data).

[0094] If the label information of the sample image image_11 corresponding to the label label_1 indicates that the sample image image_11 is a positive sample image of the label label_1, then the label information does not need to be corrected.

[0095] If P 1 [image_11] is less than or equal to thr_10 - , then it is determined that in the current iteration, the sample image image_11 is a noise-free negative sample image of the label label_1. If the label value corresponding to the label label_1 in the label information of the sample image image_11 indicates that the sample image image_11 is a positive sample image of the label label_1, then the label information corresponding to the label label_1 of the sample image is corrected so that the corrected label information indicates that the sample image image_11 is a negative sample image of the label label_1; since in the current iteration, the sample image image_11 is a negative sample image of the label label_1, it is the positive sample threshold thr_10 updated by the previous iteration. - The judgment is made with a high confidence level, so the sample image image_11 can be called a noise-free negative sample image of the label label_1.

[0096] If the sample image image_11 actually belongs to the negative sample image of the label label_1, but the label information of the sample image image_11 corresponding to the label label_1 before correction cannot represent that the sample image image_11 is the negative sample image of the label label_1, then the sample image image_11 can be called an unlabeled negative sample (negative unlabel data); if the label information of the sample image image_11 corresponding to the label label_1 after correction can represent that the sample image image_11 is the negative positive sample image of the label label_1, then the sample image image_11 can be called a labeled negative sample (labeled negative data).

[0097] If the label information of the sample image image_11 corresponding to the label label_1 indicates that the sample image image_11 is a negative sample image of the label label_1, then the label information does not need to be corrected.

[0098] If P 1 [image_11] is greater than thr_10 - and less than thr_11 - , it is difficult to determine whether the sample image image_11 is a positive sample image of the label label_1 or a negative sample image of the label label_1. The sample image image_11 belongs to the uncertain sample image of the label label_1. Therefore, the sample image image_11 is regarded as a noise sample image of the label label_1. Correspondingly, the annotation information of the sample image corresponding to the label label_1 can be corrected so that the corrected annotation information represents that the sample image image_11 is a noise sample image of the label label_1 and does not participate in the loss calculation of the label label_1 in the current iteration.

[0099] Similarly, for other sample images used in the current iteration, the server 104 can determine whether the other sample images in the current iteration are noise-free sample images of label label_1; in addition, for other labels, the noise-free sample images of other labels in the current iteration can be determined.

[0100] If the sample images used in the iteration include image_11, image_12, image_21, image_22, ..., image_N1, image_N2, and the labels are label_1, label_2, label_3, ..., label_m, image_11 and image_12 form a similar sample image pair, image_21 and image_22 form a similar sample image pair, and image_N1 and image_N2 form a similar sample image pair.

[0101] After obtaining the multi-label probability of each sample image in the current iteration using the image processing model updated in the previous iteration, server 104 determines the noise-free sample image of each label based on the positive and negative sample thresholds of each label updated in the previous iteration and the multi-label prediction probability of each sample image in the current iteration, and corrects the annotation information of the noise-free sample image. The corrected annotation information of the noise-free sample image of each label is shown in Table 1.

[0102] image_11 image_12 image_21 image_22 …… image_N1 image_N2 label_1 1 1 0 -1 …… 0 1 label_2 0 0 1 1 …… 1 1 label_3 -1 1 1 -1 …… -1 0 …… …… …… …… …… …… …… …… label_m 1 -1 1 1 …… -1 0 …… …… …… …… …… …… …… ……

[0103] Table 1

[0104] Among them, the annotation information of "1" indicates that the corresponding sample image is the noise-free positive sample image of the corresponding label in the current iteration, for example, the sample image image_11 is the noise-free positive sample image of the label label_1 in the current iteration; the annotation information of "0" indicates that the corresponding sample image is the noise-free negative sample image of the corresponding label in the current iteration, for example, the sample image image_21 is the noise-free negative sample image of the label label_1 in the current iteration; the annotation information of "-1" indicates that the corresponding sample image is the noisy sample image of the corresponding label in the current iteration, for example, the sample image image_11 is the noisy sample image of the label label_1 in the current iteration.

[0105] It can be seen from Table 1 that the noise-free positive sample images with label label_1 in the current iteration include sample image image_11, sample image image_12, …, sample image image_N2, and the noise-free negative sample images with label label_1 in the current iteration include sample image image_21 and sample image imageN1, among which the noise-free positive sample images with label label_1 in the current iteration and the noise-free negative sample images with label label_1 in the current iteration can be referred to as noise-free sample images with label label_1 in the current iteration.

[0106] The server 104 obtains the prediction loss of label label_1 in the current iteration based on the corrected annotation information of the noise-free sample image of label label_1 in the current iteration and the multi-label prediction probability of each noise-free sample image of label label_1 in the current iteration obtained by using the image processing model updated in the previous iteration; similarly, the server 104 can obtain the prediction losses of other labels in the current iteration, and obtain the multi-label prediction loss based on the prediction losses of each label in the current iteration.

[0107] Step S208, based on the multi-label prediction probability of the noise-free sample image, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration.

[0108] The positive and negative sample thresholds of each label updated in the current iteration are used to judge the noise-free positive and negative sample images of each label in the next iteration, so as to determine the noise-free sample images of each label in the next iteration, and obtain the prediction loss of each label in the next iteration based on the noise-free sample images of each label in the next iteration.

[0109] The noise-free sample images used to update the positive and negative sample thresholds of each label updated in the previous iteration are the noise-free sample images of each label in the current iteration, and the multi-label prediction probability of the noise-free sample images used is obtained by using the image processing model updated in the previous iteration. Combined with Table 1:

[0110] Among the noise-free sample images of each label in the current iteration, the noise-free sample images of label label_1 are sample image image_11, sample image image_12, sample image image_21, …, sample image imageN2. The server 104 can use the image processing model updated in the previous iteration to obtain the predicted probabilities of sample images image_11, sample image image_12, sample image image_21, …, sample image N2 corresponding to label label_1, and update the positive and negative sample thresholds of label label_1 updated in the previous iteration to obtain the positive and negative sample thresholds of label label_1 updated in the current iteration.

[0111] Step S210, determining similar sample images and dissimilar sample images of sample images from the sample image set, and determining the quantization loss of the current iteration based on the quantization features of the sample images, similar sample images and dissimilar sample images, and the quantization features of the sample images and the corresponding quantization targets.

[0112] The sample image set of this iteration includes multiple similar sample image pairs, such as sample image image_11 and sample image image_12, image_21 and image_22, and image_N1 and image_N2. Among them, the two sample images of the same similar sample image pair are similar, such as sample image image_11 and sample image image_12 are similar.

[0113] The server 104 may sequentially use each similar sample image pair used in the current iteration as a target similar sample image pair, and determine a sample image from other similar sample image pairs except the target similar sample image in the multiple similar sample image pairs used in the current iteration as a sample image that is dissimilar to the two sample images of the target similar sample image pair; illustratively, if the sample image image_11 and the sample image image_12 used in the current iteration are the target similar sample image pair, then the server 104 may determine a sample image from other similar sample image pairs (such as image_21 and image_22, image_N1 and image_N2), and use the sample image as the sample image that is dissimilar to the two sample images of the target similar sample image pair. A sample image that is dissimilar to sample image image_11 and sample image image_12; if sample image image_22 is taken as a sample image that is dissimilar to sample image image_11 and sample image image_12, then one of the sample images image_11 and sample image image_12 can be taken as an anchor sample image (anchor, abbreviated as a), the other sample image can be taken as a similar sample image (positive, abbreviated as p), and sample image image_22 can be taken as a non-similar sample image (negative, abbreviated as n), forming a triplet apn corresponding to sample image image_11 and sample image image_12.

[0114] Among them, among the sample image, the similar sample image of the sample image, and the non-similar sample image of the sample image, the sample image can be regarded as an anchor sample image, the similar sample image of the sample image can be regarded as a similar positive sample image, and the non-similar sample image of the sample image can be regarded as a non-similar sample image; the quantitative features of the anchor sample image, the similar positive sample image and the non-similar sample image are obtained using the image processing model updated in the previous iteration.

[0115] The quantization loss determined based on the quantization features of the sample image, the similar sample image and the non-similar sample image can be called a triplet quantization loss. The triplet quantization loss includes the loss used when the image processing model performs metric learning, so that the distance between similar sample images characterized by the quantization features of the anchor point sample image and the quantization features of the similar positive sample image is as large as possible, and the distance between non-similar sample images characterized by the quantization features of the anchor point sample image and the quantization features of the non-similar sample image is as small as possible.

[0116] The quantization target corresponding to the quantization feature of the sample image is to input the quantization feature of the sample image into a preset sign function to obtain the sign of the quantization feature of the sample image; for example, input each quantization feature value of the quantization feature Q of the sample image into a preset sign function to obtain the quantization target value corresponding to the quantization feature value of different bits, and the quantization target value corresponding to the quantization feature value of different bits forms the quantization target of the quantization feature. That is, the i-th quantization feature value Q of the quantization feature Q of the sample image is input into the preset sign function to obtain the quantization target value corresponding to the quantization feature value of different bits. i Input the preset symbol function and get the value corresponding to Q i The quantitative target value B i ; The preset symbol function can be a sign function, and the corresponding formula is

[0117] The quantization loss determined based on the quantization features of the sample image and the corresponding quantization target can be called image quantization loss. The image quantization loss belongs to the loss used when the image processing model performs quantization learning to make each bit of the quantization feature close enough to 1 or -1 of the preset sign function.

[0118] If the quantized feature of the sample image is a 256-dimensional feature vector, which includes a 256-bit quantized feature value, then the image quantization loss of the sample image is The image quantization loss is calculated using a regression loss function, and the image quantization loss is used to constrain the direction of parameter adjustment of the image quantization model so that the distance between the quantization features of the sample image and the quantization target gradually decreases.

[0119] After server 104 determines at least one corresponding triplet for each similar sample image pair used in the current iteration, it determines the quantization loss of each triplet. If the number of triplets corresponding to the same similar sample image pair is greater than or equal to 2, the triplet quantization loss of the same similar sample image pair is obtained based on the average of the multiple triplet quantization losses corresponding to the same similar sample image pair. In addition, server 104 obtains the image quantization loss between the quantization features of each sample image used in the current iteration and the corresponding quantization target, sums the triplet quantization loss and the image quantization loss of each similar sample image pair, and obtains the quantization loss used in the current iteration.

[0120] Step S212: Based on the multi-label prediction loss and the quantization loss, the image processing model updated in the previous iteration is updated to obtain the image processing model updated in the current iteration.

[0121] Updating the image processing model is to update the parameters of each network of the image processing model; in one iteration, based on the loss determined by the forward processing process of the iteration, the parameters of each network of the image processing model are updated so that the parameters of each network are adjusted in the direction of minimum loss. Among them, the forward processing process of one iteration is such as the process described in step S204; the process of updating the parameters of each network of the image processing model based on the loss determined by the forward processing process of the iteration can be called the backward processing process of the iteration.

[0122] The above-mentioned multi-label prediction loss and quantization loss are losses determined based on the forward process of the current iteration. The server 104 uses the loss to update the parameters of each network in the image processing model updated in the previous iteration, and uses the image processing model including the updated parameters of each network as the image processing model updated in the current iteration.

[0123] Specifically, when the sample image set used in the iteration is bs similar sample image pairs and each similar sample image pair has 10 triplets, the multi-label prediction loss of each similar sample image pair can be obtained: and the quantization loss of each similar sample image pair Get the loss of each similar sample image pair In this formula, j represents the two sample images constituting the similar sample image pair, th represents the triplet corresponding to the similar sample image pair, w1 can be set to 0.5, and w2 can be set to 1-w1. Next, the average of the losses of each similar sample image pair is used as the total loss used in the current iteration.

[0124] The multi-label prediction loss of each similar sample image pair is obtained based on the mean of the multi-label prediction losses of the two sample images constituting the similar sample image pair and the corresponding weight coefficient w1. The multi-label prediction loss of each sample image is the sum of the prediction losses corresponding to each label of each sample image.

[0125] The prediction loss of each sample image corresponding to each label is obtained based on the corrected label information corresponding to the label of the sample image and the predicted probability corresponding to the label obtained by the sample image in the current iteration. Taking the prediction loss of sample image image_11 corresponding to label label_m as an example, let the corrected label information of sample image image_11 corresponding to label label_m be tm [image_11], the predicted probability of the sample image image_11 corresponding to the label label_m is P m [image_11], then the prediction loss of the sample image image_11 corresponding to the label label_m is

[0126] L label_m =t m [image 11 ]×log(P m [image 11 ])+(1-t m [image 11 ])×log(1-P m [image 11 ])), if t m [image_11]≠-1.

[0127] The quantization loss of each similar sample image pair is obtained based on the mean of the loss of the corresponding triplet of the similar sample image pair and the corresponding weight coefficient w2; the loss of each triplet L 2 is the quantization loss L of the triple triplet And the image quantization loss L of each sample image constituting the triplet coding , that is, L 2 =w 21 L triplet +w 22 L coding ;w 21 is the weight coefficient assigned to the triplet quantization loss, which can be set to 1; w 22 is the weight coefficient assigned to the image quantization loss, which can be set to 0.5; and w 21 Greater than w 22 It can ensure that when the regression loss learning converges faster than the triplet metric learning, the triplet metric learning is dominant in the overall learning, thus ensuring that the deep features output by the convolutional neural network always have the ability to measure similarity. triplet And the image quantization loss L of each sample image constituting the triplet coding Introduced below.

[0128] If the current iteration is the last iteration of the training process of the image processing model, then after the iteration is completed, no further iteration is required to complete the training of the image processing model.

[0129] If the current iteration is not the last iteration of the image processing model training process, then the sample image set for the next iteration can be obtained, and the image processing model updated in the current iteration can be iteratively updated in accordance with steps S204 to S212 to obtain the image processing model updated in the next iteration until the last iteration of the training process is completed.

[0130] Among them, when the current iteration is not the last iteration of the image processing model training process, if the next iteration and the current iteration belong to the same full training round, that is, the next iteration and the current iteration belong to the same epoch; correspondingly, the method of obtaining the sample image set used for the next iteration can be: using the similar sample image pairs of the corresponding batches used in the iterations before the current iteration and the similar sample image pairs used in the current iteration as the similar sample image pairs of the used batches of the current full training round: among the similar sample image pairs of multiple batches of the current full training round, the similar sample image pairs of the used batches of the current full training round are eliminated to obtain the similar sample image pairs of the unused batches of the current full training round, and one of the similar sample image pairs of the unused batches of the current full training round is selected as the sample image set used for the next iteration.

[0131] If the next iteration and the current iteration do not belong to the same full training round, that is, the next iteration belongs to the iteration of the next epoch, then the similar sample image pairs of each batch obtained from the full sample image set have not been used in the next full training round. At this time, one of the similar sample image pairs of each batch obtained from the full sample image set can be selected as the sample image set used for the next iteration.

[0132] It can be understood that the method for acquiring the sample image set used in the next iteration described above is applicable to acquiring the sample image set used in each iteration of the training process of the image processing model.

[0133] In the processing method of the above-mentioned image processing model, the noise-free sample images of each label are determined from the sample image set through the positive and negative sample thresholds corresponding to each label updated in the previous iteration, and the multi-label prediction probability of the sample image obtained based on the image processing model updated in the previous iteration, so as to realize the judgment of the noise-free sample images corresponding to each label in the current iteration, so as to remove the noise sample images in the calculation of the multi-label prediction loss of the current iteration, so as to avoid the overfitting of the parameters used for image quantization processing in the image processing model to the noise sample images, and realize the effective training of the image processing model in a large number of noise sample images; and, through the multi-label prediction probability of the noise-free sample images of the current iteration, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated, and the positive and negative sample thresholds corresponding to each label updated in the current iteration are obtained. This threshold is used as the threshold for judging the noise-free sample image corresponding to each label in the next iteration, so as to realize the dynamic update of the positive and negative sample thresholds of each label used in different iterations, and further improve the effectiveness of training the image processing model in a large number of noisy sample images; in addition, according to the quantitative features of the sample image, similar sample images and non-similar sample images, as well as the quantitative features of the sample image and the corresponding quantitative targets, the quantitative loss of the current iteration is obtained, and the image processing model updated in the previous iteration is updated through the multi-label prediction loss and the quantization loss, so that in the current iteration, the parameters in the image processing model are adjusted in the direction of minimizing the multi-label loss and the quantization loss, so that the quantitative features of the image obtained based on the image processing model after the training have the ability to measure semantic relevance and the degree of similarity.

[0134] In one embodiment, the server 104 may also take each label in the multiple labels as the target label in turn; according to the noise-free positive sample image of the target label in the current iteration and the mean value of the predicted probability of the corresponding target label, the positive sample threshold of the corresponding target label updated in the previous iteration is updated to obtain the positive sample threshold of the corresponding target label updated in the current iteration; according to the noise-free negative sample image of the target label in the current iteration and the mean value of the predicted probability of the corresponding target label, the negative sample threshold of the corresponding target label updated in the previous iteration is updated to obtain the negative sample threshold of the corresponding target label updated in the current iteration.

[0135] Take the target label as label_m as an example, and combine it with Table 1 to introduce:

[0136] In this iteration, the noise-free positive sample images of label label_m include sample images image_11, sample images image_12, ..., sample images image_N2. The server 104 can obtain the mean of the predicted probabilities of sample images image_11, sample images image_12, ..., sample images image_N2 corresponding to label label_m, and use the mean to update the positive sample threshold thr_m1 of label label_m obtained in the previous iteration. - Make corrections and use the correction result as the positive sample threshold thr_m1 of the label label_m updated in the current iteration + ; Among them, the predicted probability of each sample image corresponding to the label label_m in the current iteration is obtained by the forward processing process of the current iteration. Since the forward processing process of the current iteration is based on the image processing model updated in the previous iteration, the predicted probability of each sample image corresponding to the label label_m in the current iteration can also be regarded as obtained by the image processing model updated in the previous iteration.

[0137] In this iteration, the noise-free negative sample images of label label_1 include sample image image_21 and sample image image_N1. The server 104 can obtain the mean of the predicted probabilities of sample image_21 and sample image_N1 corresponding to label label_1, and use the mean to update the negative sample threshold thr_10 of label label_1 obtained in the previous iteration. - Make corrections and use the correction result as the negative sample threshold thr_10 of the label label_1 updated in the current iteration + ; Among them, the predicted probability of each sample image corresponding to the label label_1 in the current iteration is obtained through the image processing model updated in the previous iteration.

[0138] In the above embodiment, the positive and negative sample thresholds of each label updated in the previous iteration are updated according to the mean of the predicted probabilities of the noise-free sample images of each label in the current iteration corresponding to the label, so as to realize the dynamic update of the positive and negative sample thresholds of each label, and gradually add the noise-free positive and negative sample images with higher confidence to calculate the multi-label prediction loss in each iteration, so that the image processing model can learn more effectively based on the more effective multi-label prediction loss. In addition, the positive sample threshold and negative sample threshold of each label updated in the previous iteration are updated respectively by using the mean of the predicted probabilities of the noise-free positive sample images of each label in the current iteration corresponding to the label and the mean of the predicted probabilities of the noise-free negative sample images of each label in the current iteration corresponding to the label, so as to improve the effectiveness of the positive and negative sample thresholds of each label in determining the noise-free sample images.

[0139] In one embodiment, the server 104 may also perform a weighted summation of the positive sample threshold of the corresponding target label updated in the previous iteration and the mean of the predicted probability of the target label of the noise-free positive sample image corresponding to the target label in the current iteration to obtain the positive sample threshold of the corresponding target label updated in the current iteration.

[0140] Among them, the weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean value of the predicted probability of the target label corresponding to the noise-free positive sample image in the current iteration. The weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration can be 0.9, and the mean value of the predicted probability of the target label corresponding to the noise-free positive sample image in the current iteration can be 0.1.

[0141] The target label is label_m, and the positive sample threshold of label_m updated in the previous iteration is recorded as thr_m1 - Introduction: In the noise-free positive sample images of label label_m in the iteration, the predicted probability of each noise-free positive sample image corresponding to label label_m is greater than or equal to thr_m1 - , the server 104 can obtain the average value of the predicted probability (the predicted probability is obtained in the forward processing of the iteration) of the noise-free positive sample image of the label label_m corresponding to the label label_m in the iteration Among them, j is the noise-free positive sample image of label label_m in the current iteration, P m [j] is the predicted probability that the noise-free positive sample image with label label_m in the current iteration corresponds to label label_m. The server 104 calculates thr_m1 according to the above weight coefficients 0.9 and 0.1. - The weighted sum of the label label_m and average is obtained to obtain the positive sample threshold thr_m1 of the label label_m updated in the current iteration. + =0.9×thr_m1 - +0.1×average_1.

[0142] In the above embodiment, the weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean value of the predicted probability of the noise-free positive sample image of the target label corresponding to the target label in the current iteration. When weighted summation is performed, the positive sample threshold of the target label updated in the previous iteration is mainly used, and the mean value of the predicted probability of the noise-free positive sample image of the target label corresponding to the target label in the current iteration is used as a supplement to update the positive sample threshold, so that when the positive sample image of each label is judged in the next iteration, a positive sample image of each label with a higher confidence level can be obtained.

[0143] In one embodiment, the server 104 may also perform a weighted summation of the negative sample threshold of the corresponding target label updated in the previous iteration and the mean of the predicted probability of the target label corresponding to the noise-free negative sample image of the target label in the current iteration to obtain the negative sample threshold of the corresponding target label updated in the current iteration.

[0144] Among them, the weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean value of the predicted probability of the target label corresponding to the noise-free negative sample image in the current iteration. The weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration may be 0.9, and the mean value of the predicted probability of the target label corresponding to the noise-free negative sample image in the current iteration may be 0.1.

[0145] The target label is label_m, and the negative sample threshold of label_m updated in the previous iteration is recorded as thr_m0 - Introduction: In the noise-free negative sample images of label label_m in the iteration, the predicted probability of each noise-free negative sample image corresponding to label label_m is less than or equal to thr_m0 - , the server 104 can obtain the mean value of the predicted probability of the noise-free negative sample image with label label_m corresponding to the label label_m in the current iteration (the predicted probability is obtained in the forward processing process of the current iteration) Among them, j is the noise-free negative sample image of label label_m in the current iteration, P m [j] is the predicted probability that the noise-free negative sample image with label label_m in the current iteration corresponds to label label_m. The server 104 calculates thr_m0 according to the above weight coefficients 0.9 and 0.1. - The weighted sum of the label label_m and average is obtained to obtain the negative sample threshold thr_m0 of the label label_m updated in the current iteration. + =0.9×thr_m0 - +0.1×average_0.

[0146] In the above embodiment, the weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean value of the predicted probability of the noise-free negative sample image of the target label corresponding to the target label in the current iteration. When weighted summation is performed, the negative sample threshold of the target label updated in the previous iteration is mainly used, and the mean value of the predicted probability of the noise-free negative sample image of the target label corresponding to the target label in the current iteration is used as a supplement to update the negative sample threshold, so that when the negative sample image of each label is judged in the next iteration, a negative sample image of each label with a higher confidence level can be obtained.

[0147] In one embodiment, the server 104 may also take each label in the multiple labels as the target label in turn; obtain the positive sample threshold and the negative sample threshold of the corresponding target label updated in the previous iteration; when the prediction probability of the corresponding target label in the multi-label prediction probability of the sample image is greater than or equal to the positive sample threshold of the target label, the sample image is determined to be a noise-free positive sample image of the corresponding target label in the current iteration, and the labeling information of the sample image corresponding to the target label is corrected to the first value; when the prediction probability of the corresponding target label in the multi-label prediction probability of the sample image is less than or equal to the negative sample threshold of the target label, the sample image is determined to be a noise-free negative sample image of the corresponding target label in the current iteration, and the labeling information of the sample image corresponding to the target label is corrected to the second value.

[0148] Among them, when the sample image is a noise-free positive sample image of the target label of the current iteration, the label information of the sample image corresponding to the target label is corrected to a first value, indicating that the sample image can participate in the loss calculation of the target label, and the loss of the sample image corresponding to the target label is the loss between the predicted probability of the sample image determined by the forward processing process of the current iteration and the first value; the first value can be 1.

[0149] Among them, when the sample image is a noise-free negative sample image of the target label of the current iteration, the label information of the sample image corresponding to the target label is corrected to a second value, indicating that the sample image can participate in the loss calculation of the target label, and the loss of the sample image corresponding to the target label is the loss between the predicted probability of the sample image determined by the forward processing process of the current iteration and the second value; the second value can be 0.

[0150] The target label is label_1 and the sample image is sample image_11.

[0151] During the forward processing of this iteration, the predicted probability of the sample image image_11 corresponding to the label label_1 is P 1 [image_11], if the P 1 [image_11] is greater than or equal to the positive sample threshold thr_11 of label label_1 updated in the previous iteration - , then the label information of the sample image image_11 corresponding to the label label_1 in the current iteration is corrected to the first value, such as 1; the P 1 [image_11] is less than or equal to the negative sample threshold thr_10 of the label label_1 updated in the previous iteration - , then the label information of the sample image image_11 corresponding to the label label_1 in the current iteration is corrected to the second value, such as 0.

[0152] Similarly, the annotation information of other labels and other sample images used in the current iteration is also corrected.

[0153] In the above embodiment, based on the positive sample threshold and negative sample threshold of the target label updated in the previous iteration, and the multi-label prediction probability of the sample image, the annotation information of the target label corresponding to the sample image is corrected to avoid the noise sample image from participating in the subsequent target label loss calculation, thereby improving the training effectiveness of the image processing model.

[0154] In one embodiment, the server 104 may also take each label in the multi-label as the target label in turn; for each noise-free sample image corresponding to the target label, the prediction loss of the target label is calculated according to the annotation information of the corrected noise-free sample image and the prediction probability of the corresponding target label in the multi-label prediction probability of the noise-free sample image; the prediction losses corresponding to each label are summed to obtain the multi-label prediction loss of the current iteration.

[0155] The target label is label_m, and the corrected label information of the noise-free sample image j corresponding to the label label_m of the current iteration is recorded as t m [j], the predicted probability of the noise-free sample image j corresponding to the label label_m obtained by the forward processing of the current iteration is recorded as P m [j], to introduce:

[0156] In the forward processing of the current iteration, the server 104 obtains the predicted probability of each sample image under the label label_m of the bs similar sample image pairs used in the current iteration, and calculates the predicted probability P of each sample image under the label label_m. m [j] The positive sample threshold thr_m1 of the label label_m updated in the previous iteration - And negative sample threshold thr_m0 - If the predicted probability P of sample image j under label label_m is m [j] greater than or equal to thr_m1 - , then the sample image is determined to be a noise-free positive sample image with label label_m in the current iteration, and the label information t corresponding to the sample image label label_m is m [j] is corrected to 1; if the predicted probability P of sample image j under label label_m m [j] less than or equal to thr_m0 - , then the sample image is determined to be a noise-free negative sample image of the label label_m in the current iteration, and the label information of the sample image corresponding to the label label_m is corrected by t m[j] is 0; if the predicted probability P of sample image j under label label_m m [j] greater than thr_m0 - and less than thr_m1 - , then the sample image is determined to be a noise sample image with label label_m in the current iteration, and the label information t corresponding to the sample image label label_m is m [j] is corrected to -1 and does not participate in the calculation of the prediction loss of label label_m in the current iteration. In the current iteration, the prediction loss calculation formula of label label_m is:

[0157] In this formula, sample image j is: the noise-free sample image with label label_m among the 2×bs sample images used in the iteration; N 2 It indicates the number of noise-free sample images with label label_m in the current iteration among the 2×bs sample images used in the current iteration.

[0158] The server 104 calculates the prediction loss of each label in the current iteration in the above manner, sums the prediction loss of each label in the current iteration, and obtains the multi-label prediction loss of the current iteration.

[0159] In the above embodiment, for each noise-free sample image of the target label in the current iteration, the prediction loss of the target label is obtained according to the annotation information of the corrected noise-free sample image and the predicted probability of the target label corresponding to the noise-free sample image. The noise sample image does not participate in the calculation of the prediction loss of the target label of the current iteration, thereby avoiding overfitting of the image processing model to the noise sample image in the current iteration and achieving the effectiveness of model training in a large number of noise sample images.

[0160] In one embodiment, the sample image set includes a pair of similar sample images, and the two sample images included in the similar sample image pair are similar. For each similar sample image pair in the sample image set, the server 104 may also generate a triple corresponding to the similar sample image pair; the triple includes a target sample image, a similar sample image of the target sample image, and a non-similar sample image of the target sample image; for each triple, according to the difference in quantization features between the target sample image and the similar sample image, and the difference between the quantization features of the target sample image and the non-similar sample image, a triple quantization loss is calculated; according to the difference between the quantization features of each sample image in the sample image set and the corresponding quantization target, an image quantization loss is calculated; and a weighted sum is performed on the triple quantization loss and the image quantization loss to obtain the quantization loss of the current iteration.

[0161] Among them, the target sample image included in the triplet can be called an anchor sample image and is denoted as a, the similar sample image of the target sample image can be called a similar positive sample image and is denoted as p, and the dissimilar sample image of the target sample image can be called a dissimilar sample image and is denoted as n.

[0162] The difference between the quantitative features of the target sample image and the quantitative features of the similar sample images, and the difference between the quantitative features of the target sample image and the quantitative features of the non-similar sample images can be represented by distance.

[0163] The server 104 generates a corresponding triplet for each of the bs similar sample image pairs used in the current iteration; for a, p and n of the same triplet, the server 104 can determine the distance between a and p in the quantization space and the distance between a and n in the quantization space according to the quantization features of a, p and n, and obtain the quantization loss of the triplet as L triplet =max(‖Q a -Q p ‖-‖Q a -Q n ‖+α,0); in this formula, α is the difference between distances (margin), and α can be set to 160. The purpose of this triple quantization loss is to make the distance between a and n in the quantization space larger than the distance between a and p in the quantization space by α.

[0164] For the bs similar sample image pairs used in the current iteration, if each similar sample image pair has at least two corresponding triplets, the server 104 can obtain the triplet quantization loss of the similar sample image pair based on the mean of the triplet quantization loss corresponding to the same similar sample image pair, and obtain the triplet quantization loss of each similar sample image pair. For the 2×bs sample images used in the current iteration, the server 104 can obtain the difference between the quantization feature of each sample image and the corresponding quantization target, and obtain the image quantization loss of each sample image; and use the triplet quantization loss of each similar sample image pair and the image quantization loss of each sample image as the quantization loss used in the current iteration.

[0165] In the above embodiment, based on the triplet quantization loss of each similar sample image pair used in the iteration and the image quantization loss of each sample image used in the iteration, the quantization loss of the iteration is obtained, so that the image processing model can perform quantitative learning while performing metric learning.

[0166] In one embodiment, the server 104 may also use two sample images in the similar sample image pair as a target sample image and similar sample images of the target sample image; determine at least one sample image that is dissimilar to the target sample image from the sample image set based on the image features extracted this time as a non-similar sample image of the target sample image; and form at least one triplet of the target sample image, the similar sample image of the target sample image, and the non-similar sample image of the target sample image.

[0167] The image features extracted this time are features extracted by the feature extraction network of the image processing model during the forward processing of the current iteration, and may be deep features of the image.

[0168] After obtaining bs similar sample image pairs used in the current iteration, the server 104 obtains the image features of each sample image formed in the forward processing process of the current iteration; takes the sample images included in other sample image pairs to which the similar sample image to which the target sample image belongs as candidate sample images, obtains the image features of the candidate sample images, and obtains the distance between the image features of the target sample image and the image features of the candidate sample images; determines the non-similar sample images of the target sample images in the candidate sample images according to the size of the distance; and forms at least one triple with the target sample image, the similar sample images of the target sample image, and the non-similar sample images of the target sample image, to obtain the triples shown in Table 2.

[0169] a p n image_11 image_12 image_21 image_11 image_12 image_32 image_11 image_12 …… image_11 image_12 image_N1 image_21 image_22 image_12 image_21 image_22 image_31 image_21 image_22 …… image_21 image_22 image_N2 …… …… ……

[0170] Table 2

[0171] In the above embodiment, the triples of each similar sample image pair are mined from the similar sample image pairs used in the current iteration, without the need to find non-similar sample images of the triples, thereby improving the efficiency of triple mining.

[0172] In one embodiment, the server 104 may also obtain storyboards of the video; use images of the same storyboard as similar sample images, and obtain multiple similar sample image pairs corresponding to the same storyboard.

[0173] Figure 3 This is the first storyboard diagram. Figure 4 is a schematic diagram of the second storyboard. The same storyboard may include multiple images, such as Figure 3 The first storyboard includes 4 images, Figure 4 The second split mirror includes 3 images. Images of the same split mirror can be used as similar sample images, and multiple similar sample image pairs can be obtained from the same split mirror.

[0174] When multiple similar sample image pairs are obtained from the same split screen, the server 104 can also obtain the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time; sort the candidate sample images in ascending order of distance based on the distances between the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time; determine the candidate sample images ranked after a set number of places, and determine sample images that are dissimilar to the target sample image from the candidate sample images ranked after the set number of places as non-similar sample images of the target sample image.

[0175] The candidate sample images are sample images of similar sample image pairs other than the similar sample image pair to which the target sample image belongs in the sample image set. The distances between the image features extracted at the time of the target sample image and the image features extracted at the time of each candidate sample image can be Euclidean distances. The set number of digits can be determined according to the number of similar sample image pairs used in the current iteration and the ranking percentage K%. Specifically, the set number of digits is determined according to (2×bs-2)×K / 100. For example, if bs is 64 and K is 3, then the set number of digits is 4.

[0176] Take the target sample image as sample image_11 as an example:

[0177] When the target sample image is sample image_11, the candidate sample images are other sample images except sample image_11 and sample image_12 among the sample images used in the current iteration, such as sample image_21, sample image_22, and sample image_N1. The server 104 may obtain the image features extracted at the current time of sample image image_11 and the image features extracted at the current time of each candidate sample image, and sort the candidate sample images in ascending order of distance according to the distance between the image features extracted at the current time of sample image image_11 and the image features extracted at the current time of each candidate sample image, remove the candidate sample images ranked before the set number of digits, and determine the sample images that are not similar to the sample image image_11 from the candidate sample images ranked after the set number of digits as the non-similar sample images of the sample image_11.

[0178] In the above embodiment, when multiple similar sample image pairs are obtained from the same split screen, sample images that are not similar to the target sample image are determined from the candidate sample images ranked after the set number of bits in the order of distance from small to large, so as to avoid selecting sample images of the same split screen as the target sample image when the similar sample image pairs used in the next iteration include at least two similar sample image pairs of the same split screen, thereby reducing the noise of the triplet quantization loss.

[0179] In one embodiment, the server 104 may also input the quantization features of the sample image into a preset sign function to obtain the sign of the quantization features of the sample image as the quantization target of the sample image; and calculate the image quantization loss of the sample image based on the difference between the quantization features of the sample image and the corresponding sign.

[0180] The preset sign function may be a sign function. The quantization target corresponding to the quantized feature of the sample image is the sign of the quantized feature of the sample image obtained by inputting the quantized feature of the sample image into the preset sign function.

[0181] The server 104 can input each quantization feature value of the quantization feature Q of the sample image into a preset sign function to obtain the quantization target values ​​corresponding to the quantization feature values ​​of different bits, and the quantization target values ​​corresponding to the quantization feature values ​​of different bits form the quantization target of the quantization feature. That is, the i-th quantization feature value Q of the quantization feature Q of the sample image is i Input the sign function and get the value corresponding to Q i The quantitative target value B i ;The formula corresponding to the sign function is

[0182] The quantization loss determined based on the quantization features of the sample image and the corresponding quantization target can be called image quantization loss. The image quantization loss belongs to the loss used when the image processing model performs quantization learning to make each bit of the quantization feature close enough to 1 or -1 of the preset sign function.

[0183] If the quantized feature of the sample image is a 256-dimensional feature vector, which includes a 256-bit quantized feature value, then the image quantization loss of the sample image is The image quantization loss is calculated using a regression loss function, and the image quantization loss is used to constrain the direction of parameter adjustment of the image quantization model so that the distance between the quantization features of the sample image and the quantization target gradually decreases.

[0184] In the above embodiment, the quantized features of the sample image are input into a preset sign function to obtain the sign of the quantized features of the sample image as the quantization target of the sample image, and the image quantization loss of the sample image is obtained to realize the quantitative learning of the image processing model.

[0185] In one embodiment, the image processing model includes a quantization network and a multi-label prediction network; the server 104 can also perform a weighted summation of the multi-label prediction loss and the quantization loss to obtain the total loss of the current iteration; based on the total loss of the current iteration, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in turn to obtain the image processing model updated in the current iteration.

[0186] Among them, the gradient direction of the total loss of the current iteration is from the multi-label prediction network to the quantization network.

[0187] When the sample image set used in the iteration is bs similar sample image pairs, the server 104 can obtain the average of the multi-label prediction losses of bs similar sample image pairs, and the average result is recorded as loss 多标签 ; The server 104 obtains the average of the quantization losses of bs similar sample image pairs, and the average result is recorded as loss 量化 , the total loss of the iteration is obtained = loss 多标签 +loss 量化 ; When using the total loss of the iteration for gradient backward, the loss in the total loss of the iteration 多标签 After being transmitted back to the quantization network through the multi-label prediction network, the loss in the total loss of the iteration 量化 The server 104 directly returns to the quantization network. The server 104 updates the parameters of the multi-label prediction network according to the loss returned to the multi-label prediction network, and updates the parameters of the quantization network according to the loss returned to the quantization network, thereby obtaining the updated image processing model for the current iteration.

[0188] In the above embodiment, according to the total loss of the iteration obtained by weighted summing of the multi-label prediction loss and the quantization loss, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in turn to realize metric learning and semantic learning of the image processing model; and, since metric learning converges slowly (it is more difficult to find similar and dissimilar parts on two images than to find commonalities of the same category in multiple images of the same category), adding multi-label prediction loss for semantic learning can accelerate the convergence of metric learning and avoid missed recall of semantically related images due to improper local representation in metric learning.

[0189] In one embodiment, the learning rate for the quantization network is smaller than the learning rate for the multi-label prediction network, which can avoid the multi-label prediction loss being completely transferred to the quantization network, causing semantic learning to excessively affect the measurement effect of the quantization network, and avoid improper similarity sorting during image retrieval.

[0190] Specifically, the learning rate for the quantization network can be 0.1 times the learning rate for the multi-label prediction network to avoid overfitting of quantization features due to too fast convergence of semantic learning.

[0191] In one embodiment, each iteration is any iteration in a full training round, specifically, each iteration is any iteration in each set full training round; the sample image set used in each iteration is a subset of the full sample image set.

[0192] Among them, the sample images used for the multi-label prediction loss of each iteration of each set full training round are: the noise-free sample images of each label of this iteration. Among them, a full training round with a round greater than the set round can be used as the set full training round. For example, if the round is set to 5, then the full training round after the 5th full training round is the set full training round, and in each iteration of the full training round after the 5th full training round, the multi-label prediction loss used in this iteration is determined based on the noise-free sample images of each label of each iteration. The sample images used for the multi-label prediction loss of each iteration of the non-set full training round are: the positive and negative sample images of each label determined based on an existing or pre-trained multi-label annotation model.

[0193] In the first set full training round of multiple set full training rounds, if the current iteration is the first iteration of the first set full training round, then the current iteration is the first iteration to determine the noise-free sample image based on the positive and negative sample thresholds of each label. At this time, the positive and negative sample thresholds used in the current iteration are determined as follows: Figure 5 The full training round to which the current iteration belongs is called the current full training round. For the full training round before the current full training round, the server 104 may perform the following steps: Step S502, when the current iteration is the first iteration in the full training round for determining the noise-free sample image based on the positive and negative sample thresholds of each label, obtain the original annotation information of each sample image in the full sample image set, and the multi-label prediction probability of each sample image after completing the previous full training round; Step S504, take each label in the multi-label as the target label in turn; Step S50 6. For each target label, determine the positive sample image and the negative sample image of the target label according to the original annotation information; step S508, initialize the positive sample threshold of the target label required for the current iteration according to the mean value of the prediction probability of the target label in the multi-label prediction probability of the positive sample image of the target label after completing the previous full training round; step S510, initialize the negative sample threshold of the target label required for the current iteration according to the mean value of the prediction probability of the target label in the multi-label prediction probability of the negative sample image of the target label after completing the previous full training round.

[0194] The first full training round that determines the noise-free sample image based on the positive and negative sample thresholds of each label can be called the first set full training round. In each iteration of the first set full training round, the noise-free sample image is determined based on the positive and negative sample thresholds of each label; the full training round before the first set full training round is a non-set full training round. The sample images used for the multi-label prediction loss of each iteration of the non-set full training round are positive and negative sample images of each label determined based on an existing or pre-trained multi-label annotation model.

[0195] Among them, the original annotation information of each sample image in the full sample image set can be determined based on an existing or pre-trained multi-label annotation model or manually annotated; the original annotation information of each sample image includes the original annotation information of each sample image corresponding to each label, which is used to characterize whether the sample image is a positive sample image or a negative sample image of the corresponding label. When the non-set full training round performs multi-label loss calculation, it is determined based on the original annotation information of the sample image and the multi-label prediction probability of the sample image obtained by the non-set full training round.

[0196] Exemplarily, if the number of rounds is set to 5, then the first full training round is set to be the 6th epoch, and the first iteration of the first full training round is the first iteration of determining the noise-free sample image based on the positive and negative sample thresholds of each label; correspondingly, the full training round before the first full training round is the 5th epoch.

[0197] The server 104 can obtain the multi-label prediction probability of each sample image after the completion of the 5th epoch; the server 104 takes each label of the multi-label as the target label in turn, and determines the positive sample image and negative sample image of the target label according to the pre-trained multi-label annotation model, obtains the average of the prediction probabilities of each positive sample image of the target label obtained in the 5th epoch, and uses it as the positive sample threshold thr_11 used in the first iteration process of the 6th epoch to determine the noise-free positive sample image of the target label; the server 104 can also obtain the average of the prediction probabilities of each negative sample image of the target label obtained in the 5th epoch, and uses it as the negative sample threshold thr_10 used in the first iteration process of the 6th epoch to determine the noise-free negative sample image of the target label.

[0198] In the above embodiment, when the current iteration is the first iteration to determine the noise-free sample image based on the positive and negative sample thresholds of each label, for the target label in the multiple labels, the positive and negative sample thresholds used in the current iteration are obtained by using the predicted probability corresponding to the target label in the previous full training round to achieve the judgment of the noise-free sample image of the current iteration.

[0199] Figure 6 The flowchart of a processing method of an image processing model in implementation is shown, wherein the current iteration is a non-first iteration of the first set full training round or any iteration of the non-first set full training round; the processing method of the image processing model can be executed by the terminal 102 or the server 104, or can be executed by the terminal 102 and the server 104 in collaboration. In the embodiment of the present application, the method is executed by the server 104 as an example. When performing the current iteration, the steps executed by the server 104 specifically include:

[0200] Step S602, obtaining a plurality of similar sample image pairs used in the current iteration;

[0201] Step S604, extracting image features of each sample image of multiple similar sample image pairs used in the current iteration through the image processing model updated in the previous iteration, quantizing the image features to obtain quantized features, and performing multi-label prediction based on the quantized features to obtain multi-label prediction probabilities of the sample images;

[0202] Step S606, taking each tag in the multiple tags as a target tag in turn;

[0203] Step S608, obtaining the positive sample threshold and the negative sample threshold of the corresponding target label updated in the previous iteration;

[0204] Step S610, when the prediction probability of the corresponding target label in the multi-label prediction probability of the sample image is greater than or equal to the positive sample threshold of the target label, the sample image is determined to be a noise-free positive sample image of the corresponding target label in the current iteration, and the label information of the sample image corresponding to the target label is corrected to the first value;

[0205] Step S612, when the prediction probability of the corresponding target label in the multi-label prediction probability of the sample image is less than or equal to the negative sample threshold of the target label, the sample image is determined to be a noise-free negative sample image of the corresponding target label in the current iteration, and the label information of the sample image corresponding to the target label is corrected to the second value;

[0206] Step S614, for each noise-free sample image corresponding to the target label, the prediction loss of the target label is calculated according to the annotation information of the corrected noise-free sample image and the prediction probability of the target label in the multi-label prediction probability of the noise-free sample image;

[0207] Step S616, summing the prediction losses corresponding to each label to obtain the multi-label prediction loss of the current iteration;

[0208] Step S618, for the triplet corresponding to each similar sample image pair, the triplet quantization loss is calculated according to the difference between the quantized features of the target sample image and the similar sample image, and the difference between the quantized features of the target sample image and the non-similar sample image;

[0209] Step S620, inputting the quantized features of the sample image into a preset sign function to obtain the sign of the quantized features of the sample image as a quantized target of the sample image;

[0210] Step S622, calculating the image quantization loss of the sample image based on the difference between the quantization feature of the sample image and the corresponding symbol;

[0211] Step S624, performing weighted summation of the triplet quantization loss and the image quantization loss to obtain the quantization loss of the current iteration;

[0212] Step S626, performing weighted summation of the multi-label prediction loss and the quantization loss to obtain the total loss of the iteration;

[0213] Step S628, according to the total loss of the current iteration, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in sequence to obtain the image processing model updated in the current iteration.

[0214] In the above embodiment, the noise-free sample images of each label are determined from the sample image set through the positive and negative sample thresholds corresponding to each label updated in the previous iteration, and the multi-label prediction probability of the sample image obtained based on the image processing model updated in the previous iteration, so as to realize the judgment of the noise-free sample images corresponding to each label in the current iteration, so as to remove the noise sample images in the calculation of the multi-label prediction loss of the current iteration, so as to avoid the overfitting of the parameters used for image quantization processing in the image processing model to the noise sample images, and realize the effective training of the image processing model in a large number of noise sample images; and, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated through the multi-label prediction probability of the noise-free sample images in the current iteration, and the positive and negative sample thresholds corresponding to each label updated in the current iteration are obtained. The threshold used for judging the noise-free sample image corresponding to each label in the next iteration is used to realize the dynamic update of the positive and negative sample thresholds of each label used in different iterations, so as to further improve the effectiveness of training the image processing model in a large number of noisy sample images; in addition, the quantization loss of the current iteration is obtained according to the quantization features of the sample image, similar sample images and non-similar sample images, as well as the quantization features of the sample image and the corresponding quantization targets, and the image processing model updated in the previous iteration is updated through the multi-label prediction loss and the quantization loss, so that in the current iteration, the parameters in the image processing model are adjusted in the direction of minimizing the multi-label loss and the quantization loss, so that the quantization features of the image obtained based on the image processing model after the training have the ability to measure semantic relevance and the degree of similarity.

[0215] In order to better understand the above method, the following Figure 7 and Figure 8 An embodiment of a processing method of the image processing model of the present application is described in detail.

[0216] Figure 7The training process of the image processing model is shown; when training the image processing model, the image processing model during training may include a convolutional neural network, a quantization network, and a multi-label prediction network connected in sequence; after the training is completed, the multi-label prediction network may be removed, and the convolutional neural network and the quantization network may be retained to obtain a pre-trained image processing network, that is, the pre-trained image processing model includes a convolutional neural network and a quantization network connected in sequence.

[0217] The training process of the image processing model includes: (1) data preparation: the formation of a full set of sample images, the formation of triplets, and the formation of multi-label annotation information corresponding to each sample image of a non-set full training round; (2) training process.

[0218] (1) Data preparation involves the formation of a full set of sample images, the formation of triplets, and the formation of multi-label annotation information for each sample image:

[0219] 1) Obtain similar sample image pairs, and obtain a full set of sample images based on multiple similar sample image pairs:

[0220] The similar sample image pair includes two sample images and annotation information that characterizes the similarity between the two sample images. The server 104 can extract each frame of the image from the pre-stored video, randomly select two frames of the image from each frame of the image, and use the two manually determined similar frames of the image as the similar sample image pair. Since the image similarity of the same storyboard is high, any two images of the same storyboard are similar images. Therefore, in order to improve the efficiency of obtaining similar sample image pairs, the server 104 can also storyboard the video, use any two images of the same storyboard as similar sample images, and obtain similar sample image pairs.

[0221] When the same storyboard includes three or more images, at least two similar sample image pairs can be obtained from the same storyboard. For example, when the same storyboard includes four frames of images, image_11, image_12, image_13 and image_14 in order of frames, the server 104 can form similar sample image pairs with the first frame image image_11 of the storyboard and the other frames of the storyboard, and the obtained similar sample image pairs are image_11 and image_12, image_11 and image_13, image_11 and image_14.

[0222] By combining the first frame image of the storyboard with the other frames of the storyboard to form similar sample image pairs, the image processing model can adapt to the slight changes between the first frame image image_11 and image_13 and image_14, so that the image processing model can recognize image_11 and image_13 as similar images, and image_11 and image_14 as similar images, thereby improving the generalization effect of the image processing model.

[0223] The server 104 combines the obtained multiple similar sample image pairs into a full sample image set.

[0224] Among them, similar sample image pairs can also be obtained manually to obtain a full set of sample images.

[0225] 2) Triplet mining during model training to form triples:

[0226] If the image processing model training includes multiple epochs, each epoch includes multiple iterations, each iteration includes a forward processing process and a backward processing process, and the number of similar sample image pairs used in each iteration is bs, that is, a batch of similar sample image pairs is bs.

[0227] In each iteration, triple mining of this iteration is performed to form the triples used in this iteration. Specifically, for any iteration, the server 104 can select a sample image from the two sample images included in one of the bs similar sample image pairs used in this iteration; suppose that the two sample images included in one of the similar sample image pairs are image_11 and image_12, and the selected sample image is image_11; then, for the remaining bs-1 similar sample image pairs, the server 104 can randomly select a sample image from the sample images included in each similar sample image pair as a candidate sample image, and use the deep features of each candidate sample image obtained by the convolutional neural network updated in the previous iteration of this iteration; the server 104 can also use the img obtained by the convolutional neural network updated in the previous iteration of this iteration The depth features of age_11 are obtained, and the distances between the depth features of image_11 and the depth features of each candidate sample image are calculated. The first K% of candidate sample images are removed in ascending order according to the distance, and the top 10 candidate sample images among the remaining candidate sample images after removing the first K% of candidate sample images are used as non-similar sample images of image_11, and each non-similar sample image of image_11 is formed into a triple with image_11 and image_12, respectively. In the triple, image_11 is the anchor sample image, image_12 is the similar sample image, and the other image is the non-similar sample image; therefore, each similar sample image pair has 10 triplets accordingly, and the number of triplets in this iteration is 10×bs.

[0228] The purpose of removing the first K% of candidate sample images is to reduce the noise of metric learning: in metric learning, the metric distance between similar sample images is as small as possible, and the metric distance between dissimilar sample images is as large as possible (it is also necessary to satisfy the order preservation effect, that is, the more dissimilar the sample images are, the farther the distance is). Except for the similarity of the previous and next frame images of the same storyboard of the video, or the similarity of the images obtained by adding attack methods such as hue change to the image, the probability of most images being similar to each other is low. In this embodiment, if the bs similar sample image pairs used in the same iteration include two similar sample image pairs obtained from the same storyboard, such as image_11 and image_12, image_11 and image_13, then when the triples of this iteration are mined, triples image_11, image_12 and image_13 will be formed. The sample images of this triple are similar to each other and belong to the noise of metric learning. At this time, the first K% of candidate sample images can be removed (it can be considered that there may be similar sample images in the first K% of candidate sample images, and corresponding triples should not be formed), and non-similar sample images are selected from the remaining candidate sample images to reduce the noise of metric learning. Among them, K is a controllable value. The greater the noise in the full sample image set, the larger K is.

[0229] 3) For each non-set full training round before the set full training round, the labeling information corresponding to the multi-label of each sample image used in each non-set full training round can be obtained based on an existing or pre-trained multi-label labeling model:

[0230] The server 104 can use the general object recognition open source dataset ImageNet or Open Image to obtain a pre-trained multi-label annotation model, and the multi-label annotation model can use a swin-transformer structure. In addition, other existing multi-label annotation models can also be used to perform multi-label annotation on sample images that are not set for the full training round to obtain the annotation information corresponding to the multi-label for each sample image.

[0231] The server 104 sequentially inputs each sample image of the non-set full training round into the multi-label annotation model to obtain the predicted probability of each sample image corresponding to multiple labels output by the multi-label annotation model. When the predicted probability of the sample image corresponding to a certain label is greater than or equal to the set threshold, the sample image can be considered as a positive sample image of the label, otherwise the sample image is considered as a negative sample image of the label. The set threshold can be larger than the commonly used threshold. For example, if the commonly used threshold is 0.5, the set threshold can be 0.7 to minimize the noise sample images of each label.

[0232] However, since the training sample images used to train the multi-label annotation model are not necessarily the same as the full set of sample images used to train the image processing model (for example, the training sample images used to train the multi-label annotation model are animal images, and the full set of sample images used to train the image processing model are plant images), or the annotation accuracy of the multi-label annotation model is not high, the positive sample images of each label determined by the multi-label annotation model are not necessarily the positive sample images of the label; or the negative sample images of each label determined by the multi-label annotation model are not necessarily the negative sample images of the label; or some sample images that actually belong to the positive sample images of label label_1 are not detected by the multi-label annotation model, and the multi-label annotation model considers the sample images to be negative sample images of label label_1. Therefore, the annotation information of each sample image corresponding to the multi-label given by the multi-label annotation model can be called weakly supervised multi-label.

[0233] like Figure 7 As shown, the image processing model during training may include a convolutional neural network, a quantization network, and a multi-label prediction network connected in sequence; the deep features of the sample image obtained by the convolutional neural network are input into the quantization network to obtain the quantization features of the sample image (the quantization features may be quantized floating-point features, each feature value in the quantized floating-point features is mapped to -1 and 1 through a sign function, and the vector composed of the mapping results corresponding to each value of the quantized floating-point features may be called a hash quantization feature or a binary quantization feature); the multi-label prediction network may predict labels of N categories, and the dimension of the multi-label prediction probability of a certain sample image outputted is 1×N; in addition, due to the setting of the full The full training round involves the correction of the annotation information of each sample image and the noise-free sample judgment of each label in each iteration. Therefore, a sample image annotation information maintenance unit is introduced. With the help of the sample image annotation information maintenance unit, based on the positive and negative sample thresholds obtained by updating in the previous iteration, the noise-free sample images of each label are judged for the sample images of each iteration of the full training round, and the noise-free sample images with higher confidence levels for each label are obtained to participate in the prediction loss calculation of each label. With the help of the sample image annotation information maintenance unit, the corrected annotation information of the sample images in each iteration is maintained to dynamically update the positive and negative sample thresholds used in each iteration, so as to realize the optimization learning of weakly supervised multi-label.

[0234] (2) Training process:

[0235] 1) Parameter initialization of image processing model:

[0236] Among them, the parameters of the convolutional neural network including Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x can adopt the corresponding parameters of the multi-label annotation model ResNet101 obtained based on the general object recognition open source dataset imagenet, and the other layers included in the convolutional neural network, such as the feature layer, can be initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. The parameters of the quantization network and the multi-label prediction network can be initialized with a random normal distribution with a variance of 0.01 and a mean of 0.

[0237] 2) Set the parameters to be iteratively updated: all parameters in Table 3 to Table 5.

[0238] Among them, Table 3 is the structure table of the convolutional neural network, Table 4 is the structure table of the quantization network, and Table 5 is the structure table of the multi-label prediction network. The input of the quantization network is the output of the pooling layer of the convolutional neural network, and the output of the quantization network is a quantized 1x256 floating-point feature. The quantized floating-point feature is mapped to a binary value (-1 or 1) through a sign function to obtain a binary quantized feature. The input of the multi-label prediction network is the output of the quantization layer of the quantization network, and the output of the multi-label prediction network is the predicted probability of each sample image corresponding to each label.

[0239]

[0240] Table 3

[0241]

[0242] Table 4

[0243]

[0244] Table 5

[0245] 3) Set the learning rate:

[0246] The learning rate of the convolutional neural network and the quantization network can be lr=0.0005, and the learning rate of the multi-label prediction network can be lr=0.005. Among them, after every 10 iterations, the learning rate of each network becomes 0.1 times the original. Among them, the multi-label prediction loss is transmitted back to the multi-label prediction network to update the parameters of the multi-label prediction network, and the multi-label prediction loss is also transmitted back to the quantization network and the convolutional neural network via the multi-label prediction network to update the parameters of the quantization network and the convolutional neural network. Among them, the learning rate of the quantization network is less than the learning rate of the multi-label prediction network, which can avoid the multi-label prediction loss from being fully transmitted to the quantization network, resulting in the semantic learning of multi-labels excessively affecting the measurement effect of the quantization network, thereby avoiding improper ranking of similarities during the final retrieval.

[0247] 4) Learning process: Perform full training for multiple epochs on the entire sample image set; process the entire sample image set once in each epoch until the loss of a certain epoch no longer decreases compared to the loss of the previous epoch, and the model training is completed; the loss of each epoch is determined based on the average of the losses of each iteration of the epoch.

[0248] 5) In each iteration of each full training round, similar sample image pairs used in the current iteration are determined from multiple batches of similar sample image pairs obtained by dividing the full sample image set, and triple mining is performed based on the deep features obtained in the forward processing of the current iteration to obtain 10 triplets corresponding to each similar sample image pair used in the current iteration;

[0249] 5.1) If the number of rounds is set to 5, when the 6th epoch arrives, it can be determined that when the first set full training round arrives, at this time, based on the multi-label prediction probability of each sample image obtained in the 5th epoch, and the labeling information corresponding to the multi-label of each sample image determined based on the multi-label labeling model, the positive sample threshold and negative sample threshold of each label used in the first iteration of the 6th epoch are initialized; taking the positive sample threshold and negative sample threshold of the label label_1 used in the first iteration of the 6th epoch as an example: the server 104 can obtain the predicted probability of each sample image corresponding to the label label_1 obtained in the 5th epoch, and according to the multi-label The labeling model determines that each sample image corresponds to the labeling information of label label_1. In each sample image, the positive sample image of label label_1 and the negative sample image of label label_1 are determined; then, the mean value of the predicted probability of the sample images of the positive sample images of label label_1 corresponding to label label_1 is obtained, and used as the positive sample threshold of label label_1 used in the first iteration of the 6th epoch; the mean value of the predicted probability of the sample images of the negative sample images of label label_1 corresponding to label label_1 is obtained, and used as the negative sample threshold of label label_1 used in the first iteration of the 6th epoch.

[0250] The positive and negative sample thresholds of each label used in each iteration after the first iteration of the sixth epoch are the positive and negative sample thresholds of each label updated in the previous iteration of this iteration.

[0251] 5.2) Forward processing of each iteration: All parameters of the image processing model are set to the state that needs to be learned. During training, the image processing model performs forward calculation on an input sample image to obtain the quantization features of the sample image output by the quantization layer of the quantization network, and the multi-label prediction probability of the sample image output by the classification layer of the multi-label prediction network; wherein the quantization feature can be represented by Q, which can be a 1x256 hash quantization feature; the multi-label prediction probability can be used for P[image]={P 1 [image],P 2 [image],…,P m [image], ...} means that when the output size of the multi-label prediction network is 1×N, the dimension of P[image] is 1×N, P m [image] indicates the predicted probability that the image corresponds to the label label_m.

[0252] 5.3) For each iteration of each set full training round, based on the positive and negative sample thresholds of each label used in this iteration, the noise-free sample images of each label of this iteration are judged, and the labeling information of the noise-free sample images corresponding to each label is corrected according to the judgment results:

[0253] Based on the forward processing process of the current iteration, the multi-label prediction probability of each sample image in the current iteration is obtained, and the multi-label is taken as the target label in turn, and the prediction probability corresponding to the target label of each sample image in the current iteration is compared with the positive and negative sample thresholds of the target label used in the current iteration; if the prediction probability of the sample image corresponding to the target label in the current iteration is greater than or equal to the positive sample threshold of the target label used in the current iteration, then the sample image is determined to be a noise-free positive sample image of the target label in the current iteration, and the annotation information of the sample image corresponding to the target label is corrected to 1; if the prediction probability of the sample image corresponding to the target label in the current iteration is less than or equal to the negative sample threshold of the target label used in the current iteration, then the sample image is determined to be a noise-free negative sample image of the target label in the current iteration. The sample image is selected, and the annotation information of the sample image corresponding to the target label is corrected to 0; if the predicted probability of the sample image corresponding to the target label in the current iteration is greater than the negative sample threshold of the target label used in the current iteration and less than the positive sample threshold of the target label used in the current iteration, the sample image is determined to be a noise sample image of the target label in the current iteration, and the annotation information of the sample image corresponding to the target label is corrected to -1, indicating that it is difficult to determine whether the sample image is a positive sample image or a negative sample image of the target label. The uncertain sample image belonging to the target label does not participate in the prediction loss calculation of the target label. The prediction loss of the sample image in the target label in the current iteration will not be fed back to the multi-label prediction network, the quantization network and the convolutional neural network to avoid affecting the learning of each network.

[0254] 5.4) When calculating the loss used in each iteration of each set full training round, the annotation information of the sample image used is the revised annotation information in the corresponding iteration; the specific calculation process is described later.

[0255] When calculating the loss used in each iteration of the non-set full training round, the labeling information of the sample image used is the labeling information determined by the multi-label labeling model.

[0256] 5.5) Update the positive and negative sample thresholds of each label used in each iteration of each set full training round to serve as the positive and negative sample thresholds of each label used in the next iteration; take the update of the positive and negative sample thresholds of the label label_m used in the current iteration as an example: the positive sample threshold of the label label_m used in the current iteration is recorded as thr_m1 - , the positive sample threshold of label label_m used in the next iteration is updated and the positive sample threshold of label label_m is recorded as thr_m1 + , the negative sample threshold of label label_m used in the next iteration is recorded as thr_m0 - , the positive sample threshold of label label_m used in the next iteration is updated and the positive sample threshold of label label_m is recorded as thr_m0 + The formula used to update the positive and negative sample thresholds of the label label_m used in the current iteration is:

[0257]

[0258]

[0259] In formula ①, j is the noise-free positive sample image of label label_m in the current iteration, P m [j] is the predicted probability of the noise-free positive sample image with label label_m corresponding to label label_m in the current iteration; N 1 is the number of noise-free positive sample images with label label_m in the current iteration.

[0260] In formula ②, j is the noise-free negative sample image of label label_m in the current iteration, P m [j] is the predicted probability that the noise-free negative sample image with label label_m corresponds to label label_m in the current iteration; N 0 is the number of noise-free negative sample images with label label_m in the current iteration.

[0261] 5.6) In each iteration, the parameters of each network of the image processing model are updated: the SGD stochastic gradient descent method is used to perform gradient backward on the total loss determined in this iteration, and the gradient is accordingly fed back to the multi-label prediction network, quantization network and convolutional neural network of the image processing model to obtain the updated values ​​of the parameters of each network and update the parameters of each network.

[0262] Specifically, the total loss determined at each iteration includes loss 多标签 and loss 量化 , the return process of each part of the loss is:

[0263] loss 多标签 Directly feed back to the multi-label prediction network; determine the update value of the parameters of the multi-label prediction network according to the loss fed back to the multi-label prediction network, and update the parameters of the multi-label prediction network.

[0264] loss 多标签 It is also passed back to the quantization network and convolutional neural network in turn through the multi-label prediction network, as well as the loss 量化 It does not go through the multi-label prediction network, but directly passes it back to the quantization network and the convolutional neural network in sequence; the update value of the parameters of the quantization network is determined according to the loss passed back to the quantization network, and the parameters of the quantization network are updated; the update value of the parameters of the convolutional neural network is determined according to the loss passed back to the convolutional neural network, and the parameters of the convolutional neural network are updated.

[0265] (6) Set the loss calculation in each iteration of the full training round:

[0266] For the bs similar sample image pairs used in the current iteration, obtain the multi-label prediction loss of each similar sample image pair and quantify the loss Get the loss of each similar sample image pair

[0267]

[0268] In the formula, j represents the two sample images constituting the similar sample image pair, th represents the triplet corresponding to the similar sample image pair, w1 can be set to 0.5, and w2 can be set to 1-w1.

[0269] Among them, the multi-label prediction loss of each similar sample image pair is obtained according to the mean of the multi-label prediction losses of the two sample images constituting the similar sample image pair and the corresponding weight coefficient w1; the quantization loss of each similar sample image pair is obtained according to the mean of the losses of the corresponding triplets of the similar sample image pair and the corresponding weight coefficient w2.

[0270] Next, calculate the mean of the multi-label prediction loss of bs similar sample image pairs used in the iteration, and the loss result is recorded as loss 多标签 ; Calculate the mean of the quantization loss of bs similar sample image pairs used in the iteration, and the loss result is recorded as loss 量化 , the total loss of the iteration is obtained = loss 多标签 +loss 量化 .

[0271] Among them, since the metric learning of the quantization network converges slowly (that is, the difficulty of finding similar and dissimilar parts in two sample images is higher than the difficulty of finding the commonality of the same category in multiple images of the same category), the loss passed back to the quantization network includes not only loss 量化 Also includes loss 多标签 , which can accelerate convergence and avoid missed recall of semantically related images due to improper local representation in metric learning.

[0272] Among them, the learning rate of the quantization network can be 0.1 times the learning rate of the multi-label prediction network to avoid overfitting of the quantization features due to too fast convergence of semantic learning.

[0273] 6.1) Triplet loss L 2 Including triplet quantization loss L triplet And the image quantization loss L of each sample image constituting the triplet coding , that is, L 2 =w 21 L triplet +w 22 L coding ; Among them, the triple quantization loss L triplet And the image quantization loss L of each sample image constituting the triplet coding The inputs of are all quantized features output by the quantization layer of the quantization network. 21 is the weight coefficient assigned to the triplet quantization loss, which can be set to 1; w 22 is the weight coefficient assigned to the image quantization loss, which can be set to 0.5; and w 21 Greater than w 22 It can ensure that when the learning of regression loss converges faster than the metric learning of triples, the metric learning of triples will dominate the overall learning, thereby ensuring that the deep features output by the convolutional neural network always have the measurement ability of similarity.

[0274] Triplet quantization loss L triplet:Since the quantization features output by the quantization layer of the quantization network are 256-dimensional, each bit needs to learn a value of -1 or 1. The distance between the quantization features of the anchor sample image in the triplet and the quantization features of the non-similar sample image needs to be large enough to ensure that the triplet can be distinguished in the quantization space. Therefore, the margin needs to be set larger, for example, it can be set to margin = 160. Triplet quantization loss L triplet The formula is as follows, where α is margin, set to 160, ‖Q a -Q p ‖ represents the distance between the quantized features of the anchor sample image and the quantized features of similar sample images in the triplet, ‖Q a -Q n ‖ represents the distance between the quantized features of the anchor sample image and the quantized features of the non-similar sample image in the triplet. Triplet quantization loss L triplet The purpose is to make the distance between the anchor sample image and the non-similar sample image greater than the distance between the anchor sample image and the similar sample image by margin.

[0275] L triplet =max(‖Q a -Q p ‖-‖Q a -Q n ‖+α,0)

[0276] The image quantization loss L of each sample image constituting the triplet coding :Based on the quantized features output by the quantized network, the loss of the quantized effect of the quantized network (whether each quantized feature value of the quantized feature is close enough to -1 or 1) is calculated; since after the quantized network outputs the quantized features, each quantized feature value needs to be mapped to a -1 / 1 binary value in the application. If each quantized feature value is close enough to 1 or -1, the triplet quantization loss L can be determined triplet The loss can describe the application of the quantized features, otherwise the triple quantization loss L triplet No matter how good the measurement effect corresponding to the loss is, it does not mean that the quantitative representation in the application is good enough, so it is hoped that each quantized feature value is close enough to 1 or -1.

[0277] The image quantization loss L of each sample image constituting the triplet coding , the loss function used is:

[0278]

[0279] Among them, Q i is the quantized feature value Q of the sample image at the i-th position i , B i corresponds to Q i The quantitative target value, Bi By Q i Generated by the sign function - the sign function is used to generate the target code of the quantitative learning task; for each quantized feature value of the sample image, the corresponding quantized target value is obtained through the sign function, and then the quantized target is obtained. Among them, the sign function is:

[0280] The regression loss function is used to obtain the image quantization loss of the sample image, so that the distance between the quantization feature Q of the sample image output by the quantization network and the quantization target B becomes smaller.

[0281] In the application, the sign function is directly used to generate a quantized binary vector for the quantized features output by the quantization network. The quantized binary vector can be used for image retrieval.

[0282] 6.2) Calculation of multi-label prediction loss: The multi-label prediction loss of a similar sample image pair is obtained based on the average of the multi-label prediction losses of the two sample images that constitute the similar sample image pair; the multi-label prediction loss of a sample image includes the prediction loss of the sample image for each label; taking the calculation of the prediction loss of the sample image image_11 corresponding to the label label_m as an example, the calculation of the prediction loss of the sample image for each label is introduced:

[0283] In a certain iteration of the full training round, let the corrected label information of the sample image image_11 corresponding to the label label_m be t m [image_11], the predicted probability of the sample image image_11 corresponding to the label label_m is P m [image_11], then the prediction loss of the sample image image_11 corresponding to the label label_m is

[0284] Among them, the corrected annotation information of the sample image image_11 corresponding to the label label_m is t m [image_11] is corrected in this iteration; the predicted probability of the sample image image_11 corresponding to the label label_m is P m [image_11] is the output of the multi-label prediction network in this iteration.

[0285] When the corrected annotation information t m When [image_11] is 1, the left part of the plus sign in the above formula takes effect, and the prediction loss of the sample image image_11 corresponding to the label label_m is obtained; when the corrected label information t mWhen [image_11] is 0, the right side of the plus sign in the above formula takes effect, and the prediction loss of the sample image image_11 corresponding to the label label_m is obtained; when the corrected label information t m When [image_11] is -1, the prediction loss of the sample image image_11 corresponding to the label label_m is not calculated.

[0286] Similarly, if the label information of the sample image image_11 corresponding to other labels is 1 or 0, the prediction loss of the sample image under the label can be obtained; the server 104 can obtain the sum of the prediction losses of the sample image under the corresponding labels, and obtain the multi-label prediction loss of the sample image image_11 in the current iteration. Among them, if the label information of the sample image image_11 corresponding to a certain label is -1, then the multi-label prediction loss of the sample image image_11 in the current iteration does not include the prediction loss of the sample image image_11 corresponding to the label.

[0287] After the training is completed, the multi-label prediction network is removed, and the convolutional neural network and the quantization network are retained to obtain a pre-trained image processing model. The server 104 can use the pre-trained image processing model for image retrieval.

[0288] Specifically, the server 104 obtains the quantitative features of each stock image using the pre-trained image processing model, and performs binary activation on the quantitative features of each stock image through the sign function to obtain the binary quantitative features of each stock image, and uses the binary quantitative features of each stock image as the index of the stock image itself and stores it in the data storage system. In addition, the server 104 can also perform kmeans clustering on each stock image according to the set number of clusters based on the quantitative features or binary quantitative features of the stock image to obtain multiple clusters, and uses the binary quantitative features of the stock image at the center of the cluster as the query index of the cluster, and establishes an association relationship between the binary quantitative features of the stock image at the center of the cluster and the cluster.

[0289] When the server 104 receives the input image sent by the terminal 102, the quantized features of the input image are obtained by using the pre-trained image processing model, and the binary quantized features of the input image are activated by binarization through the sign function to obtain the binary quantized features of the input image. The server 104 obtains the distance between the binary quantized features of the input image and the binary quantized features as the query index of each cluster; the server 104 determines the binary quantized features with the shortest distance to the binary quantized features of the input image, or the binary quantized features with a distance less than the preset distance from the binary quantized features of the input image among the binary quantized features as the query index of each cluster, and uses the cluster associated with the binary quantized features as the target cluster; the stock image of the target cluster is used as the retrieval image matched with the input image; the server 104 obtains the distance between the binary quantized features of the input image and the binary quantized features of each retrieval image, and determines the retrieval images ranked before the preset number of digits according to the order of distance from small to large, and feeds back the retrieval images ranked before the preset number of digits in sequence according to the order of distance from small to large.

[0290] Figure 8 The schematic diagram of the architecture of the processing method of the image processing model. The multi-label annotation model is used to obtain the multi-label annotation information of each image. After obtaining the predicted probability of each label of the sample image output by the multi-label prediction network, if the multi-label annotation information of each image is obtained directly based on the multi-label annotation model, the loss is determined. The loss noise is large and can be called noisy loss. If the image processing model is directly iteratively updated based on the noisy loss during training, the image processing model will overfit the noisy sample images of each label.

[0291] Figure 8 The upper right diagram shows the distribution of predicted probabilities of positive and negative samples under a certain label, including the distribution of predicted probabilities of unlabeled positive samples determined by the multi-label annotation model, and the distribution of predicted probabilities of labeled positive samples after noise-free sample image judgment. This embodiment implements the judgment of the noise-free sample image of the label based on the negative sample threshold and positive sample threshold of the label (wherein the horizontal coordinate corresponding to the peak of the curve of the negative sample is the negative sample threshold of the label, and the horizontal coordinate of the peak of the curve of the labeled positive sample is the positive sample threshold of the label), thereby improving the multi-label classification effect.

[0292] In the above processing, the distance between the quantitative features of different images, or the distance between the depth features of different images, can be represented by the Euclidean distance.

[0293] During the training of the image processing model, the result of mapping through the sign function can be set to -1 and 1, and a binary quantized feature consisting of -1 and 1 can be obtained; in the application scenario of image retrieval, the result of mapping through the sign function can be set to 0 and 1, and a binary quantized feature consisting of 0 and 1 can be obtained.

[0294] The distance between binary quantized features of different images is obtained by counting the number of different values ​​at corresponding positions between binary quantized features. For example, in binary quantized feature 1000 and binary quantized feature 0011, there are 3 different values ​​at positions, and the distance between these two binary quantized features is 3. The distance between binary quantized features of an image can be called the number of bits or the Hamming distance.

[0295] In this embodiment, by obtaining large-scale weakly supervised multi-label annotation information, unlabeled positive sample images and unlabeled negative sample images of each label are mined through dynamic label prediction in quantitative learning, and the annotation information of the corresponding label of the sample image is corrected, so that the image processing model can feedback and correct the weakly supervised multi-label annotation information of the sample image according to the learned information in the process of improving the semantic learning of multi-labels, and gradually converge in the iteration of the multi-label annotation information of the corrected sample image and metric learning. By comparing the positive and negative sample thresholds of each label with the weakly supervised multi-label annotation information of each sample image, the positive and negative sample images with higher accuracy of each label can be obtained, and the annotation information of the corresponding label of the sample image can be corrected. By dynamically maintaining the positive and negative sample thresholds of each label, positive and negative sample images with higher confidence are gradually added in each iteration based on the active learning mechanism for learning, so that the image processing model can finally learn more effective label data to obtain the final quantitative features.

[0296] In this embodiment, the multi-label prediction loss and the triplet loss are integrated to perform iterative parameter update to achieve multi-label semantic learning and metric learning, which can avoid the semantic missed recall of metric learning only and the improper sorting of semantic learning only. In addition, with the help of an existing or pre-trained multi-label annotation model, the full sample image set is annotated with multi-labels, and the large-scale weakly supervised multi-label annotation information required for image processing model learning is obtained through weak supervision based on prior knowledge, etc., thereby saving the annotation cost. With the help of a priori methods, similar sample image pairs required for metric learning are obtained from the video, so that a large-scale full sample image set can be quickly obtained. With the help of tracking of noise-free positive and negative sample images of each label and correction of multi-label annotation information of sample images, more clean sample images are mined to avoid the noise information in the weakly supervised multi-label annotation information affecting the overall model learning effect, thereby improving the multi-label accuracy.

[0297] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0298] In one embodiment, Fig. 9 As shown, a processing device for an image processing model is provided, comprising:

[0299] The sample image set acquisition module 902 is used to acquire the sample image set of the current iteration;

[0300] The forward processing module 904 of the current iteration is used to extract the image features of the sample images in the sample image set through the image processing model updated in the previous iteration, quantize the image features to obtain quantized features, and perform multi-label prediction based on the quantized features to obtain the multi-label prediction probability of the sample images;

[0301] A multi-label prediction loss determination module 906 is used to determine the noise-free sample images corresponding to each label from the sample image set and correct the labeling information of the noise-free sample images based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image, and determine the multi-label prediction loss of the current iteration according to the labeling information of the noise-free sample images and the multi-label prediction probability;

[0302] The positive and negative sample threshold updating module 908 is used to update the positive and negative sample thresholds corresponding to each label updated in the previous iteration according to the multi-label prediction probability of the noise-free sample image, and obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration;

[0303] A quantization loss determination module 910 is used to determine similar sample images and non-similar sample images of the sample image from the sample image set, and determine the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization target;

[0304] The backward processing module 912 of the current iteration is used to update the image processing model updated in the previous iteration based on the multi-label prediction loss and the quantization loss to obtain the image processing model updated in the current iteration.

[0305] In one embodiment, the multi-label prediction loss determination module 906 is also used to take each label in the multi-label as a target label in turn; obtain the positive sample threshold and the negative sample threshold corresponding to the target label updated in the previous iteration; when the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is greater than or equal to the positive sample threshold of the target label, then the sample image is determined to be the noise-free positive sample image corresponding to the target label in the current iteration, and the labeling information of the sample image corresponding to the target label is corrected to the first value; when the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is less than or equal to the negative sample threshold of the target label, then the sample image is determined to be the noise-free negative sample image corresponding to the target label in the current iteration, and the labeling information of the sample image corresponding to the target label is corrected to the second value.

[0306] In one embodiment, the positive and negative sample threshold update module 908 is also used to take each label in the multiple labels as the target label in turn; according to the noise-free positive sample image of the target label in the current iteration, the mean value of the predicted probability corresponding to the target label is used to update the positive sample threshold corresponding to the target label updated in the previous iteration, so as to obtain the positive sample threshold corresponding to the target label updated in the current iteration; according to the noise-free negative sample image of the target label in the current iteration, the mean value of the predicted probability corresponding to the target label is used to update the negative sample threshold corresponding to the target label updated in the previous iteration, so as to obtain the negative sample threshold corresponding to the target label updated in the current iteration.

[0307] In one embodiment, the positive and negative sample threshold updating module 908 is further used to perform weighted summation of the positive sample threshold corresponding to the target label updated in the previous iteration and the mean of the predicted probability of the target label corresponding to the noise-free positive sample image of the target label in the current iteration to obtain the positive sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

[0308] In one embodiment, the positive and negative sample threshold updating module 908 is further used to perform weighted summation of the negative sample threshold corresponding to the target label updated in the previous iteration and the mean of the predicted probability of the target label corresponding to the noise-free negative sample image of the target label in the current iteration to obtain the negative sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

[0309] In one embodiment, the multi-label prediction loss determination module 906 is also used to take each label in the multi-label as the target label in turn; for each noise-free sample image corresponding to the target label, the prediction loss of the target label is calculated according to the corrected annotation information of the noise-free sample image and the prediction probability corresponding to the target label in the multi-label prediction probability of the noise-free sample image; the prediction losses corresponding to each label are summed to obtain the multi-label prediction loss of the current iteration.

[0310] In one embodiment, the sample image set includes a similar sample image pair, and the two sample images included in the similar sample image pair are similar;

[0311] The quantization loss determination module 910 is also used to generate a triplet corresponding to each similar sample image pair of the sample image set; the triplet includes a target sample image, a similar sample image of the target sample image and a non-similar sample image of the target sample image; for each triplet, the triplet quantization loss is calculated according to the difference in quantization features between the target sample image and the similar sample images, and the difference between the quantization features of the target sample image and the non-similar sample images; the image quantization loss is calculated according to the difference between the quantization features of each sample image of the sample image set and the corresponding quantization target; and the triplet quantization loss and the image quantization loss are weightedly summed to obtain the quantization loss of the current iteration.

[0312] In one embodiment, the quantization loss determination module 910 is further used to use the two sample images in the similar sample image pair as the target sample image and the similar sample images of the target sample image; based on the image features extracted this time, determine at least one sample image that is not similar to the target sample image from the sample image set as the non-similar sample image of the target sample image; and form at least one triplet with the target sample image, the similar sample image of the target sample image, and the non-similar sample image of the target sample image.

[0313] In one embodiment, the device further comprises: a similar sample image pair acquisition module, which is used to acquire the storyboards of the video; using the images of the same storyboard as similar sample images to obtain a plurality of similar sample image pairs corresponding to the same storyboard;

[0314] The quantization loss determination module 910 is also used to obtain the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time; the candidate sample images are sample images of other similar sample image pairs in the sample image set except the similar sample image pair to which the target sample image belongs; based on the distances between the image features of the target sample image extracted at that time and the image features of each candidate sample image extracted at that time, the candidate sample images are sorted in ascending order of distance; the candidate sample images ranked after a set number of digits are determined, and sample images that are dissimilar to the target sample image are determined from the candidate sample images ranked after the set number of digits as non-similar sample images of the target sample image.

[0315] In one embodiment, the quantization loss determination module 910 is further used to input the quantization feature of the sample image into a preset sign function to obtain the sign of the quantization feature of the sample image as the quantization target of the sample image; and calculate the image quantization loss of the sample image based on the difference between the quantization feature of the sample image and the corresponding sign.

[0316] In one embodiment, the image processing model includes a quantization network and a multi-label prediction network;

[0317] The backward processing module 912 of the current iteration is used to perform weighted summation of the multi-label prediction loss and the quantization loss to obtain the total loss of the current iteration; based on the total loss of the current iteration, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in sequence to obtain the image processing model updated in the current iteration.

[0318] In one embodiment, each iteration is any iteration in a full training round, and the sample image set used in each iteration is a subset of the full sample image set;

[0319] The device also includes a positive and negative sample threshold determination module, which is used to obtain the original annotation information of each sample image in the full sample image set and the multi-label prediction probability of each sample image after completing the previous full training round when the current iteration is the first iteration in the full training round in which the noise-free sample image is determined based on the positive and negative sample thresholds of each label; take each label in the multi-label as the target label in turn; for each target label, determine the positive sample image and the negative sample image of the target label according to the original annotation information; initialize the positive sample threshold of the target label required for the current iteration according to the mean value of the prediction probability corresponding to the target label in the multi-label prediction probability of the positive sample image of the target label after completing the previous full training round; initialize the negative sample threshold of the target label required for the current iteration according to the mean value of the prediction probability corresponding to the target label in the multi-label prediction probability of the negative sample image of the target label after completing the previous full training round.

[0320] In one embodiment, the device also includes an image retrieval module, which is used to obtain a pre-trained image processing model when a training stop condition is met; obtain an input image for image retrieval; extract image features of the input image through the pre-trained image processing model, and quantify the image features of the input image to obtain quantitative features of the input image; compare the quantitative features of the input image with the quantitative features of each stock image in an image library; and determine a retrieval image that matches the input image based on the comparison result.

[0321] For the specific limitations and technical effects of the processing device of the image processing model, please refer to the limitations and technical effects of the processing method of the image processing model mentioned above, which will not be repeated here. The various modules in the processing device of the above-mentioned image processing model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0322] In one embodiment, a computer device is provided. The computer device may be a terminal 102 or a server 104. The internal structure diagram thereof may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store processing data of the image processing model. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer device also includes an input and output interface, which is a connection circuit for exchanging information between the processor and the external device, and they are connected to the processor through a bus, referred to as an I / O interface. When the computer program is executed by the processor, a processing method of an image processing model is implemented.

[0323] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0324] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0325] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0326] In one embodiment, a computer program product is provided, on which a computer program is stored, and the computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0327] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0328] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0329] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0330] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A processing method of an image processing model, characterized in that: The method comprises: Get the sample image set of the current iteration; Extracting image features of sample images in the sample image set through the image processing model updated in the previous iteration, quantizing the image features to obtain quantized features, and performing multi-label prediction based on the quantized features to obtain multi-label prediction probabilities of the sample images; Based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image, determine the noise-free sample image corresponding to each label from the sample image set and correct the label information of the noise-free sample image, and determine the multi-label prediction loss of the current iteration according to the label information of the noise-free sample image and the multi-label prediction probability; According to the multi-label prediction probability of the noise-free sample image, the positive and negative sample thresholds corresponding to each label updated in the previous iteration are updated to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration; Determine similar sample images and non-similar sample images of the sample image from the sample image set, and determine the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization targets; Based on the multi-label prediction loss and the quantization loss, the image processing model updated in the previous iteration is updated to obtain the image processing model updated in the current iteration.

2. The method according to claim 1, characterized in that The method of determining a noise-free sample image corresponding to each label from the sample image set and correcting the labeling information of the noise-free sample image based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image comprises: Taking each label in the multiple labels as the target label in turn; Obtaining the positive sample threshold and the negative sample threshold corresponding to the target label updated in the previous iteration; When the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is greater than or equal to the positive sample threshold of the target label, the sample image is determined to be a noise-free positive sample image corresponding to the target label in the current iteration, and the label information of the sample image corresponding to the target label is corrected to a first value; When the prediction probability corresponding to the target label in the multi-label prediction probability of the sample image is less than or equal to the negative sample threshold of the target label, the sample image is determined to be a noise-free negative sample image corresponding to the target label in the current iteration, and the labeling information of the sample image corresponding to the target label is corrected to a second value.

3. The method according to claim 2, characterized in that The method of updating the positive and negative sample thresholds corresponding to each label updated in the previous iteration according to the multi-label prediction probability of the noise-free sample image to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration includes: Taking each label in the multiple labels as the target label in turn; According to the noise-free positive sample image of the target label in the current iteration and the mean of the predicted probabilities corresponding to the target label, the positive sample threshold corresponding to the target label updated in the previous iteration is updated to obtain the positive sample threshold corresponding to the target label updated in the current iteration; According to the noise-free negative sample image of the target label in the current iteration and the mean of the predicted probabilities corresponding to the target label, the negative sample threshold corresponding to the target label updated in the previous iteration is updated to obtain the negative sample threshold corresponding to the target label updated in the current iteration.

4. The method according to claim 3, characterized in that The method of updating the positive sample threshold corresponding to the target label updated in the previous iteration according to the noise-free positive sample image of the target label in the current iteration and the mean value of the predicted probability corresponding to the target label to obtain the positive sample threshold corresponding to the target label updated in the current iteration includes: The positive sample threshold corresponding to the target label updated in the previous iteration is weightedly summed with the mean of the predicted probabilities of the target label corresponding to the noise-free positive sample images of the target label in the current iteration to obtain the positive sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the positive sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

5. The method according to claim 3, characterized in that: The method of updating the negative sample threshold corresponding to the target label updated in the previous iteration according to the noise-free negative sample image of the target label in the current iteration and the mean of the predicted probabilities corresponding to the target label to obtain the negative sample threshold corresponding to the target label updated in the current iteration includes: The negative sample threshold corresponding to the target label updated in the previous iteration is weightedly summed with the mean of the predicted probabilities of the target label corresponding to the noise-free negative sample images of the target label in the current iteration to obtain the negative sample threshold corresponding to the target label updated in the current iteration; wherein the weight coefficient of the negative sample threshold corresponding to the target label updated in the previous iteration is greater than the weight coefficient of the mean.

6. The method according to claim 1, characterized in that The step of determining the multi-label prediction loss of the current iteration according to the annotation information of the noise-free sample image and the multi-label prediction probability includes: Taking each label in the multiple labels as the target label in turn; For each noise-free sample image corresponding to the target label, calculating the prediction loss of the target label according to the corrected annotation information of the noise-free sample image and the prediction probability corresponding to the target label in the multi-label prediction probability of the noise-free sample image; Sum the prediction losses corresponding to each label to get the multi-label prediction loss of the iteration.

7. The method according to claim 1, characterized in that The sample image set includes a similar sample image pair, and the two sample images included in the similar sample image pair are similar; The determining of similar sample images and non-similar sample images of the sample image from the sample image set, and determining the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image, and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization target, includes: For each similar sample image pair of the sample image set, generating a triplet corresponding to the similar sample image pair; the triplet includes a target sample image, a similar sample image of the target sample image, and a non-similar sample image of the target sample image; For each triplet, calculating a triplet quantization loss according to a difference between the quantized features of the target sample image and the similar sample image, and a difference between the quantized features of the target sample image and the non-similar sample image; Calculating the image quantization loss according to the difference between the quantization feature of each sample image in the sample image set and the corresponding quantization target; The triplet quantization loss and the image quantization loss are weightedly summed to obtain the quantization loss of the current iteration.

8. The method according to claim 7, characterized in that For each similar sample image pair of the sample image set, generating a triplet corresponding to the similar sample image pair comprises: Using two sample images in the similar sample image pair as a target sample image and similar sample images of the target sample image; Based on the image features extracted this time, determining at least one sample image that is dissimilar to the target sample image from the sample image set as a non-similar sample image to the target sample image; The target sample image, a sample image similar to the target sample image, and a sample image not similar to the target sample image form at least one triple.

9. The method according to claim 8, characterized in that The method further comprises: Get the storyboard of the video; Taking images of the same storyboard as similar sample images, a plurality of similar sample image pairs corresponding to the same storyboard are obtained; The step of determining, based on the image features extracted this time, at least one sample image that is dissimilar to the target sample image from the sample image set as a non-similar sample image to the target sample image comprises: Obtaining the image features extracted at the time of the target sample image and the image features extracted at the time of each candidate sample image; the candidate sample images are sample images of other similar sample image pairs in the sample image set except the similar sample image pair to which the target sample image belongs; Based on the distances between the image features of the target sample image extracted at that time and the image features of the candidate sample images extracted at that time, the candidate sample images are sorted in ascending order of distance; The candidate sample images ranked after the set number of bits are determined, and sample images that are dissimilar to the target sample image are determined from the candidate sample images ranked after the set number of bits as non-similar sample images to the target sample image.

10. The method according to claim 7, characterized in that The calculating the image quantization loss according to the difference between the quantization feature of each sample image in the sample image set and the corresponding quantization target comprises: Inputting the quantized feature of the sample image into a preset sign function to obtain the sign of the quantized feature of the sample image as a quantized target of the sample image; An image quantization loss of the sample image is calculated based on a difference between the quantization feature of the sample image and the corresponding symbol.

11. The method according to claim 1, characterized in that: The image processing model includes a quantization network and a multi-label prediction network; Based on the multi-label prediction loss and the quantization loss, the image processing model updated in the previous iteration is updated to obtain the image processing model updated in the current iteration, including: Perform a weighted summation of the multi-label prediction loss and the quantization loss to obtain the total loss of the iteration; According to the total loss of the current iteration, the parameters of the multi-label prediction network and the parameters of the quantization network in the image processing model updated in the previous iteration are updated in sequence to obtain the image processing model updated in the current iteration.

12. The method according to claim 1, characterized in that Each iteration is any iteration in a full training round, and the sample image set used in each iteration is a subset of the full sample image set; the method further includes: When the current iteration is the first iteration in the first full training round in which the noise-free sample images are determined based on the positive and negative sample thresholds of each label, the original annotation information of each sample image in the full sample image set and the multi-label prediction probability of each sample image after completing the previous full training round are obtained; Taking each label in the multiple labels as the target label in turn; For each target label, determining a positive sample image and a negative sample image of the target label according to the original annotation information; Initialize the positive sample threshold of the target label required for the current iteration according to the mean of the prediction probabilities corresponding to the target label in the multi-label prediction probabilities of the positive sample images of the target label after completing the previous full training round; Initialize the negative sample threshold of the target label required for the current iteration according to the mean of the predicted probabilities corresponding to the target label in the multi-label predicted probabilities of the negative sample images of the target label after completing the previous full training round.

13. The method according to claim 1, characterized in that The method further comprises: When the training stop condition is met, a pre-trained image processing model is obtained; Obtain an input image for image retrieval; Extracting image features of the input image through the pre-trained image processing model, and performing quantization processing on the image features of the input image to obtain quantized features of the input image; Comparing the quantitative features of the input image with the quantitative features of each stock image in the image library; A search image matching the input image is determined according to the comparison result.

14. A processing device for an image processing model, characterized in that: The device comprises: A sample image set acquisition module is used to acquire the sample image set of the current iteration; The forward processing module of the current iteration is used to extract image features of the sample images in the sample image set through the image processing model updated in the previous iteration, quantize the image features to obtain quantized features, and perform multi-label prediction based on the quantized features to obtain the multi-label prediction probability of the sample images; A multi-label prediction loss determination module is used to determine the noise-free sample images corresponding to each label from the sample image set and correct the labeling information of the noise-free sample images based on the positive and negative sample thresholds corresponding to each label updated in the previous iteration and the multi-label prediction probability of the sample image, and determine the multi-label prediction loss of the current iteration according to the labeling information of the noise-free sample images and the multi-label prediction probability; A positive and negative sample threshold updating module is used to update the positive and negative sample thresholds corresponding to each label updated in the previous iteration according to the multi-label prediction probability of the noise-free sample image, so as to obtain the positive and negative sample thresholds corresponding to each label updated in the current iteration; a quantization loss determination module, used to determine similar sample images and non-similar sample images of the sample image from the sample image set, and determine the quantization loss of the current iteration based on the quantization features of the sample image, the similar sample image and the non-similar sample image, and the quantization features of the sample image and the corresponding quantization targets; The backward processing module of the current iteration is used to update the image processing model updated in the previous iteration based on the multi-label prediction loss and the quantization loss to obtain the image processing model updated in the current iteration.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.