Search model updating method and device, equipment and computer readable storage medium

Abstract

The application provides a search model updating method and device, equipment and a computer readable storage medium. The method comprises the following steps: obtaining an old search model currently used online, a new search model to be deployed online and training data, wherein the training data comprises a plurality of training images and label information corresponding to each training image; extracting features of the plurality of training images by using the old search model to obtain old training features; performing non-degradation compatible training on the new search model by using the training data and the old training features to obtain a trained new search model; deploying the trained new search model online, and performing candidate image library updating and image searching by using the trained new search model. Through the application, the waiting period of model deployment can be shortened, and the prediction accuracy of the search model can be ensured.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, device, and computer-readable storage medium for updating a search model. Background Technology

[0002] Image retrieval refers to the process where, after a user inputs an image of interest (the query image), the search system first extracts the image's features and then retrieves related images from a candidate image library by comparing feature similarity. With the optimization of deep learning frameworks and the increase in training data, the industry typically requires regular system updates to improve system performance and user experience. System updates include cold refresh and hot refresh. The "cold refresh" update mode involves re-extracting features from the candidate image library using the updated model, and the new model cannot be deployed until the feature update is complete. This update cycle can last for weeks or even months, severely impacting system upgrade progress. With the advent of backward-compatible learning, the features extracted by the new model for the query image are directly comparable to those in the candidate library, making the "hot refresh" update mode possible. "Hot refresh" refers to deploying the new model directly, dynamically and gradually refreshing the candidate image library during this period, greatly shortening the model update and deployment cycle. However, analysis of the experimental results revealed that the "hot refresh" process can lead to model degradation, which is caused by "negative inversion." This means that images that the old model could retrieve correctly cannot be retrieved by the new model, thus reducing search accuracy. Summary of the Invention

[0003] This application provides a method, apparatus, and computer-readable storage medium for updating a search model, which can shorten the waiting period for model deployment and ensure the predictive accuracy of the search model.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for updating a search model, including:

[0006] Acquire the old search model currently in use online, the new search model to be deployed online, and training data, wherein the training data includes multiple training images and label information corresponding to each training image;

[0007] The old search model is used to extract features from the multiple training images to obtain old training features;

[0008] The new search model is trained using the training data and the old training features to obtain a degeneracy-free compatible new search model.

[0009] The trained new search model is deployed online, and the candidate image library is updated and images are searched using the trained new search model.

[0010] This application provides a search model updating device, comprising:

[0011] The first acquisition module is used to acquire the old search model currently in use online, the new search model to be deployed online, and training data. The training data includes multiple training images and label information corresponding to each training image.

[0012] The feature extraction module is used to extract features from the multiple training images using the old search model to obtain old training features;

[0013] The model training module is used to perform degradation-free compatible training on the new search model using the training data and the old training features to obtain a trained new search model.

[0014] The model deployment module is used to deploy the trained new search model online and use the trained new search model to update the candidate image library and perform image search.

[0015] In some embodiments, the new search model includes a new feature extraction sub-model, and the model training module is further configured to:

[0016] The new feature extraction sub-model of the new search model is used to extract features from the multiple training images to obtain new training features;

[0017] The loss function of the new search model is determined using the old and new training features.

[0018] The new search model is trained using the new training features, the old training features, and the loss function to obtain a degeneracy-free compatible search model.

[0019] In some embodiments, the model training module is further configured to:

[0020] The degradation-free compatible loss function of the new search model is determined using the old and new training features.

[0021] The classification loss function of the new search model is determined using the new training features and the label information corresponding to each training image;

[0022] The loss function of the new search model is constructed based on the degradation-free compatibility loss function and the classification loss function.

[0023] In some embodiments, the model training module is further configured to:

[0024] Construct multiple new-old positive example pairs, multiple new-new negative example pairs, and multiple new-old negative example pairs using the new training features and the old training features;

[0025] A baseline loss function is constructed based on the first dot product result of the new-old positive example pair and the second dot product result of the new-old negative example pair.

[0026] Based on the third dot product result of the new-new negative example pair, the baseline loss function is subjected to compatibility constraints to obtain the degeneracy-free compatibility loss function of the new search model.

[0027] In some embodiments, the new search model further includes a new classification sub-model, and the model training module is further configured to:

[0028] The new training features are input into the new classification sub-model to obtain the classification result vector of the new training features;

[0029] The classification result vector, the new training features, and the label information are subjected to cross-entropy calculation to obtain the classification loss function of the new search model.

[0030] In some embodiments, the model deployment module is further configured to:

[0031] Determine the uncertainty value of each old image feature in the candidate image library;

[0032] Based on the uncertainty value of each old image feature, the candidate images corresponding to each old image feature are sorted to obtain the sorting result;

[0033] Based on the ranking results, the trained new search model is used to extract the image features of each candidate image in the candidate image library to obtain new image features.

[0034] In some embodiments, the model deployment module is further configured to:

[0035] The old image features are input into the classification sub-model of the trained new search model to obtain the classification result vector corresponding to each old image feature.

[0036] The uncertainty value of each old image feature is determined based on the classification result vector corresponding to each old image feature.

[0037] In some embodiments, the model deployment module is further configured to:

[0038] The category probabilities in the classification result vectors corresponding to each old image feature are sorted in descending order to obtain the sorting results;

[0039] The uncertainty value of each old image feature is determined based on the maximum probability value among the sorting results; or,

[0040] Based on the difference between the first and second probability values ​​in each sorting result, the uncertainty value of each old image feature is determined; or,

[0041] The entropy of each old image feature is determined based on the classification result vector corresponding to each old image feature, and the uncertainty value of each old image feature is determined based on the entropy corresponding to each old image feature.

[0042] In some embodiments, the device further includes:

[0043] The fourth acquisition module is used to receive a video publishing request and acquire the video data to be published carried in the video publishing request;

[0044] The frame extraction module is used to perform frame extraction processing on the video data to be published to obtain multiple target video frames;

[0045] The second determining module is used to extract image features of the multiple target video frames using the feature extraction sub-model in the trained classification model, obtain multiple video frame features, and determine the video features of the video data to be published based on the multiple video frame features.

[0046] The third determining module is used to determine the similarity between the video features and the video features of each published video in the video library;

[0047] The video publishing module is used to determine that there are no published videos similar to the video data to be published when there is no target similarity greater than the similarity threshold, and then publish the video data to be published.

[0048] This application provides a computer device, including:

[0049] Memory, used to store executable instructions;

[0050] The processor, when executing executable instructions stored in the memory, implements the search model update method provided in the embodiments of this application.

[0051] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the search model update method provided in this application.

[0052] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method for updating the search model provided in this application.

[0053] The embodiments of this application have the following beneficial effects:

[0054] When updating the search model in the search system, the following steps are taken: First, the old search model currently in use, the new search model to be deployed, and training data are obtained. The training data includes multiple training images and their corresponding label information. The new search model to be deployed is an untrained initial neural network model. Then, the old search model is used to extract features from the multiple training images to obtain old training features. The new search model is then trained using the training data and the old training features to obtain a degeneracy-free compatible training model. After that, the trained new search model is deployed to achieve hot refresh of the search system. Finally, the trained new search model is used to update the candidate image library and perform image search. Because the old image features are used when training the new search model, and degeneracy-free compatible training is performed, the model degradation phenomenon can be mitigated. The new model can be deployed immediately after training, thus improving the model deployment efficiency. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the network architecture of the search system provided in the embodiments of this application;

[0056] Figure 2 This is a schematic diagram of the structure of the server 400 provided in the embodiment of this application;

[0057] Figure 3 This is a schematic diagram illustrating an implementation flow of the search model update method provided in an embodiment of this application;

[0058] Figure 4 This is a schematic diagram illustrating another implementation flow of the search model update method provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram illustrating another implementation of the search model update method provided in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of model updates based on cold refresh / hot refresh.

[0061] Figure 7 A schematic diagram of model degradation provided for an embodiment of this application;

[0062] Figure 8 A schematic diagram illustrating the implementation process of the model update method provided in this application embodiment;

[0063] Figure 9This application provides an embodiment of the retrieval performance trend chart using the Google Landmark v2 test dataset during hot refresh.

[0064] Figure 10 A trend chart showing the retrieval performance during hot refresh using the Oxford test dataset is provided in this embodiment of the application.

[0065] Figure 11 A trend chart showing the retrieval performance during hot refresh using the Paris test dataset is provided in this embodiment of the application.

[0066] Figure 12 This is a trend chart showing the changes in different uncertainty metrics refresh strategies provided in the embodiments of this application during the hot refresh process;

[0067] Figure 13 This is another trend chart showing the retrieval performance during hot refresh using the Google Landmark v2 test dataset, provided as an embodiment of this application.

[0068] Figure 14 This is another trend chart showing the retrieval performance during hot refresh using the Oxford test dataset, provided as an embodiment of this application.

[0069] Figure 15 This is another trend chart showing the retrieval performance during hot refresh using the Paris test dataset, provided as an embodiment of this application.

[0070] Figure 16 This is another trend chart showing the different uncertainty metrics refresh strategies provided in the embodiments of this application during the hot refresh process;

[0071] Figure 17 This is a graph showing the performance trends of different methods in hot refresh model updates. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] In the implementation of this application, the collection and processing of relevant data should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0074] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0075] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0077] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0078] 1) Hot-refresh: After obtaining a trained new model, it is deployed online directly, and then the candidate image library is dynamically and gradually refreshed by the trained new model.

[0079] 2) Cold refresh: The trained new model is used to re-extract features from the candidate image library. The trained new model can only be deployed online after the feature update is complete.

[0080] 3) Backward Compatible Training (BCT) refers to the training of a new model that is directly compatible with the old model, meaning that the features extracted by the new model can be directly used to search in the database indexed by the old model.

[0081] 4) Mean Average Precision (mAP) is a performance metric for this class of algorithms that predict the location and category of a target.

[0082] In related technologies, hot updates to search systems are typically achieved using backward compatibility learning (BCT) or local distillation. BCT aims to allow direct comparison of new and old features extracted by the new and old models; specifically, BCT uses the old classifier to supervise the training of the new features. Local distillation was the first to investigate model degradation in classification models. This method achieves consistent model classification performance by assigning a larger weight to correctly classified samples.

[0083] While existing backward-compatible methods can achieve model updates based on "hot refresh," they ignore the model regression problem that occurs during the update process. Local distillation addresses the regression problem in image classification; directly transferring this method to image retrieval tasks only yields a suboptimal solution and fails to achieve significant performance gains.

[0084] Based on this, embodiments of this application provide a model update method, apparatus, device, and computer-readable storage medium, which adopts a degradation-free compatible training update mode to obtain a trained new model, and updates the image features in the image library sequentially based on an uncertainty metric during the hot refresh process, thereby achieving rapid performance convergence.

[0085] The following describes exemplary applications of the computer device provided in the embodiments of this application. The computer device provided in the embodiments of this application can be implemented as a server. Exemplary applications when the device is implemented as a server will be described below.

[0086] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the search system 100 provided in an embodiment of this application, as shown below. Figure 1 As shown, the search system 100 includes a terminal 200, a network 300, and a server 400. The terminal 200 is connected to the server 400 through the network 300, which can be a wide area network, a local area network, or a combination of both.

[0087] The terminal 200 can have various applications installed, such as instant messaging applications, shopping applications, and video viewing applications. The terminal 200 can use the instant messaging application to chat with friends and family via text, voice, and video; it can use the shopping application to shop online; and it can use the video viewing application to watch or post videos. In this embodiment, an online shopping search system is used as an example for explanation.

[0088] When a user shops online through the shopping application in terminal 200, they can search for products using keywords. If the exact keywords cannot be determined, they can search for products using images. In the server, each product for sale corresponds to at least one image. The server 400 stores an image database, corresponding to the candidate image library in other embodiments. This image database includes each image of the product for sale and the image features of each image. These image features are extracted based on the feature extraction sub-model in the current online search model. When the search model needs to be updated, in this embodiment, after obtaining the new search model to be deployed, the old training features of the training images are extracted using the current online old search model. Based on the old training features and training images, the new search model is trained using degradation-free compatibility training to obtain a trained new search model. The trained search model is then directly deployed online, and the image database is refreshed and searched using the trained search model. When server 400 receives the query image sent by terminal 200, it performs feature extraction based on the newly trained search model to obtain the query image features, determines the similarity between the query image features and the features of each candidate image in the image database, determines the search results based on the determined similarity, and returns the search results to terminal 200, which then presents the search results in a graphical interface.

[0089] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 200 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart TV, smart vehicle device, etc., but is not limited thereto. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0090] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 400 provided in this embodiment of the application. Figure 2 The server 400 shown includes at least one processor 410, at least one network interface 420, a bus system 430, and memory 440. The various components in the server 400 are coupled together via the bus system 430. It is understood that the bus system 430 is used to implement communication between these components. In addition to a data bus, the bus system 430 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2The general labeled all buses as Bus System 430.

[0091] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0092] The memory 440 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 440 may optionally include one or more storage devices physically located away from the processor 410.

[0093] The memory 440 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 440 described in this application embodiment is intended to include any suitable type of memory.

[0094] In some embodiments, memory 440 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0095] Operating system 441 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0096] The network communication module 442 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0097] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2An update device 443 for the search model stored in memory 440 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a first acquisition module 4431, a first extraction module 4432, a model training module 4433, and a model deployment module 4434. These modules are logically linked and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0098] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the update method of the search model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0099] To better understand the methods provided in the embodiments of this application, we will first explain artificial intelligence, its various branches, and the application fields involved in the methods provided in the embodiments of this application.

[0100] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0101] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. The solutions provided in this application mainly relate to machine learning techniques in artificial intelligence, which will be described below.

[0102] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0103] Artificial intelligence cloud services, also commonly known as AI as a Service (AIaaS), are a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them independently or as bundled services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through API interfaces. Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services. In the information recommendation method provided in this embodiment of the invention, the data scheduling and distribution system can be deployed and maintained through the AI ​​framework and AI infrastructure provided by the artificial intelligence cloud service.

[0104] The method for updating the search model provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0105] This application provides a method for updating a search model, applied to a server. Figure 3 This is a flowchart illustrating the search model update method provided in this application embodiment. The following is a summary of the process. Figure 3 The steps of the search model update method provided in the embodiments of this application will be described.

[0106] Step S101: Obtain the old search model currently in use online, the new search model to be deployed online, and the training data.

[0107] In practice, the server determines whether the model update conditions have been met. These conditions can be either receiving a model update command or receiving a new search model to be deployed. This new search model is a neural network model with default parameters. The existing online search model is a pre-trained neural network model. The old and new search models can be of the same type of neural network model or different types.

[0108] Training data can be obtained from large-scale public datasets, such as Google Landmark v2, while test data can be obtained from one or more public datasets, such as Google Landmark v2, Oxford, and Paris. The training data includes multiple training images and their corresponding labels. These labels can be the category of the training images, such as animals, plants, landscapes, or even cats, dogs, tigers, and squirrels.

[0109] Step S102: Use the old search model to extract features from the multiple training images to obtain old training features.

[0110] In this embodiment of the application, the old search model includes an old feature extraction sub-model and an old classification sub-model. Step S102 can be implemented by using the old feature extraction sub-model in the old search model to extract features from each training image, thereby obtaining the old training features of each training image.

[0111] Step S103: Use the training data and the old training features to perform degradation-free compatible training on the new search model to obtain a trained new search model.

[0112] Here, similar to the old search model, the new search model includes a new feature extraction sub-model and a new classification sub-model. In implementation, this step first uses the new feature extraction sub-model to extract features from each training image, obtaining new training features. Then, based on the new and old training features, a degeneracy-free compatible loss function is constructed for the feature extraction sub-model. A classification loss function is constructed based on the new training features and the label information in the training data. Finally, a joint loss function for the new search model is determined based on the degeneracy-free compatible loss function and the classification loss function. Based on this joint loss function and the training data, the new search model is trained using degeneracy-free compatibility, resulting in a well-trained new search model.

[0113] Because the old training features extracted by the old search model are used as part of the training data during the training of the new search model, and the old training features are also considered when constructing the loss function, it can be ensured that the new image features extracted by the new search model are comparable to the old image features. This ensures that the old image features extracted by the old search model can be correctly retrieved by the new search model before the features of the candidate images in the candidate feature library are updated, thus avoiding model degradation.

[0114] Step S104: Deploy the trained new search model online, and use the trained new search model to update the candidate image library and perform image search.

[0115] In this embodiment of the application, after obtaining the trained new search model, the trained new search model is deployed online. Then, the features of each candidate image in the candidate image library are updated using the trained new search model. When an image search request is obtained, the new search model is used to extract features from the search image, and the extracted features are compared with the features of the candidate images in the candidate image library to calculate similarity, thereby achieving hot refresh of the model.

[0116] In the search model update method provided in this application embodiment, when it is necessary to update the search model in the search system, the old search model currently in use, the new search model to be deployed, and training data are first obtained. The training data includes multiple training images and label information corresponding to each training image. The new search model to be deployed is an untrained initial neural network model. Then, the old search model is used to extract features from the multiple training images to obtain old training features. The new search model is then trained using the training data and the old training features to obtain a degeneracy-free compatible training, resulting in a trained new search model. After that, the trained new search model is deployed online to achieve hot refresh of the search system. Finally, the trained new search model is used to update the candidate image library and perform image search. Since the old image features are used when training the new search model, and degeneracy-free compatible training is performed, the model degradation phenomenon can be alleviated. The new model can be deployed online immediately after training, thus improving the model deployment efficiency.

[0117] In some embodiments, the new search model includes a new feature extraction sub-model. Correspondingly, the above step S103, "using the training data and the old training features to perform degradation-free compatible training on the new search model to obtain a trained new search model," can be achieved through the following steps:

[0118] Step S1031: Use the new feature extraction sub-model of the new search model to extract features from the multiple training images to obtain new training features.

[0119] Step S1032: Determine the loss function of the new search model using the old training features and the new training features.

[0120] In this embodiment, the loss function of the new search model is a joint loss function. It is called a joint loss function because it is obtained by weighting a non-degradation compatible loss function and a classification loss function. The non-degradation compatible loss function is determined by constructing a regularization term based on old and new training features, while the classification loss function is determined based on the new training features, the label information of the training images, and the cross-entropy loss function.

[0121] Step S1033: Based on the training data and the loss function, perform degradation-free compatible training on the new search model to obtain the trained new search model.

[0122] In implementing step S1033, the training images from the training data can be input into the new search model to obtain the predicted category information of the training images. Then, the difference between the predicted category information and the corresponding label information is determined, and this difference information is propagated to the new search model. The new search model is then trained using the loss function through backpropagation to obtain a trained search model. Since the loss function is determined based on the non-degradation compatible loss function and the classification loss function during training, it can be considered that the new search model has undergone non-degradation compatible training, ensuring that the trained new search model avoids model degradation and guarantees search accuracy.

[0123] In some embodiments, step S1032 above can be implemented by the following steps:

[0124] Step S321: Determine the degradation-free compatible loss function of the new search model using the old training features and the new training features.

[0125] Considering that the goal of compatibility learning is to ensure direct comparability between new and old features, metric learning is performed at the feature level when constructing the degeneracy-free compatibility loss function. This step can be achieved through the following steps:

[0126] Step S3211: Construct multiple new-old positive pairs, multiple new-new negative pairs, and multiple new-old negative pairs using the new training features and the old training features.

[0127] In this step, it can be implemented by constructing new-old positive pairs using the new training features and old training features obtained by the new feature extraction sub-model and the old feature extraction sub-model on the same training image; constructing new-old negative pairs using the new training features and old training features obtained by the new feature extraction sub-model and the old feature extraction sub-model on different training images; and constructing new-new negative pairs using the new training features and new training features obtained by the new feature extraction sub-model on different training images.

[0128] Step S3212: Construct a baseline loss function based on the first dot product result of the new-old positive example pair and the second dot product result of the new-old negative example pair.

[0129] In implementation, the first dot product result of each new-old positive example pair and the second dot product result of each new-old negative example pair are first determined, and then the baseline loss function is constructed based on formula (1-1).

[0130] (1-1);

[0131] in, This is the result of the first dot product. This is the result of the second dot product.

[0132] Step S3213: Apply compatibility constraints to the baseline loss function based on the third dot product result of the new-new negative example pair to obtain the degeneracy-free compatibility loss function of the new search model.

[0133] Model degradation occurs because the similarity between the query image features extracted by the new search model and the positive examples formed by the candidate library extracted by the old search model is less than the similarity between the query image features extracted by the new search model and the negative examples formed by the candidate library extracted by the new search model. Therefore, in order to improve the similarity between new and old positive example pairs and reduce the similarity between new and new negative example pairs, the baseline loss function is improved. In implementation, the new and new negative example pairs can be added as an additional regularization term to the denominator of the baseline loss function, which yields formula (1-2):

[0134] (1-2);

[0135] in, This is the result of the second dot product of the new negative pair.

[0136] By adding the constraint of new-new negative example pairs, the distance between new-old positive example pairs can be shortened, while the distance between new-new negative example pairs can be widened. This makes the similarity between new-old positive example pairs exceed the similarity between new-new negative example pairs, thereby alleviating the negative inversion (model degradation) problem that occurs during model update.

[0137] Step S322: Determine the classification loss function of the new search model using the new training features and the label information corresponding to each training image.

[0138] In implementing this step, the new training features can first be input into the new classification sub-model to obtain the classification result vector of the new training features. Then, the classification result vector, the new training features, and the label information are subjected to cross-entropy calculation according to formula (1-3) to obtain the classification loss function of the new search model:

[0139] (1-3);

[0140] in Represents the cross-entropy loss function. Image Corresponding category tags, This is the classification result vector.

[0141] Step S323: Construct the loss function of the new search model based on the degradation-free compatible loss function and the classification loss function.

[0142] In implementation, the weights corresponding to the degradation-free compatibility loss function and the classification loss function can be obtained first. Then, the degradation-free compatibility loss function and the classification loss function can be weighted and summed based on their respective weights to obtain the loss function of the new search model. The loss function of the new search model can then be used as the training and optimization target of the new search model.

[0143] Based on the foregoing embodiments, this application provides a method for updating a search model, applied to... Figure 1 The network architecture shown is Figure 4 This is a schematic diagram illustrating another implementation flow of the search model update method provided in this application embodiment, such as... Figure 4 As shown, the process includes:

[0144] Step S301: The server obtains the old search model currently in use online, the new search model to be deployed online, and training data.

[0145] The training data includes multiple training images and the label information corresponding to each training image;

[0146] In step S302, the server uses the old search model to extract features from the multiple training images to obtain old training features.

[0147] In step S303, the server uses the training data and the old training features to perform degradation-free compatible training on the new search model, thereby obtaining a trained new search model.

[0148] It should be noted that the implementation process of steps S301 to S303 is similar to that of steps S101 to S103, and the implementation process of steps S101 to S103 can be referred to.

[0149] Step S304: The server determines the uncertainty value of each old image feature in the candidate image library.

[0150] In this embodiment, the uncertainty value of an image feature reflects whether the image feature is easily categorized; the higher the uncertainty value, the greater the difficulty of identification. When determining the uncertainty value of an old image feature, it can be determined based on at least one of the minimum confidence level, the interval method, and the entropy method given in step S3042 below.

[0151] Step S305: The server sorts the candidate images corresponding to each old image feature based on the uncertainty value of each old image feature, and obtains the sorting result.

[0152] In implementation, candidate images can be sorted in descending order of uncertainty value to obtain the sorting result; alternatively, candidate images can be sorted in ascending order of uncertainty value to obtain the sorting result.

[0153] Step S306: Based on the ranking result, the server uses the trained new search model to extract image features of each candidate image in the candidate image library to obtain new image features.

[0154] To accelerate the convergence of accuracy during feature updates in the candidate image library, the feature refresh order should follow the principle of "worst first," meaning that features that are difficult to distinguish should be refreshed first, or features with higher uncertainty values ​​should be refreshed first. Therefore, in this step, if the sorting result is obtained by sorting the uncertainty values ​​from largest to smallest, then in step S306, the image features of each candidate image in the candidate image library are extracted using the trained new search model according to the sorting result; if the sorting result is obtained by sorting the uncertainty values ​​from smallest to largest, then in step S306, the image features of each candidate image in the candidate image library are extracted using the trained new search model in reverse order of the sorting result.

[0155] Step S307: The terminal sends a video publishing request to the server.

[0156] The video publishing request carries the video data to be published. This video data can be recorded by the terminal itself, edited from downloaded video data, or created by editing multiple images into a video. The above is merely an illustrative example of the process for obtaining the video data to be published.

[0157] Step S308: The server receives the video publishing request and obtains the video data to be published carried in the video publishing request.

[0158] Step S309: The server performs frame extraction processing on the video data to be published to obtain multiple target video frames.

[0159] This step can be implemented by performing frame extraction at equal intervals or by extracting keyframes from the video data to be published.

[0160] In step S310, the server uses the feature extraction sub-model in the trained classification model to extract image features of the multiple target video frames, obtains multiple video frame features, and determines the video features of the video data to be published based on the multiple video frame features.

[0161] Determining the video features of the video data to be published based on multiple video frame features can be achieved by arithmetically averaging the features of multiple video frames, or by weighted averaging the features of multiple video frames. The weights of the video frame features can be determined by the frame type of the corresponding target video frame. For example, the target video frame of an I-frame has a higher weight than the target video frame of a P-frame.

[0162] Step S311: The server determines the similarity between the video features and the video features of each published video in the video library.

[0163] In the video publishing scenario shown in this application embodiment, the video library corresponds to the candidate image library in other embodiments. The video library stores multiple published videos and their video features. The video features of the published videos can be determined using a new search model or an old search model.

[0164] In implementing step S311, the similarity can be determined by calculating the Hamming distance between the video features of the video to be published and the video features of each published video, with the larger the Hamming distance, the lower the similarity.

[0165] In step S312, the server determines whether there is a target similarity greater than the similarity threshold.

[0166] When there is no target similarity greater than the similarity threshold, it means that there is no published video data similar to the video data to be published, and the process proceeds to step S313; when there is a target similarity greater than the similarity threshold, it means that there is published video data similar to the video data to be published, and the process proceeds to step S315.

[0167] In step S313, the server determines that there is no published video similar to the video data to be published, and then publishes the video data to be published.

[0168] If it is determined that there are no already published videos similar to the video data to be published, then the video data to be published will be published.

[0169] Step S314: The server sends a notification message to the terminal indicating successful publication.

[0170] In step S315, the server determines that there is a published video that is similar to the video data to be published.

[0171] Step S316: The server sends a notification message to the terminal indicating that the publication failed.

[0172] In some embodiments, the notification message indicating a failed publication may also include a link to view a previously published video that is similar to the video data to be published.

[0173] The search model update method provided in this application is applied to the scenario of terminal video publishing. The search model is used to search for whether there are published videos similar to the video to be published. In implementation, when the server determines that a model update is needed, it obtains the old search model currently in use online, the new search model to be deployed online, and training data. Then, it uses the old search model to extract features from the multiple training images to obtain old training features. Then, it uses the training data and the old training features to perform degradation-free compatibility training on the new search model to obtain a trained new search model. After that, the trained new search model is deployed online to realize hot refresh of the search system, thereby reducing the time cost of model upgrade. In the process of updating the candidate image library features using the trained new search model, the uncertainty value of the candidate image features in each candidate image library is first determined. The update order is determined based on the uncertainty value, and the candidate image features with high uncertainty are updated first. This can ensure the accuracy of target search using the new search model and improve the search performance of the search model. After the server receives the video to be published from the terminal, it first obtains the video features of the video to be published, then determines the similarity between the video features and the video features of already published videos, and finally determines whether there are any already published videos similar to the video to be published based on the similarity, thereby determining whether to publish the video to be published, reducing the duplication of video files in the video library and protecting the copyright of already published videos.

[0174] It should be noted that, in this embodiment of the application, when a terminal publishes video data to be published, the server will only publish the video if it determines that there are no similar published videos. In some embodiments, after receiving the video to be published, the server may perform a legality check on the video data to be published and, if it determines that the legality conditions are met, publish the video and store it in the video library. Then, a new search model will deduplicate the published videos in the video library. In actual implementation, the similarity between the video features of the published videos can be calculated. If the similarity between two or more published videos is higher than the similarity threshold, the published videos that need to be removed are determined based on the publication time of the two or more published videos. The selected published videos are then removed from the video library, and a removal notification message is sent to the terminal that published the selected published videos.

[0175] In some embodiments, step S304, "determining the uncertainty value of each old image feature in the candidate image library," can be achieved through the following steps:

[0176] Step S3041: Input the various old image features into the classification sub-model of the trained new search model to obtain the classification result vector corresponding to each old image feature.

[0177] Assuming there are N categories, the dimension of the classification result vector is 1*N. The i-th element in the classification result vector represents the probability of belonging to the i-th category, which is a real number between 0 and 1, and the sum of the N probability values ​​in the classification result vector is 1.

[0178] Step S3042: Determine the uncertainty value of each old image feature based on the classification result vector corresponding to each old image feature.

[0179] In practical applications, when determining the uncertainty value of old image features, there are at least three calculation methods:

[0180] The first calculation method involves determining the entropy of each old image feature based on the classification result vector corresponding to each old image feature, and then determining the uncertainty value of each old image feature based on the entropy corresponding to each old image feature.

[0181] In implementation, the uncertainty value of each old image feature can be determined using formula (1-4):

[0182] (1-4);

[0183] in, Representing old image features f g The classification result vector, This represents the probability of the k-th ranked category.

[0184] The second calculation method involves sorting the category probabilities in the classification result vectors corresponding to each old image feature in descending order to obtain each sorting result; and determining the uncertainty value of each old image feature based on the maximum probability value in each sorting result.

[0185] In implementation, the uncertainty level of the old image feature can be determined by subtracting the maximum probability value from 1. In other words, the smaller the maximum probability value of the old image feature, the greater the uncertainty level.

[0186] The third calculation method determines the uncertainty value of each old image feature based on the difference between the first probability value and the second probability value in each sorting result.

[0187] In implementation, the uncertainty value of the old image feature can be obtained by subtracting the difference corresponding to the old image feature from 1. In other words, the smaller the difference between the maximum probability value and the second highest probability value of an old image feature, the more difficult it is to distinguish, and therefore the greater the uncertainty value.

[0188] Based on the foregoing embodiments, this application provides a method for updating a search model, applied to... Figure 1 The network architecture shown is Figure 5 This is a schematic diagram illustrating another implementation of the search model update method provided in the embodiments of this application, such as... Figure 5 As shown, the process includes:

[0189] In step S401, when the server determines that a model update is needed, it obtains the old search model currently in use online, the new search model to be deployed online, and the training data.

[0190] The training data includes multiple training images and the label information corresponding to each training image.

[0191] In step S402, the server uses the old search model to extract features from the multiple training images to obtain old training features.

[0192] Step S403: The server uses the training data and the old training features to perform degradation-free compatible training on the new search model to obtain a trained new search model.

[0193] Step S404: The server determines the uncertainty value of each old image feature in the candidate image library.

[0194] Step S405: The server sorts the candidate images corresponding to each old image feature based on the uncertainty value of each old image feature, and obtains the sorting result.

[0195] Step S406: Based on the ranking result, the server uses the trained new search model to extract image features of each candidate image in the candidate image library to obtain new image features.

[0196] The implementation process of steps S401 to S406 is similar to that of steps S301 to S306. You can refer to the implementation process of steps S101 to S103.

[0197] In step S407, the terminal sends a query request to the server.

[0198] The query request carries a query image, and the query request can be used to search for product information that matches the query image.

[0199] In step S408, the server receives the query request and retrieves the query image carried in the query request.

[0200] Step S409: The server uses the feature extraction sub-model in the trained search model to extract the query image features of the query image.

[0201] In step S410, the server determines the similarity between the query image features and the candidate image features of each listed product image in the candidate image library.

[0202] When determining the corresponding product information by querying images, the candidate image library stores images of each listed product and their candidate image features. This step, similar to step S311 above, can involve determining the Hamming distance between the query image features and the candidate image features, using the Hamming distance to determine a similarity value, with a smaller Hamming distance indicating lower similarity.

[0203] In some embodiments, the absolute values ​​of the query image features and the corresponding feature values ​​of each candidate image feature can be subtracted, and then the mean of the absolute values ​​of the differences corresponding to each feature value can be calculated to obtain the average difference. The smaller the average difference, the higher the similarity.

[0204] Step S411: The server determines the query results based on each similarity score.

[0205] In this step, the server sorts the listed products based on each similarity level, and obtains the sorting results. The listed products with higher similarity to the query image are sorted higher. Then, the product information corresponding to the top M products in the sorting results is determined as the query results.

[0206] In step S412, the server returns the query result to the terminal.

[0207] In some embodiments, after receiving the query results, the terminal displays multiple product information carried in the query results on its own display interface. This information may include product images, product names, links to view details, etc., enabling the terminal user to determine the product they need from the multiple product information carried in the query results.

[0208] The search model update method provided in this application is applied to the scenario of terminal video publishing. The search model is used to search for whether there are published videos similar to the video to be published. In implementation, when the server determines that a model update is needed, it obtains the old search model currently in use online, the new search model to be deployed online, and training data. Then, it uses the old search model to extract features from the multiple training images to obtain old training features. Then, it uses the training data and the old training features to perform degradation-free compatibility training on the new search model to obtain a trained new search model. After that, the trained new search model is deployed online to realize hot refresh of the search system, thereby reducing the time cost of model upgrade. In the process of updating the candidate image library features using the trained new search model, the uncertainty value of the candidate image features in each candidate image library is first determined. The update order is determined based on the uncertainty value, and the candidate image features with high uncertainty are updated first. This can ensure the accuracy of target search using the new search model and improve the search performance of the search model.

[0209] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0210] In this embodiment, image search is used as an example to illustrate the model update method. Image search refers to the server's ability to search a large-scale candidate image library (Gallery, denoted as...). Correctly retrieve and query images (Query, denoted as) in ) Images that have similar content or objects.

[0211] Assuming the search system (corresponding to the search model in other embodiments) includes a feature extractor (corresponding to the feature extraction sub-model in other embodiments) and a classifier (corresponding to the classification sub-model in other embodiments), in this embodiment, the performance of the retrieval system, i.e., the mean average precision (mAP), is denoted as... The new / old image feature extractor is denoted as , The goal of compatible training can be expressed using inequality (2-1):

[0212] (2-1);

[0213] in, This indicates the performance of the search system when both the query image features and the candidate image library features are extracted by the old image feature extractor; This indicates the performance of the search system when the features of the query image and the candidate image library are extracted by the new and old feature extractors, respectively. This indicates the performance of the search system when both the features of the query image and the features of the candidate image library are extracted by the new image feature extractor.

[0214] In this embodiment of the application, the ideal hot refresh model update should satisfy the constraint shown in inequality (2-2):

[0215] (2-2);

[0216] Figure 6 This is a schematic diagram of model updates based on cold refresh / hot refresh, such as... Figure 6 As shown, in the cold refresh process, the new model is deployed only after the offline feature refresh is completed (i.e., refreshed to 100%), while in the hot refresh process, the new model is deployed first, and the new model performs online feature refresh until the refresh is complete.

[0217] During hot refresh, some query images that could be correctly retrieved by the old model may fail to be correctly recalled by the new model, a phenomenon known as model degradation. Analysis revealed that this degradation is due to negative inversion; that is, the similarity between the query image features extracted by the new model and the candidate image features extracted by the old model (new-to-old positive pairs) is less than the similarity between the query image features extracted by the new model and the candidate image features extracted by the new model (new-to-new negative pairs). This process can be formalized as inequality (2-3):

[0218] (2-3);

[0219] in, This indicates the correct candidate library sample. represents an incorrect candidate library sample, and <·, ·> represents the vector inner product. Figure 7 This is a schematic diagram of model degradation provided in an embodiment of this application, wherein, Figure 7 In the diagram, point O represents the mAP of the old model; point A represents the mAP that is compatible with the old model after updating to the new model; point C represents the mAP that is negatively inverted during the process of refreshing the features in the candidate image library by the new model; and point B represents the mAP that is compatible with the new model after the update. Figure 7 It can be seen that during the refresh process, the mAP will be lower than before the refresh, which means that model degradation has occurred.

[0220] The model update method provided in the embodiments of this application will be described below. Figure 8 This is a schematic diagram illustrating the implementation process of the model update method provided in the embodiments of this application, as shown below. Figure 8As shown, to obtain a new compatible model through training, given an input image, features are first extracted from the training image using both the new and old feature extractors, resulting in new training features 801 and old training features 802, respectively. Then, a degradation-free compatible loss function is constructed using the new and old training features 801 and 802, and a classification loss function is constructed using the new training features, thereby achieving the hot-refresh model upgrade process. Furthermore, during the upgrade process, the distance between new and old positive example pairs is shortened, while the distance between new and new negative example pairs and new and old negative example pairs is widened.

[0221] When constructing a degradation-free compatible loss function using the obtained new and old training features, considering that the goal of compatible learning is to ensure direct comparability between the new and old features, this problem can be transformed into a feature-level metric learning problem. In this embodiment, the InfoNCE loss function can be selected as the baseline model, defined as shown in formula (2-4):

[0222] (2-4);

[0223] in, It's a hyperparameter - temperature. To train new training features for image x, To train the old training features of image x, For the new-old positive example pair, This is the inner product of the new and old negative pairs.

[0224] To mitigate the performance degradation caused by negative inversion while achieving feature compatibility, this embodiment improves the baseline model by adding new-new negative example pairs as an additional regularization term, resulting in the regression-free compatibility regularization loss function shown in formula (2-5):

[0225] (2-5);

[0226] in, This is a new-new negative example pair. Through this constraint, the similarity between new-old positive example pairs will exceed the similarity between new-new negative example pairs, thereby alleviating the negative inversion (model degradation) problem that occurs during model update.

[0227] In addition to constraining the compatibility between features, a regularization term adapted to the retrieval task is also needed to improve the feature representation capability of the model. In this embodiment, the cross-entropy loss function is used as the classification function, defined as shown in formula (2-6):

[0228] (2-6);

[0229] in Represents the cross-entropy loss function. Image Corresponding category tags, This is the classification vector obtained using the classifier in the new model.

[0230] After determining the degradation-free compatibility loss function and the classification function, the loss function of the entire new model can be expressed by formula (2-7):

[0231] (2-7);

[0232] in, This is a hyperparameter.

[0233] After determining the loss function of the entire model, the new model can be trained by backpropagation based on the training images, their labels, and the loss function. This will result in a trained new model, which can then be deployed online for hot refresh of the candidate image library.

[0234] In the process of hot-refreshing the candidate image library using a new model, in order to further accelerate model convergence and improve accuracy, this application proposes that the refresh order of features should follow the "poor-first" principle, that is, features that are difficult to distinguish should be refreshed first. Faced with massive amounts of industry data, a lightweight and fast way to determine the priority of feature refresh is needed. Therefore, this application proposes a refresh strategy based on uncertainty. In practical applications, it is believed that the predicted probability of a feature for each category by the classifier to a certain extent characterizes the "goodness" or "badness" of that feature. After compatible training, it is assumed that the new and old feature spaces can be directly compared, so the classifier of the new model can be used to predict the category probability of the old features. Although the predicted categories of the new and old models may not overlap, it can still be assumed that "good features" should have sharper and sparser category probabilities. In this application, features with higher uncertainty scores are given higher refresh priority. In actual implementation, the uncertainty of each old image feature in the candidate image library can be determined in at least the following three ways:

[0235] The first type, the minimum confidence level, can be determined using formula (2-8):

[0236] (2-8);

[0237] Second, the interval method can be determined using formula (2-9):

[0238] (2-9);

[0239] Third, the entropy method can be determined using formula (2-10):

[0240] (2-10);

[0241] in, Representing old image features f g The classification result vector, This represents the probability of the k-th ranked category.

[0242] The degradation-free compatible training method proposed in this application can effectively achieve hot-refresh model updates, alleviate model degradation, and significantly shorten the model deployment waiting period. Validation was performed on three public datasets: Google Landmark v2, Oxford, and Paris.

[0243] In this application, the following three different types of model training data were studied, covering most scenarios in practical applications:

[0244] First, Expansion: The new training dataset is an expansion of the old training dataset. For example, 30% of the images in Google Landmark can be randomly selected as the training set for the old model, and 100% of the images can be used as the training set for the new model.

[0245] Second, Open-data: The new and old training sets do not overlap, but contain the same categories.

[0246] Third, Open-class: The new and old training sets have no duplicate classes. 30% of the classes are randomly selected for training the old model, and the remaining 70% of the classes are used for training the new model.

[0247] In actual implementation, the collection information for various types of datasets is shown in Table 1:

[0248] Table 1. Information Collection Table for Different Types of Datasets

[0249]

[0250] Evaluation Metric: Mean Precision (mAP) is used to measure the performance of the retrieval system. Here, we use mAP@100 on Google Landmark and mAP@10 on Oxford and Paris, where mAP@100 represents the mean precision of the top 100 retrieved instances. Because mAP measures overall performance, which is affected by both positive and negative flip rates, this application defines another metric, Negative Flip Rate (NFR), to measure the degree of model degradation during hot refresh, defined as in formula (2-11):

[0251] (2-11);

[0252] in, This represents the set of query images that can be correctly recalled from the old candidate library among the first k candidate images; This represents the set of query images that could not be correctly recalled from the first k candidate images.

[0253] Regarding the structure of the feature extractor, two combinations are compared in the embodiments of this application:

[0254] The first combination, R50-R101: ResNet-50 is used for the old model, and ResNet-101 is used for the new model.

[0255] The second combination, R50-R50: Both the new and old models use the ResNet-50 structure.

[0256] All models were trained using the Google Landmark v2 training dataset and tested on three test datasets: Google Landmark v2, Oxford, and Paris. The results of updating the model according to the first combination of R50-R101 are as follows: Figures 9 to 11 As shown, where, Figure 9 The image provided in this application illustrates the performance change trend of retrieval during hot refresh using the Google Landmark v2 test dataset. Figure 10 The performance trend chart of retrieval during hot refresh using the Oxford test dataset provided in this embodiment of the application is as follows. Figure 11 The performance change trend chart of retrieval during hot refresh using the Paris test dataset provided in the embodiments of this application is obtained through... Figures 9 to 11 It can be seen that the new model obtained by using the model update method provided in the embodiments of this application has a better mAP than the vanilla model, and the new model has a lower NFR than the vanilla model.

[0257] Figure 12 This is a trend chart showing the changes in different uncertainty metrics during the hot refresh process using the refresh strategies provided in the embodiments of this application. Figure 12 (a) shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Oxford dataset using the extended collection type. Figure 12 (b) shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model was updated by collecting training data in the Oxford dataset using the open data collection type. Figure 12 (c) In the Oxford dataset, after collecting training data using the open-class collection type for model updates, the trend chart of the change in refresh order is determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy. Figure 12 (d) in the figure shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Pairs 6k dataset using the extended collection type. Figure 12 (e) in the figure shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Pairs 6k dataset using the open data collection type. Figure 12 (f) in the figure shows the trend of the refresh order determined by random refresh strategy, minimum confidence strategy, interval strategy and entropy strategy after the training data is collected in the Pairs 6k dataset through open class collection type for model update.

[0258] The result of updating the model according to the second combination R50-R50 is as follows: Figures 13 to 15 As shown, where, Figure 13 This is another trend chart showing the retrieval performance during hot refresh using the Google Landmark v2 test dataset, provided as an embodiment of this application. Figure 14 This is another trend chart showing the retrieval performance during hot refresh using the Oxford test dataset, provided as an embodiment of this application. Figure 15 This is another trend chart showing the retrieval performance during hot refresh using the Paris test dataset, provided in an embodiment of this application. Figures 13 to 15 It can be seen that the new model obtained by using the model update method provided in the embodiments of this application has a better mAP than the vanilla model, and the new model has a lower NFR than the vanilla model.

[0259] Figure 16This is another trend chart showing the different uncertainty metrics used in the hot refresh process according to the embodiments of this application. Figure 16 (a) shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Oxford dataset using the extended collection type. Figure 16 (b) shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model was updated by collecting training data in the Oxford dataset using the open data collection type. Figure 16 (c) In the Oxford dataset, after collecting training data using the open-class collection type for model updates, the trend chart of the change in refresh order is determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy. Figure 16 (d) in the figure shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Pairs 6k dataset using the extended collection type. Figure 16 (e) in the figure shows the trend of the refresh order determined by the random refresh strategy, minimum confidence strategy, interval strategy, and entropy strategy after the model is updated by collecting training data in the Pairs 6k dataset using the open data collection type. Figure 16 (f) in the figure shows the trend of the refresh order determined by random refresh strategy, minimum confidence strategy, interval strategy and entropy strategy after the training data is collected in the Pairs 6k dataset through open class collection type for model update.

[0260] Figure 17 This is a graph showing the performance trends of different methods in hot refresh model updates (model structure uses R50-R101). Figure 17 It can be seen that the mAP change trends of the old model, the new model obtained by backward compatibility training, the new model obtained by local distillation, the new model obtained by the model update method provided in the embodiments of this application, and the new model obtained by the model update method and backward compatibility training provided in the embodiments of this application are all obtained by training data obtained from the training data in the three training datasets. Among them, the performance of the new model obtained by the model update method and backward compatibility training provided in the embodiments of this application is the best.

[0261] In this embodiment, compatible learning is used in the retrieval system to achieve hot-refresh model upgrades, enabling immediate deployment of the new model. This saves significant time compared to traditional cold-refresh model upgrades. Furthermore, the model update is based on a degradation-free compatible training update mode, which can alleviate model degradation. Additionally, an uncertainty metric algorithm is applied to feature updates, enabling rapid performance convergence during the feature update process.

[0262] The following description continues to illustrate the exemplary structure of the search model update device 443 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module in the search model update device 443 stored in memory 440 may include:

[0263] The first acquisition module 4431 is used to acquire the old search model currently in use online, the new search model to be deployed online, and training data. The training data includes multiple training images and label information corresponding to each training image.

[0264] Feature extraction module 4432 is used to extract features from the multiple training images using the old search model to obtain old training features;

[0265] The model training module 4433 is used to perform degradation-free compatible training on the new search model using the training data and the old training features to obtain a trained new search model.

[0266] The model deployment module 4434 is used to deploy the trained new search model online and use the trained new search model to update the candidate image library and perform image search.

[0267] In some embodiments, the new search model includes a new feature extraction sub-model, and the model training module is further configured to:

[0268] The new feature extraction sub-model of the new search model is used to extract features from the multiple training images to obtain new training features;

[0269] The loss function of the new search model is determined using the old and new training features.

[0270] The new search model is trained using the new training features, the old training features, and the loss function to obtain a degeneracy-free compatible search model.

[0271] In some embodiments, the model training module is further configured to:

[0272] The degradation-free compatible loss function of the new search model is determined using the old and new training features.

[0273] The classification loss function of the new search model is determined using the new training features and the label information corresponding to each training image;

[0274] The loss function of the new search model is constructed based on the degradation-free compatibility loss function and the classification loss function.

[0275] In some embodiments, the model training module is further configured to:

[0276] Construct multiple new-old positive example pairs, multiple new-new negative example pairs, and multiple new-old negative example pairs using the new training features and the old training features;

[0277] A baseline loss function is constructed based on the first dot product result of the new-old positive example pair and the second dot product result of the new-old negative example pair.

[0278] Based on the third dot product result of the new-new negative example pair, the baseline loss function is subjected to compatibility constraints to obtain the degeneracy-free compatibility loss function of the new search model.

[0279] In some embodiments, the new search model further includes a new classification sub-model, and the model training module is further configured to:

[0280] The new training features are input into the new classification sub-model to obtain the classification result vector of the new training features;

[0281] The classification result vector, the new training features, and the label information are subjected to cross-entropy calculation to obtain the classification loss function of the new search model.

[0282] In some embodiments, the model deployment module is further configured to:

[0283] Determine the uncertainty value of each old image feature in the candidate image library;

[0284] Based on the uncertainty value of each old image feature, the candidate images corresponding to each old image feature are sorted to obtain the sorting result;

[0285] Based on the ranking results, the trained new search model is used to extract the image features of each candidate image in the candidate image library to obtain new image features.

[0286] In some embodiments, the model deployment module is further configured to:

[0287] The old image features are input into the classification sub-model of the trained new search model to obtain the classification result vector corresponding to each old image feature.

[0288] The uncertainty value of each old image feature is determined based on the classification result vector corresponding to each old image feature.

[0289] In some embodiments, the model deployment module is further configured to:

[0290] The category probabilities in the classification result vectors corresponding to each old image feature are sorted in descending order to obtain the sorting results;

[0291] The uncertainty value of each old image feature is determined based on the maximum probability value among the sorting results; or,

[0292] Based on the difference between the first and second probability values ​​in each sorting result, the uncertainty value of each old image feature is determined; or,

[0293] The entropy of each old image feature is determined based on the classification result vector corresponding to each old image feature, and the uncertainty value of each old image feature is determined based on the entropy corresponding to each old image feature.

[0294] In some embodiments, the device further includes:

[0295] The fourth acquisition module is used to receive a video publishing request and acquire the video data to be published carried in the video publishing request;

[0296] The frame extraction module is used to perform frame extraction processing on the video data to be published to obtain multiple target video frames;

[0297] The second determining module is used to extract image features of the multiple target video frames using the feature extraction sub-model in the trained classification model, obtain multiple video frame features, and determine the video features of the video data to be published based on the multiple video frame features.

[0298] The third determining module is used to determine the similarity between the video features and the video features of each published video in the video library;

[0299] The video publishing module is used to determine that there are no published videos similar to the video data to be published when there is no target similarity greater than the similarity threshold, and then publish the video data to be published.

[0300] It should be noted that the description of the search model update device in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment of this application for understanding.

[0301] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the search model update method described above in this application.

[0302] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the search model update method provided in this application, for example... Figure 3 , Figure 4 and Figure 5 The update method for the search model is shown.

[0303] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0304] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0305] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0306] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0307] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training a search model, characterized in that, The method comprises: obtaining an old search model currently used online, a new search model to be deployed online, and training data, the training data comprising a plurality of training images and label information corresponding to each training image; extracting features of the plurality of training images using the old search model to obtain old training features; extracting features of the plurality of training images using a new feature extraction submodel of the new search model to obtain new training features; constructing a plurality of new-old positive pairs, a plurality of new-new negative pairs, and a plurality of new-old negative pairs using the new training features and the old training features; constructing a benchmark loss function based on a first dot product result of the new-old positive pairs and a second dot product result of the new-old negative pairs; performing compatibility constraint on the benchmark loss function based on a third dot product result of the new-new negative pairs to obtain a non-degradation compatibility loss function of the new search model; determining a classification loss function of the new search model using the new training features and the label information corresponding to each training image; constructing a loss function of the new search model based on the non-degradation compatibility loss function and the classification loss function; training the new search model based on the new training features, the old training features, and the loss function of the new search model to obtain a trained new search model; deploying the trained new search model online, and performing candidate image library updating and image searching using the trained new search model.

2. The method of claim 1, wherein, The new search model further comprises a new classification submodel, and the determination of the classification loss function of the new search model using the new training features and the label information corresponding to each training image comprises: inputting the new training features into the new classification submodel to obtain a classification result vector of the new training features; performing cross-entropy operation on the classification result vector, the new training features, and the label information to obtain the classification loss function of the new search model.

3. The method according to any one of claims 1 to 2, characterized in that, The candidate image library updating using the trained new search model comprises: determining uncertainty degree values of each old image feature in the candidate image library; sorting candidate images corresponding to each old image feature based on the uncertainty degree values of each old image feature to obtain a sorting result; extracting image features of each candidate image in the candidate image library based on the sorting result using the trained new search model to obtain new image features.

4. The method of claim 3, wherein, The determination of the uncertainty degree values of each old image feature in the candidate image library comprises: inputting each old image feature into a classification submodel in the trained new search model to obtain a classification result vector corresponding to each old image feature; determining the uncertainty degree values of each old image feature based on the classification result vector corresponding to each old image feature.

5. The method of claim 4, wherein, The determination of the uncertainty degree values of each old image feature based on the classification result vector corresponding to each old image feature comprises: sorting class probabilities in the classification result vector corresponding to each old image feature in descending order to obtain each sorting result; determine the uncertainty degree value of the respective old image feature based on a maximum probability value in the respective ranking result; or determine the uncertainty degree value of the respective old image feature based on a difference between a first probability value and a second probability value in the respective ranking result; or determine the uncertainty degree value of the respective old image feature based on an entropy of the respective old image feature corresponding to the classification result vector.

6. The method according to any one of claims 1 to 2, characterized in that, The method further comprises: receiving a video publishing request and obtaining to-be-published video data carried in the video publishing request; performing frame extraction processing on the to-be-published video data to obtain a plurality of target video frames; extracting image features of the plurality of target video frames by using a feature extraction sub-model in the trained new search model to obtain a plurality of video frame features, and determining a video feature of the to-be-published video data based on the plurality of video frame features; determining a similarity between the video feature and video features of each published video in a video library; when there is no target similarity greater than a similarity threshold value, determining that there is no published video similar to the to-be-published video data, and publishing the to-be-published video data.

7. An update apparatus of a search model, characterized by, The device comprises: a first obtaining module configured to obtain an old search model currently used online, a new search model to be deployed online, and training data, wherein the training data comprises a plurality of training images and label information corresponding to each training image; a feature extraction module configured to extract features of the plurality of training images by using the old search model to obtain old training features; a model training module configured to extract features of the plurality of training images by using a new feature extraction sub-model of the new search model to obtain new training features, construct a plurality of new-old positive example pairs, a plurality of new-new negative example pairs, and a plurality of new-old negative example pairs by using the new training features and the old training features, construct a benchmark loss function based on a first dot product result of the new-old positive example pairs and a second dot product result of the new-old negative example pairs, perform compatibility constraint on the benchmark loss function based on a third dot product result of the new-new negative example pairs to obtain a non-degradation compatibility loss function of the new search model, determine a classification loss function of the new search model by using the new training features and the label information corresponding to each training image, construct a loss function of the new search model based on the non-degradation compatibility loss function and the classification loss function, and train the new search model based on the new training features, the old training features, and the loss function of the new search model to obtain a trained new search model; a model deployment module configured to deploy the trained new search model online and perform update of a candidate image library and image search by using the trained new search model.

8. The apparatus of claim 7, wherein, The new search model further comprises a new classification sub-model, and the model training module is further configured to: input the new training features into the new classification sub-model to obtain a classification result vector of the new training features. The classification result vector, the new training feature and the label information are subjected to cross entropy operation to obtain a classification loss function of the new search model.

9. The apparatus of any one of claims 7-8, wherein, The model deployment module is further configured to: determine uncertainty degree values of each old image feature in the candidate image library; sort candidate images corresponding to each old image feature based on the uncertainty degree values of each old image feature, to obtain a sorting result; extract image features of each candidate image in the candidate image library based on the sorting result using the trained new search model, to obtain new image features.

10. The apparatus of claim 9, wherein, When determining the uncertainty degree values of each old image feature in the candidate image library, the model deployment module is specifically configured to: input each old image feature into a classification sub-model in the trained new search model, to obtain a classification result vector corresponding to each old image feature; determine the uncertainty degree values of each old image feature based on the classification result vector corresponding to each old image feature.

11. The apparatus of claim 10, wherein, When determining the uncertainty degree values of each old image feature based on the classification result vector corresponding to each old image feature, the model deployment module is specifically configured to: sort class probabilities in the classification result vector corresponding to each old image feature in descending order, to obtain each sorting result; determine the uncertainty degree values of each old image feature based on maximum probability values in the each sorting result; or determine the uncertainty degree values of each old image feature based on a difference between a first probability value and a second probability value in the each sorting result; or determine the entropy of each old image feature based on the classification result vector corresponding to each old image feature, and determine the uncertainty degree values of each old image feature based on the entropy corresponding to each old image feature.

12. The apparatus of any one of claims 7 to 8, wherein, The device further includes: a video request receiving module configured to receive a video publishing request and obtain to-be-published video data carried in the video publishing request; a frame extraction module configured to perform frame extraction processing on the to-be-published video data to obtain a plurality of target video frames; a video feature determination module configured to extract image features of the plurality of target video frames using a feature extraction sub-model in the trained classification model to obtain a plurality of video frame features, and determine a video feature of the to-be-published video data based on the plurality of video frame features; a similarity determination module configured to determine similarities between the video feature and video features of each published video in a video library; a video publishing module configured to determine that there is no published video similar to the to-be-published video data when there is no target similarity greater than a similarity threshold value, and publish the to-be-published video data.

13. A computer device, comprising: The computer device includes: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method in any one of claims 1 to 6.

14. A computer-readable storage medium storing executable instructions, the instructions being executable to: The executable instructions are executed by the processor to implement the method in any one of claims 1 to 6.

15. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the method in any one of claims 1 to 6.

Images