Feature fusion model processing, sample retrieval method, device and computer equipment

Abstract

The present application relates to a feature fusion model processing method, device, computer equipment, storage medium and computer program product, as well as a sample method, device, computer equipment, storage medium and computer program product. The method includes: obtaining semantic features and non-semantic features corresponding to training samples; taking at least one of the semantic features and non-semantic features as candidate features, and determining the activated feature components from the feature components contained in each candidate feature; performing statistics on the activated feature components belonging to the same feature bit, and determining the activation degree of each feature bit according to the statistical results; setting the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each candidate feature to inactive values, respectively, to obtain each target feature; based on each target feature and using the feature fusion model to fuse, to obtain each fused feature; training the feature fusion model based on each fused feature. The use of this method can improve the accuracy of sample features.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a feature fusion model processing method, apparatus, computer equipment, storage medium and computer program product, as well as a sample retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, machine learning technology has emerged. Various machine learning models can be trained through machine learning. For example, a model for obtaining sample features can be trained. The model can obtain the features corresponding to the input sample, and similar samples can be retrieved in the database based on the features.

[0003] In traditional technology, samples can be input into the model to extract underlying features. However, since the underlying features do not have semantic measurement capabilities, they cannot accurately represent the samples and have low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a feature fusion model processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of the obtained features in response to the above technical problems.

[0005] On the one hand, the present application provides a feature fusion model processing method. The method includes: obtaining semantic features and non-semantic features corresponding to training samples in a training sample set; taking at least one of the semantic features and the non-semantic features as candidate features, and determining the activated feature components from the feature components contained in each of the candidate features; performing statistics on the activated feature components belonging to the same feature bit, and determining the activation levels of each feature bit corresponding to each of the candidate features according to the statistical results; respectively setting the feature components corresponding to the feature bits whose activation levels meet preset conditions in each of the candidate features to non-activated values ​​to obtain each target feature; performing fusion processing based on each of the target features and using the feature fusion model to be trained to obtain each fusion feature; training the feature fusion model based on each of the fusion features, and obtaining a target fusion model when the training stop condition is met; the target fusion model is used to perform fusion processing on the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0006] On the other hand, the present application also provides a feature fusion model processing device. The device includes: a feature acquisition module, which is used to acquire semantic features and non-semantic features corresponding to training samples in a training sample set; an activation component determination module, which is used to take at least one of the semantic features and the non-semantic features as candidate features, and determine the activation feature components from the feature components contained in each of the candidate features; an activation degree determination module, which is used to count the activation feature components belonging to the same feature bit, and determine the activation degree of each feature bit corresponding to each of the candidate features according to the statistical results; a target feature acquisition module, which is used to set the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each of the candidate features to non-activation values, and obtain each target feature; a fusion processing module, which is used to perform fusion processing based on each of the target features and using the feature fusion model to be trained to obtain each fusion feature; a parameter adjustment module, which is used to train the feature fusion model based on each of the fusion features, and obtain the target fusion model when the training stop condition is met; the target fusion model is used to perform fusion processing on the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0007] On the other hand, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the feature fusion model processing method when executing the computer program.

[0008] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned feature fusion model processing method when executed by a processor.

[0009] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the feature fusion model processing method are implemented.

[0010] The feature fusion model processing method, device, computer equipment, storage medium and computer program product mentioned above, on the one hand, can obtain the target fusion feature by fusing the semantic features and non-semantic features of the input sample through the target fusion model. The target fusion feature has the semantic measurement ability and non-semantic measurement ability because it fuses the semantic features and non-semantic features. It can characterize the sample more accurately with high accuracy; on the other hand, in the process of training to obtain the target fusion model, at least one of the semantic features and the non-semantic features is used as a candidate feature, and the activated feature component is determined from the feature components contained in each candidate feature, and the activated feature components belonging to the same feature bit are counted, and the activation degree of each feature bit corresponding to each candidate feature is determined according to the statistical result, and the feature components corresponding to the feature bits whose activation degree meets the preset conditions in each candidate feature are set to non-activated values ​​to obtain each target feature, and then based on each target feature and using the feature fusion model to be trained for fusion processing, the influence of the features of the low-activated feature bits on the training process can be reduced, so that the model can better learn the key feature bits, and the trained target fusion model can better perform feature fusion, further improving the accuracy of the target fusion feature.

[0011] On the other hand, the present application provides a sample retrieval method. The method includes: obtaining semantic features and non-semantic features corresponding to a query sample; inputting the semantic features and non-semantic features into a target fusion model for fusion processing to obtain a target fusion feature corresponding to the query sample; the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met; each of the fusion features is obtained by fusion processing based on each target feature and using the feature fusion model; each of the target features is obtained by setting the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each of the candidate features to non-activation values; the activation levels of each feature bit are determined by statistically analyzing the activated feature components belonging to the same feature bit according to the statistical results; the activated feature components are determined by taking at least one of the semantic features and the non-semantic features as candidate features from the feature components contained in each of the candidate features; the semantic features and non-semantic features correspond to training samples in a training sample set; and the target retrieval sample corresponding to the query sample is retrieved from a sample database based on the target fusion feature.

[0012] On the other hand, the present application provides a sample retrieval device. The device includes: a feature acquisition module, which is used to acquire semantic features and non-semantic features corresponding to training samples in a training sample set; an activation component determination module, which is used to take at least one of the semantic features and the non-semantic features as candidate features, and determine the activation feature components from the feature components contained in each of the candidate features; an activation degree determination module, which is used to count the activation feature components belonging to the same feature bit, and determine the activation degree of each feature bit corresponding to each of the candidate features according to the statistical results; a target feature acquisition module, which is used to set the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each of the candidate features to non-activation values, and obtain each target feature; a fusion processing module, which is used to perform fusion processing based on each of the target features and using the feature fusion model to be trained to obtain each fusion feature; a parameter adjustment module, which is used to train the feature fusion model based on each of the fusion features, and obtain the target fusion model when the training stop condition is met; the target fusion model is used to perform fusion processing on the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0013] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above sample retrieval method are implemented.

[0014] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned sample retrieval method when executed by a processor.

[0015] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above sample retrieval method are implemented.

[0016] The above-mentioned sample retrieval method, device, computer equipment, storage medium and computer program product, on the one hand, can obtain the target fusion feature by fusing the semantic features and non-semantic features of the query sample through the target fusion model. The target fusion feature has both semantic measurement ability and non-semantic measurement ability because it fuses the semantic features and non-semantic features. It can characterize the sample more accurately, and the retrieval based on the target fusion feature is highly accurate. On the other hand, since the target fusion model is obtained by training the feature fusion model to be trained based on each fusion feature until the training stop condition is met, each fusion feature is obtained based on each target feature and is fused using the feature fusion model. Each target feature is obtained by setting the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each candidate feature to inactive values. The activation levels of each feature bit are determined by statistically analyzing the activated feature components belonging to the same feature bit. The activated feature components are determined from the feature components contained in each candidate feature by taking at least one of the semantic features and the non-semantic features as candidate features. This can reduce the impact of the features of the low-activated feature bits on the training process during model training, so that the model can better learn the key feature bits. The target fusion model obtained through training can better perform feature fusion, further improving the accuracy of sample retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An application environment diagram of a feature fusion model processing method in an embodiment;

[0018] Figure 2 It is a flowchart of a feature fusion model processing method in one embodiment;

[0019] Figure 3 A schematic diagram of feature bit distribution of a training sample in one embodiment;

[0020] Figure 4 A schematic diagram of a flow chart of a feature fusion model processing method in another embodiment;

[0021] Figure 5 A schematic diagram of a fusion process in an embodiment;

[0022] Figure 6 A schematic diagram of a flow chart of a training step of a feature extraction model in one embodiment;

[0023] Figure 7 A schematic diagram of a similar sample in an embodiment;

[0024] Figure 8 A schematic diagram of a similar sample in another embodiment;

[0025] Fig. 9 A schematic diagram of a sample generated by an image attack in one embodiment;

[0026] Fig.10 is a flow chart of a sample retrieval method in one embodiment;

[0027] Fig.11 is a schematic diagram of a model structure in an embodiment;

[0028] Fig.12 is a structural block diagram of a feature fusion model processing device in an embodiment;

[0029] Fig.13 is a structural block diagram of a sample retrieval device in one embodiment;

[0030] Fig.14 is an internal structure diagram of a computer device in one embodiment;

[0031] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0035] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0036] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computer devices simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computer devices intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0037] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, machine learning and other technologies, which are specifically described by the following embodiments:

[0038] The feature fusion model processing method and sample retrieval method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, a laptop, a smart phone, a tablet computer, a desktop computer, a smart TV, a vehicle-mounted terminal, and a portable wearable device. The terminal can be provided with an application, through which the input sample can be retrieved to obtain the target retrieval sample. The application can refer to a client installed in the terminal, and the client (also referred to as an application client, APP client) refers to a program installed and running in the terminal; the application can also refer to an installation-free application, that is, an application that can be used without downloading and installing. This type of application is also commonly known as a mini-program, which usually runs in the client as a subroutine; the application can also refer to a web application opened through a browser; and so on. The server 104 can be implemented with an independent server or a server cluster or a cloud server composed of multiple servers. It can be understood that the embodiment of the present application does not limit the number of terminals and servers. The terminal can be one or more, and the server can be one or more, which can be set as needed. Among them, multiple refers to at least two.

[0039] Both the terminal 102 and the server 104 can be used independently to execute the feature fusion model processing and sample retrieval method provided in the embodiments of the present application.

[0040] For example, the server 104 can obtain the semantic features and non-semantic features corresponding to the training samples in the training sample set, take at least one of the semantic features and non-semantic features as a candidate feature, determine the activated feature component from the feature components included in each candidate feature, count the activated feature components belonging to the same feature bit, determine the activation degree of each feature bit corresponding to each candidate feature according to the statistical results, set the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each candidate feature to non-activated values, obtain each target feature, perform fusion processing based on each target feature and using the feature fusion model to be trained to obtain each fused feature, train a feature fusion model based on each fused feature, and obtain a target fusion model when the training stop condition is met; the target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0041] The server 104 can obtain semantic features and non-semantic features corresponding to the query sample, input the semantic features and non-semantic features into the target fusion model for fusion processing, obtain the target fusion features corresponding to the query sample, and retrieve the target retrieval sample corresponding to the query sample from the sample database based on the target fusion features.

[0042] The terminal 102 and the server 104 can also be used together to execute the feature fusion model processing and sample retrieval method provided in the embodiments of the present application.

[0043] For example, the server 104 may obtain a training sample from the terminal. After training the target fusion model, the server 104 may send the trained target fusion model to the terminal. The terminal may fuse the semantic features and non-semantic features of the input sample based on the target fusion model to obtain the target fusion feature. The terminal may further retrieve the target retrieval sample corresponding to the input sample from the database of the server based on the extracted target fusion feature.

[0044] In one embodiment, Figure 2 As shown, a feature fusion model processing method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be the server 104. In this embodiment, the feature fusion model processing method includes the following steps:

[0045] Step 202: Acquire semantic features and non-semantic features corresponding to training samples in the training sample set.

[0046] Among them, the training sample set includes multiple training samples, and the training sample refers to the content sample used to train the feature extraction model, and the content can be any one of text, audio or image. Non-semantic features refer to features that do not have semantic measurement capabilities, and can be features from the bottom layer of the deep learning model. Taking the training sample as an image as an example, non-semantic features can include, for example, representation information describing image texture and feature layout. Semantic features refer to features with semantic measurement capabilities. Because of their semantic measurement capabilities, semantic features can be used to classify input samples. Taking the training sample as an image as an example, semantic features can include representations that describe parts of the image with certain specified semantic content. For example, features used to describe a dog can be extracted by extracting features of the dog's location in the image as semantic features.

[0047] Specifically, the computer device can obtain a trained target semantic feature extraction model and a target non-semantic feature extraction model, input the training samples in the training sample set into the target semantic feature extraction model and the target non-semantic feature extraction model respectively, and obtain the semantic features and non-semantic features corresponding to the training samples.

[0048] In one embodiment, both semantic features and non-semantic features are embedding features, which can be floating point features or binary quantized features, i.e., hash features. Binary quantization refers to the process of binary encoding a feature, for example, the feature can be encoded into a binary code with values ​​of 0 or 1.

[0049] Step 204: taking at least one of the semantic feature and the non-semantic feature as a candidate feature, and determining an activated feature component from the feature components included in each candidate feature.

[0050] Here, feature components refer to feature values ​​used to form features. For example, assuming that the feature is a 1x N feature vector, each dimension is a feature component, and the feature contains N feature components in total. Activated feature components refer to activated feature components.

[0051] In one embodiment, the computer device may use semantic features as candidate features, and determine the activated feature components from the feature components contained in the semantic features corresponding to each training sample in the training sample set. In one embodiment, the computer device may use non-semantic features as candidate features, and determine the activated feature components from the feature components contained in the non-semantic features corresponding to each training sample in the training sample set.

[0052] In other embodiments, the computer device may use both semantic features and non-semantic features as candidate features, determine the activated feature components from the feature components contained in the semantic features corresponding to each training sample in the training sample set, and determine the activated feature components from the feature components contained in the non-semantic features corresponding to each training sample in the training sample set. In specific implementation, the computer device may map the feature components contained in each semantic feature and the feature components contained in each non-semantic feature in accordance with a preset mapping method to obtain the mapping feature values ​​of each feature component, compare the mapping feature values ​​of each feature component with the activation value set by the preset mapping method, and determine the feature components corresponding to the mapping feature values ​​that match the consistency as the activated feature components.

[0053] Step 206 , statistics are collected on the activated feature components belonging to the same feature bit, and the activation levels of the feature bits corresponding to the candidate features are determined according to the statistical results.

[0054] Among them, the feature bit refers to the position of the feature component in the feature. The activated feature components belonging to the same feature bit refer to the activated feature components belonging to the same position in different features. For example, assuming that feature 1 is (a1, a2, a3, a4), and feature 2 is (b1, b2, b3, b4), wherein feature component a1 and feature component b1 belong to the same feature bit, feature component a2 and feature component b2 belong to the same feature bit, feature component a3 and feature component b3 belong to the same feature bit, feature component a4 and feature component b4 belong to the same feature bit, assuming that a1 and b1 are both activated feature components, then a1 and b1 are activated feature components belonging to the same feature bit. The feature bits corresponding to each candidate feature refer to the feature bits contained in these candidate features, for example, the feature bits corresponding to the above-mentioned features 1 and 2 are the positions of the feature components of the first dimension in the feature vector, the positions of the feature components of the second dimension in the feature vector, the positions of the feature components of the third dimension in the feature vector, and the positions of the feature components of the third dimension in the feature vector. The activation degree corresponding to the feature bit is used to characterize the influence of the feature component belonging to the feature bit in the model learning process. The higher the activation degree, the greater the influence of the feature component belonging to the feature bit in the model learning process. Conversely, the lower the activation degree, the smaller the influence of the feature component belonging to the feature bit in the model learning process.

[0055] Specifically, after the computer device determines the activated feature components from the feature components included in each candidate feature, it counts the activated feature components belonging to the same feature bit among these activated feature components, and the activation degree of each feature bit corresponding to each candidate feature can be determined based on the statistical results.

[0056] In one embodiment, the computer device performs statistics on the activated feature components belonging to the same feature bit, and specifically calculates the number of activated feature components belonging to the same feature bit. The statistical result obtained can specifically be the total number of activated feature components corresponding to each feature bit. For example, Figure 3 As shown, assuming that the training sample set includes 4 training samples, the candidate features are semantic features, the semantic features of training sample 1 are (a1, a2, a3, a4), the semantic features of training sample 2 are (b1, b2, b3, b4), the semantic features of training sample 3 are (c1, c2, c3, c4), and the semantic features of training sample 4 are (d1, d2, d3, d4), among which b1, c1, a2, b2, d2, b4 are activated feature components. The total number of activated feature components belonging to feature position 1 is 2, the total number of activated feature components belonging to feature position 2 is 3, the total number of activated feature components belonging to feature position 3 is 0, and the total number of activated feature components belonging to feature position 4 is 1.

[0057] In one embodiment, the computer device can count the number of activated feature components belonging to the same feature bit, obtain the target number of each feature bit corresponding to each candidate feature, count the total number of feature components belonging to each feature bit, and determine the activation degree of each feature bit based on the target number and total number of each feature bit.

[0058] Step 208, in each candidate feature, the feature components corresponding to the feature bits whose activation levels meet the preset conditions are set to inactive values ​​to obtain each target feature.

[0059] Among them, the inactive value refers to a predefined value for distinguishing inactive feature components. The active value is a predefined value for distinguishing inactive feature components. For example, assuming that both semantic features and non-semantic features are hash features, the active value can be 1 and the inactive value can be -1. The feature bit whose activation degree meets the preset conditions, i.e., the low-activation feature bit, can be a feature bit whose activation degree is less than a preset activation degree threshold, or a feature bit that is arranged before the preset sorting threshold when the activation degrees of each feature bit are sorted from small to large. Among them, the activation degree threshold can be set as needed.

[0060] Taking into account that semantic features and non-semantic features may contain meaningless feature information during feature fusion, in order to reduce the loss of feature bits with poor effects in learning and the impact on the learning of key feature bits, when splicing and fusion of features, low-activation feature bits can be screened out by whether the features are effective, and then after splicing, by setting the low-activation feature bits to inactive values, it is equivalent to discarding these feature bits in the learning process, thereby avoiding the influence of more low-activation feature bits on the high-activation part of the spliced ​​features.

[0061] In one embodiment, the computer device may compare the activation level of each feature bit with a preset activation level threshold to determine the feature bits whose activation level is less than the preset activation level threshold, and randomly select a preset number of feature bits from these feature bits to obtain target feature bits, and set the feature components of these target feature bits in each candidate feature to inactive values.

[0062] In another embodiment, the computer device may sort the activation levels of the respective feature bits from small to large, randomly select a preset number of feature bits from the feature bits corresponding to the activation levels of the top N bits, and obtain the target feature bits. For each candidate feature, the feature components of these target feature bits are set to inactive values.

[0063] Step 210, based on each target feature and using the feature fusion model to be trained, fusion processing is performed to obtain each fusion feature.

[0064] The feature fusion model to be trained refers to a feature fusion model that needs to be trained. The feature fusion model refers to a machine learning model used to perform feature fusion processing.

[0065] In one embodiment, the computer device only uses semantic features as candidate features, and can obtain the target features corresponding to the semantic features. The computer device then splices the target features and non-semantic features belonging to the same training sample to obtain spliced ​​features, and inputs the spliced ​​features into the feature fusion model for fusion processing to obtain the fusion features corresponding to each training sample.

[0066] In one embodiment, the computer device only uses non-semantic features as candidate features, and can obtain target features corresponding to the non-semantic features. The computer device then splices the target features and semantic features belonging to the same training sample to obtain spliced ​​features, and inputs the spliced ​​features into a feature fusion model for fusion processing to obtain fused features corresponding to each training sample.

[0067] In one embodiment, a computer device may take both semantic features and non-semantic features as candidate features, and then obtain target features corresponding to the non-semantic features and target features corresponding to the semantic features. The computer device then splices the two target features belonging to the same training sample to obtain spliced ​​features, and inputs the spliced ​​features into a feature fusion model for fusion processing to obtain fused features corresponding to each training sample.

[0068] Step 212, training a feature fusion model based on each fusion feature, and obtaining a target fusion model when a training stop condition is met; the target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0069] Among them, the training stop condition can be that the model parameters no longer change, the loss reaches the minimum value, the number of training iterations reaches the maximum number of iterations, the training time reaches the preset time, etc.

[0070] Specifically, the computer device can determine the target loss based on each fusion feature, adjust the model parameters of the feature fusion model based on the target loss and continue training, and obtain the target fusion model when the training stop condition is met. The target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion feature, and the target fusion feature is used for sample retrieval. The sample retrieval refers to retrieving similar samples corresponding to the input sample in the database. The database can be deduplicated through sample retrieval.

[0071] In one embodiment, during the training process, the computer device may use any one of the stochastic gradient descent algorithm, Adagrad (Adaptive Gradient) algorithm, Adadelta (an improvement of the AdaGrad algorithm), RMSprop (an improvement of the AdaGrad algorithm), Adam (Adaptive Moment Estimation) algorithm, etc. to adjust the model parameters of the feature fusion model.

[0072] In one embodiment, a computer device may obtain a comparison training sample corresponding to a training sample, obtain a comparison semantic feature and a comparison non-semantic feature corresponding to the comparison training sample, use at least one of the comparison semantic feature and the comparison non-semantic feature as a comparison candidate feature, determine an activated feature component from the feature components contained in each comparison candidate feature, perform statistics on the activated feature components belonging to the same feature bit, determine the activation degree of each feature bit corresponding to each comparison candidate feature according to the statistical result, set the feature components corresponding to the feature bit whose activation degree meets the preset conditions in each comparison candidate feature to an inactive value, obtain each comparison target feature, perform fusion processing based on each comparison target feature and using the feature fusion model to be trained to obtain each comparison fusion feature, and calculate the target loss based on the difference between the fusion feature corresponding to the same training sample and the comparison fusion feature. The comparison training sample may be at least one of a positive comparison training sample and a negative comparison training sample. The fusion feature and comparative fusion feature corresponding to the same training sample refer to the fusion feature corresponding to the training sample and the comparative fusion feature of the comparative training sample corresponding to the training sample. For example, assuming that the fusion feature of a training sample A is feature 1, the comparative training sample of the training sample A is training sample B, and the fusion feature of training sample B is feature 2, then both feature 1 and feature 2 correspond to training sample A.

[0073] In the above-mentioned feature fusion model processing method, on the one hand, since the semantic features and non-semantic features of the input samples can be fused through the target fusion model to obtain the target fusion features, the target fusion features have the semantic measurement ability and non-semantic measurement ability because they fuse the semantic features and non-semantic features, and can more accurately characterize the samples with high accuracy; on the other hand, since in the process of training the target fusion model, at least one of the semantic features and the non-semantic features is used as a candidate feature, the activated feature component is determined from the feature components contained in each candidate feature, and the activated feature components belonging to the same feature bit are counted, and the activation degree of each feature bit corresponding to each candidate feature is determined according to the statistical results, and the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each candidate feature are set to non-activated values ​​to obtain each target feature, and then based on each target feature and using the feature fusion model to be trained for fusion processing, the influence of the features of the low-activated feature bits on the training process can be reduced, so that the model can better learn the key feature bits, and the trained target fusion model can better perform feature fusion, further improving the accuracy of the target fusion feature.

[0074] In one embodiment, statistics are taken for activated feature components belonging to the same feature bit, and the activation levels of the feature bits corresponding to the candidate features are determined based on the statistical results, including: statistics are taken for the number of activated feature components belonging to the same feature bit to obtain the activation numbers of the feature bits corresponding to the candidate features; statistics are taken for the total number of features of the feature components belonging to each feature bit; and the activation levels of the feature bits are determined based on the activation numbers and the total number of features of the feature bits.

[0075] The activation number corresponding to the feature bit is used to represent the number of activated feature components belonging to the feature bit. The total number of features of the feature bit is used to represent the number of feature components belonging to the feature bit. For example, continue to refer to Figure 3 , the number of activations corresponding to feature bit 1 is 2, and the total number of features is 4.

[0076] Specifically, the computer device can count the number of activated feature components belonging to the same feature bit, obtain the activation number of the activated feature components of each feature bit contained in the candidate feature, and further count the total number of features of the feature components belonging to each feature bit. Based on the activation number of each feature bit and the total number of features, the influence of the feature bit in the model learning process can be determined, thereby obtaining the activation degree of each feature bit.

[0077] In one embodiment, the computer device can calculate the ratio between the number of activations and the total number of features to determine the activation ratio corresponding to each feature bit, and then determine the activation degree of each feature bit based on the activation ratio corresponding to each feature bit. The activation ratio corresponding to the feature bit can represent how many proportions of training samples can be distinguished by the feature component belonging to the feature bit. When the distinguishable ratio is closer to 0.5, it means that the influence of the feature bit is greater and the activation degree is higher.

[0078] In the above embodiment, by counting the activation number of the activated feature components of each feature bit and the total number of features of the feature components of each feature bit, the activation degree of each feature bit can be quickly determined based on the activation number and the total number of features.

[0079] In one embodiment, based on the activation number of each feature bit and the total number of features, the activation degree of each feature bit is determined, including: based on the activation number of each feature bit and the total number of features, determining the activation ratio corresponding to each feature bit; based on the activation ratio corresponding to each feature bit, calculating the activation entropy corresponding to each feature bit, and determining the activation entropy corresponding to each feature bit as the activation degree corresponding to each feature bit.

[0080] The activation ratio of the feature bit represents the ratio of the activated feature component of the feature bit to all the feature components belonging to the feature bit. For example, Figure 3 In , the activation ratio corresponding to feature bit 1 is 2 / 4=0.5.

[0081] Specifically, after the computer device determines the activation ratio corresponding to each feature bit based on the activation number of each feature bit and the total number of features, the activation entropy corresponding to each feature bit can be calculated by referring to the following formula (1), where H is used to represent the activation entropy, p a Indicates the activation ratio.

[0082] H=-p a *log(1-p a ) (1)

[0083] The activation entropy can be used to characterize the amount of information of each feature bit. The larger the amount of information, the greater the influence in the model learning process. Therefore, in this embodiment, the computer device can determine the activation entropy corresponding to each feature bit as the activation degree corresponding to each feature bit.

[0084] In the above embodiment, the activation ratio is determined by calculation, and the activation entropy is determined based on the activation ratio. Since the activation entropy can characterize the amount of information, the activation entropy is used as the degree of activation to accurately express the influence of the feature bit characteristics in the model learning process.

[0085] In one embodiment, an activated feature component is determined from the feature components included in each candidate feature, including: mapping the feature components included in each candidate feature according to a preset mapping method to obtain a mapping feature value of each feature component; comparing the mapping feature value of each feature component with the activation value set by the preset mapping method; and determining the feature component corresponding to the mapping feature value that matches the comparison as the activated feature component.

[0086] In this embodiment, both semantic features and non-semantic features are 1*N vectors whose values ​​tend to -1 or 1. The preset mapping method refers to a pre-set method for mapping the values ​​of semantic features and non-semantic features to activation values ​​or inactivation values. The preset mapping method can be set as needed. In one embodiment, the computer device can map the feature components contained in each candidate feature with reference to the following formula (2) to obtain the mapping feature values ​​of each feature component, where Q i is the i-th feature component in the candidate feature, B i is the mapping eigenvalue of the i-th eigencomponent:

[0087]

[0088] After mapping to obtain the mapping characteristic value, the mapping characteristic value of each characteristic component of the computer device is compared with the activation value set by the preset mapping method, and the characteristic component corresponding to the mapping characteristic value that matches the comparison is determined as the activated characteristic component. For example, when the activation value set by the preset mapping method is 1, that is, when the characteristic component is 1, the characteristic component is the activated characteristic component.

[0089] In the above embodiment, by obtaining the mapping characteristic value of each feature component, comparing the mapping characteristic value of each feature component with the activation value set by the preset mapping method to determine the activated feature component, the activated feature component can be determined from the feature components included in each candidate feature.

[0090] In one embodiment, the feature components corresponding to the feature bits whose activation levels meet preset conditions in each candidate feature are set to inactive values ​​to obtain each target feature, including: sorting the feature bits based on their respective activation levels, determining the feature bits whose activation levels meet preset conditions based on the sorting results, and obtaining the target feature bits; and setting the feature components corresponding to the target feature bits in each candidate feature to inactive values ​​to obtain each target feature.

[0091] Specifically, the computer device can sort the feature bits from small to large based on the activation levels of each feature bit, determine a preset proportion of feature bits with the smallest activation levels based on the sorting results, obtain the target feature bits, and set the feature components of the target feature bits in each candidate feature to inactive values ​​to obtain each target feature.

[0092] For example, assuming that the candidate feature is a 1X64 vector and the corresponding feature bit is 64 bits, the 64 feature bits can be sorted from small to large according to the activation degree, and the 10% feature bits with the smallest activation degree are determined based on the sorting result as the target feature bits. For example, the target feature bits can be the feature bits with serial numbers 0, 1, 2, 61, 62, and 63. It should be noted that the serial number here refers to the position order of the feature bit in the vector. For example, the feature bit with serial number 0 refers to the position of the leftmost feature component in the candidate feature.

[0093] In the above embodiment, by sorting each feature bit, determining the feature bit whose activation degree meets the preset conditions based on the sorting result, and obtaining the target feature bit, it is possible to quickly obtain the feature bit with a low activation degree, and then set the feature components of these feature bits to inactive values ​​to reduce the impact of the low-activation feature bits on the key feature bits during the training process.

[0094] In one embodiment, Figure 4 As shown, a feature fusion model processing method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be the server 104. In this embodiment, the feature fusion model processing method includes the following steps:

[0095] Step 402: Acquire semantic features and non-semantic features corresponding to training samples in the training sample set.

[0096] Step 404: determining an activated feature component corresponding to the semantic feature from the feature components included in each semantic feature, and determining an activated feature component corresponding to the non-semantic feature from the feature components included in each non-semantic feature.

[0097] Specifically, for each training sample, after obtaining the semantic features and non-semantic features of the training sample, the computer device can determine the activated feature components corresponding to the semantic features from the feature components contained in the semantic features, and determine the activated feature components corresponding to the non-semantic features from the feature components contained in the non-semantic features.

[0098] Step 406 , statistics are collected on the activated feature components belonging to the same feature bit in each semantic feature, and the activation degree of each feature bit corresponding to each semantic feature is determined according to the statistical result.

[0099] Step 408 , statistics are collected on the activated feature components belonging to the same feature bit in each non-semantic feature, and the activation levels of each feature bit corresponding to each non-semantic feature are determined according to the statistical results.

[0100] Step 410 , in each semantic feature, the feature components corresponding to the feature bits whose activation levels meet preset conditions are respectively set to inactive values ​​to obtain each first target feature.

[0101] Step 412, in each non-semantic feature, the feature components corresponding to the feature bits whose activation levels meet the preset conditions are respectively set to inactive values ​​to obtain each second target feature.

[0102] Step 414, based on each first target feature and each second target feature and using the feature fusion model to be trained, a fusion process is performed to obtain each fusion feature.

[0103] Specifically, the computer device can splice the semantic features and non-semantic features corresponding to the same training sample to obtain spliced ​​features, input the spliced ​​features into a feature fusion model, perform fusion processing through the feature fusion model, and obtain fusion features corresponding to each training sample.

[0104] Step 416, training a feature fusion model based on each fusion feature, and obtaining a target fusion model when a training stop condition is met; the target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0105] In the above embodiment, by performing feature selection on both semantic features and non-semantic features and setting the feature components of low-activated feature bits to non-activated values, the influence of the features of low-activated feature bits on the key features can be avoided to the greatest extent during the fusion process, so that the fused features can better characterize the training samples.

[0106] In one embodiment, fusion processing is performed based on each target feature and using a feature fusion model to be trained to obtain each fused feature, including: splicing the first target feature and the second target feature corresponding to the same training sample to obtain a spliced ​​feature; inputting the spliced ​​feature into the feature fusion model; the feature fusion model includes a first fully connected layer and a second fully connected layer; mapping the spliced ​​feature through the first fully connected layer to obtain an intermediate feature; mapping the intermediate feature through the second fully connected layer to obtain a fused feature; the intermediate feature has the same dimension as the spliced ​​feature, and the dimension of the fused feature is less than the dimension of the spliced ​​feature.

[0107] Specifically, the computer device can splice the first target feature and the second target feature corresponding to the same training sample to obtain a spliced ​​feature, and input the spliced ​​feature into the first fully connected layer and the second fully connected layer of the feature fusion model in sequence, map the spliced ​​feature through the first fully connected layer to obtain an intermediate feature, and map the intermediate feature through the second fully connected layer to obtain a fused feature. The intermediate feature obtained by the first fully connected layer has the same dimension as the spliced ​​feature, so information fusion of features within the spliced ​​feature can be achieved. The dimension of the fused feature obtained by the second fully connected layer is less than the dimension of the spliced ​​feature, so the feature can be compressed.

[0108] For example, see Figure 5 , after setting the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in the 1x64 semantic feature 502 and the 1x64 non-semantic feature 504 of a training sample to inactive values, a 1x64 first target feature 506 and a 1x64 second target feature 508 are obtained, wherein the shaded parts in the first target feature 506 and the second target feature 508 are inactive values, and the first target feature 506 and the second target feature 508 are concatenated to obtain a 1x128 concatenated feature 510, which is input into a feature fusion model, the feature fusion model includes a first fully connected layer and a second fully connected layer, wherein the parameters of the first fully connected layer are a parameter matrix of 128x128, and the parameters of the second fully connected layer are a parameter matrix of 128x96, and after mapping through the first fully connected layer and the second fully connected layer, a fused feature is finally obtained, and the fused feature is a 1x96 vector.

[0109] In the above embodiment, by mapping the concatenated features through two fully connected layers, the feature information between the feature bits can be more integrated, and finally the fused features with compressed feature dimensions are obtained. The feature fusion model trained in this way can reduce the consumption of existing resources when fusing the input samples.

[0110] In one embodiment, the semantic features of the training samples are obtained by extracting features from the training samples using a trained target semantic feature extraction model, and the non-semantic features corresponding to the training samples are obtained by extracting features from the training samples using a trained target non-semantic feature extraction model; Figure 6 As shown, the training steps of the target semantic feature extraction model and the target non-semantic feature extraction model include:

[0111] Step 602: Obtain a comparison training sample corresponding to a training sample in a training sample set.

[0112] The comparison training sample includes at least one of a positive comparison training sample and a negative comparison training sample. The similarity between the training sample and the positive comparison training sample is greater than the similarity between the training sample and the negative comparison training sample. For example, the positive comparison training sample corresponding to the training sample may be a training sample similar to the positive comparison training sample, and the negative comparison training sample may be a training sample that is not similar to the training sample.

[0113] In one embodiment, the trained target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion features, and the target fusion features are used for image deduplication retrieval. In the deduplication system, two samples need to be extremely similar to be considered similar samples, such as Figure 7 and 8 As shown, or other images generated by image attacks, such as Fig. 9 Image attacks are the process of modifying images using image enhancement methods, such as hue changes, histogram equalization, watermarking, cropping, rotation, watermarking, black edges, etc. Specifically, refer to Figure 7 ,in Figure 7 In the figure (a), (b), (c) and (d) all contain the same object, (a) and (b) both contain text, (c) and (d) do not contain text, and the facial postures and expressions of (a), (b), (c) and (d) are different. These images are all extremely similar samples. Figure 8 ,in Figure 8 Figures (a), (b), (c) and (d) all contain the same object. Figures (a), (b), (c) and (d) are images of the object in different postures and expressions. These images are all extremely similar samples. Fig. 9 ,in Fig. 9 Figures (a), (b), (c), and (d) all contain the same object. Fig. 9 Figures (b) and (c) are obtained by adding watermarks to Figure (a), and Figure (d) is obtained by adding watermarks to Figure (a) and performing tone transformation. Figures (b), (c) and (d) are all images generated through image attacks.

[0114] Step 604: input the training sample and the comparison training sample as target samples into the initial semantic feature extraction model to be trained and the initial non-semantic feature extraction model to be trained respectively.

[0115] Specifically, the computer device will use the training samples and the comparison training samples as target samples respectively, that is, the training samples will be input into the initial semantic feature extraction model to be trained and the initial non-semantic feature extraction model to be trained respectively, and the comparison training samples will be input into the initial semantic feature extraction model to be trained and the initial non-semantic feature extraction model to be trained respectively.

[0116] Step 606: output the semantic training features corresponding to each target sample through the initial semantic feature extraction model, and output the non-semantic training features corresponding to each target sample through the initial non-semantic feature extraction model.

[0117] The feature extraction model is a machine learning model used to extract features and output feature vectors. The semantic feature extraction model refers to a feature extraction model used to extract semantic features, while the non-semantic feature extraction model refers to a feature extraction model used to extract non-semantic features. The initial semantic feature extraction model is a model obtained by parameter initialization. The initial non-semantic feature extraction model can be a model obtained by parameter initialization or a pre-trained model.

[0118] In one embodiment, the feature extraction model includes at least an embedding model, and the embedding model is used to output a feature vector. The feature vector output by the embedding model can be called an embedding. The embedding model can be a model composed of one or more fully connected layers. The output features of the embedding model are the features extracted by the feature extraction model.

[0119] In one embodiment, the feature vector output by the embedding model may be normalized so that each component of the feature vector has a value range of -1 to 1. In other embodiments, the feature vector may be binary quantized to obtain a binary quantized feature, i.e., a hash feature. In this case, the embedding model may be called a hash quantization model.

[0120] In one embodiment, the feature extraction model may only include an embedding model, and the input end of the embedding model may be connected to the output end of the trained basic neural network model, and the output of the basic neural network model is received as input. The basic neural network model may be a model for extracting feature information contained in the content, and the basic neural network model may be a neural network based on artificial intelligence, such as a convolutional neural network (CNN), or a network such as ResNet101 (deep residual network 101) or ResNet18 (deep residual network 18). The convolutional neural network is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. In other embodiments, the feature extraction model may include a basic neural network model and an embedding model, and the basic neural network model and the embedding model are trained together as feature extraction models. It can be understood that different feature extraction models can be trained for different types of samples. For example, if the sample is an image type sample, then an image feature extraction model can be trained, and if the sample is a speech type sample, then a speech feature extraction model can be trained.

[0121] Specifically, the computer device extracts features from the training samples through an initial semantic feature extraction model to obtain semantic training features corresponding to the training samples, extracts features from the training samples through an initial non-semantic feature extraction model to obtain non-semantic training features corresponding to the training samples, extracts features from the comparison training samples through an initial semantic feature extraction model to obtain semantic training features corresponding to the comparison training samples, and extracts features from the comparison training samples through an initial non-semantic feature extraction model to obtain non-semantic training features corresponding to the comparison training samples.

[0122] Step 608, obtaining a semantic feature loss based on the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample;

[0123] Specifically, the computer device may obtain the semantic feature loss based on the difference between the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample.

[0124] In one embodiment, the computer device can calculate the cosine similarity between the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample, and use the obtained cosine similarity to characterize the difference between the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample, so as to calculate the difference between the cosine similarity and the training label to obtain the semantic feature loss. Wherein, when the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.

[0125] In one embodiment, the computer device can perform similarity classification on the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample, and divide them into two categories, similar and dissimilar, to obtain the probability of similarity and the probability of dissimilarity, and take the larger probability value as the classification result, and use the probability value of the classification result to characterize the difference between the training sample feature and the extracted feature of the comparison sample, and then calculate the difference between the probability of the classification result and the training label to obtain the semantic feature loss. Wherein, when the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.

[0126] In one embodiment, a computer device may calculate a feature distance between semantic training features corresponding to a training sample and semantic training features corresponding to a comparison training sample, use the feature distance to characterize the difference between the semantic training features corresponding to the training sample and semantic training features corresponding to the comparison training sample, and use the feature distance as a semantic feature loss, wherein the feature distance may be, for example, a Euclidean distance or an L2 distance.

[0127] Step 610: Obtain a first non-semantic feature loss based on the non-semantic training features corresponding to the training sample and the non-semantic training features corresponding to the comparison training sample.

[0128] Specifically, the computer device may obtain a first non-semantic feature loss based on a difference between a non-semantic training feature corresponding to the training sample and a non-semantic training feature corresponding to the comparison training sample.

[0129] Step 612: classify the training samples based on the semantic training features of the training samples to obtain classification results, and determine the classification loss based on the classification results.

[0130] Specifically, the computer device classifies the training samples based on the semantic training features of the training samples to obtain classification results, and determines the classification loss based on the difference between the classification results and the category labels of the training samples, wherein the category labels refer to pre-labeled labels used to characterize the categories to which the training samples belong.

[0131] In one embodiment, the computer device may classify the training samples using a classification model, that is, input the semantic training features of the training samples into the classification model, and obtain the output of the classification model as the classification result. Among them, the classification model refers to a machine learning model that can perform category recognition. In one embodiment, the classification result may be a category identifier used to characterize the category to which the training sample belongs. For example, if the classification model classifies N categories, the first category may be represented by an N-dimensional vector (1, 0, 0, 0, ..., 0), and the second category may be represented by a vector (0, 1, 0, 0, ..., 0), and so on. In other embodiments, the classification result may be a probability used to characterize the category to which the training sample belongs. For example, the classification model may be used to classify N categories, and the classification model may output a probability vector including an N dimension, wherein the probability of each dimension is used to characterize the likelihood that the training sample belongs to the category.

[0132] In other embodiments, the computer device may calculate the feature distance between the semantic training features of the training sample and the category features corresponding to each candidate category. When the feature distance between the semantic training features of the training sample and the category features is less than a distance threshold, the candidate category corresponding to the category features is determined as the classification category to which the training sample belongs. At this time, the classification result may be the category identifier of the classification category. The category features corresponding to the candidate category may be determined in the following manner: collect multiple content samples corresponding to the training category, extract the feature vectors of these content samples through an embedding model, and then calculate an average vector as the category feature corresponding to the candidate category.

[0133] Step 614, based on the semantic feature loss, the first non-semantic feature loss and the classification loss, the initial semantic feature extraction model and the initial non-semantic feature extraction model are trained respectively. When the training stop condition is met, the trained target semantic feature extraction model and the trained target non-semantic feature extraction model are obtained.

[0134] Specifically, the computer device can count the semantic feature loss, the first non-semantic feature loss, and the classification loss corresponding to each training sample to obtain the target loss, adjust the model parameters of the initial semantic feature extraction model and the initial non-semantic feature extraction model based on the target loss, and continue training. When the training stop condition is met, the trained target semantic feature extraction model and the trained target non-semantic feature extraction model are obtained. For example, the computer device can perform weighted summation of the semantic feature loss, the first non-semantic feature loss, and the classification loss corresponding to the same training sample to obtain the target loss.

[0135] In the above embodiment, by jointly training the initial semantic feature extraction model and the initial non-semantic feature extraction model, the training effect can be improved, and the obtained target semantic feature extraction model and target non-semantic feature extraction model can perform better feature extraction.

[0136] In one embodiment, there is a shared network layer between the initial semantic feature extraction model and the initial non-semantic feature extraction model; the semantic training features corresponding to each target sample are outputted through the initial semantic feature extraction model, and the non-semantic training features corresponding to each target sample are outputted through the initial non-semantic feature extraction model, including: performing convolution processing on each target sample through the shared network layer to obtain the shared features corresponding to each target sample; performing feature processing on each shared feature through the first feature processing layer of the initial semantic feature extraction model to obtain the semantic training features corresponding to each target sample; performing feature processing on each shared feature through the second feature processing layer of the initial non-semantic feature extraction model to obtain the non-semantic training features corresponding to each target sample; the network depth of the first feature processing layer is greater than that of the second feature processing layer.

[0137] The existence of a shared network layer between the initial semantic feature extraction model and the initial non-semantic feature extraction model means that the initial semantic feature extraction model and the initial non-semantic feature extraction model have a common underlying basic feature extraction network, the output of which is connected to the input of the first feature processing layer to form the initial semantic feature extraction model, and the output of which is also connected to the input of the second feature processing layer to form the initial non-semantic feature extraction model. The initial semantic feature extraction model and the initial non-semantic feature extraction model can form a dual-branch feature extraction model.

[0138] Considering that the results of the basic feature extraction network output support the global underlying feature learning, and the high-level abstract semantic features require a deeper texture representation based on the bottom layer for effective extraction, in order to make the extraction effective, it is necessary to design the semantic features to be at a deeper network position than the non-semantic features. Therefore, the network depth of the first feature processing layer is greater than that of the second feature processing layer. In one embodiment, the first feature processing layer can adopt a single convolution layer, and the second feature processing layer can adopt multiple convolution layers. By setting multiple convolution layers for the second feature processing layer, on the one hand, it can avoid the classification gradient from being transmitted back too quickly to the basic feature extraction network, thereby affecting the effect of the non-semantic features; on the other hand, it can enable the classification module to further perform classification-related key part feature extraction on the global features of the basic feature extraction network, so that the semantic representation is more complete. For example, if Fig.11As shown, it is a schematic diagram of the model structure in one embodiment. It can be seen that the semantic feature branch and the non-semantic feature branch share the CNN network input end, and the output end of the CNN network is connected to the input ends of the two branches at the same time. The branch where the semantic feature is located includes multiple layers of convolutional layers, which can extract high-level abstract semantic features.

[0139] In the above embodiment, by setting a shared network layer, the complexity of the model can be reduced, the amount of data processing of the model can be reduced, and the parameters that the model needs to learn can be reduced.

[0140] In one embodiment, the initial non-semantic feature extraction model is obtained by the following steps: inputting the training samples and the comparison training samples into the candidate non-semantic feature extraction model respectively, and outputting the non-semantic training features corresponding to the training samples and the non-semantic training features corresponding to the comparison training samples; obtaining the second non-semantic feature loss based on the non-semantic training features corresponding to the training samples of the candidate non-semantic feature extraction model and the non-semantic training features corresponding to the comparison training samples; based on the second non-semantic feature loss, adjusting the model parameters of the candidate non-semantic feature extraction model and continuing the training, and when the training stop condition is met, obtaining the initial non-semantic feature extraction model.

[0141] Specifically, after obtaining the non-semantic training features corresponding to the training samples and the non-semantic training features corresponding to the comparison training samples, the computer device can calculate the second non-semantic feature loss based on the difference between the non-semantic training features corresponding to the training samples and the non-semantic training features corresponding to the comparison training samples, adjust the model parameters of the candidate non-semantic feature extraction model based on the second non-semantic feature loss and continue training, and when the training stop condition is met, obtain the initial non-semantic feature extraction model, and then jointly train the initial non-semantic feature extraction model and the initial semantic feature extraction model to obtain the target non-semantic feature extraction model and the target semantic feature extraction model. Among them, the initial semantic feature extraction model is obtained after parameter initialization.

[0142] In the above embodiment, considering that the non-semantic feature extraction model converges slowly, the non-semantic feature extraction model can be trained in advance, and the trained model can be jointly trained with the semantic feature extraction model to improve the training efficiency and training effect.

[0143] In one embodiment, the fused features are binary quantized features, and a feature fusion model is trained based on each fused feature. When a training stop condition is met, a target fusion model is obtained, including: obtaining a quantization target corresponding to a feature component in the fused feature, and determining a quantization loss based on a difference between the feature component in the fused feature and the corresponding quantization target; obtaining a contrast fusion feature corresponding to a contrast training sample of the training sample, and determining a fusion loss based on a difference between the fused feature and the contrast fusion feature; the contrast fusion feature is obtained based on the feature fusion model; statistically calculating the quantization loss and the fusion loss to obtain the target loss; adjusting the model parameters of the feature fusion model based on the target loss and continuing training until the training stop condition is met to obtain the target fusion model.

[0144] Among them, quantization loss refers to the loss of calculating the quantization effect (whether it is close enough to -1 or 1). During the training process, it is expected that each quantized component in the fusion feature is close enough to 1 or -1. Quantization loss is positively correlated with the difference between the quantized component and the quantization target corresponding to the quantized component.

[0145] Specifically, the computer device can obtain the quantization target corresponding to each feature component in the fusion feature, determine the quantization loss based on the difference between each feature component and the respective quantization target, further obtain the comparative fusion feature corresponding to the comparative training sample of each training sample, determine the fusion loss based on the difference between the fusion feature and the comparative fusion feature of the same training sample, perform weighted summation of the quantization loss and the fusion loss to obtain the target loss, and finally adjust the model parameters of the feature fusion model based on the target loss and continue training until the training stop condition is met to obtain the target fusion model.

[0146] In one embodiment, the computer device may calculate the quantization loss by referring to the following formula (3), where Q i is the value of the fusion feature Q at the i-th position, B i is the quantization target of the i-th position, B i By Q i After the preset symbol function - sign function - is generated, the sign function can combine each bit Q of the fusion feature Q i Calculate the target code Bi respectively, and the sign function can refer to the above formula (2) for details.

[0147]

[0148] In one embodiment, for each training sample in a training sample set, a computer device can obtain a comparative training sample of each training sample, obtain semantic features and non-semantic features corresponding to the comparative training sample, take at least one of the semantic features and non-semantic features as a candidate feature, and determine an activated feature component from the feature components contained in each candidate feature; perform statistics on the activated feature components belonging to the same feature bit, determine the activation degree of each feature bit corresponding to each candidate feature according to the statistical results, set the feature components corresponding to the feature bits whose activation degrees meet preset conditions in each candidate feature to non-activated values, obtain each target feature, perform fusion processing based on each target feature and using the feature fusion model to be trained, and obtain each comparative fusion feature.

[0149] In the above embodiment, the quantization loss and the fusion loss are calculated, and the target loss is obtained by combining the two losses. Then, the feature fusion model is trained based on the target loss. The obtained target fusion model can obtain accurate binary quantization features through fusion processing.

[0150] In one embodiment, the contrast fusion features include positive contrast fusion features corresponding to positive contrast training samples and negative contrast fusion features corresponding to negative contrast training samples; determining the fusion loss based on the difference between the fusion features and the contrast fusion features includes: obtaining a positive feature difference value, the positive feature difference value is a feature difference value between the fusion feature and the corresponding positive contrast fusion feature; obtaining a negative feature difference value, the negative feature difference value is a feature difference value between the fusion feature and the corresponding negative contrast fusion feature; determining the fusion loss based on the positive feature difference value and the negative feature difference value.

[0151] Specifically, the computer device can obtain the fusion features corresponding to the positive contrast training samples to obtain the positive contrast fusion features, and obtain the fusion features corresponding to the negative contrast training samples to obtain the negative contrast fusion features, obtain the feature difference value between the fusion features corresponding to the same training sample and the corresponding positive contrast fusion features to obtain the positive feature difference value, obtain the feature difference value between the fusion features corresponding to the same training sample and the corresponding negative contrast fusion features to obtain the negative feature difference value, and finally determine the fusion loss value based on the positive feature difference value and the negative feature difference value. It can be understood that the positive contrast fusion features corresponding to the training samples are the fusion features of the positive contrast training samples corresponding to the training samples, and the negative contrast fusion features corresponding to the training samples are the fusion features of the negative contrast training samples corresponding to the training samples.

[0152] In one embodiment, the computer device may determine the fusion loss value by referring to the following formula (4), where x a is the fusion feature, x p is the positive contrast fusion feature, x nis the negative contrast fusion feature, ||x a -x p || represents x a and x p The L2 distance between them, that is, the positive feature difference value, ||x a -x n || represents x a and x n The L2 distance between them, that is, the negative feature difference value, the purpose of formula (4) is to make the distance between the training sample and the negative comparison training sample larger than the distance between the training sample and the positive comparison training sample by α, where α is the margin (interval term), and the value of α can be set as needed, for example, it can be set to 56.

[0153] L tri =max(||x a -x p ||-||x a -x n ||+α,0) (4)

[0154] It can be seen from formula (4) that the fusion loss value is 0 only when the distance between the training sample and the negative comparison training sample is greater than the distance between the training sample and the positive comparison training sample by α, otherwise, the fusion loss value is greater than 0. Therefore, in the process of reducing the loss value, the distance between the training sample and the negative comparison training sample develops in the direction of being greater than the distance between the current training sample and the positive comparison training sample by α, so that the fusion features obtained by the feature fusion model can better focus on the preservation of semantics.

[0155] In the above embodiment, since the fusion loss is determined based on the positive feature difference value and the negative feature difference value, the feature extraction model takes into account the impact of too small inter-class feature distance on classification when performing similarity metric learning, thereby improving the semantic accuracy of the features extracted by the feature extraction model.

[0156] In one embodiment, the above method also includes: determining the activated feature components from the feature components contained in each fused feature, and performing statistics on the activated feature components belonging to the same feature bit, and determining the activation degree of each feature bit corresponding to each fused feature according to the statistical results; respectively taking the feature components corresponding to the feature bits whose activation degrees meet preset conditions in each fused feature as target feature components; obtaining the quantization target corresponding to the feature component in the fused feature, and determining the quantization loss based on the difference between the feature component and the corresponding quantization target, including: obtaining the quantization target corresponding to the non-target feature component in the fused feature, and determining the quantization loss based on the difference between the non-target feature component and the corresponding quantization target; the non-target feature component is the feature component other than the target feature component in the fused feature.

[0157] Among them, the characteristic bit whose activation degree meets the preset conditions is the low activation characteristic bit, and the details can refer to the description in the above embodiment.

[0158] Specifically, the computer device can determine the feature components corresponding to the low-activation feature bits from the fused features to obtain the target feature components. When calculating the loss, these target feature components are discarded, and only the quantization loss is calculated for the non-target feature components other than the target feature components, thereby further reducing the impact of the features of the low-activation feature bits on the key features during the model training process.

[0159] In one embodiment, Fig.10 As shown, a sample retrieval method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be the server 104. In this embodiment, the feature fusion model processing method includes the following steps:

[0160] Step 1002: Acquire semantic features and non-semantic features corresponding to the query sample.

[0161] The query sample refers to a content sample for which similar samples need to be recalled from the sample database. For example, given an image A, similar images of image A are recalled from the database, and image A is the query sample.

[0162] Step 1004 , inputting the semantic features and the non-semantic features into the target fusion model for fusion processing to obtain the target fusion features corresponding to the query sample.

[0163] Among them, the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met; each fusion feature is obtained by fusion processing based on each target feature and using the feature fusion model; each target feature is obtained by setting the feature component corresponding to the feature bit whose activation degree meets the preset condition in each candidate feature to an inactive value; the activation degree of each feature bit is determined by counting the activated feature components belonging to the same feature bit and determining it according to the statistical results; the activated feature component is determined from the feature components contained in each candidate feature by taking at least one of the semantic feature and the non-semantic feature as a candidate feature; the semantic feature and the non-semantic feature correspond to the training samples in the training sample set.

[0164] Step 1006: retrieve a target retrieval sample corresponding to the query sample from the sample database based on the target fusion feature.

[0165] In one embodiment, the computer device can obtain the target fusion features of each content sample in each sample database based on the same method as the query sample, respectively calculate the feature distance between the target fusion features of the query sample and the target fusion features of each content sample, and determine the content sample whose feature distance meets the distance condition as the target retrieval sample. The distance condition is that the feature distance is less than a preset distance threshold, or the feature distance is arranged in ascending order before the sorting threshold. Both the distance threshold and the sorting threshold can be set as needed.

[0166] In the above sample retrieval method, on the one hand, the target fusion feature can be obtained by fusing the semantic features and non-semantic features of the query sample through the target fusion model. The target fusion feature has semantic measurement capability and non-semantic measurement capability because it fuses semantic features and non-semantic features. It can characterize the sample more accurately. Retrieval based on the target fusion feature can obtain a high accuracy retrieval result. On the other hand, since the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met, each fusion feature is obtained by fusing each target feature using the feature fusion model. Each target feature is obtained by setting the feature component corresponding to the feature bit whose activation degree meets the preset condition in each candidate feature to an inactive value. The activation degree of each feature bit is determined by statistically analyzing the activated feature components belonging to the same feature bit and determining the activated feature component according to the statistical result. The activated feature component is determined by taking at least one of the semantic features and non-semantic features as a candidate feature from the feature components contained in each candidate feature. Therefore, the influence of the feature of the low-activated feature bit on the training process can be reduced during the model training process, so that the model can better learn the key feature bit. The trained target fusion model can better perform feature fusion, further improving the accuracy of sample retrieval.

[0167] In one embodiment, the training steps of the target fusion model include: obtaining semantic features and non-semantic features corresponding to training samples in a training sample set; taking at least one of the semantic features and non-semantic features as a candidate feature, and determining an activated feature component from the feature components contained in each candidate feature; performing statistics on the activated feature components belonging to the same feature bit, and determining the activation levels of each feature bit corresponding to each candidate feature according to the statistical results; setting the feature components corresponding to the feature bits whose activation levels meet preset conditions in each candidate feature to inactive values ​​to obtain each target feature; performing fusion processing based on each target feature and using the feature fusion model to be trained to obtain each fused feature; training a feature fusion model based on each fused feature, and obtaining a target fusion model when a training stop condition is met.

[0168] The present application also provides an application scenario, which can be applied to the above-mentioned feature extraction model processing method and sample retrieval method. In this application scenario, the content is an image, and the training sample is an image sample. The feature extraction model processing method and sample retrieval method can be applied to image retrieval, and the sample database is deduplicated through image retrieval to remove duplicate images in the sample database.

[0169] The method provided in the embodiment of the present application realizes effective learning of fusion embedding by designing a dual-branch feature extraction network, a feature fusion method, a multi-stage learning method, etc., mainly including: 1) A network structure in which the dual-branch feature extraction network shares underlying parameters: considering that different network depths can learn different features, the shallow network learns global fine-grained textures, and the deep network can learn more abstract information of multiple local features (such as semantic information: learning dogs requires joint judgment of the head, body hair, limbs and tail, etc.), the network structure of the present application uses shallow and deep outputs to represent global features and semantic features respectively for the global similarity features and semantic features to be learned; 2) Feature extraction: Feature selection: Considering that if features are directly fused, it is easy to bring low-level activated hash bits into the target, the embodiment of the present application first performs feature screening before fusion, first screens out high-activated hash bits, and then splices such hash bits that are more meaningful to image representation as the basis for subsequent learning; 3) Learning method: Since features of different depths converge at different speeds, deep semantic features converge quickly, shallow metric features converge slowly, and it is more effective to learn fused features based on relatively stable basic features, the method provided in the embodiment of the present application designs a three-stage learning method, pre-learning shallow features that converge slowly, and then learning deep semantic features. After the features of the two tasks are stable, the fused features are learned.

[0170] Specifically, the following description is made by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a computer device 104. The above-mentioned feature extraction model processing method and sample retrieval method are applied in the application scenario as follows:

[0171] 1. Data Preparation

[0172] The training process uses triplet samples as input, but it is difficult to find suitable triplets from massive data, so positive sample pairs are marked, and then negative samples are mined through positive sample pairs to obtain triplet samples.

[0173] 1.1. Prepare labeled data. Obtain positive sample pairs: Label whether the image sample pairs are similar. For example, extract two images from a large amount of data as a pair and label whether each pair is similar enough. Since the model is used for duplicate elimination, two samples need to be extremely similar to be considered similar samples, as follows Figure 7 , 8 , or other images generated by image attack such as 9. The sample pairs marked as similar are positive sample pairs, and the sample pairs marked as dissimilar are negative sample pairs. In this embodiment, positive sample pairs are mainly collected, and negative sample pairs may not be collected. The negative samples of triples can be obtained by the following mining method.

[0174] 1.2. Triple data mining: Since the training metric learning features need to be composed of a triple sample (a, p, n) consisting of an anchor (i.e. the training sample in the previous text, hereinafter referred to as a), a positive sample (i.e. the positive contrast training sample in the previous text, hereinafter referred to as p), and a negative sample (i.e. the negative contrast training sample in the previous text, hereinafter referred to as n) for loss function learning, a and p in the triple sample constitute a positive sample pair, and a and n constitute a negative sample pair. In the learning task, the positive sample pair needs to have features that are close enough - the L2 distance is small enough (so that they can be retrieved from each other), while the negative sample pair needs to be far enough away. Each sample pair that has been obtained in the above annotation can be used as the anchor and positive of the triple (randomly select an image as the anchor). How to further mine negative samples (including difficult negative samples and global negative samples) is as follows:

[0175] Since the GPU (graphics processing unit) of computer equipment has limited memory and all positive samples need to be sent to GPU training in batches during the overall training, the mining of negative samples is carried out within a batch.

[0176] For each batch of positive sample pairs (assuming there are bs pairs), the following method is used to mine negative samples to obtain triple samples: For the x-anchor in a sample x pair (randomly select one as the anchor): calculate the distance between it and the x-anchor from the remaining bs-1 sample pairs (randomly select one image for each pair), sort them from small to large by distance, remove the top 5 images, and take the first 20 samples as difficult negative samples (because what needs to be learned is the characteristics of extremely similar samples, it is believed that the smaller the distance, the more similar, and in massive data, the probability of two images being extremely similar is relatively low, so the top 5 similar samples are directly removed, and the remaining samples can constitute the difficult negative samples in the triple), and form triples with x respectively, so each sample pair generates 20 triplets, and the entire batch obtains 20*bs triplets. In order to ensure the effectiveness of mining negative sample pairs, bs needs to be set to a relatively large value, such as 1024.

[0177] Triplet-based metric learning requires more difficult samples. If they are all simple samples, the model cannot learn a discriminative representation. In fact, it is not guaranteed that all of the first 20 negative samples are difficult negative samples, but it is guaranteed that there are more difficult samples, which is beneficial to learning.

[0178] 2. Training Process

[0179] The model to be trained includes three parts: semantic feature extraction model, non-semantic feature extraction model and feature fusion model. The semantic feature extraction model and non-semantic feature extraction model have a shared basic feature extraction network. The basic feature extraction network uses resnet101, which includes five parts: convolution layer 1, convolution layer 2-convolution layer 5. Convolution layer 1 is a 7×7×64 convolution with a stride of 2. Convolution layer 2 includes a 3×3 maximum pooling layer and 3 ResNet modules. Convolution layer 3-convolution layer 5 include 3 ResNet modules, 4 ResNet modules, 23 ResNet modules and 3 ResNet modules respectively. The parameters of the basic feature extraction network can be seen in Table 1.

[0180] Table 1

[0181]

[0182] The non-semantic feature extraction model also includes a first feature extraction layer, the input of the first feature extraction layer is the output of the convolution layer 5, and the output of the first feature extraction layer is a 1*64 vector whose value tends to -1 or 1. The first feature extraction layer includes a pooling layer and a hash quantization layer, and the parameters are shown in Table 2.

[0183] Table 2

[0184]

[0185]

[0186] The semantic feature extraction model also includes a second feature extraction layer, which includes a convolution layer 6 and a hash quantization layer, wherein the input of the convolution layer 6 is the output of the convolution layer 5, and the shallow features output by the convolution layer 5 are used to extract deeper semantic information. The input of the hash quantization layer is the output of the convolution layer 6 in Table 3, and the hash quantization layer is a 1*64 vector whose value tends to -1 or 1. In addition, a classification layer is included, and the number of classifications 100 is the 100 categories that need to be distinguished in this deduplication task (which can be adjusted according to actual conditions). The parameters of the second feature extraction layer are shown in Tables 3 and 4.

[0187] Table 3

[0188]

[0189] Table 4

[0190] The name of the layer Output size layer Hash Quantization Layer 1x64 Fully connected layer Classification layer 1x100 Fully connected layer

[0191] The input of the feature fusion model (i.e., the input of fusion layer 1) is the result of the concatenation of the outputs of Tables 2 and 4. The output of fusion layer 1 is further input into fusion layer 2, and the hash feature finally obtained by fusion layer 2 is output as a 1*96 vector with values ​​tending to -1 or 1. Considering that there may be useless or redundant hash bits in the non-semantic hash features output by the first feature extraction layer and the semantic hash features output by the second feature extraction layer, the actual effective bits of the final fusion feature are generally less than those of direct concatenation, so it is set to 96 (<128) and can be adjusted according to actual conditions. The parameters of the feature fusion model can be found in Table 5.

[0192] Table 5

[0193]

[0194]

[0195] It should be noted that the basic feature extraction network, hash quantization layer and classification layer can also use other model structures, such as the basic feature extraction network using the resnet18CNN module, and the hash quantization layer using a multi-layer fully connected layer connection. The specific training process is as follows:

[0196] 1.1 Parameter initialization: Convolutional layers 1 to 5 use the parameters of ResNet101 pre-trained on the preset data set. Other network layers such as the fully connected layer and convolutional layer 6 are initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0197] 1.2. Overall process: For a total of M sample pairs, each bs sample pairs is a batch, and there are M / bs batches in total. The above-mentioned triple mining is first performed on each batch of samples, and the triple samples are input into the model after being obtained. The model forward calculation is performed to calculate the corresponding loss value, and the gradient of each parameter is calculated with the help of gradient backpropagation; the model parameters are updated according to the gradient, and an epoch is completed after completing the full M / bs updates; the training ends after a total of K epochs (or when the average loss of 10 consecutive epochs has not decreased).

[0198] 1.3. Parameter learning settings: For the first stage: set Tables 1 and 2 as parameters to be learned; for the second stage, set Tables 1, 2, 3, and 4 as parameters to be learned; for the third stage, set Table 5 as parameters to be learned. When learning in each stage, only the parameters to be learned in that stage are updated.

[0199] 1.4. Model forward: During training, the neural network performs forward calculations on each image of the input triplet sample to obtain the outputs of Table 2, Table 4, and Table 5. Different outputs are used at different stages to calculate different losses, as follows:

[0200] 1) In the first stage, the output of Table 2 is used to calculate Loss1, refer to Fig.11 , each sample in the triplet sample is input as an input sample to the CNN network (i.e., convolutional layer 1 to convolutional layer 5). The CNN network is a network layer shared by the semantic feature extraction model and the non-semantic feature extraction model. The output features of the CNN network are input into the non-semantic feature extraction branch to extract the non-semantic hash feature. Based on the non-semantic hash feature, non-semantic embedding metric learning is performed and the triplet loss corresponding to the non-semantic feature is obtained by referring to the above formula (4). Then, the total loss of the first stage is calculated by referring to the following formula (5):

[0201] Loss1=w1L hash1-triplet +w2L hash1-coding (5)

[0202] Among them, Loss1 is the total loss of the first stage, L hash1-triplet is the triple loss corresponding to the non-semantic feature calculated with reference to the above formula (4), L hash1-coding The loss (i.e., quantization loss) used to calculate the quantization effect (whether it is close enough to -1 or 1) of the vector output by the hash quantization layer. Since the final application needs to be mapped to a binary value of -1 / 1 after quantization output, if each bit of Q output is close enough to 1 or -1, then the loss corresponding to the above metric learning is sufficient to describe the application of the quantization feature. Otherwise, even if the metric effect is good, it does not mean that the quantization representation in the application is good enough. Therefore, it is hoped that each bit of Q output is close enough to 1 or -1. For the quantization result of each image, refer to formula (2) and formula (3) in the previous article to calculate L coding loss, where Q i is the value of the quantized Q of the image at the i-th bit (a total of 256 bits in this embodiment), B i is the quantization target of the i-th position, B i Q i The target encoding of the quantized learning task generated by the sign function.

[0203] It should be noted that the application directly uses the sign function to generate a quantized binary vector (the binary value is selected as 0 / 1, so that it is convenient to represent it with bits with values ​​of 0 or 1 in computer devices) for the output Q, which can be used for image retrieval.

[0204] 2) In the second stage, the output of Table 2 and Table 4 are used to calculate Loss2. Fig.11, each sample in the triple sample is input as an input sample into the CNN network trained in the first stage, the output feature of CNN is input into the non-semantic feature extraction branch trained in the first stage to extract the non-semantic hash feature, and then input into the semantic feature extraction branch to extract the semantic hash feature, non-semantic embedding metric learning is performed based on the non-semantic hash feature and the triple loss corresponding to the non-semantic feature is obtained by referring to the above formula (4), semantic embedding metric learning is performed based on the semantic hash feature and the triple loss corresponding to the semantic feature is obtained by referring to the above formula (4), semantic information learning is performed based on the semantic hash feature, and the classification loss is calculated by referring to the classification loss function of the following formula (6), where p in formula (6) is ic represents the predicted probability that sample i belongs to category c, y ic Indicates whether the label of sample i is c. If it is c, yic=1, otherwise it is 0. N is the number of images in the batch. The classification loss of each batch is: Calculate the classification loss for each image and then average it.

[0205]

[0206] The computer equipment then calculates the total loss of the second stage by referring to the following formula (7):

[0207] Loss2=w1L hash1-triplet +w2L hash1-coding +w3L hash2-triplet +w4L hash2-coding +w5L class (7)

[0208] Among them, Loss2 is the total loss of the second stage, L hash1-triplet is the triple loss corresponding to the non-semantic feature calculated with reference to the above formula (4), L hash1-coding is the quantization loss corresponding to the non-semantic features calculated by referring to the above formulas (2) and (3), L hash2-triplet is the triple loss corresponding to the non-semantic feature calculated with reference to the above formula (4), L hash2-coding is the quantization loss corresponding to the non-semantic features calculated by referring to the above formulas (2) and (3), L class is the classification loss.

[0209] 3) In the third stage, the output of Table 5 is used to calculate Loss3. Fig.11, each sample in the triplet sample is input as an input sample into the CNN network trained in the second stage, the output feature of the CNN network is input into the non-semantic feature extraction branch trained in the second stage to extract the non-semantic hash feature, and then input into the semantic feature extraction branch trained in the second stage to extract the semantic hash feature, perform feature selection on the semantic hash feature and the non-semantic hash feature, that is, select the feature component of the low activation hash feature bit from the semantic hash feature and the non-semantic hash feature and set it to -1, then concatenate the selected features, and then input them into the feature fusion model for fusion learning to obtain the fusion feature, perform fusion embedding metric learning based on the fusion feature, and refer to the above formula (4) to obtain the triplet loss corresponding to the fusion feature, and then refer to the following formula (8) to calculate the total loss of the third stage:

[0210] Loss3=w6L hashall-triplet +w7L hashall-coding (8)

[0211] Among them, Loss3 is the total loss of the third stage, L hashall-triplet is the triple loss corresponding to the non-semantic feature calculated with reference to the above formula (4), L hashall-coding is the quantization loss corresponding to the non-semantic features calculated with reference to the above formulas (2) and (3).

[0212] It should be noted that in the above loss calculation, the weight of the metric learning loss can be set to w1=w3=w6=1, the weight of the hash code to w2=w4=w7=0.01, and the weight of the classification loss to w5=1.

[0213] 1.5. Model parameter update process: The loss calculated at each stage is used to perform gradient backward calculation to obtain the gradient of the network parameters that need to be updated at this stage, and the parameter values ​​are updated. The network parameters involved in learning use a learning rate of 0.0005, and the learning rate is adjusted to 0.1 times the original rate every 10 rounds; after each round of learning, the gradient is obtained by the loss feedback, and the network weights are updated according to different learning rates.

[0214] 3. Application of Quantitative Feature Retrieval

[0215] For all stock images, semantic features are extracted by the trained semantic feature extraction model, and non-semantic features are extracted by the trained non-semantic feature extraction model. The semantic features and non-semantic features are fused by the trained fusion model to obtain fused features. The fused features are activated by the sign function to obtain binary quantized features and stored. For the query image, semantic features are extracted by the trained semantic feature extraction model, and non-semantic features are extracted by the trained non-semantic feature extraction model. The semantic features and non-semantic features are fused by the trained fusion model to obtain fused features. The fused features are activated by the sign function to obtain binary quantized features, which are then compared with the binary quantized vectors in the stock. The Hamming distance calculation for the binary quantized features can accelerate the calculation efficiency (compared with the floating-point embedding features). After calculating the distance, the K most similar ones are sorted from small to large and returned, thereby realizing image deduplication retrieval.

[0216] This embodiment can produce the following beneficial effects:

[0217] 1. Reduce maintenance costs: By combining two feature extraction models to generate a unified hash feature, only one set of retrieval features and retrieval system needs to be maintained in the application, thereby reducing the maintenance cost of the application.

[0218] 2. Reduce computing resource consumption: By fusing features into one model, when features need to be extracted in an application, the computing resources required for model inference are reduced by nearly half compared to the method of using two models to generate two features.

[0219] 3. Improve application efficiency and effect: By fusing dual features into one feature, the recall of similar samples is improved compared with a single representation. On the other hand, by compressing the original two features (each 1x64 bits) into 1x96, the consumption of ongoing resources is reduced.

[0220] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0221] Based on the same inventive concept, the embodiment of the present application also provides a feature fusion model processing device for implementing the feature fusion model processing method involved above, and a sample retrieval device for implementing the sample retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more feature fusion model processing device embodiments provided below can refer to the limitations on the feature fusion model processing method above, and will not be repeated here.

[0222] In one embodiment, Fig.12 As shown, a feature fusion model processing device 1200 is provided, comprising:

[0223] A feature acquisition module 1202 is used to acquire semantic features and non-semantic features corresponding to training samples in a training sample set;

[0224] An activated component determination module 1204 is used to take at least one of the semantic feature and the non-semantic feature as a candidate feature, and determine an activated feature component from the feature components included in each candidate feature;

[0225] The activation level determination module 1206 is used to perform statistics on the activated feature components belonging to the same feature bit, and determine the activation level of each feature bit corresponding to each candidate feature according to the statistical results;

[0226] The target feature acquisition module 1208 is used to set the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each candidate feature to inactive values, so as to obtain each target feature;

[0227] A fusion processing module 1210 is used to perform fusion processing based on each target feature and using a feature fusion model to be trained to obtain each fusion feature;

[0228] The parameter adjustment module 1212 is used to train a feature fusion model based on each fusion feature, and obtain a target fusion model when the training stop condition is met; the target fusion model is used to fuse the semantic features and non-semantic features of the input sample to obtain the target fusion feature.

[0229] The feature fusion model processing device can obtain the target fusion feature by fusing the semantic features and non-semantic features of the input sample through the target fusion model. The target fusion feature has the semantic measurement capability and the non-semantic measurement capability because it fuses the semantic features and non-semantic features. It can characterize the sample more accurately with high accuracy. On the other hand, in the process of training the target fusion model, at least one of the semantic features and the non-semantic features is used as a candidate feature, and the activated feature component is determined from the feature components contained in each candidate feature. The activated feature components belonging to the same feature bit are counted, and the activation degree of each feature bit corresponding to each candidate feature is determined according to the statistical result. The feature components corresponding to the feature bits whose activation degree meets the preset conditions in each candidate feature are set to non-activated values ​​to obtain each target feature. Then, based on each target feature and using the feature fusion model to be trained, the fusion process is performed, which can reduce the influence of the features of the low-activated feature bits on the training process, so that the model can better learn the key feature bits, and the trained target fusion model can better perform feature fusion, further improving the accuracy of the target fusion feature.

[0230] In one embodiment, the activation degree determination module is used to count the number of activated feature components belonging to the same feature bit to obtain the activation number of each feature bit corresponding to each candidate feature; count the total number of features of the feature components belonging to each feature bit respectively; and determine the activation degree of each feature bit based on the activation number and the total number of features of each feature bit.

[0231] In one embodiment, the activation degree determination module is further used to determine the activation ratio corresponding to each feature bit based on the activation number of each feature bit and the total number of features; based on the activation ratio corresponding to each feature bit, calculate the activation entropy corresponding to each feature bit, and determine the activation entropy corresponding to each feature bit as the activation degree corresponding to each feature bit.

[0232] In one embodiment, an activation component determination module is used to map the feature components contained in each candidate feature according to a preset mapping method to obtain a mapping feature value of each feature component; compare the mapping feature value of each feature component with the activation value set by the preset mapping method; and determine the feature component corresponding to the mapping feature value that matches the comparison as the activated feature component.

[0233] In one embodiment, a target feature acquisition module is used to sort each feature bit based on the activation level of each feature bit, determine the feature bit whose activation level meets the preset conditions based on the sorting result, and obtain the target feature bit; set the feature component corresponding to the target feature bit in each candidate feature to an inactive value to obtain each target feature.

[0234] In one embodiment, an activation component determination module is used to determine the activation feature component corresponding to the semantic feature from the feature components included in each semantic feature, and to determine the activation feature component corresponding to the non-semantic feature from the feature components included in each non-semantic feature; an activation degree determination module is used to count the activation feature components belonging to the same feature bit in each semantic feature, and determine the activation degree of each feature bit corresponding to each semantic feature according to the statistical result; count the activation feature components belonging to the same feature bit in each non-semantic feature, and determine the activation degree of each feature bit corresponding to each non-semantic feature according to the statistical result; a target feature acquisition module is used to respectively set the feature components corresponding to the feature bits of each semantic feature whose activation degree meets the preset conditions to inactive values, so as to obtain each first target feature; and respectively set the feature components corresponding to the feature bits of each non-semantic feature whose activation degree meets the preset conditions to inactive values, so as to obtain each second target feature.

[0235] In one embodiment, a fusion processing module is used to splice the first target feature and the second target feature corresponding to the same training sample to obtain a spliced ​​feature; input the spliced ​​feature into a feature fusion model; the feature fusion model includes a first fully connected layer and a second fully connected layer; the spliced ​​feature is mapped through the first fully connected layer to obtain an intermediate feature; the intermediate feature is mapped through the second fully connected layer to obtain a fused feature; the intermediate feature has the same dimension as the spliced ​​feature, and the fused feature has a smaller dimension than the spliced ​​feature.

[0236] In one embodiment, the semantic features of the training samples are obtained by extracting features from the training samples using a trained target semantic feature extraction model, and the non-semantic features corresponding to the training samples are obtained by extracting features from the training samples using a trained target non-semantic feature extraction model; the above-mentioned device also includes: a feature extraction model training module, which is used to obtain comparative training samples corresponding to the training samples in the training sample set; the training samples and comparative training samples are respectively input as target samples into the initial semantic feature extraction model to be trained and the initial non-semantic feature extraction model to be trained; the semantic training features corresponding to each target sample are outputted through the initial semantic feature extraction model, and the semantic training features corresponding to each target sample are outputted through the initial non-semantic feature extraction model. The invention discloses a method for extracting non-semantic features corresponding to target samples respectively; obtaining semantic feature loss based on semantic training features corresponding to training samples and semantic training features corresponding to comparison training samples; obtaining first non-semantic feature loss based on non-semantic training features corresponding to training samples and non-semantic training features corresponding to comparison training samples; classifying training samples based on semantic training features of training samples to obtain classification results, and determining classification loss based on classification results; training initial semantic feature extraction model and initial non-semantic feature extraction model based on semantic feature loss, first non-semantic feature loss and classification loss, and obtaining trained target semantic feature extraction model and trained target non-semantic feature extraction model when training stop condition is met.

[0237] In one embodiment, there is a shared network layer between the initial semantic feature extraction model and the initial non-semantic feature extraction model; the feature extraction model training module is also used to perform convolution processing on each target sample through the shared network layer to obtain the shared features corresponding to each target sample; perform feature processing on each shared feature through the first feature processing layer of the initial semantic feature extraction model to obtain the semantic training features corresponding to each target sample; perform feature processing on each shared feature through the second feature processing layer of the initial non-semantic feature extraction model to obtain the non-semantic training features corresponding to each target sample; the network depth of the first feature processing layer is greater than that of the second feature processing layer.

[0238] In one embodiment, the feature extraction model training module is also used to input the training samples and the comparison training samples into the candidate non-semantic feature extraction model respectively, and output the non-semantic training features corresponding to the training samples and the non-semantic training features corresponding to the comparison training samples; based on the non-semantic training features corresponding to the training samples of the candidate non-semantic feature extraction model and the non-semantic training features corresponding to the comparison training samples, a second non-semantic feature loss is obtained; based on the second non-semantic feature loss, the model parameters of the candidate non-semantic feature extraction model are adjusted and training is continued, and when the training stop condition is met, an initial non-semantic feature extraction model is obtained.

[0239] In one embodiment, the fusion feature is a binary quantization feature, and the parameter adjustment module is used to obtain the quantization target corresponding to the feature component in the fusion feature, and determine the quantization loss based on the difference between the feature component in the fusion feature and the corresponding quantization target; obtain the comparative fusion feature corresponding to the comparative training sample of the training sample, and determine the fusion loss based on the difference between the fusion feature and the comparative fusion feature; the comparative fusion feature is obtained based on the feature fusion model; the quantization loss and the fusion loss are statistically calculated to obtain the target loss; the model parameters of the feature fusion model are adjusted based on the target loss and the training is continued until the training stop condition is met to obtain the target fusion model.

[0240] In one embodiment, the contrast fusion features include positive contrast fusion features corresponding to positive contrast training samples and negative contrast fusion features corresponding to negative contrast training samples; the parameter adjustment module is also used to obtain positive feature difference values, which are feature difference values ​​between the fusion feature and the corresponding positive contrast fusion feature; obtain negative feature difference values, which are feature difference values ​​between the fusion feature and the corresponding negative contrast fusion feature; and determine the fusion loss based on the positive feature difference values ​​and the negative feature difference values.

[0241] In one embodiment, the parameter adjustment module is also used to determine the activated feature components from the feature components included in each fused feature, and to perform statistics on the activated feature components belonging to the same feature bit, and to determine the activation degree of each feature bit corresponding to each fused feature according to the statistical results; the feature components corresponding to the feature bits whose activation degrees meet preset conditions in each fused feature are respectively used as target feature components; the quantization targets corresponding to the non-target feature components in the fused feature are obtained, and the quantization loss is determined based on the difference between the non-target feature components and the corresponding quantization targets; the non-target feature components are feature components other than the target feature components in the fused feature.

[0242] Each module in the feature fusion model processing device and the sample retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0243] In one embodiment, Fig.13 As shown, a sample retrieval device 1300 is provided, comprising:

[0244] A feature acquisition module 1302 is used to acquire semantic features and non-semantic features corresponding to the query sample;

[0245] The feature fusion module 1304 is used to input the semantic features and the non-semantic features into the target fusion model for fusion processing to obtain the target fusion features corresponding to the query sample; the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met; each fusion feature is obtained by fusion processing based on each target feature using the feature fusion model; each target feature is obtained by setting the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each candidate feature to non-activation values; the activation levels of each feature bit are determined by counting the activated feature components belonging to the same feature bit according to the statistical results; the activated feature components are determined by taking at least one of the semantic features and the non-semantic features as candidate features from the feature components contained in each candidate feature; the semantic features and the non-semantic features correspond to the training samples in the training sample set;

[0246] The sample retrieval module 1306 is used to retrieve the target retrieval sample corresponding to the query sample from the sample database based on the target fusion feature.

[0247] In the above sample retrieval method, on the one hand, the target fusion feature can be obtained by fusing the semantic features and non-semantic features of the query sample through the target fusion model. The target fusion feature has semantic measurement capability and non-semantic measurement capability because it fuses semantic features and non-semantic features. It can characterize the sample more accurately. Retrieval based on the target fusion feature can obtain a high accuracy retrieval result. On the other hand, since the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met, each fusion feature is obtained by fusing each target feature using the feature fusion model. Each target feature is obtained by setting the feature component corresponding to the feature bit whose activation degree meets the preset condition in each candidate feature to an inactive value. The activation degree of each feature bit is determined by statistically analyzing the activated feature components belonging to the same feature bit and determining the activated feature component according to the statistical result. The activated feature component is determined by taking at least one of the semantic features and non-semantic features as a candidate feature from the feature components contained in each candidate feature. Therefore, the influence of the feature of the low-activated feature bit on the training process can be reduced during the model training process, so that the model can better learn the key feature bit. The trained target fusion model can better perform feature fusion, further improving the accuracy of sample retrieval.

[0248] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device can be used to store training sample data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a feature fusion model processing method or a sample retrieval method is implemented.

[0249] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a feature fusion model processing method or a sample retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0250] Those skilled in the art will understand that Fig.14 , 15The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0251] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0252] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0253] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0254] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0255] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0256] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0257] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A feature fusion model processing method, characterized in that: The method comprises: Acquire semantic features and non-semantic features corresponding to training samples in a training sample set; the training samples are any one of text, audio or image; Taking at least one of the semantic feature and the non-semantic feature as a candidate feature, and determining an activated feature component from the feature components included in each of the candidate features; Counting the number of activated feature components belonging to the same feature bit to obtain the number of activations of each feature bit corresponding to each of the candidate features; counting the total number of features of the feature components belonging to each of the feature bits; determining the degree of activation of each of the feature bits based on the number of activations and the total number of features of each of the feature bits; The feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the candidate features are respectively set to inactive values ​​to obtain the target features; Based on each of the target features and using the feature fusion model to be trained, a fusion process is performed to obtain each fusion feature; The feature fusion model is trained based on each of the fusion features, and when a training stop condition is met, a target fusion model is obtained; the target fusion model is used to fuse the semantic features and non-semantic features of the input samples to obtain the target fusion features.

2. The method according to claim 1, characterized in that The determining the activation degree of each of the feature bits based on the activation quantity of each of the feature bits and the total quantity of features includes: Determine the activation ratio corresponding to each of the feature bits based on the activation quantity of each of the feature bits and the total number of features; Based on the activation ratios corresponding to the respective feature bits, the activation entropies corresponding to the respective feature bits are calculated, and the activation entropies corresponding to the respective feature bits are determined as the activation degrees corresponding to the respective feature bits.

3. The method according to claim 1, characterized in that The step of determining an activated feature component from the feature components included in each of the candidate features includes: Mapping the feature components contained in each of the candidate features according to a preset mapping method to obtain a mapping feature value of each feature component; Comparing the mapping characteristic value of each characteristic component with the activation value set by the preset mapping method; The feature components corresponding to the mapped feature values ​​that are consistent with the comparison are determined as activated feature components.

4. The method according to claim 1, characterized in that The step of respectively setting the feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the candidate features to inactive values ​​to obtain the target features includes: Based on the activation degree of each of the feature bits, the feature bits are sorted, and based on the sorting result, the feature bits whose activation degree meets the preset conditions are determined to obtain the target feature bits; The feature components corresponding to the target feature bits in each of the candidate features are respectively set to inactive values ​​to obtain each target feature.

5. The method according to claim 1, characterized in that Taking at least one of the semantic feature and the non-semantic feature as a candidate feature, and determining an activated feature component from feature components included in each of the candidate features, including: Determining an activated feature component corresponding to a semantic feature from the feature components included in each of the semantic features, and determining an activated feature component corresponding to a non-semantic feature from the feature components included in each of the non-semantic features; The counting of the number of activated feature components belonging to the same feature bit to obtain the number of activations of each feature bit corresponding to each candidate feature; counting the total number of features of the feature components belonging to each feature bit; and determining the degree of activation of each feature bit based on the number of activations and the total number of features of each feature bit, comprises: Counting the number of activated feature components belonging to the same feature bit in each of the semantic features to obtain the number of activations of each feature bit corresponding to each of the semantic features; counting the total number of features of the feature components belonging to each feature bit corresponding to each of the semantic features; determining the degree of activation of each feature bit corresponding to each of the semantic features based on the number of activations and the total number of features of each feature bit corresponding to each of the semantic features; Counting the number of activated feature components belonging to the same feature bit in each of the non-semantic features to obtain the number of activations of each feature bit corresponding to each of the non-semantic features; counting the total number of features of the feature components belonging to each feature bit corresponding to each of the non-semantic features; determining the degree of activation of each feature bit corresponding to each of the non-semantic features based on the number of activations and the total number of features of each feature bit corresponding to each of the non-semantic features; The feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the candidate features are respectively set to inactive values, and the target features are obtained as follows: The feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the semantic features are respectively set to inactive values ​​to obtain the first target features; The feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the non-semantic features are respectively set to inactive values ​​to obtain the second target features.

6. The method according to claim 5, characterized in that The fusion processing is performed based on each of the target features and using the feature fusion model to be trained to obtain each fusion feature, including: splicing the first target feature and the second target feature corresponding to the same training sample to obtain a spliced ​​feature; Inputting the concatenated features into the feature fusion model; the feature fusion model comprises a first fully connected layer and a second fully connected layer; Mapping the concatenated features through the first fully connected layer to obtain intermediate features; Mapping the intermediate features through the second fully connected layer to obtain the fused features; The intermediate feature has the same dimension as the splicing feature, and the fusion feature has a smaller dimension than the splicing feature.

7. The method according to claim 1, characterized in that The semantic features of the training samples are obtained by extracting features from the training samples using a trained target semantic feature extraction model, and the non-semantic features corresponding to the training samples are obtained by extracting features from the training samples using a trained target non-semantic feature extraction model; The training steps of the target semantic feature extraction model and the target non-semantic feature extraction model include: Obtaining a comparison training sample corresponding to a training sample in a training sample set; Input the training sample and the comparison training sample as target samples into an initial semantic feature extraction model to be trained and an initial non-semantic feature extraction model to be trained respectively; Outputting semantic training features corresponding to each of the target samples through the initial semantic feature extraction model, and outputting non-semantic training features corresponding to each of the target samples through the initial non-semantic feature extraction model; Obtaining a semantic feature loss based on the semantic training features corresponding to the training sample and the semantic training features corresponding to the comparison training sample; Obtaining a first non-semantic feature loss based on the non-semantic training features corresponding to the training sample and the non-semantic training features corresponding to the comparison training sample; Classifying the training samples based on the semantic training features of the training samples to obtain classification results, and determining classification losses based on the classification results; The initial semantic feature extraction model and the initial non-semantic feature extraction model are trained based on the semantic feature loss, the first non-semantic feature loss and the classification loss. When a training stop condition is met, a trained target semantic feature extraction model and a trained target non-semantic feature extraction model are obtained.

8. The method according to claim 7, characterized in that The initial semantic feature extraction model and the initial non-semantic feature extraction model have a shared network layer; the outputting of semantic training features corresponding to each of the target samples through the initial semantic feature extraction model, and the outputting of non-semantic training features corresponding to each of the target samples through the initial non-semantic feature extraction model include: Performing convolution processing on each target sample through the shared network layer to obtain the shared features corresponding to each target sample; Performing feature processing on each of the shared features through the first feature processing layer of the initial semantic feature extraction model to obtain semantic training features corresponding to each of the target samples; Performing feature processing on each of the shared features through the second feature processing layer of the initial non-semantic feature extraction model to obtain non-semantic training features corresponding to each of the target samples; The network depth of the first feature processing layer is greater than the network depth of the second feature processing layer.

9. The method according to claim 7, characterized in that: The initial non-semantic feature extraction model is obtained by the following steps: Inputting the training sample and the comparison training sample into a candidate non-semantic feature extraction model respectively, and outputting the non-semantic training features corresponding to the training sample and the non-semantic training features corresponding to the comparison training sample; Obtaining a second non-semantic feature loss based on the non-semantic training features corresponding to the training sample of the candidate non-semantic feature extraction model and the non-semantic training features corresponding to the comparison training sample; Based on the second non-semantic feature loss, the model parameters of the candidate non-semantic feature extraction model are adjusted and training is continued. When a training stop condition is met, the initial non-semantic feature extraction model is obtained.

10. The method according to claim 1, characterized in that The fusion feature is a binary quantized feature, and the feature fusion model is trained based on each of the fusion features. When a training stop condition is met, a target fusion model is obtained, including: Obtaining a quantization target corresponding to a feature component in the fused feature, and determining a quantization loss based on a difference between the feature component in the fused feature and the corresponding quantization target; Obtaining a comparative fusion feature corresponding to a comparative training sample of the training sample, and determining a fusion loss based on a difference between the fusion feature and the comparative fusion feature; the comparative fusion feature is obtained based on the feature fusion model; Counting the quantization loss and the fusion loss to obtain the target loss; The model parameters of the feature fusion model are adjusted based on the target loss and the training is continued until a training stop condition is met to obtain a target fusion model.

11. The method according to claim 10, characterized in that The contrast fusion feature includes a positive contrast fusion feature corresponding to a positive contrast training sample and a negative contrast fusion feature corresponding to a negative contrast training sample; and determining the fusion loss based on the difference between the fusion feature and the contrast fusion feature includes: Obtaining a positive feature difference value, where the positive feature difference value is a feature difference value between the fusion feature and a corresponding positive contrast fusion feature; Obtain a negative feature difference value, where the negative feature difference value is a feature difference value between the fusion feature and a corresponding negative contrast fusion feature; A fusion loss is determined based on the positive feature difference value and the negative feature difference value.

12. The method according to claim 10, characterized in that The method further comprises: Determine the activated feature components from the feature components included in each of the fused features, and perform statistics on the activated feature components belonging to the same feature bit, and determine the activation degree of each feature bit corresponding to each of the fused features according to the statistical results; Respectively taking the feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the fused features as the target feature components; The obtaining of the quantization target corresponding to the feature component in the fused feature, and determining the quantization loss based on the difference between the feature component in the fused feature and the corresponding quantization target, includes: A quantization target corresponding to a non-target feature component in the fused feature is obtained, and a quantization loss is determined based on a difference between the non-target feature component and the corresponding quantization target; the non-target feature component is a feature component other than the target feature component in the fused feature.

13. A sample retrieval method, characterized in that: The method comprises: Obtaining semantic features and non-semantic features corresponding to the query sample; The semantic features and non-semantic features are input into the target fusion model for fusion processing to obtain the target fusion features corresponding to the query sample; the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met; each of the fusion features is obtained by fusion processing based on each target feature and using the feature fusion model; each of the target features is obtained by setting the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each candidate feature to non-activation values; the activation levels of each feature bit are determined based on the activation numbers and the total number of features of each feature bit; the activation numbers of each feature bit are obtained by counting the number of activated feature components belonging to the same feature bit; the total number of features is obtained by counting the feature components belonging to each feature bit; the activated feature components are determined from the feature components contained in each of the candidate features by taking at least one of the semantic feature and the non-semantic feature as a candidate feature; the semantic features and non-semantic features correspond to training samples in the training sample set; the training samples are any one of text, audio or image; A target retrieval sample corresponding to the query sample is retrieved from a sample database based on the target fusion feature.

14. A feature fusion model processing device, characterized in that: The device comprises: A feature acquisition module, used to acquire semantic features and non-semantic features corresponding to training samples in a training sample set; the training samples are any one of text, audio or image; an activation component determination module, configured to take at least one of the semantic feature and the non-semantic feature as a candidate feature, and determine an activation feature component from feature components included in each of the candidate features; The activation degree determination module is used to count the number of activated feature components belonging to the same feature bit to obtain the activation number of each feature bit corresponding to each candidate feature; respectively count the total number of features of the feature components belonging to each feature bit; and determine the activation degree of each feature bit based on the activation number and the total number of features of each feature bit; A target feature acquisition module is used to set the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each of the candidate features to inactive values, so as to obtain each target feature; A fusion processing module, used for performing fusion processing based on each of the target features and using a feature fusion model to be trained to obtain each fusion feature; The parameter adjustment module is used to train the feature fusion model based on each of the fusion features, and obtain a target fusion model when the training stop condition is met; the target fusion model is used to fuse the semantic features and non-semantic features of the input samples to obtain the target fusion features.

15. The device according to claim 14, characterized in that The activation degree determination module is also used to determine the activation ratio corresponding to each of the feature bits based on the activation number of each of the feature bits and the total number of features; based on the activation ratio corresponding to each of the feature bits, calculate the activation entropy corresponding to each of the feature bits, and determine the activation entropy corresponding to each of the feature bits as the activation degree corresponding to each of the feature bits.

16. The device according to claim 14, characterized in that The activation component determination module is also used to map the feature components contained in each of the candidate features according to a preset mapping method to obtain a mapping feature value of each feature component; compare the mapping feature value of each feature component with the activation value set by the preset mapping method; and determine the feature component corresponding to the mapping feature value that matches the comparison as the activated feature component.

17. The device according to claim 14, characterized in that The target feature acquisition module is also used to sort the feature bits based on the activation degree of each feature bit, determine the feature bit whose activation degree meets the preset conditions based on the sorting result, and obtain the target feature bit; respectively, The feature components corresponding to the target feature bits are set to inactive values ​​to obtain each target feature.

18. The device according to claim 14, characterized in that The activation component determination module is further used to determine the activation feature component corresponding to the semantic feature from the feature components included in each of the semantic features, and to determine the activation feature component corresponding to the non-semantic feature from the feature components included in each of the non-semantic features; The activation degree determination module is also used to count the number of activated feature components belonging to the same feature bit in each of the semantic features to obtain the activation number of each feature bit corresponding to each of the semantic features; respectively count the total number of features of the feature components of each feature bit corresponding to each of the semantic features; determine the activation degree of each feature bit corresponding to each of the semantic features based on the activation number and the total number of features of each feature bit corresponding to each of the semantic features; count the number of activated feature components belonging to the same feature bit in each of the non-semantic features to obtain the activation number of each feature bit corresponding to each of the non-semantic features; respectively count the total number of features of the feature components of each feature bit corresponding to each of the non-semantic features; determine the activation degree of each feature bit corresponding to each of the non-semantic features based on the activation number and the total number of features of each feature bit corresponding to each of the non-semantic features; The target feature acquisition module is also used to set the feature components corresponding to the feature bits whose activation degrees meet the preset conditions in each of the semantic features to inactive values, so as to obtain each first target feature; The feature components corresponding to the feature bits whose activation degrees satisfy the preset conditions in the non-semantic features are respectively set to inactive values ​​to obtain the second target features.

19. The device according to claim 18, characterized in that The fusion processing module is also used to splice the first target feature and the second target feature corresponding to the same training sample to obtain a spliced ​​feature; input the spliced ​​feature into the feature fusion model; the feature fusion model includes a first fully connected layer and a second fully connected layer; the spliced ​​feature is mapped through the first fully connected layer to obtain an intermediate feature; the intermediate feature is mapped through the second fully connected layer to obtain the fused feature; the intermediate feature has the same dimension as the spliced ​​feature, and the fused feature has a smaller dimension than the spliced ​​feature.

20. The device according to claim 14, characterized in that The semantic features of the training samples are obtained by extracting features from the training samples using a trained target semantic feature extraction model, and the non-semantic features corresponding to the training samples are obtained by extracting features from the training samples using a trained target non-semantic feature extraction model; The device also includes: A feature extraction model training module is used to obtain comparative training samples corresponding to training samples in a training sample set; input the training samples and the comparative training samples as target samples into an initial semantic feature extraction model to be trained and an initial non-semantic feature extraction model to be trained; output the semantic training features corresponding to each of the target samples through the initial semantic feature extraction model, and output the non-semantic training features corresponding to each of the target samples through the initial non-semantic feature extraction model; obtain a semantic feature loss based on the semantic training features corresponding to the training samples and the semantic training features corresponding to the comparative training samples; obtain a first non-semantic feature loss based on the non-semantic training features corresponding to the training samples and the non-semantic training features corresponding to the comparative training samples; classify the training samples based on the semantic training features of the training samples to obtain a classification result, and determine a classification loss based on the classification result; train the initial semantic feature extraction model and the initial non-semantic feature extraction model based on the semantic feature loss, the first non-semantic feature loss and the classification loss, and obtain a trained target semantic feature extraction model and a trained target non-semantic feature extraction model when a training stop condition is met.

21. The device according to claim 20, characterized in that The initial semantic feature extraction model and the initial non-semantic feature extraction model have a shared network layer; the feature extraction model training module is also used to perform convolution processing on each target sample through the shared network layer to obtain the shared features corresponding to each target sample; perform feature processing on each shared feature through the first feature processing layer of the initial semantic feature extraction model to obtain the semantic training features corresponding to each target sample; perform feature processing on each shared feature through the second feature processing layer of the initial non-semantic feature extraction model to obtain the non-semantic training features corresponding to each target sample; the network depth of the first feature processing layer is greater than the network depth of the second feature processing layer.

22. The device according to claim 20, characterized in that The feature extraction model training module is also used to input the training sample and the comparison training sample into a candidate non-semantic feature extraction model respectively, and output the non-semantic training features corresponding to the training sample and the non-semantic training features corresponding to the comparison training sample; based on the non-semantic training features corresponding to the training sample and the non-semantic training features corresponding to the comparison training sample of the candidate non-semantic feature extraction model, a second non-semantic feature loss is obtained; Based on the second non-semantic feature loss, the model parameters of the candidate non-semantic feature extraction model are adjusted and training is continued. When a training stop condition is met, the initial non-semantic feature extraction model is obtained.

23. The device according to claim 14, characterized in that The fused feature is a binary quantized feature, and the parameter adjustment module is further used to obtain a quantization target corresponding to a feature component in the fused feature, and determine a quantization loss based on a difference between the feature component in the fused feature and the corresponding quantization target; Obtain a comparative fusion feature corresponding to a comparative training sample of the training sample, and determine a fusion loss based on a difference between the fusion feature and the comparative fusion feature; the comparative fusion feature is obtained based on the feature fusion model; the quantization loss and the fusion loss are counted to obtain a target loss; based on the target loss, the model parameters of the feature fusion model are adjusted and training is continued until a training stop condition is met to obtain a target fusion model.

24. The device according to claim 23, characterized in that The contrast fusion features include positive contrast fusion features corresponding to positive contrast training samples and negative contrast fusion features corresponding to negative contrast training samples; the parameter adjustment module is also used to obtain positive feature difference values, which are feature difference values ​​between the fusion features and the corresponding positive contrast fusion features; obtain negative feature difference values, which are feature difference values ​​between the fusion features and the corresponding negative contrast fusion features; and determine the fusion loss based on the positive feature difference values ​​and the negative feature difference values.

25. The device according to claim 23, characterized in that The parameter adjustment module is also used to determine the activated feature components from the feature components included in each of the fused features, and to perform statistics on the activated feature components belonging to the same feature bit, and to determine the activation levels of each feature bit corresponding to each of the fused features according to the statistical results; to respectively use the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each of the fused features as target feature components; to obtain the quantization targets corresponding to the non-target feature components in the fused features, and to determine the quantization loss based on the difference between the non-target feature components and the corresponding quantization targets; the non-target feature components are feature components other than the target feature components in the fused features.

26. A sample retrieval device, characterized in that: The device comprises: A feature acquisition module, used to acquire semantic features and non-semantic features corresponding to the query sample; A feature fusion module, used for inputting the semantic features and non-semantic features into a target fusion model for fusion processing to obtain a target fusion feature corresponding to the query sample; the target fusion model is obtained when the feature fusion model to be trained is trained based on each fusion feature until the training stop condition is met; each of the fusion features is obtained by fusion processing based on each target feature and using the feature fusion model; each of the target features is obtained by setting the feature components corresponding to the feature bits whose activation levels meet the preset conditions in each candidate feature to inactive values; the activation levels of each feature bit are determined based on the activation numbers and the total number of features of each feature bit; the activation numbers of each feature bit are obtained by counting the number of activated feature components belonging to the same feature bit; the total number of features is obtained by counting the feature components belonging to each feature bit; the activated feature components are determined from the feature components contained in each of the candidate features by taking at least one of the semantic feature and the non-semantic feature as a candidate feature; the semantic features and non-semantic features correspond to training samples in a training sample set; the training samples are any one of text, audio or image; The sample retrieval module is used to retrieve the target retrieval sample corresponding to the query sample from the sample database based on the target fusion feature.

27. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

29. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images