Feature fusion model training and sample retrieval method, device and computer equipment

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for feature fusion model training and sample retrieval. The method comprises: inputting training samples into an initial feature fusion model to obtain training semantic features and training global features, and classifying and identifying based on the training semantic features to obtain training initial categories, splicing the training semantic features and the training global features to obtain training splicing features, calculating autocorrelation features based on the training splicing features to obtain autocorrelation features, calculating self-attention weights based on the autocorrelation features to obtain self-attention weights, and adjusting the training splicing features by the self-attention weights to obtain training fusion features. The initial feature fusion model is updated based on the training global features, training semantic features, training fusion features, training initial categories and training sample category labels, and a target fusion model is obtained by iterating in a loop. The use of this method can improve the accuracy of feature fusion.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a feature fusion model training and sample retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of artificial intelligence technology, sample recognition technology has emerged. When performing sample recognition, it is usually necessary to characterize the sample first, and then classify and recognize it through the sample characterization. In order to improve the accuracy of classification and recognition, different representations of the sample are usually identified, and then the different representations are fused, and the fused features are used for classification and recognition. At present, different representations of the sample are usually directly spliced ​​to obtain the final sample features, for example, two 1x64-dimensional features are spliced ​​into 1x128-dimensional features. However, directly splicing different representations will result in the problem of information redundancy between different representations, making the obtained sample features not accurate enough. Summary of the invention

[0003] Based on this, it is necessary to provide a feature fusion model training and sample retrieval method, device, computer equipment, computer-readable storage medium and computer program product that can reduce information fusion and improve the accuracy of sample features in response to the above technical problems.

[0004] In one aspect, the present application provides a method for training a feature fusion model. The method comprises:

[0005] Obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model;

[0006] The training samples are subjected to feature extraction through the initial feature fusion model to obtain training semantic features and training global features, and classification and recognition are performed based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0007] The training semantic features and the training global features are spliced ​​through the initial feature fusion model to obtain the training spliced ​​features, the autocorrelation features are calculated based on the training spliced ​​features to obtain the autocorrelation features, the self-attention weights are calculated based on the autocorrelation features to obtain the self-attention weights, and the training spliced ​​features are adjusted by the self-attention weights to obtain the training fusion features;

[0008] The feature loss is calculated based on the training global features, the training semantic features and the training fusion features to obtain the initial feature loss, and the category loss is calculated based on the training initial category and the training sample category label to obtain the initial category loss;

[0009] The initial feature fusion model is updated based on the initial feature loss and the initial category loss, and the training samples and the corresponding training sample category labels are returned. The steps of inputting the training samples into the initial feature fusion model are iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0010] On the other hand, the present application also provides a feature fusion model training device. The device comprises:

[0011] An input module is used to obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model;

[0012] The initial recognition module is used to extract features from the training samples through the initial feature fusion model to obtain training semantic features and training global features, and to perform classification and recognition based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0013] An initial fusion module is used to splice the training semantic features and the training global features through an initial feature fusion model to obtain training spliced ​​features, calculate autocorrelation features based on the training spliced ​​features to obtain autocorrelation features, calculate self-attention weights based on the autocorrelation features to obtain self-attention weights, and adjust the training spliced ​​features through the self-attention weights to obtain training fusion features;

[0014] A loss calculation module is used to calculate feature loss based on training global features, training semantic features and training fusion features to obtain initial feature loss, and to calculate category loss based on training initial categories and training sample category labels to obtain initial category loss;

[0015] The iteration module is used to update the initial feature fusion model based on the initial feature loss and the initial category loss, and return the training samples and the corresponding training sample category labels. The step of inputting the training samples into the initial feature fusion model is iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0016] On the other hand, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0017] Obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model;

[0018] The training samples are subjected to feature extraction through the initial feature fusion model to obtain training semantic features and training global features, and classification and recognition are performed based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0019] The training semantic features and the training global features are spliced ​​through the initial feature fusion model to obtain the training spliced ​​features, the autocorrelation features are calculated based on the training spliced ​​features to obtain the autocorrelation features, the self-attention weights are calculated based on the autocorrelation features to obtain the self-attention weights, and the training spliced ​​features are adjusted by the self-attention weights to obtain the training fusion features;

[0020] The feature loss is calculated based on the training global features, the training semantic features and the training fusion features to obtain the initial feature loss, and the category loss is calculated based on the training initial category and the training sample category label to obtain the initial category loss;

[0021] The initial feature fusion model is updated based on the initial feature loss and the initial category loss, and the training samples and the corresponding training sample category labels are returned. The steps of inputting the training samples into the initial feature fusion model are iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0022] On the other hand, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0023] Obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model;

[0024] The training samples are subjected to feature extraction through the initial feature fusion model to obtain training semantic features and training global features, and classification and recognition are performed based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0025] The training semantic features and the training global features are spliced ​​through the initial feature fusion model to obtain the training spliced ​​features, the autocorrelation features are calculated based on the training spliced ​​features to obtain the autocorrelation features, the self-attention weights are calculated based on the autocorrelation features to obtain the self-attention weights, and the training spliced ​​features are adjusted by the self-attention weights to obtain the training fusion features;

[0026] The feature loss is calculated based on the training global features, the training semantic features and the training fusion features to obtain the initial feature loss, and the category loss is calculated based on the training initial category and the training sample category label to obtain the initial category loss;

[0027] The initial feature fusion model is updated based on the initial feature loss and the initial category loss, and the training samples and the corresponding training sample category labels are returned. The steps of inputting the training samples into the initial feature fusion model are iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0028] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0029] Obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model;

[0030] The training samples are subjected to feature extraction through the initial feature fusion model to obtain training semantic features and training global features, and classification and recognition are performed based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0031] The training semantic features and the training global features are spliced ​​through the initial feature fusion model to obtain the training spliced ​​features, the autocorrelation features are calculated based on the training spliced ​​features to obtain the autocorrelation features, the self-attention weights are calculated based on the autocorrelation features to obtain the self-attention weights, and the training spliced ​​features are adjusted by the self-attention weights to obtain the training fusion features;

[0032] The feature loss is calculated based on the training global features, the training semantic features and the training fusion features to obtain the initial feature loss, and the category loss is calculated based on the training initial category and the training sample category label to obtain the initial category loss;

[0033] The initial feature fusion model is updated based on the initial feature loss and the initial category loss, and the training samples and the corresponding training sample category labels are returned. The steps of inputting the training samples into the initial feature fusion model are iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0034] The above-mentioned feature fusion model training method, device, computer equipment, storage medium and computer program product, when training the initial feature fusion model, splices the extracted training semantic features and training global features to obtain training spliced ​​features, then calculates the autocorrelation features of the training spliced ​​features to obtain the autocorrelation features, and then uses the autocorrelation features to calculate the self-attention weights to obtain the self-attention weights, and adjusts the training spliced ​​features by the self-attention weights to obtain training fusion features, so that the obtained training fusion features can avoid information redundancy, thereby improving the accuracy of the obtained training fusion features, and then continuously iterates to train the initial feature fusion model, so that the target fusion model obtained by training improves the accuracy of fusing semantic features and global features.

[0035] In one aspect, the present application provides a sample retrieval method. The method comprises:

[0036] Obtaining query fusion features corresponding to the query sample, where the query fusion features are obtained by inputting the query sample into the target fusion model to extract fusion features;

[0037] Extracting features of query samples through the target fusion model to obtain query semantic features and query global features, splicing the query semantic features and query global features to obtain query splicing features, calculating autocorrelation features based on the query splicing features to obtain query autocorrelation features, calculating self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjusting the query splicing features through the query self-attention weights to obtain query fusion features;

[0038] Sample retrieval is performed based on the query fusion features to obtain the target retrieval sample corresponding to the query sample.

[0039] On the other hand, the present application also provides a sample retrieval device. The device comprises:

[0040] A feature acquisition module is used to acquire query fusion features corresponding to the query sample. The query fusion features are obtained by inputting the query sample into the target fusion model to extract fusion features.

[0041] A feature extraction module is used to extract features from query samples through a target fusion model to obtain query semantic features and query global features, concatenate the query semantic features and query global features to obtain query concatenation features, calculate autocorrelation features based on the query concatenation features to obtain query autocorrelation features, calculate self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjust the query concatenation features through the query self-attention weights to obtain query fusion features;

[0042] The retrieval module is used to perform sample retrieval based on the query fusion features to obtain the target retrieval sample corresponding to the query sample.

[0043] On the other hand, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0044] Obtaining query fusion features corresponding to the query sample, where the query fusion features are obtained by inputting the query sample into the target fusion model to extract fusion features;

[0045] Extracting features of query samples through the target fusion model to obtain query semantic features and query global features, splicing the query semantic features and query global features to obtain query splicing features, calculating autocorrelation features based on the query splicing features to obtain query autocorrelation features, calculating self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjusting the query splicing features through the query self-attention weights to obtain query fusion features;

[0046] Sample retrieval is performed based on the query fusion features to obtain the target retrieval sample corresponding to the query sample.

[0047] On the other hand, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0048] Obtaining query fusion features corresponding to the query sample, where the query fusion features are obtained by inputting the query sample into the target fusion model to extract fusion features;

[0049] Extracting features of query samples through the target fusion model to obtain query semantic features and query global features, splicing the query semantic features and query global features to obtain query splicing features, calculating autocorrelation features based on the query splicing features to obtain query autocorrelation features, calculating self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjusting the query splicing features through the query self-attention weights to obtain query fusion features;

[0050] Sample retrieval is performed based on the query fusion features to obtain the target retrieval sample corresponding to the query sample.

[0051] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Obtaining query fusion features corresponding to the query sample, where the query fusion features are obtained by inputting the query sample into the target fusion model to extract fusion features;

[0053] The target fusion model extracts features based on query samples to obtain query semantic features and query global features, concatenates the query semantic features and query global features to obtain query concatenation features, calculates autocorrelation features based on the query concatenation features to obtain query autocorrelation features, calculates self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjusts the query concatenation features by the query self-attention weights to obtain query fusion features;

[0054] Sample retrieval is performed based on the query fusion features to obtain the target retrieval sample corresponding to the query sample.

[0055] The above-mentioned sample retrieval method, apparatus, computer equipment, storage medium and computer program product perform sample retrieval by using query fusion features. Since the query fusion features are obtained by using a feature fusion model to perform feature extraction based on query samples to obtain query semantic features and query global features, the query semantic features and query global features are spliced ​​to obtain query splicing features, autocorrelation features are calculated based on the query splicing features to obtain query autocorrelation features, self-attention weights are calculated based on the query autocorrelation features to obtain query self-attention weights, and the query splicing features are adjusted by the query self-attention weights, which can more accurately characterize the query samples, thereby making the obtained target retrieval samples more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 An application environment diagram of a feature fusion model training method in an embodiment;

[0057] Figure 2 A schematic diagram of a flow chart of a feature fusion model training method in one embodiment;

[0058] Figure 3 A schematic diagram of a process for obtaining an initial feature fusion model in one embodiment;

[0059] Figure 4 A schematic diagram of a process for obtaining initial categories for training in one embodiment;

[0060] Figure 5 A schematic diagram of a process for obtaining training fusion features in one embodiment;

[0061] Figure 6 A schematic diagram of obtaining training fusion features in a specific embodiment;

[0062] Figure 7 A schematic diagram of a process for obtaining an initial feature loss in one embodiment;

[0063] Figure 8 A schematic diagram of a process for obtaining global loss information in one embodiment;

[0064] Fig. 9 is a schematic diagram of a process for obtaining negative samples in one embodiment;

[0065] Fig.10 is a schematic diagram of an image sample pair in a specific embodiment;

[0066] Fig.11 A schematic diagram of a process for obtaining semantic loss information in one embodiment;

[0067] Fig.12 A schematic diagram of a process for obtaining fusion loss information in one embodiment;

[0068] Fig.13 A schematic diagram of a process of obtaining an input sample index in one embodiment;

[0069] Fig.14 A schematic diagram of a flow chart of obtaining an input sample index in another embodiment;

[0070] Fig.15 is a flow chart of a sample retrieval method in one embodiment;

[0071] Fig.16 A flowchart of a feature fusion model training method in a specific embodiment;

[0072] Fig.17 A schematic diagram of a training framework of a target fusion model in a specific embodiment;

[0073] Fig.18 is a structural block diagram of a feature fusion model training device in an embodiment;

[0074] Fig.19 is a structural block diagram of a sample retrieval device in one embodiment;

[0075] Fig. 20 is an internal structure diagram of a computer device in one embodiment;

[0076] Fig.21 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0078] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0079] The solution provided in the embodiments of the present application involves technologies such as image recognition and image retrieval by artificial intelligence, which are specifically described by the following embodiments:

[0080] The feature fusion model training method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can receive the model training instruction sent by the terminal 102, and the server 104 obtains the training sample and the corresponding training sample category label from the data storage system according to the model training instruction, and inputs the training sample into the initial feature fusion model; the initial feature fusion model extracts features based on the training samples to obtain training semantic features and training global features, and performs classification and recognition based on the training semantic features to obtain the training initial category corresponding to the training sample; the initial feature fusion model splices the training semantic features and the training global features to obtain the training splicing features, calculates the autocorrelation features based on the training splicing features to obtain the autocorrelation features, calculates the self-attention weights based on the autocorrelation features to obtain the self-attention weights, and adjusts the self-attention weights through the self-attention weights Training splicing features to obtain training fusion features; the server 104 calculates feature loss based on training global features, training semantic features and training fusion features to obtain initial feature loss, and calculates category loss based on training initial categories and training sample category labels to obtain initial category loss; the server 104 updates the initial feature fusion model based on the initial feature loss and initial category loss, and returns to obtain training samples and corresponding training sample category labels, and iterates the steps of inputting training samples into the initial feature fusion model until the training completion condition is met, and obtains the target fusion model, which is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and uses the target fusion features as the index of the input samples. Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc., and the terminal 102 can also be installed with a client, such as a model training service client, a data retrieval service client, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0081] In one embodiment, Figure 2 As shown in the figure, a feature fusion model training method is provided, which is applied to Figure 1 The server in the example is used for explanation. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the following steps are included:

[0082] Step 202: Obtain training samples and corresponding training sample category labels, and input the training samples into the initial feature fusion model.

[0083] Among them, the training sample refers to the sample used in training, and there can be multiple samples. The sample can be samples of different types, such as images, texts, videos, audios, etc. The training sample category label refers to the classification label corresponding to the training sample, and the classification label refers to the category to which the sample object belongs, such as the category to which the object in the image sample belongs, such as the category to which the content in the text belongs. The initial feature fusion model refers to the feature fusion model with model parameter initialization. The feature fusion model is used to fuse different features to obtain fused features. The parameter initialization can be random initialization or pre-trained parameter initialization, etc.

[0084] Specifically, the server can obtain training samples and corresponding training sample category labels from the database, or from the data service provider, or from the terminal. Then, the neural network is used to establish an initial feature fusion model, and the training samples are input into the initial feature fusion model for training, that is, forward calculation is performed first, and then reverse iteration is performed.

[0085] Step 204 , extracting features from the training samples through the initial feature fusion model to obtain training semantic features and training global features, and performing classification and recognition based on the training semantic features to obtain training initial categories corresponding to the training samples.

[0086] Among them, the training semantic feature refers to the feature extracted by the initial feature fusion model during training to characterize the semantics of the training sample. The training semantic feature is a feature that characterizes the details. The training global feature refers to the feature extracted by the initial feature fusion model during training to characterize the entire training sample. The training global feature is a feature that characterizes the global whole. The training initial category refers to the category corresponding to the training sample identified during training.

[0087] Specifically, the initial feature fusion model in the server performs feature extraction based on the training samples to obtain training semantic features and training global features, respectively. The semantic features can be extracted through a semantic branch network, and the training global features can be extracted through a global branch network. The training speech features and the training global features can be extracted hash features, which refer to sample representations composed of binary feature vectors. The use of hash features can compress feature storage space while using hash features for sample retrieval can improve retrieval efficiency. The training speech features and the training global features can also be floating-point features. The training semantic features are then used to perform classification and recognition through a classification network to obtain the probabilities of each category corresponding to the training samples, and then the initial training categories are obtained by screening according to the probabilities of each category.

[0088] Step 206, concatenate the training semantic features and the training global features through the initial feature fusion model to obtain the training concatenated features, calculate the autocorrelation features based on the training concatenated features to obtain the autocorrelation features, calculate the self-attention weights based on the autocorrelation features to obtain the self-attention weights, and adjust the training concatenated features through the self-attention weights to obtain the training fusion features.

[0089] Among them, the training splicing feature refers to the feature obtained by splicing the training semantic feature and the training global feature head to tail. The autocorrelation feature is used to characterize the autocorrelation relationship within the feature bit, which is used to characterize the distribution difference of each feature bit in the sample space. The correlation between each feature and each feature bit in the autocorrelation feature, the larger the feature value, the more relevant it is. The self-attention weight refers to the adjustment parameter obtained by self-attention calculation. This parameter is a hyperparameter after training is completed and is used to adjust the training splicing feature. The training fusion feature refers to the unified feature obtained in the end. The training fusion feature integrates the information of the training semantic feature and the training global feature, and there is no information redundancy.

[0090] Specifically, the initial feature fusion model concatenates the training semantic features and the training global features to obtain the training concatenated features, and then uses the training concatenated features to extract fusion features through self-attention. That is, the training concatenated features are used to calculate the autocorrelation features to obtain the autocorrelation features, and then the self-attention parameters are used to adjust the autocorrelation features to obtain the self-attention weights, and the training concatenated features are weighted by the self-attention weights to obtain the training fusion features.

[0091] Step 208, feature loss is calculated based on the training global features, the training semantic features, and the training fusion features to obtain an initial feature loss, and category loss is calculated based on the training initial category and the training sample category label to obtain an initial category loss.

[0092] Among them, the initial feature loss is used to characterize the error of the features obtained by training. The initial category loss is used to characterize the error between the initial category of training and the category label of the training sample.

[0093] Specifically, the server calculates the loss of training global features, calculates the loss of training speech features, calculates the loss of training fusion features, and finally calculates the sum of all losses to obtain the initial feature loss, and then calculates the error between the training initial category and the training sample category label to obtain the initial category loss.

[0094] Step 210, based on the initial feature loss and the initial category loss, the initial feature fusion model is updated, and the training samples and the corresponding training sample category labels are returned. The step of inputting the training samples into the initial feature fusion model is iteratively executed until the training completion condition is met, and the target fusion model is obtained. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and the target fusion features are used as the index of the input samples.

[0095] Specifically, the server calculates the sum of the initial feature loss and the initial category loss to obtain the sum of all losses, uses the sum of losses to reversely update the initial feature fusion model, that is, updates the parameters in the initial feature fusion model to obtain the updated feature fusion model, and then uses the updated feature fusion model as the initial feature fusion model, and returns to obtain the training sample and the corresponding training sample category label, and iterates the steps of inputting the training sample into the initial feature fusion model until the training completion condition is met, and the target fusion model is obtained, wherein the training completion condition may be that the number of trainings reaches the maximum number of iterations or the training loss reaches the preset loss threshold or the parameters of the model no longer change. Then the obtained target fusion model can be deployed and used. That is, when the input sample is obtained, the input sample is input into the target fusion model, and the target fusion model extracts semantic features and global features, fuses them, and inputs the target fusion feature. The target fusion feature is used to characterize the input sample, that is, the target fusion feature can be used as an index of the input sample, that is, the input sample can be retrieved using the target fusion feature.

[0096] The above-mentioned feature fusion model training method splices the extracted training semantic features and training global features to obtain training spliced ​​features when training the initial feature fusion model, and then calculates the autocorrelation features of the training spliced ​​features to obtain the autocorrelation features, and then uses the autocorrelation features to calculate the self-attention weights to obtain the self-attention weights, and adjusts the training spliced ​​features by the self-attention weights to obtain the training fusion features, so that the obtained training fusion features can avoid information redundancy, thereby improving the accuracy of the obtained training fusion features, and then continuously iterates to train the initial feature fusion model, so that the target fusion model obtained by training improves the accuracy of fusing semantic features and global features.

[0097] In one embodiment, Figure 3 As shown, before step 202, before the training sample is input into the initial feature fusion model, it also includes:

[0098] Step 302, obtain pre-training samples and corresponding pre-training sample category labels, input the pre-training samples into the pre-training model, extract features of the pre-training samples through the pre-training model, obtain pre-training semantic features and pre-training global features, and perform classification and recognition based on the pre-training semantic features to obtain the pre-training categories corresponding to the pre-training samples.

[0099] Step 304, performing feature loss calculation based on the pre-trained semantic features and the pre-trained global features to obtain the pre-trained feature loss, and performing category loss calculation based on the pre-trained categories and the pre-trained sample category labels to obtain the pre-trained category loss.

[0100] Step 306, updates the pre-trained model based on the pre-trained feature loss and the pre-trained category loss, and returns to obtain the pre-trained samples and the corresponding pre-trained sample category labels, and iterates the step of inputting the pre-trained samples into the pre-trained model until the pre-training completion condition is met, thereby obtaining the target pre-trained model.

[0101] Among them, the pre-training samples and the corresponding pre-training sample category labels refer to the training samples and the corresponding training sample category labels used in pre-training. The pre-training samples can be the same as or different from the training samples. The pre-training model refers to a neural network model with model parameter initialization. The parameter initialization can be random parameterization, zero initialization, Gaussian distribution initialization, etc. The target pre-training model refers to a pre-trained model that has been trained. The target pre-training model has pre-trained semantic feature extraction parameters, global feature extraction parameters, and classification parameters.

[0102] Specifically, the server establishes a pre-training model, and then pre-trains the pre-training sample in the pre-training model. When the pre-training completion condition is met, the target pre-training model is obtained. Among them, the pre-training completion condition can be that the number of pre-training reaches the maximum number of iterations or the pre-training loss reaches the preset loss threshold or the pre-training parameters no longer change, etc. When training the target feature fusion model, two stages of training are required. The first stage can train the feature extraction parameters and classification parameters in the target feature fusion model, and the second stage trains all parameters in the target feature fusion model. The first stage can be pre-training. The feature extraction parameters refer to the model parameters used when extracting semantic features and global features. The classification parameters refer to the model parameters used when classifying. In a specific embodiment, the network parameters participating in the learning in the first stage adopt a learning rate of 0.0005, and the learning rate is adjusted to 0.1 times the original every 10 rounds; in the second stage, except for the parameters in Table 1, the learning rate of 0.00005 is adopted in the network participating in the learning, and the other learning rates are 0.0005, and the learning rate is adjusted to 0.1 times the original every 10 rounds.

[0103] Step 308: Obtain an initial feature fusion model based on the target pre-trained model.

[0104] Specifically, the parameters trained in the target pre-trained model are used as the corresponding initialization parameters in the initial feature fusion model, and then other parameters in the initial feature fusion model, such as self-attention parameters, are initialized to obtain the initial feature fusion model.

[0105] In the above embodiment, by performing pre-training to obtain a target pre-training model, and then using the target pre-training model to establish an initial feature fusion model, the convergence speed during model training can be improved, thereby improving training efficiency.

[0106] In one embodiment, the initial feature fusion model includes an initial basic feature extraction network, an initial semantic feature extraction network, an initial global feature extraction network, and an initial classification network;

[0107] like Figure 4 As shown, step 204, i.e., extracting features based on the training samples to obtain training semantic features and training global features, and performing classification and recognition based on the training semantic features to obtain the training initial categories corresponding to the training samples, includes:

[0108] Step 402: input the training sample into the initial basic feature extraction network to obtain the initial first basic feature and the initial second basic feature, wherein the dimension of the initial first basic feature is smaller than the dimension of the initial second basic feature.

[0109] The initial basic feature extraction network refers to a basic feature extraction network with network parameters initialized, and the basic feature extraction network is suitable for extracting basic features of samples. The initial first basic feature and the initial second basic feature refer to the features output by different network layers of the initial basic feature extraction network, wherein the initial first basic feature is the output of the last network layer in the initial basic feature extraction network, and the initial second basic feature refers to the output of the second last network layer in the initial basic feature extraction network.

[0110] Specifically, the server uses the initial basic feature extraction network to extract basic features from the training samples, and obtains the initial first basic features and the initial second basic features output by different network layers, and the dimension of the initial first basic features is smaller than the dimension of the initial second basic features. In a specific embodiment, the basic feature extraction network can be a pre-trained ResNet101 (residual network 101). The network structure of the basic feature extraction network can be shown in Table 1 below.

[0111] Table 1 Network structure of basic feature extraction network

[0112]

[0113]

[0114] Among them, the features output by the 4th convolutional layer are used as the initial second basic features, and the features output by the 4th convolutional layer are used as the initial first basic features.

[0115] Step 404: input the initial first basic feature into the initial global feature extraction network to extract the global hash feature to obtain the training global feature.

[0116] The initial global feature extraction network refers to the global feature extraction network with initialized network parameters.

[0117] Specifically, the server uses the initial first basic feature to perform global hash feature extraction, that is, inputs the initial first basic feature into the initial global feature extraction network to perform global hash feature extraction to obtain a training global feature. In a specific embodiment, the network structure of the global feature extraction network can be shown in Table 2 below.

[0118] Table 2 Network structure of global feature extraction network

[0119]

[0120] The input of the global feature extraction network is the output of the last layer in Table 1, and the output is a 1X64-dimensional global feature. The global feature is a vector whose eigenvalues ​​tend to -1 or 1.

[0121] Step 406, input the initial second basic features into the initial semantic feature extraction network to perform semantic hash feature extraction to obtain training semantic features, and input the training semantic features into the initial classification network to perform classification and recognition to obtain the training initial category corresponding to the training sample.

[0122] The initial semantic feature extraction network refers to the semantic feature extraction network with network parameters initialized, and the initial classification network refers to the classification network with network parameters initialized.

[0123] Specifically, the server uses the initial semantic feature extraction network to extract semantic hash features, that is, the initial second basic features are input into the initial semantic feature extraction network for semantic hash feature extraction to obtain training semantic features. The training semantic features are then input into the initial classification network for classification and recognition to obtain the training initial categories corresponding to the training samples. The second basic features are used as the input of the semantic feature extraction network to avoid information interference and make the extracted semantic features more accurate.

[0124] In a specific embodiment, the network structures of the initial semantic feature extraction network and the initial classification network are shown in Table 3 below.

[0125] Table 3 Network structure of the initial semantic feature extraction network and the initial classification network

[0126]

[0127] The number of categories in the classification layer can be adjusted according to actual needs, and the number of categories in the classification layer is 100. By taking the second basic feature as input, the second basic feature is extracted with deeper semantic information, and the hash layer 2 outputs a 1X64-dimensional semantic feature, which is a vector whose feature values ​​tend to be -1 or 1. The classification layer then uses the semantic features for classification and outputs the probability of each category, ultimately determining the category of the sample.

[0128] In the above embodiment, by training the initial basic feature extraction network, the initial semantic feature extraction network, the initial global feature extraction network and the initial classification network, the feature fusion model obtained by training is more accurate.

[0129] In one embodiment, Figure 5 As shown, step 206, based on the training splicing feature, autocorrelation feature is calculated to obtain the autocorrelation feature, self-attention weight is calculated based on the autocorrelation feature to obtain the self-attention weight, and the training splicing feature is adjusted by the self-attention weight to obtain the training fusion feature, including:

[0130] Step 502, calculate the transposition of the training concatenated feature to obtain the transposed feature, and multiply the transposed feature by the training concatenated feature to obtain the autocorrelation feature.

[0131] The transposed feature refers to the vector obtained by transposing the training concatenated feature.

[0132] Specifically, the server transposes the training concatenated features corresponding to the training samples to obtain the transposed features corresponding to the training samples, and then multiplies the transposed features by the training concatenated features to obtain the autocorrelation features.

[0133] Step 504, perform mean pooling based on the autocorrelation features to obtain pooled features, obtain self-attention parameters, perform weighted calculation on the pooled features based on the self-attention parameters, and obtain self-attention weights.

[0134] Among them, pooled features refer to the features obtained after pooling. Self-attention parameters refer to the network parameters of the initial self-attention layer.

[0135] Specifically, the server performs mean pooling on the autocorrelation features to obtain pooled features, and then uses the network parameters of the initial self-attention layer to perform weighted calculations on the pooled features, that is, perform feature matrix dot multiplication operations to obtain self-attention weights.

[0136] Step 506, adjusting the training concatenated features by self-attention weights to obtain training adjusted features, performing feature crossover based on the training adjusted features to obtain training fusion features.

[0137] Specifically, the server uses the self-attention weight to perform weighted calculation on the training splicing features, and obtains the final self-attention fusion result after data and feature autocorrelation, namely the training adjustment feature, so that the obtained training adjustment feature reduces the redundant information in the feature. Then, the feature crossover is performed through the neural network to obtain the training fusion feature, so that the feature redundant information in the obtained training fusion feature can be further compressed through feature crossover, further improving the accuracy of the obtained training fusion feature.

[0138] In a specific embodiment, Figure 6As shown in FIG. 1 , a schematic diagram of obtaining training fusion features is shown. Among them, when the training samples are samples of a batch, that is, the training samples have bs (positive integer, the number of samples in each batch), the obtained 1x128-dimensional training splicing features have bs, that is, the feature vector of bsx128 dimension is obtained. Then the autocorrelation matrix of each feature vector is calculated to obtain a 128x128-dimensional autocorrelation matrix, which represents the correlation between each feature and the other 128 bits. The larger the correlation matrix value, the more correlated it is, and the values ​​on the diagonal are all 1. Then, the bs autocorrelation matrices are averaged and pooled along the bs dimension to obtain a pooling matrix. For example, for the autocorrelation matrix of (i, j, n), n=1…bs, the average value of the pooling matrix at the position (i, j) after pooling is calculated, which represents the average correlation between the i-th bit and the j-th bit on the batch data. When the pooling matrix (i, j) is close to 1, it means that for bs samples, there is almost a situation where the i-th bit and the j-th bit have the same value trend. When two bits are correlated under the average performance of the sample, they are redundant. Then, the pooling matrix is ​​multiplied by the weight condition parameter, that is, the self-attention parameter of the self-attention layer, to obtain the self-attention weight. Then, the training adjustment matrix is ​​used to perform matrix multiplication with the training splicing feature to obtain the final self-attention fusion result after data and feature autocorrelation, that is, the training adjustment matrix. Finally, feature crossover is performed through the neural network to obtain the training fusion feature, so that feature redundant information can be further compressed through feature crossover.

[0139] In one embodiment, the initial feature fusion model includes an initial self-attention network and an initial feature intersection network;

[0140] Step 504, obtaining a self-attention parameter, performing weighted calculation on the pooled feature based on the self-attention parameter, and obtaining a self-attention weight, including the steps of:

[0141] The pooled features are input into the initial self-attention network, the initial self-attention network obtains the self-attention parameters, and performs weighted calculation on the pooled features based on the self-attention parameters to obtain the self-attention weights.

[0142] Among them, the initial self-attention network refers to the self-attention network with initialized network parameters.

[0143] Specifically, the server can calculate the self-attention weight through the initial self-attention network, and use the self-attention parameters in the initial self-attention network to calculate and obtain the self-attention weight.

[0144] In a specific embodiment, the network structure of the self-attention network is shown in Table 4 below.

[0145] Table 4 Network structure of self-attention network

[0146]

[0147]

[0148] Among them, the input is the result of splicing the outputs of Table 2 and Table 3, wherein the output of Table 2 can be put in front and the output of Table 3 can be put in the back for head-to-tail splicing, or the output of Table 3 can be put in front and the output of Table 2 can be put in the back for head-to-tail splicing to obtain the self-attention weight, and then the self-attention weight is used to adjust the training splicing features, and finally the training adjusted features are output.

[0149] Step 506, i.e., performing feature crossover based on the training adjustment features to obtain the training fusion features, includes the following steps:

[0150] The training adjustment features are input into the initial feature cross network for feature crossover to obtain the training fusion features.

[0151] The initial feature cross network refers to a feature cross network with network parameters initialized, and the feature cross network is used to compress redundant hash bits.

[0152] Specifically, the server performs feature crossover on the training adjustment features, using the initial feature crossover network, to obtain the training fusion features.

[0153] In a specific embodiment, the network structure table of the feature cross network is shown in Table 5 below.

[0154] Table 5 Feature cross network structure table

[0155]

[0156] Among them, the feature cross network outputs the final fused hash feature, which is a 1*96-dimensional vector whose values ​​tend to -1 or 1. Due to the reduction of useless or redundant hash bits, the actual valid bits of all the hash features finally fused are less than the valid bits of the features obtained by direct splicing.

[0157] In one embodiment, Figure 7 As shown, step 208, based on the training global features, the training semantic features and the training fusion features, the feature loss is calculated to obtain the initial feature loss, including:

[0158] Step 702, performing triplet loss calculation based on the trained global features to obtain global feature loss information, and performing quantization loss calculation based on the trained global features to obtain global quantization loss information.

[0159] The global feature loss information refers to the triplet loss of the global feature training, which is used to characterize the error between the distance between the positive sample pair and the distance between the negative sample pair. The global quantization loss information refers to the quantization error corresponding to the global feature training.

[0160] Specifically, the server uses the triplet loss function to calculate the global feature loss information corresponding to the training global feature. The server uses the training global feature for symbolic quantization to obtain a quantized label, and then uses the regression loss function to calculate the error between the quantized label and the training global feature to obtain the global quantization loss information.

[0161] Step 704, performing triplet loss calculation based on the training semantic features to obtain semantic feature loss information, and performing quantization loss calculation based on the training semantic features to obtain semantic quantization loss information.

[0162] Among them, the semantic feature loss information refers to the triple loss of training semantic features, and the semantic quantization loss information refers to the quantization error corresponding to training global features.

[0163] Specifically, the server uses the triplet loss function to calculate the semantic feature loss information corresponding to the training semantic feature. The server uses the training semantic feature to perform symbolic quantization to obtain a quantized label, and then uses the regression loss function to calculate the error between the quantized label and the training semantic feature to obtain the semantic quantization loss information.

[0164] Step 706, performing triplet loss calculation based on the training fusion features to obtain fusion feature loss information, and performing quantization loss calculation based on the training fusion features to obtain fusion quantization loss information.

[0165] Among them, the fusion feature loss information refers to the triplet loss of training fusion features, and the fusion quantization loss information refers to the quantization error corresponding to the training fusion features.

[0166] Specifically, the server uses the triplet loss function to calculate the fusion feature loss information corresponding to the training fusion feature. The server uses the training fusion feature to perform symbol quantization to obtain a quantization label, and then uses the regression loss function to calculate the error between the quantization label and the training fusion feature to obtain the fusion quantization loss information.

[0167] Step 708 , performing weighted calculation based on the global feature loss information, the global quantization loss information, the semantic feature loss information, the semantic quantization loss information, the fused feature loss information and the fused quantization loss information to obtain an initial feature loss.

[0168] Specifically, the server can directly calculate the loss information and the loss information of global feature loss information, global quantization loss information, semantic feature loss information, semantic quantization loss information, fused feature loss information and fused quantization loss information to obtain the initial feature loss. The server can also weight the global feature loss information, global quantization loss information, semantic feature loss information, semantic quantization loss information, fused feature loss information and fused quantization loss information, and calculate the weighted sum to obtain the initial feature loss.

[0169] In the above embodiment, by calculating global feature loss information, global quantization loss information, semantic feature loss information, semantic quantization loss information, fused feature loss information and fused quantization loss information, and then obtaining the initial feature loss, the accuracy of the initial feature loss is improved.

[0170] In one embodiment, the training samples include each positive sample pair;

[0171] like Figure 8 As shown, step 702, i.e., performing triplet loss calculation based on the training global features to obtain global feature loss information, and performing quantization loss calculation based on the training global features to obtain global quantization loss information, includes:

[0172] Step 802 , performing negative sample screening based on each positive sample pair to obtain negative samples corresponding to each positive sample pair, and obtaining each sample triplet based on each positive sample pair and the corresponding negative sample.

[0173] Among them, a positive sample pair includes two positive samples, and the two positive samples correspond to the same category, or the two positive samples are highly similar. Negative samples refer to samples that are different from the positive sample category or are not highly similar.

[0174] Specifically, the server screens negative samples from the samples in each positive sample pair, selects the negative sample corresponding to the positive sample in each positive sample pair, and then obtains each sample triplet according to each positive sample pair and the corresponding negative sample, wherein the sample triplet includes a first positive sample, a second positive sample and a negative sample.

[0175] Step 804, calculate the first similarity distance between the training global feature corresponding to the first positive sample in each sample triplet and the training global feature corresponding to the second positive sample in each sample triplet, and calculate the second similarity distance between the training global feature corresponding to the first positive sample in each sample triplet and the training global feature corresponding to the negative sample in each sample triplet.

[0176] The first similarity distance is used to characterize the similarity distance between the global features of positive sample training in the sample triplet, and the second similarity distance is used to characterize the similarity distance between the global features of positive sample training and the global features of negative sample training in the sample triplet. The similarity distance can be calculated using a similarity algorithm.

[0177] Specifically, the server calculates the first similarity distance between the training global feature corresponding to the first positive sample in each sample triple and the training global feature corresponding to the second positive sample in each sample triple, and calculates the second similarity distance between the training global feature corresponding to the first positive sample in each sample triple and the training global feature corresponding to the negative sample in each sample triple, and can also calculate the similarity distance between the training global feature corresponding to the second positive sample and the training global feature corresponding to the negative sample to obtain the second similarity distance. The Euclidean distance algorithm can be used to calculate the similarity distance between feature vectors.

[0178] Step 806, calculate the error between the first similarity distance and the second similarity distance, obtain the triplet global loss information corresponding to each sample triplet, and perform average calculation based on the triplet global loss information corresponding to each sample triplet to obtain the global feature loss information.

[0179] The triplet global loss information refers to the loss information of the global features calculated by the sample triplet. Each sample triplet has corresponding triplet global loss information.

[0180] Specifically, the server calculates the error between the first similarity distance and the second similarity distance to obtain the triplet global loss information corresponding to each sample triplet, and performs average calculation based on the triplet global loss information corresponding to each sample triplet to obtain the global feature loss information.

[0181] Step 808, symbolically quantize the training global features corresponding to each sample in the training sample to obtain the global quantization features, calculate the error between the training global features and the global quantization features, obtain the initial global quantization loss information corresponding to each sample in the training sample, and perform average calculation based on the initial global quantization loss information corresponding to each sample in the training sample to obtain the global quantization loss information.

[0182] Among them, the global quantization feature refers to the quantization feature obtained after symbolic quantization of the training global feature. The initial global quantization loss information refers to the quantization loss corresponding to the training global feature, and each training global feature has corresponding initial global quantization loss information.

[0183] Specifically, the server performs symbolic quantization on all training global features corresponding to the training samples to obtain global quantization features corresponding to each training global feature. Among them, the symbolic function sign function can be used to generate the quantized target code. Then, the regression loss function is used to calculate the error between the training global feature and the global quantization feature to obtain the initial global quantization loss information corresponding to each sample in the training sample, and the initial global quantization loss information corresponding to each sample in the training sample is averaged to obtain the global quantization loss information. By calculating the initial global quantization loss information corresponding to each sample, and then averaging all the initial global quantization loss information, the global quantization loss information is obtained, thereby improving the accuracy of the obtained global quantization loss information.

[0184] In one embodiment, Fig. 9 As shown, step 802, based on each positive sample pair, negative samples are screened to obtain negative samples corresponding to each positive sample pair, including:

[0185] Step 902: determine the current positive sample pair and each positive sample pair to be mined based on each positive sample pair.

[0186] Among them, the positive sample pairs to be mined refer to the positive sample pairs that need to be screened for negative samples, which are all positive sample pairs except the current positive sample pairs.

[0187] Specifically, the server takes each positive sample pair as the current positive sample and determines the corresponding positive sample pair to be mined. The two positive samples in the positive sample pair can be completely identical samples, or extremely similar samples or attack samples generated by an attack. Fig.10 In the image sample diagram shown in Fig. Fig.10 The first row of the figure shows two very similar person image samples. Fig.10 The second row of the figure shows two image samples generated by image attacks. The attacks also include various types such as color change, chroma change, cropping, brightness, filters, etc.

[0188] Step 904: randomly select from the current positive sample pair to obtain the current positive sample, and randomly select from each pair of positive samples to be mined to obtain each sample to be mined.

[0189] The current positive sample refers to the sample in the current positive sample pair, and the sample to be mined refers to the sample in the positive sample pair to be mined.

[0190] Specifically, the server randomly selects samples from each positive sample pair to obtain the current sample and each sample to be mined. The current sample and each sample to be mined can also be obtained by selecting from each positive sample pair according to a preset selection object. For example, it can be specified that the first sample from the positive sample pair is selected as the current sample. Alternatively, it can be specified that the second sample from the positive sample pair is selected as the current sample.

[0191] Step 906, calculating the similarity between the current positive sample and each sample to be mined, and determining the current negative sample corresponding to the current positive sample from each sample to be mined based on the similarity.

[0192] The similarity is used to characterize the similarity between the current positive sample and the sample to be mined. The current negative sample refers to the negative sample corresponding to the current positive sample obtained by screening.

[0193] Specifically, the server uses a similarity algorithm to calculate the similarity between the current sample and each sample to be mined, wherein the similarity algorithm can use a distance similarity algorithm, such as a Euclidean distance algorithm and a Manhattan distance algorithm. A cosine similarity algorithm can also be used. Then, the samples to be mined are sorted from small to large using the similarity, and the current negative sample corresponding to the current positive sample is selected. Among them, the server can also remove samples according to a pre-set denoising quantity after sorting each sample to be mined from small to large, for example, removing the top 5 samples to be mined in the sorting. Then, the current negative sample corresponding to the current positive sample is selected from the remaining samples to be mined. It is also possible to select 20 current negative samples from the remaining samples to be mined in turn, and obtain sample triplets mined from 20 current sample pairs.

[0194] Step 908, traverse each positive sample pair to obtain the negative sample corresponding to each positive sample pair.

[0195] Specifically, the server screens negative samples for each positive sample pair to obtain negative samples corresponding to each positive sample pair. The negative samples may be multiple, and the multiple may be at least two, and finally all sample triplets are obtained.

[0196] In the above embodiment, by screening negative samples based on each positive sample pair, negative samples corresponding to each positive sample pair are obtained, thereby obtaining sample triplets, which can make the obtained sample triplets of higher quality, thereby improving the accuracy of training.

[0197] In one embodiment, the training samples include respective sample triplets;

[0198] like Fig.11 As shown, step 704, performing triple loss calculation based on the training semantic features to obtain semantic feature loss information, and performing quantization loss calculation based on the training semantic features to obtain semantic quantization loss information, including:

[0199] Step 1102, calculate the third similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the second positive sample in each sample triple, and calculate the fourth similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the negative sample in each sample triple.

[0200] Among them, the third similarity distance is used to characterize the similarity distance between the positive sample training semantic features in the sample triplet, and the fourth similarity distance is used to characterize the similarity distance between the positive sample training semantic features and the negative sample training semantic features in the sample triplet.

[0201] Specifically, when the server obtains training samples, it directly obtains the training samples in the form of triplets, that is, it obtains a batch of samples, that is, each sample triple. Then, when calculating the triple loss of the feature, the triple loss corresponding to each sample triple can be directly calculated. That is, the server can use the Euclidean distance algorithm to calculate the third similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature of the second positive sample corresponding to the first positive sample, and use the Euclidean distance algorithm to calculate the fourth similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature of the negative sample corresponding to the first positive sample.

[0202] Step 1104, calculate the error between the third similarity distance and the fourth similarity distance to obtain triple semantic loss information corresponding to each sample triple, and perform average calculation based on the triple semantic loss information corresponding to each sample triple to obtain semantic feature loss information.

[0203] The triplet global loss information refers to the loss information of the global features calculated by the sample triplet. Each sample triplet has corresponding triplet global loss information.

[0204] Specifically, the server calculates the error between the third similarity distance and the fourth similarity distance, and then performs an average calculation to obtain semantic feature loss information, so that the obtained semantic feature loss information can be made more accurate, thereby improving the accuracy of training.

[0205] Step 1106, symbolically quantize the training semantic features corresponding to each sample in the training sample to obtain semantic quantization features, calculate the error between the training semantic features and the semantic quantization features, obtain initial semantic quantization loss information corresponding to each sample in the training sample, and perform average calculation based on the initial semantic quantization loss information corresponding to each sample in the training sample to obtain semantic quantization loss information.

[0206] The semantic quantization feature refers to the quantization feature obtained after symbolic quantization of the training semantic feature. The initial semantic quantization loss information refers to the quantization error corresponding to the training semantic feature, and each training semantic feature has corresponding initial semantic quantization loss information.

[0207] Specifically, when calculating the quantization loss, the server first symbolically quantizes the training semantic features, where the symbolic function sign can be used for symbolic quantization to obtain semantic quantization features. Then, the regression loss can be used to calculate the error between the training semantic features and the semantic quantization features, so as to obtain the initial semantic quantization loss information corresponding to each sample, and then average calculation is performed to obtain the semantic quantization loss information. Using the semantic quantization loss information for reverse iterative update can make each bit in the fusion feature output by the trained target fusion feature model close to the quantization value, for example, close to 1 or -1, thereby improving the accuracy of feature fusion of the target fusion feature model.

[0208] In one embodiment, Fig.12 As shown, step 706, performing triplet loss calculation based on the training fusion feature to obtain fusion feature loss information, and performing quantization loss calculation based on the training fusion feature to obtain fusion quantization loss information, including:

[0209] Step 1202, calculate the fifth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triplet and the training fusion feature corresponding to the second positive sample in each sample triplet, and calculate the sixth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triplet and the training fusion feature corresponding to the negative sample in each sample triplet.

[0210] Among them, the third similarity distance is used to characterize the similarity distance between the positive sample training fusion features in the sample triplet, and the fourth similarity distance is used to characterize the similarity distance between the positive sample training fusion features and the negative sample training fusion features in the sample triplet.

[0211] Specifically, the server uses the Euclidean distance algorithm to calculate the distance between the positive sample training fusion features in each sample triplet to obtain the fifth similarity distance, and simultaneously uses the Euclidean distance algorithm to calculate the distance between the positive and negative sample training fusion features in each sample triplet to obtain the sixth similarity distance.

[0212] Step 1204, calculate the error between the fifth similarity distance and the sixth similarity distance, obtain the triplet fusion loss information corresponding to each sample triplet, and perform average calculation based on the triplet fusion loss information corresponding to each sample triplet to obtain the fusion feature loss information.

[0213] Among them, the triple fusion loss information refers to the loss information of the fusion features calculated by the sample triple. Each sample triple has a corresponding triple fusion loss information

[0214] Specifically, the server calculates the error between the fifth similarity distance and the sixth similarity distance, and then performs an average calculation to obtain fused feature loss information, so that the obtained fused feature loss information can be more accurate, thereby improving the accuracy of training.

[0215] Step 1206, symbolically quantize the training fusion features corresponding to each sample in the training sample to obtain a fused quantized feature, calculate the error between the training fusion feature and the fused quantized feature, obtain the initial fused quantized loss information corresponding to each sample in the training sample, and perform an average calculation based on the initial fused quantized loss information corresponding to each sample in the training sample to obtain the fused quantized loss information.

[0216] The fused quantized feature refers to the quantized feature obtained after symbolic quantization of the training fused feature. The initial fused quantized loss information refers to the quantization error corresponding to the training fused feature, and each training fused feature has corresponding initial fused quantized loss information.

[0217] Specifically, when the server calculates the quantization loss of the training fusion feature, it first symbolically quantizes the training fusion feature, where the symbolic function sign can be used for symbolic quantization to obtain the fusion quantized feature. Then, the regression loss can be used to calculate the error between the training fusion feature and the fusion quantized feature, so as to obtain the initial fusion quantization loss information corresponding to each sample, and then the average calculation is performed to obtain the fusion quantization loss information. The fusion quantization loss information is used for reverse iterative update, so that each bit in the fusion feature output by the trained target fusion feature model is close to the quantized value, for example, close to 1 or -1, thereby improving the accuracy of feature fusion of the target fusion feature model.

[0218] In a specific embodiment, the server calculates the triplet loss using formula (1) shown below.

[0219] l tri =max(||x a -x p ||-||x a -x n ||+α, 0) Formula (1)

[0220] Among them, (a, p, n) is a triple sample, l tri Refers to the global loss information of the triple. a is the first positive sample, p is the second positive sample, and n is the negative sample. arepresents the feature corresponding to the first positive sample, which can be the initial global feature, the initial semantic feature, or the training fusion feature. p represents the feature corresponding to the second positive sample, x n Indicates the features corresponding to negative samples. α is a hyperparameter used to control the spatial distance between positive and negative samples. It can be set as needed. For example, global features and semantic features can be set to 56 when calculating the loss, and fusion features can be set to 80 when calculating the loss. The training goal of the triplet loss information is to make the distance between the first positive sample and the second positive sample greater than the distance between the first positive sample and the negative sample image. The value set by the hyperparameter.

[0221] In one embodiment, step 208, performing category loss calculation based on the training initial category and the training sample category label to obtain the initial category loss includes:

[0222] Calculate the classification loss information between the training initial category and the training sample category label corresponding to each sample in the training sample to obtain the training category loss corresponding to each sample; perform average calculation based on the training category loss corresponding to each sample to obtain the initial category loss corresponding to the training sample.

[0223] Specifically, the server can use the cross entropy loss function to calculate the classification loss, calculate the loss between the initial training category and the training sample category label corresponding to each sample, and then perform an average calculation to obtain the initial category loss, thereby improving the accuracy of the obtained initial category loss, and then use the initial category loss to perform reverse iterative updates to improve the accuracy of model training.

[0224] In a specific embodiment, the classification loss information can be calculated using formula (2) shown below.

[0225]

[0226] Where N is the number of samples. ic represents the predicted probability that sample i belongs to category c, y ic Indicates whether the training category label of sample i is c. If it is c, yic=1, otherwise it is 0.

[0227] In a specific embodiment, the total loss in the training process can be calculated using the formula (3) shown below, and then the total loss is used to perform reverse iterative update on the initial feature fusion model.

[0228]

[0229] Among them, L total represents the total loss. Ntriplet represents the total number of sample triplets, and Nimage refers to the total number of samples.hash-triplet1 refers to the triplet loss of global features, L coding1 It refers to the quantization loss of global features. hash-triplet2 refers to the triple loss of semantic features, L coding2 It refers to the quantization loss of semantic features. hash-triplet3 refers to the triplet loss of fused features, L coding3 Refers to the quantization loss of fused features. L class Refers to classification loss. W refers to the pre-set weight. Different losses have different weights. For example, the weights can be set to w1=w3=w5=1, w2=w4=0.01, w6=1, w7=0.01.

[0230] In a specific embodiment, the total pre-training loss in the pre-training process can be calculated using the formula (4) shown below, and then the pre-training model can be reversely iterated and updated using the total pre-training loss.

[0231]

[0232] Among them, L total1 Represents the total pre-training loss, which is the loss during the first stage of training.

[0233] In one embodiment, Fig.13 As shown, after step 210, that is, after updating the initial feature fusion model based on the initial feature loss and the initial category loss, returning to obtain the training sample and the corresponding training sample category label, and inputting the training sample into the initial feature fusion model, the steps are iteratively executed until the training completion condition is met and the target fusion model is obtained, the following further includes:

[0234] Step 1302, obtaining an input sample;

[0235] Step 1304: input the input sample into the target fusion model, extract features of the input sample through the target fusion model to obtain input semantic features and input global features, concatenate the input semantic features and the input global features to obtain input concatenated features, calculate autocorrelation features based on the input concatenated features to obtain input autocorrelation features, calculate self-attention weights based on the input autocorrelation features to obtain input self-attention weights, and adjust the input concatenated features by the input self-attention weights to obtain a first input fusion feature corresponding to the input sample;

[0236] Step 1306: Use the first input fusion feature as the index corresponding to the input sample.

[0237] The input sample refers to the sample input when the target fusion model is deployed and used. The first input fusion feature refers to the fused feature extracted from the input sample.

[0238] Specifically, when the server obtains the target fusion model through training, it can be deployed and used. When in use, input samples can be obtained from the database, input samples uploaded by the terminal, and input samples sent by the business party. The input samples are then input into the target fusion model for fusion feature extraction to obtain the first input fusion feature corresponding to the output input sample. The obtained first input fusion feature can then be used as the index corresponding to the input sample, thereby making the obtained index more accurate. The index can then be associated with the corresponding input sample and saved for subsequent use. When performing sample retrieval, the index can be used for similar sample detection, thereby improving efficiency and accuracy.

[0239] In one embodiment, the target fusion model includes a semantic feature extraction network, a global feature extraction network, a self-attention network, and a feature intersection network;

[0240] like Fig.14 As shown, step 1304, input the input sample into the target fusion model, extract the input sample through the target fusion model to obtain input semantic features and input global features, splice the input semantic features and the input global features to obtain input splicing features, calculate the autocorrelation features based on the input splicing features to obtain the input autocorrelation features, calculate the self-attention weight based on the input autocorrelation features to obtain the input self-attention weight, and adjust the input splicing features through the input self-attention weight to obtain the first input fusion feature corresponding to the input sample, including:

[0241] Step 1402, input the input samples into the semantic feature extraction network and the global feature extraction network in the target fusion model respectively, to obtain input semantic features and input global features.

[0242] Step 1404, concatenate the input semantic features and the input global features to obtain input concatenated features, and input the input concatenated features into the self-attention network. The self-attention network calculates autocorrelation features based on the input concatenated features to obtain input autocorrelation features, calculates self-attention weights based on the input autocorrelation features to obtain input self-attention weights, and adjusts the input concatenated features by the input self-attention weights to obtain input adjusted features corresponding to the input samples.

[0243] Step 1406: Input the input adjustment feature to the feature cross network for feature crossover to obtain the first input fusion feature corresponding to the input sample.

[0244] Specifically, the target fusion model includes a semantic feature extraction network, a global feature extraction network, a self-attention network and a feature intersection network, and the target fusion model does not include the classification network used during training. When the target fusion model performs fusion feature extraction, it uses the semantic feature extraction network and the global feature extraction network to extract features, and then performs feature fusion through the self-attention network and the feature intersection network to obtain the output first input fusion feature, and then uses the first input fusion feature as the index corresponding to the input sample, thereby improving the accuracy of the obtained first input fusion feature, making the index corresponding to the input sample more accurate, and using the first input fusion feature as the index corresponding to the input sample can reduce the consumption of storage resources and improve retrieval efficiency.

[0245] In one embodiment, Fig.15 As shown, a sample retrieval method is provided, which is applied to Figure 1 The server in the example is used for explanation. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the following steps are included:

[0246] Step 1502: Obtain query fusion features corresponding to the query sample. The query fusion features are obtained by inputting the query sample into a target fusion model to extract fusion features.

[0247] Step 1504, extract features from the query sample through the target fusion model to obtain query semantic features and query global features, concatenate the query semantic features and the query global features to obtain query concatenation features, calculate autocorrelation features based on the query concatenation features to obtain query autocorrelation features, calculate self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjust the query concatenation features by the query self-attention weights to obtain query fusion features.

[0248] The query sample refers to the original sample used when similarity retrieval is required, that is, the sample similar to the query sample is to be retrieved. The target fusion model can be trained using any embodiment of the feature fusion model training method described above. The query fusion feature refers to the query sample obtained by extracting the fusion feature through the feature fusion model.

[0249] Specifically, the server can obtain the query sample from the database, then call the feature fusion model, input the query sample into the target fusion model for feature extraction, and obtain the output query fusion feature. The query sample can be an image, text, video, audio, etc.

[0250] Step 1506: perform sample retrieval based on the query fusion feature to obtain a target retrieval sample corresponding to the query sample.

[0251] Specifically, the server can use the query fusion features and the fusion features of each sample to be retrieved stored in the database to calculate the similarity, obtain each similarity, and then select the most similar sample to be retrieved from each sample to be retrieved as the target retrieval sample based on the similarity, or select multiple samples to be retrieved that are ranked at the top of the similarity as each target retrieval sample.

[0252] In one embodiment, the fusion feature extracted by the target fusion model is a hash feature. In this case, the query fusion feature is used to calculate the similarity with the fusion features of each sample to be retrieved and the target retrieval sample is obtained, which can improve the efficiency of the retrieval.

[0253] The above-mentioned sample retrieval method uses query fusion features to perform sample retrieval. Since the query fusion features are obtained by using the target fusion model to perform feature extraction based on the query sample to obtain query semantic features and query global features, the query semantic features and the query global features are spliced ​​to obtain query splicing features, autocorrelation features are calculated based on the query splicing features to obtain query autocorrelation features, self-attention weights are calculated based on the query autocorrelation features to obtain query self-attention weights, and the query splicing features are adjusted by the query self-attention weights, which can more accurately characterize the query samples, thereby making the obtained target retrieval samples more accurate.

[0254] In a specific embodiment, Fig.16 As shown, it is a flowchart of the feature fusion model training method, specifically:

[0255] Step 1602, obtaining training samples and corresponding training sample category labels, and inputting the training samples into the initial feature fusion model.

[0256] Step 1604: Input the training sample into the initial basic feature extraction network through the initial feature fusion model to obtain the initial first basic feature and the initial second basic feature. Input the initial first basic feature into the initial global feature extraction network to extract the global hash feature to obtain the training global feature.

[0257] Step 1606, input the initial second basic features into the initial semantic feature extraction network through the initial feature fusion model to extract semantic hash features to obtain training semantic features, and input the training semantic features into the initial classification network for classification and recognition to obtain the training initial category corresponding to the training sample.

[0258] Step 1608, calculate the transposition of the training splicing feature through the initial feature fusion model to obtain the transposed feature, multiply the transposed feature with the training splicing feature to obtain the autocorrelation feature; perform mean pooling based on the autocorrelation feature to obtain the pooled feature.

[0259] Step 1610, input the pooled features into the initial self-attention network through the initial feature fusion model, the initial self-attention network obtains the self-attention parameters, performs weighted calculation on the pooled features based on the self-attention parameters to obtain the self-attention weights, inputs the training adjustment features into the initial feature cross network for feature crossover, and obtains the training fusion features.

[0260] Step 1612: Calculate triplet loss based on the trained global features to obtain global feature loss information, and calculate quantization loss based on the trained global features to obtain global quantization loss information; calculate triplet loss based on the trained semantic features to obtain semantic feature loss information, and calculate quantization loss based on the trained semantic features to obtain semantic quantization loss information.

[0261] Step 1614, performing triplet loss calculation based on the training fusion feature to obtain fusion feature loss information, performing quantization loss calculation based on the training fusion feature to obtain fusion quantization loss information, and performing category loss calculation based on the training initial category and the training sample category label to obtain the initial category loss;

[0262] Step 1616, perform weighted calculation based on global feature loss information, global quantization loss information, semantic feature loss information, semantic quantization loss information, fused feature loss information, fused quantization loss information and initial category loss to obtain the total model loss.

[0263] Step 1618, based on the total loss of the model, the initial feature fusion model is updated, and the training samples and corresponding training sample category labels are returned. The step of inputting the training samples into the initial feature fusion model is iteratively executed until the training completion condition is met to obtain the target fusion model.

[0264] In a specific embodiment, Fig.17 As shown, a schematic diagram of the training framework of a target fusion model is provided, specifically:

[0265] The server obtains the full amount of samples M, and then divides the full amount of samples into different batches for training. Each batch has bs samples. If they are provided in the form of sample pairs, negative samples are screened to obtain sample triplets. Then all sample triplets of a batch are input into the feature fusion model to be trained. The parameters in the feature fusion model to be trained are initialized, including the networks in Table 1, Table 2, Table 3, Table 4 and Table 5. Among them, the network in Table 1 can be initialized using pre-trained parameters, the parameters of conv6 in Table 3 can use the pre-trained parameters of conv5 in Table 1, and other networks can be initialized using Gaussian distributions with a variance of 0.01 and a mean of 0. Then the feature fusion model with initialized parameters is trained. The feature fusion model extracts basic features from all samples through the basic feature network (such as the network in Table 1) to obtain the basic features of the last layer output and the basic features of the second last layer output. The basic features of the last layer output are used to extract features through the global feature network (such as the network in Table 2) to obtain training global features. The basic features of the second last layer output are input into the semantic module (such as the network in Table 3). The semantic features are extracted through the semantic feature network to obtain training semantic features. The training semantic features are then classified through the classification layer to obtain the initial classification results. The training semantic features and the training global features are then concatenated to obtain training concatenated features. The training concatenated features are input into the self-attention module to perform feature fusion through the self-attention network (such as the network in Table 4) to obtain training adjustment features. The training adjustment features are then cross-featured through the fusion learning network (such as the network in Table 5) to obtain training fusion features. Loss calculation is then performed, including triplet loss, quantization loss, and classification loss. The global feature loss, global quantization loss, fusion feature loss, fusion quantization loss, category loss, semantic feature loss and semantic quantization loss are calculated, and then all losses are weighted and summed to obtain the final model total loss. The model total loss is then used to update all parameters of the model through the gradient descent algorithm to obtain the updated feature fusion model. The next batch of training samples are then input for continuous iteration until all samples are trained. This completes a round, and training is performed again. When the preset number of times, such as K rounds of training or the average loss of 10 consecutive rounds has not decreased, the training is completed to obtain the target fusion model.

[0266] In a specific embodiment, the sample retrieval method is applied to an image search platform, specifically:

[0267] The server obtains the query image uploaded by the terminal through the image search platform, and then inputs the query image into the target fusion model, the target fusion model extracts features of the query image to obtain semantic features of the query image and global features of the query image, splices the semantic features of the query image and the global features of the query image to obtain spliced ​​features of the query image, calculates autocorrelation features based on the spliced ​​features of the query image to obtain autocorrelation features of the query image, calculates self-attention weights based on the autocorrelation features of the query image to obtain self-attention weights of the query image, and adjusts the spliced ​​features of the query image by the self-attention weights of the query image to obtain query image fusion features corresponding to the query image samples.

[0268] Then the server can use the query image fusion feature to perform similar recall from the existing image database, that is, calculate the fusion feature similarity between the query image semantic feature or the query image global feature or the query image fusion feature and the existing image fusion feature in the image database, and determine the retrieved existing image based on the similarity. The server can also use the query image semantic feature or the query image global feature to perform image similarity recall from the existing image database, that is, calculate the semantic feature similarity between the query image semantic feature and the corresponding semantic feature fusion feature in the image database, or calculate the global feature similarity between the query image global feature and the global feature of the existing image in the image database, and then perform image similarity recall from the existing image database based on the semantic feature similarity or the global feature similarity to obtain the retrieved existing image, and then the server returns the retrieved similar existing image to the terminal for display. Among them, by using fusion features for retrieval, the retrieval efficiency and accuracy can be improved, and the resource consumption of image index storage can be reduced.

[0269] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0270] Based on the same inventive concept, the embodiment of the present application also provides a feature fusion model training device for implementing the feature fusion model training method involved above and a sample retrieval device for the sample retrieval method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more feature fusion model training devices or sample retrieval device embodiments provided below can refer to the limitations of the feature fusion model training method or sample retrieval method above, and will not be repeated here.

[0271] In one embodiment, Fig.18 As shown, a feature fusion model training device 1800 is provided, comprising: an input module 1802, an initial recognition module 1804, an initial fusion module 1806, a loss calculation module 1808 and an iteration module 1810, wherein:

[0272] An input module 1802 is used to obtain training samples and corresponding training sample category labels, and input the training samples into an initial feature fusion model;

[0273] The initial recognition module 1804 is used to extract features from the training samples through the initial feature fusion model to obtain training semantic features and training global features, and to perform classification and recognition based on the training semantic features to obtain the training initial categories corresponding to the training samples;

[0274] An initial fusion module 1806 is used to splice the training semantic features and the training global features through an initial feature fusion model to obtain a training spliced ​​feature, calculate an autocorrelation feature based on the training spliced ​​feature to obtain an autocorrelation feature, calculate a self-attention weight based on the autocorrelation feature to obtain a self-attention weight, and adjust the training spliced ​​feature through the self-attention weight to obtain a training fusion feature;

[0275] The loss calculation module 1808 is used to calculate the feature loss based on the training global features, the training semantic features and the training fusion features to obtain the initial feature loss, and to calculate the category loss based on the training initial category and the training sample category label to obtain the initial category loss;

[0276] Iterative module 1810 is used to update the initial feature fusion model based on the initial feature loss and the initial category loss, and return to obtain training samples and corresponding training sample category labels, and iteratively execute the steps of inputting the training samples into the initial feature fusion model until the training completion condition is met, and obtain the target fusion model. The target fusion model is used to fuse the semantic features and global features extracted from the input samples to obtain the target fusion features, and use the target fusion features as the index of the input samples.

[0277] In one embodiment, the feature fusion model training device 1800 further includes:

[0278] The pre-training module is used to obtain pre-training samples and corresponding pre-training sample category labels, input the pre-training samples into the pre-training model, extract features of the training samples through the pre-training model to obtain pre-training semantic features and pre-training global features, and perform classification and recognition based on the pre-training semantic features to obtain the pre-training categories corresponding to the pre-training samples; calculate feature loss based on the pre-training semantic features and the pre-training global features to obtain pre-training feature loss, and calculate category loss based on the pre-training categories and pre-training sample category labels to obtain pre-training category loss; update the pre-training model based on the pre-training feature loss and the pre-training category loss, and return the steps of obtaining pre-training samples and corresponding pre-training sample category labels, and inputting the pre-training samples into the pre-training model, and iteratively execute until the pre-training completion conditions are met to obtain the target pre-training model; obtain the initial feature fusion model based on the target pre-training model.

[0279] In one embodiment, the initial feature fusion model includes an initial basic feature extraction network, an initial semantic feature extraction network, an initial global feature extraction network, and an initial classification network;

[0280] The initial recognition module 1804 is also used to input the training sample into the initial basic feature extraction network to obtain the initial first basic feature and the initial second basic feature, the dimension of the initial first basic feature is smaller than the dimension of the initial second basic feature; input the initial first basic feature into the initial global feature extraction network to perform global hash feature extraction to obtain the training global feature; input the initial second basic feature into the initial semantic feature extraction network to perform semantic hash feature extraction to obtain the training semantic feature, and input the training semantic feature into the initial classification network to perform classification and recognition to obtain the training initial category corresponding to the training sample.

[0281] In one embodiment, the initial fusion module 1806 is also used to calculate the transpose of the training spliced ​​features to obtain the transposed features, multiply the transposed features by the training spliced ​​features to obtain the autocorrelation features; perform mean pooling based on the autocorrelation features to obtain the pooled features, and obtain the self-attention parameters, perform weighted calculation on the pooled features based on the self-attention parameters to obtain the self-attention weights; adjust the training spliced ​​features by the self-attention weights to obtain the training adjusted features, perform feature crossover based on the training adjusted features to obtain the training fusion features.

[0282] In one embodiment, the initial feature fusion model includes an initial self-attention network and an initial feature intersection network;

[0283] The initial fusion module 1806 is also used to input the pooled features into the initial self-attention network, the initial self-attention network obtains the self-attention parameters, performs weighted calculation on the pooled features based on the self-attention parameters, and obtains the self-attention weights; the training adjustment features are input into the initial feature cross network for feature crossover to obtain the training fusion features.

[0284] In one embodiment, the loss calculation module 1808 includes:

[0285] A global loss calculation unit, used to perform triplet loss calculation based on the trained global features to obtain global feature loss information, and to perform quantization loss calculation based on the trained global features to obtain global quantization loss information;

[0286] A semantic loss calculation unit, used to perform triple loss calculation based on the training semantic features to obtain semantic feature loss information, and perform quantization loss calculation based on the training semantic features to obtain semantic quantization loss information;

[0287] A fusion loss calculation unit, used for performing triplet loss calculation based on the training fusion features to obtain fusion feature loss information, and performing quantization loss calculation based on the training fusion features to obtain fusion quantization loss information;

[0288] The feature loss obtaining unit is used to perform weighted calculation based on global feature loss information, global quantization loss information, semantic feature loss information, semantic quantization loss information, fused feature loss information and fused quantization loss information to obtain an initial feature loss.

[0289] In one embodiment, the training samples include each positive sample pair;

[0290] The global loss calculation unit is also used to perform negative sample screening based on each positive sample pair to obtain negative samples corresponding to each positive sample pair, and obtain each sample triplet based on each positive sample pair and the corresponding negative sample; calculate the first similarity distance between the training global feature corresponding to the first positive sample in each sample triplet and the training global feature corresponding to the second positive sample in each sample triplet, and calculate the second similarity distance between the training global feature corresponding to the first positive sample in each sample triplet and the training global feature corresponding to the negative sample in each sample triplet; calculate the error between the first similarity distance and the second similarity distance to obtain the triplet global loss information corresponding to each sample triplet, and perform average calculation based on the triplet global loss information corresponding to each sample triplet to obtain the global feature loss information; perform symbol quantization on the training global features corresponding to each sample in the training sample to obtain the global quantization feature, calculate the error between the training global feature and the global quantization feature, obtain the initial global quantization loss information corresponding to each sample in the training sample, and perform average calculation based on the initial global quantization loss information corresponding to each sample in the training sample to obtain the global quantization loss information.

[0291] In one embodiment, the global loss calculation unit is also used to determine the current positive sample pair and each positive sample pair to be mined based on each positive sample pair; randomly select from the current positive sample pair to obtain the current positive sample, and randomly select from each positive sample pair to be mined to obtain each sample to be mined; calculate the similarity between the current positive sample and each sample to be mined, and determine the current negative sample corresponding to the current positive sample pair from each sample to be mined based on the similarity; traverse each positive sample pair to obtain the negative sample corresponding to each positive sample pair.

[0292] In one embodiment, the training samples include respective sample triplets;

[0293] The semantic loss calculation unit is also used to calculate the third similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the second positive sample in each sample triple, and calculate the fourth similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the negative sample in each sample triple; calculate the error between the third similarity distance and the fourth similarity distance to obtain the triple semantic loss information corresponding to each sample triple, and perform average calculation based on the triple semantic loss information corresponding to each sample triple to obtain the semantic feature loss information; symbolically quantize the training semantic features corresponding to each sample in the training samples to obtain semantic quantization features, calculate the error between the training semantic features and the semantic quantization features to obtain the initial semantic quantization loss information corresponding to each sample in the training samples, and perform average calculation based on the initial semantic quantization loss information corresponding to each sample in the training samples to obtain the semantic quantization loss information.

[0294] In one embodiment, the fusion loss calculation unit is also used to calculate the fifth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the second positive sample in each sample triple, and calculate the sixth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the negative sample in each sample triple; calculate the error between the fifth similarity distance and the sixth similarity distance to obtain the triple fusion loss information corresponding to each sample triple, and perform average calculation based on the triple fusion loss information corresponding to each sample triple to obtain the fusion feature loss information; perform symbol quantization on the training fusion feature corresponding to each sample in the training sample to obtain a fusion quantized feature, calculate the error between the training fusion feature and the fusion quantized feature to obtain the initial fusion quantization loss information corresponding to each sample in the training sample, and perform average calculation based on the initial fusion quantization loss information corresponding to each sample in the training sample to obtain the fusion quantization loss information.

[0295] In one embodiment, the loss calculation module 1808 includes:

[0296] The category loss calculation unit is used to calculate the classification loss information between the training initial category and the training sample category label corresponding to each sample in the training sample to obtain the training category loss corresponding to each sample; and perform average calculation based on the training category loss corresponding to each sample to obtain the initial category loss corresponding to the training sample.

[0297] In one embodiment, the feature fusion model training device 1800 further includes:

[0298] A sample index establishment module is used to obtain input samples; input the input samples into a target fusion model, extract features of the input samples through the target fusion model to obtain input semantic features and input global features, concatenate the input semantic features and the input global features to obtain input concatenation features, calculate autocorrelation features based on the input concatenation features to obtain input autocorrelation features, calculate self-attention weights based on the input autocorrelation features to obtain input self-attention weights, and adjust the input concatenation features through the input self-attention weights to obtain a first input fusion feature corresponding to the input sample; and use the first input fusion feature as the index corresponding to the input sample.

[0299] In one embodiment, the target fusion model includes a semantic feature extraction network, a global feature extraction network, a self-attention network, and a feature intersection network;

[0300] The sample index establishment module is also used to input the input sample into the semantic feature extraction network and the global feature extraction network in the target fusion model respectively to obtain the input semantic feature and the input global feature; splice the input semantic feature and the input global feature to obtain the input splicing feature, and input the input splicing feature into the self-attention network, the self-attention network calculates the autocorrelation feature based on the input splicing feature to obtain the input autocorrelation feature, calculates the self-attention weight based on the input autocorrelation feature to obtain the input self-attention weight, and adjusts the input splicing feature by the input self-attention weight to obtain the input adjustment feature corresponding to the input sample; inputs the input adjustment feature into the feature crossing network for feature crossing to obtain the first input fusion feature, and uses the first input fusion feature as the index corresponding to the input sample.

[0301] In one embodiment, Fig.19 As shown, a sample retrieval device 1900 is provided, comprising: a feature acquisition module 1902, a feature extraction module 1904 and a retrieval module 1906, wherein:

[0302] The feature acquisition module 1902 is used to acquire the query fusion feature corresponding to the query sample. The query fusion feature is obtained by inputting the query sample into the target fusion model to extract the fusion feature;

[0303] The feature extraction module 1904 is used to extract features from the query sample through the target fusion model to obtain query semantic features and query global features, concatenate the query semantic features and the query global features to obtain query concatenation features, calculate autocorrelation features based on the query concatenation features to obtain query autocorrelation features, calculate self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjust the query concatenation features by the query self-attention weights to obtain query fusion features;

[0304] The retrieval module 1906 is used to perform sample retrieval based on the query fusion feature to obtain a target retrieval sample corresponding to the query sample.

[0305] Each module in the above sample retrieval device or feature fusion model training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0306] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 20As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data, index data, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a feature fusion model training method or a sample retrieval method is implemented.

[0307] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.21 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a feature fusion model training method or a retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0308] Those skilled in the art will understand that Fig. 20Or the structure shown in 21 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0309] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0310] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0311] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0312] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0313] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0314] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0315] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A feature fusion model training method, characterized in that: The method comprises: Acquire training samples and corresponding training sample category labels, and input the training samples into an initial feature fusion model, wherein the training samples include at least one of the following: images, texts, videos, and audios; Extracting features from the training samples through the initial feature fusion model to obtain training semantic features and training global features, and performing classification and recognition based on the training semantic features to obtain training initial categories corresponding to the training samples; The training semantic features and the training global features are spliced ​​by the initial feature fusion model to obtain training spliced ​​features, autocorrelation features are calculated based on the training spliced ​​features to obtain autocorrelation features, self-attention weights are calculated based on the autocorrelation features to obtain self-attention weights, and the training spliced ​​features are adjusted by the self-attention weights to obtain training fusion features; Performing feature loss calculation based on the training global feature, the training semantic feature and the training fusion feature to obtain an initial feature loss, and performing category loss calculation based on the training initial category and the training sample category label to obtain an initial category loss; The initial feature fusion model is updated based on the initial feature loss and the initial category loss, and the step of obtaining the training sample and the corresponding training sample category label is returned, and the step of inputting the training sample into the initial feature fusion model is iteratively executed until the training completion condition is met, and a target fusion model is obtained, and the target fusion model is used to fuse the semantic features and global features extracted from the input sample to obtain a target fusion feature, and the target fusion feature is used as the index of the input sample.

2. The method according to claim 1, characterized in that Before inputting the training sample into the initial feature fusion model, the method further includes: Obtaining pre-training samples and corresponding pre-training sample category labels, inputting the pre-training samples into a pre-training model, performing feature extraction on the pre-training samples through the pre-training model to obtain pre-training semantic features and pre-training global features, and performing classification and recognition based on the pre-training semantic features to obtain a pre-training category corresponding to the pre-training samples; Performing feature loss calculation based on the pre-trained semantic features and the pre-trained global features to obtain a pre-trained feature loss, and performing category loss calculation based on the pre-trained categories and the pre-trained sample category labels to obtain a pre-trained category loss; The pre-training model is updated based on the pre-training feature loss and the pre-training category loss, and the steps of obtaining the pre-training sample and the corresponding pre-training sample category label and inputting the pre-training sample into the pre-training model are returned and iteratively executed until the pre-training completion condition is met, thereby obtaining a target pre-training model; The initial feature fusion model is obtained based on the target pre-trained model.

3. The method according to claim 1, characterized in that The initial feature fusion model includes an initial basic feature extraction network, an initial semantic feature extraction network, an initial global feature extraction network and an initial classification network; The extracting features based on the training samples to obtain training semantic features and training global features, and classifying and identifying based on the training semantic features to obtain training initial categories corresponding to the training samples, include: Inputting the training sample into the initial basic feature extraction network to obtain an initial first basic feature and an initial second basic feature, wherein the dimension of the initial first basic feature is smaller than the dimension of the initial second basic feature; Inputting the initial first basic feature into the initial global feature extraction network to extract global hash features to obtain training global features; The initial second basic feature is input into the initial semantic feature extraction network for semantic hash feature extraction to obtain training semantic features, and the training semantic features are input into the initial classification network for classification and recognition to obtain the training initial category corresponding to the training sample.

4. The method according to claim 1, characterized in that: The step of calculating the autocorrelation feature based on the training splicing feature to obtain the autocorrelation feature, calculating the self-attention weight based on the autocorrelation feature to obtain the self-attention weight, and adjusting the training splicing feature by the self-attention weight to obtain the training fusion feature includes: Calculating the transposition of the training splicing feature to obtain a transposed feature, and multiplying the transposed feature by the training splicing feature to obtain the autocorrelation feature; Perform mean pooling based on the autocorrelation feature to obtain a pooling feature, obtain a self-attention parameter, and perform weighted calculation on the pooling feature based on the self-attention parameter to obtain a self-attention weight; The training splicing feature is adjusted by the self-attention weight to obtain a training adjustment feature, and feature crossover is performed based on the training adjustment feature to obtain a training fusion feature.

5. The method according to claim 4, characterized in that The initial feature fusion model includes an initial self-attention network and an initial feature intersection network; The obtaining of the self-attention parameter and performing weighted calculation on the pooling feature based on the self-attention parameter to obtain the self-attention weight include: Inputting the pooled features into the initial self-attention network, the initial self-attention network obtains self-attention parameters, and performs weighted calculation on the pooled features based on the self-attention parameters to obtain self-attention weights; The performing feature crossover based on the training adjustment feature to obtain the training fusion feature includes: The training adjustment features are input into the initial feature cross network for feature crossover to obtain training fusion features.

6. The method according to claim 1, characterized in that The calculating feature loss based on the training global feature, the training semantic feature and the training fusion feature to obtain the initial feature loss includes: Performing triplet loss calculation based on the training global features to obtain global feature loss information, and performing quantization loss calculation based on the training global features to obtain global quantization loss information; Performing triplet loss calculation based on the training semantic features to obtain semantic feature loss information, and performing quantization loss calculation based on the training semantic features to obtain semantic quantization loss information; Performing triplet loss calculation based on the training fusion feature to obtain fusion feature loss information, and performing quantization loss calculation based on the training fusion feature to obtain fusion quantization loss information; The initial feature loss is obtained by performing weighted calculation based on the global feature loss information, the global quantization loss information, the semantic feature loss information, the semantic quantization loss information, the fused feature loss information and the fused quantization loss information.

7. The method according to claim 6, characterized in that The training samples include each positive sample pair; The performing triplet loss calculation based on the training global features to obtain global feature loss information, and performing quantization loss calculation based on the training global features to obtain global quantization loss information, includes: Perform negative sample screening based on each positive sample pair to obtain negative samples corresponding to each positive sample pair, and obtain each sample triplet based on each positive sample pair and the corresponding negative sample; Calculating a first similarity distance between a training global feature corresponding to a first positive sample in each sample triple and a training global feature corresponding to a second positive sample in each sample triple, and calculating a second similarity distance between a training global feature corresponding to a first positive sample in each sample triple and a training global feature corresponding to a negative sample in each sample triple; Calculating the error between the first similarity distance and the second similarity distance to obtain the triplet global loss information corresponding to each sample triplet, and performing average calculation based on the triplet global loss information corresponding to each sample triplet to obtain the global feature loss information; The training global features corresponding to each sample in the training samples are symbolically quantized to obtain global quantization features, the errors between the training global features and the global quantization features are calculated to obtain initial global quantization loss information corresponding to each sample in the training samples, and the initial global quantization loss information corresponding to each sample in the training samples is averaged to obtain the global quantization loss information.

8. The method according to claim 7, characterized in that The performing negative sample screening based on each positive sample pair to obtain negative samples corresponding to each positive sample pair includes: Determine the current positive sample pair and each positive sample pair to be mined based on each positive sample pair; Randomly select from the current positive sample pair to obtain a current positive sample, and randomly select from each of the to-be-mined positive sample pairs to obtain each to-be-mined sample; Calculating the similarities between the current positive sample and each of the samples to be mined, and determining the current negative sample corresponding to the current positive sample from each of the samples to be mined based on the similarities; Traverse the positive sample pairs to obtain negative samples corresponding to the positive sample pairs.

9. The method according to claim 6, characterized in that The training samples include sample triplets; The triplet loss calculation based on the training semantic features to obtain semantic feature loss information, and the quantization loss calculation based on the training semantic features to obtain semantic quantization loss information include: Calculating a third similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the second positive sample in each sample triple, and calculating a fourth similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the negative sample in each sample triple; Calculating the error between the third similarity distance and the fourth similarity distance to obtain the triple semantic loss information corresponding to each sample triple, and performing average calculation based on the triple semantic loss information corresponding to each sample triple to obtain the semantic feature loss information; The training semantic features corresponding to each sample in the training samples are symbolically quantized to obtain semantic quantization features, the errors between the training semantic features and the semantic quantization features are calculated to obtain initial semantic quantization loss information corresponding to each sample in the training samples, and the initial semantic quantization loss information corresponding to each sample in the training samples is averaged to obtain the semantic quantization loss information.

10. The method according to claim 6, characterized in that The triplet loss calculation based on the training fusion feature to obtain fusion feature loss information, and the quantization loss calculation based on the training fusion feature to obtain fusion quantization loss information include: Calculating a fifth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the second positive sample in each sample triple, and calculating a sixth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the negative sample in each sample triple; Calculating the error between the fifth similarity distance and the sixth similarity distance to obtain triplet fusion loss information corresponding to each sample triplet, and performing average calculation based on the triplet fusion loss information corresponding to each sample triplet to obtain the fusion feature loss information; The training fusion features corresponding to each sample in the training samples are symbolically quantized to obtain fused quantized features, the errors between the training fusion features and the fused quantized features are calculated to obtain initial fused quantized loss information corresponding to each sample in the training samples, and the fused quantized loss information is averaged based on the initial fused quantized loss information corresponding to each sample in the training samples to obtain the fused quantized loss information.

11. The method according to claim 1, characterized in that: The calculating of the category loss based on the training initial category and the training sample category label to obtain the initial category loss includes: Calculating classification loss information between the training initial category corresponding to each sample in the training samples and the training sample category label to obtain the training category loss corresponding to each sample; The training category losses corresponding to the samples are averaged to obtain the initial category losses corresponding to the training samples.

12. The method according to claim 1, characterized in that After the steps of updating the initial feature fusion model based on the initial feature loss and the initial category loss, returning the steps of obtaining the training samples and the corresponding training sample category labels, and inputting the training samples into the initial feature fusion model are iteratively executed until the training completion condition is met and the target fusion model is obtained, the method further includes: Obtaining the input sample; Inputting the input sample into the target fusion model, performing feature extraction on the input sample through the target fusion model to obtain input semantic features and input global features, splicing the input semantic features and the input global features to obtain input splicing features, performing autocorrelation feature calculation based on the input splicing features to obtain input autocorrelation features, performing self-attention weight calculation based on the input autocorrelation features to obtain input self-attention weights, and adjusting the input splicing features through the input self-attention weights to obtain a first input fusion feature corresponding to the input sample; The first input fusion feature is used as the index corresponding to the input sample.

13. The method according to claim 12, characterized in that The target fusion model includes a semantic feature extraction network, a global feature extraction network, a self-attention network and a feature intersection network; The input sample is input into the target fusion model, feature extraction is performed on the input sample through the target fusion model to obtain input semantic features and input global features, the input semantic features and the input global features are spliced ​​to obtain input spliced ​​features, autocorrelation features are calculated based on the input spliced ​​features to obtain input autocorrelation features, self-attention weights are calculated based on the input autocorrelation features to obtain input self-attention weights, and the input spliced ​​features are adjusted by the input self-attention weights to obtain a first input fusion feature corresponding to the input sample, including: Inputting the input samples into the semantic feature extraction network and the global feature extraction network in the target fusion model respectively to obtain the input semantic features and the input global features; The input semantic feature and the input global feature are concatenated to obtain an input concatenated feature, and the input concatenated feature is input into the self-attention network. The self-attention network calculates an autocorrelation feature based on the input concatenated feature to obtain an input autocorrelation feature, calculates a self-attention weight based on the input autocorrelation feature to obtain an input self-attention weight, and adjusts the input concatenated feature by the input self-attention weight to obtain an input adjustment feature corresponding to the input sample; The input adjustment feature is input into a feature cross network for feature crossover to obtain a first input fusion feature corresponding to the input sample.

14. A sample retrieval method, characterized in that: The method comprises: Obtaining a query fusion feature corresponding to the query sample, wherein the query fusion feature is obtained by inputting the query sample into a target fusion model to extract the fusion feature; The query sample is subjected to feature extraction through the target fusion model to obtain query semantic features and query global features, the query semantic features and the query global features are spliced ​​to obtain query splicing features, autocorrelation features are calculated based on the query splicing features to obtain query autocorrelation features, self-attention weights are calculated based on the query autocorrelation features to obtain query self-attention weights, and the query splicing features are adjusted by the query self-attention weights to obtain query fusion features; the target fusion model is trained based on the feature fusion model training method according to any one of claims 1 to 13; Sample retrieval is performed based on the query fusion feature to obtain a target retrieval sample corresponding to the query sample.

15. A feature fusion model training device, characterized in that: The device comprises: An input module, used to obtain training samples and corresponding training sample category labels, and input the training samples into an initial feature fusion model, wherein the training samples include at least one of the following: images, texts, videos, and audios; An initial recognition module, used to extract features from the training samples through the initial feature fusion model to obtain training semantic features and training global features, and to perform classification and recognition based on the training semantic features to obtain training initial categories corresponding to the training samples; An initial fusion module, used for splicing the training semantic features and the training global features through the initial feature fusion model to obtain training splicing features, calculating autocorrelation features based on the training splicing features to obtain autocorrelation features, calculating self-attention weights based on the autocorrelation features to obtain self-attention weights, and adjusting the training splicing features through the self-attention weights to obtain training fusion features; A loss calculation module, used to perform feature loss calculation based on the training global features, the training semantic features and the training fusion features to obtain an initial feature loss, and to perform category loss calculation based on the training initial category and the training sample category label to obtain an initial category loss; An iterative module is used to update the initial feature fusion model based on the initial feature loss and the initial category loss, and return the steps of obtaining the training sample and the corresponding training sample category label, and inputting the training sample into the initial feature fusion model, until the training completion condition is met, and a target fusion model is obtained, the target fusion model is used to fuse the semantic features and global features extracted from the input sample to obtain a target fusion feature, and the target fusion feature is used as the index of the input sample.

16. The device according to claim 15, characterized in that The device further comprises: A pre-training module is used to obtain pre-training samples and corresponding pre-training sample category labels, input the pre-training samples into a pre-training model, perform feature extraction on the pre-training samples through the pre-training model to obtain pre-training semantic features and pre-training global features, and perform classification and recognition based on the pre-training semantic features to obtain the pre-training category corresponding to the pre-training samples; perform feature loss calculation based on the pre-training semantic features and pre-training global features to obtain pre-training feature loss, and perform category loss calculation based on the pre-training category and the pre-training sample category label to obtain pre-training category loss; update the pre-training model based on the pre-training feature loss and the pre-training category loss, and return the steps of obtaining pre-training samples and corresponding pre-training sample category labels, and inputting the pre-training samples into the pre-training model, and iteratively execute until the pre-training completion condition is met to obtain a target pre-training model; obtain the initial feature fusion model based on the target pre-training model.

17. The device according to claim 15, characterized in that The initial feature fusion model includes an initial basic feature extraction network, an initial semantic feature extraction network, an initial global feature extraction network and an initial classification network; The initial recognition module is also used to input the training sample into the initial basic feature extraction network to obtain an initial first basic feature and an initial second basic feature, wherein the dimension of the initial first basic feature is smaller than the dimension of the initial second basic feature; input the initial first basic feature into the initial global feature extraction network to perform global hash feature extraction to obtain a training global feature; input the initial second basic feature into the initial semantic feature extraction network to perform semantic hash feature extraction to obtain a training semantic feature, and input the training semantic feature into the initial classification network to perform classification and recognition to obtain a training initial category corresponding to the training sample.

18. The device according to claim 15, characterized in that The initial fusion module is also used to calculate the transpose of the training splicing feature to obtain the transposed feature, multiply the transposed feature by the training splicing feature to obtain the autocorrelation feature; perform mean pooling based on the autocorrelation feature to obtain the pooled feature, and obtain the self-attention parameter, perform weighted calculation on the pooled feature based on the self-attention parameter to obtain the self-attention weight; adjust the training splicing feature by the self-attention weight to obtain the training adjustment feature, perform feature crossover based on the training adjustment feature to obtain the training fusion feature.

19. The device according to claim 18, characterized in that The initial feature fusion model includes an initial self-attention network and an initial feature intersection network; The initial fusion module is also used to input the pooled features into the initial self-attention network, the initial self-attention network obtains self-attention parameters, performs weighted calculation on the pooled features based on the self-attention parameters, and obtains self-attention weights; inputs the training adjustment features into the initial feature cross network for feature crossover to obtain training fusion features.

20. The device according to claim 15, characterized in that The loss calculation module comprises: A global loss calculation unit, used to perform triplet loss calculation based on the training global features to obtain global feature loss information, and perform quantization loss calculation based on the training global features to obtain global quantization loss information; A semantic loss calculation unit, configured to perform triple loss calculation based on the training semantic features to obtain semantic feature loss information, and perform quantization loss calculation based on the training semantic features to obtain semantic quantization loss information; A fusion loss calculation unit, used to perform triplet loss calculation based on the training fusion feature to obtain fusion feature loss information, and perform quantization loss calculation based on the training fusion feature to obtain fusion quantization loss information; The feature loss obtaining unit is used to perform weighted calculation based on the global feature loss information, the global quantization loss information, the semantic feature loss information, the semantic quantization loss information, the fused feature loss information and the fused quantization loss information to obtain the initial feature loss.

21. The device according to claim 20, characterized in that The training samples include each positive sample pair; The global loss calculation unit is also used to perform negative sample screening based on the positive sample pairs to obtain negative samples corresponding to the positive sample pairs, and obtain sample triplets based on the positive sample pairs and the corresponding negative samples; calculate a first similarity distance between a training global feature corresponding to a first positive sample in each sample triple and a training global feature corresponding to a second positive sample in each sample triple, and calculate a second similarity distance between a training global feature corresponding to a first positive sample in each sample triple and a training global feature corresponding to a negative sample in each sample triple; Calculating the error between the first similarity distance and the second similarity distance to obtain the triplet global loss information corresponding to each sample triplet, and performing average calculation based on the triplet global loss information corresponding to each sample triplet to obtain the global feature loss information; The training global features corresponding to each sample in the training samples are symbolically quantized to obtain global quantization features, the errors between the training global features and the global quantization features are calculated to obtain initial global quantization loss information corresponding to each sample in the training samples, and the initial global quantization loss information corresponding to each sample in the training samples is averaged to obtain the global quantization loss information.

22. The device according to claim 21, characterized in that The global loss calculation unit is also used to determine the current positive sample pair and each positive sample pair to be mined based on each positive sample pair; randomly select from the current positive sample pair to obtain the current positive sample, and randomly select from each positive sample pair to be mined to obtain each sample to be mined; Calculate the similarity between the current positive sample and each of the samples to be mined, and determine the current negative sample corresponding to the current positive sample pair from the samples to be mined based on the similarity; traverse each of the positive sample pairs to obtain the negative samples corresponding to each of the positive sample pairs.

23. The device according to claim 20, characterized in that The training samples include sample triplets; The semantic loss calculation unit is further used to calculate a third similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the second positive sample in each sample triple, and calculate a fourth similarity distance between the training semantic feature corresponding to the first positive sample in each sample triple and the training semantic feature corresponding to the negative sample in each sample triple; Calculating the error between the third similarity distance and the fourth similarity distance to obtain the triple semantic loss information corresponding to each sample triple, and performing average calculation based on the triple semantic loss information corresponding to each sample triple to obtain the semantic feature loss information; The training semantic features corresponding to each sample in the training samples are symbolically quantized to obtain semantic quantization features, the errors between the training semantic features and the semantic quantization features are calculated to obtain initial semantic quantization loss information corresponding to each sample in the training samples, and the initial semantic quantization loss information corresponding to each sample in the training samples is averaged to obtain the semantic quantization loss information.

24. The device according to claim 20, characterized in that The fusion loss calculation unit is further used to calculate a fifth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the second positive sample in each sample triple, and calculate a sixth similarity distance between the training fusion feature corresponding to the first positive sample in each sample triple and the training fusion feature corresponding to the negative sample in each sample triple; Calculating the error between the fifth similarity distance and the sixth similarity distance to obtain triplet fusion loss information corresponding to each sample triplet, and performing average calculation based on the triplet fusion loss information corresponding to each sample triplet to obtain the fusion feature loss information; The training fusion features corresponding to each sample in the training samples are symbolically quantized to obtain fused quantized features, the errors between the training fusion features and the fused quantized features are calculated to obtain initial fused quantized loss information corresponding to each sample in the training samples, and the fused quantized loss information is averaged based on the initial fused quantized loss information corresponding to each sample in the training samples to obtain the fused quantized loss information.

25. The device according to claim 15, characterized in that The loss calculation module comprises: The category loss calculation unit is used to calculate the classification loss information between the training initial category corresponding to each sample in the training sample and the category label of the training sample to obtain the training category loss corresponding to each sample; and perform average calculation based on the training category loss corresponding to each sample to obtain the initial category loss corresponding to the training sample.

26. The device according to claim 15, characterized in that The device further comprises: A sample index establishment module is used to obtain the input sample; input the input sample into the target fusion model, perform feature extraction on the input sample through the target fusion model to obtain input semantic features and input global features, splice the input semantic features and the input global features to obtain input splicing features, calculate autocorrelation features based on the input splicing features to obtain input autocorrelation features, calculate self-attention weights based on the input autocorrelation features to obtain input self-attention weights, and adjust the input splicing features through the input self-attention weights to obtain a first input fusion feature corresponding to the input sample; and use the first input fusion feature as the index corresponding to the input sample.

27. The device according to claim 26, characterized in that The target fusion model includes a semantic feature extraction network, a global feature extraction network, a self-attention network and a feature intersection network; The sample index establishment module is also used to input the input sample into the semantic feature extraction network and the global feature extraction network in the target fusion model respectively to obtain the input semantic feature and the input global feature; splice the input semantic feature and the input global feature to obtain an input splicing feature, and input the input splicing feature into the self-attention network, the self-attention network calculates autocorrelation features based on the input splicing features to obtain input autocorrelation features, calculates self-attention weights based on the input autocorrelation features to obtain input self-attention weights, and adjusts the input splicing features by the input self-attention weights to obtain input adjustment features corresponding to the input sample; inputs the input adjustment features into the feature crossover network for feature crossover to obtain a first input fusion feature corresponding to the input sample.

28. A sample retrieval device, characterized in that: The device comprises: A feature acquisition module is used to acquire a query fusion feature corresponding to a query sample, wherein the query fusion feature is obtained by inputting the query sample into a target fusion model to extract the fusion feature; A feature extraction module, used to extract features from the query sample through the target fusion model to obtain query semantic features and query global features, concatenate the query semantic features and the query global features to obtain query concatenation features, calculate autocorrelation features based on the query concatenation features to obtain query autocorrelation features, calculate self-attention weights based on the query autocorrelation features to obtain query self-attention weights, and adjust the query concatenation features through the query self-attention weights to obtain query fusion features; the target fusion model is trained based on the feature fusion model training method according to any one of claims 1 to 13; The retrieval module is used to perform sample retrieval based on the query fusion feature to obtain a target retrieval sample corresponding to the query sample.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images