Cross-modal retrieval method, device, equipment and computer-readable storage medium

Abstract

The embodiment of the present application discloses a cross-modal retrieval method, including: obtaining first modal data and second modal data to be retrieved; inputting the first modal data and the second modal data into a trained cross-modal retrieval network; extracting a first feature of the first modal data and a second feature of the second modal data through the cross-modal retrieval network, and outputting a retrieval result according to the matching degree of the first feature and the second feature; the retrieval result is used to characterize the data retrieved from the first modal data and matching the second modal data; wherein the cross-modal retrieval network includes a network trained based on original data and generated data in a data set using a neural network; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a cross-modal retrieval method, a cross-modal retrieval apparatus, a cross-modal retrieval device, and a computer-readable storage medium. Background Art

[0002] With the popularity of smart mobile terminals, every user can publish and receive multi-modal data including text, pictures, videos, audio, etc. anytime and anywhere, which brings rich content. However, rich content often brings more pain to users in terms of selection, and it becomes increasingly difficult to obtain the information they really need. Rich information needs to be accurately screened and presented to users. For retrieval systems, it is necessary to provide accurate retrieval and services with users as the center. Therefore, cross-modal retrieval systems have become a hot topic today.

[0003] For example, image-text cross-modal retrieval is an important and challenging classic computer vision task, which has wide applications in search engines, smart online shopping malls, preschool education, etc. However, current retrieval is only in the single-modal stage. For example, although the current mainstream search engines can return a series of pictures or images when users enter text, this retrieval relies on the text information annotated in the picture itself, so it is essentially still a single-modal retrieval that searches for text with text.

[0004] The prior art discloses a cross-modal retrieval method based on a generative adversarial network, which includes step 1, first using a feature extraction method to extract features from the data of the input modality and the data of the target modality, step 2, establishing and training a generative adversarial network (GAN) model so that the GAN model can generate the data of the target modality from the data of the input modality, step 3, using the GAN model to perform similarity matching with the data of the corresponding modality obtained in step 1, that is, calculating the Euclidean distance; step 4, arranging the calculation results of the Euclidean distance from small to large, thereby obtaining the result of the cross-modal retrieval; the smaller the Euclidean distance, the higher the similarity between the top-ranked result and the retrieval target.

[0005] However, the heterogeneity between samples of different modalities makes it very difficult to measure sample similarity across modalities. At the same time, due to the small number of standard training data sets, it is impossible to learn a more discriminative embedding subspace, and therefore it is impossible to effectively distinguish and retrieve some difficult samples. Summary of the invention

[0006] The technical problem to be solved by the embodiments of the present application is to provide a cross-modal retrieval method, a cross-modal retrieval device, a cross-modal retrieval equipment and a computer-readable storage medium, which can distinguish between noise and dense textures, solve the technical problem in the prior art that there is heterogeneity between samples of different modalities, which brings great difficulties to the cross-modal measurement of sample similarity, and can improve the accuracy of cross-modal retrieval.

[0007] In order to solve the above technical problems, the present application discloses a cross-modal retrieval method, including:

[0008] Acquire first modal data and second modal data to be retrieved;

[0009] Inputting the first modality data and the second modality data into a trained cross-modality retrieval network;

[0010] Extracting a first feature of the first modality data and a second feature of the second modality data through the cross-modality retrieval network, and outputting a retrieval result according to a matching degree between the first feature and the second feature; the retrieval result is used to represent data retrieved from the first modality data and matching the second modality data;

[0011] Among them, the cross-modal retrieval network includes a network trained by a neural network based on original data and generated data in a data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data.

[0012] Another aspect of the present application embodiment discloses a cross-modal retrieval method, including:

[0013] Train the generator and discriminator in the generative adversarial network based on the original data in the dataset;

[0014] Generating data based on the original data by a trained generator; the generator is used to generate data belonging to the first modality from data belonging to the second modality in the original data;

[0015] A cross-modal retrieval network is obtained by training a neural network based on the original data and the generated data; wherein the cross-modal retrieval network is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result; the retrieval result is used to characterize the data retrieved from the first modality data and matching the second modality data.

[0016] Another aspect of the present application discloses a cross-modal search device, including:

[0017] A data acquisition unit, used to acquire the first modality data and the second modality data to be retrieved;

[0018] A data input unit, used for inputting the first modality data and the second modality data into a trained cross-modality retrieval network;

[0019] a retrieval unit, configured to extract a first feature of the first modal data and a second feature of the second modal data through the cross-modal retrieval network, and output a retrieval result according to a matching degree between the first feature and the second feature; the retrieval result is used to represent data retrieved from the first modal data and matching the second modal data;

[0020] Among them, the cross-modal retrieval network includes a network trained by a neural network based on original data and generated data in a data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data.

[0021] Another aspect of the present application discloses a cross-modal search device, including:

[0022] A first training unit is used to train a generator and a discriminator in a generative adversarial network based on original data in a data set;

[0023] A data generating unit, configured to generate data based on the original data by a trained generator; the generator is configured to generate data belonging to the first modality from data belonging to the second modality in the original data;

[0024] The second training unit is used to use a neural network to train a cross-modal retrieval network based on the original data and the generated data; wherein the cross-modal retrieval network is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result; the retrieval result is used to characterize the data retrieved from the first modality data that matches the second modality data.

[0025] On the other hand, an embodiment of the present application discloses a cross-modal retrieval device, characterized in that it includes a processor, wherein the processor is configured to call a stored cross-modal retrieval program to execute the above-mentioned cross-modal retrieval method.

[0026] On the other hand, an embodiment of the present application discloses a computer-readable storage medium, wherein the computer storage medium stores program instructions, and when the program instructions are executed by a processor, the processor executes the above-mentioned cross-modal retrieval method.

[0027] In implementing the embodiment of the present application, a cross-modal retrieval network is trained based on the original data and generated data in the data set using a neural network; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from the data belonging to the second modality in the original data. The generated data can effectively expand the data set and can be combined with many model structures, with high universality; by using more training data, a more discriminative embedding feature subspace can be learned, and fine-grained matching relationships between cross-modalities can be mined, the discrimination between samples can be enhanced, and the heterogeneity between modalities can be reduced, and better discriminative features between cross-modalities in the embedding space can be obtained, thereby reducing the difficulty of similarity judgment between different modalities, thereby effectively improving the accuracy of cross-modal retrieval. Furthermore, in order to achieve the accuracy of cross-modal retrieval in the present application, the present application introduces an enhanced embedding space model (i.e., a more discriminative embedding feature subspace) and expands the training data by generating adversarial networks combined with reinforcement learning training strategies. Compared with the existing technology, the operation efficiency is higher and the computer computing resources occupied are less, thereby improving computer performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be introduced below.

[0029] Figure 1 is a schematic block diagram of a system architecture in an embodiment of the present application;

[0030] Figure 2 It is a chip hardware structure provided by an embodiment of the present application;

[0031] Figure 3 is a flowchart of a cross-modal retrieval method provided in an embodiment of the present application;

[0032] Figure 4 is a flowchart of another embodiment of the cross-modal retrieval method provided by the present application;

[0033] Figure 5 is a flowchart of another embodiment of the cross-modal retrieval method provided by the present application;

[0034] Figure 6 is a schematic block diagram of a convolutional neural network in an embodiment of the present application;

[0035] Figure 7 is a schematic diagram of the structure of a cross-modal retrieval device provided in an embodiment of the present application;

[0036] Figure 8 is a structural schematic diagram of another embodiment of the cross-modal retrieval device provided by the present application;

[0037] Fig. 9 is a schematic diagram of the structure of a cross-modal retrieval device provided in an embodiment of the present application;

[0038] Fig.10 It is a schematic block diagram of another cross-modal retrieval device in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following is a description with reference to the accompanying drawings in the embodiments of the present application.

[0040] In order to better understand the cross-modal detection method provided in the embodiment of the present application, the technical scenarios in which the cross-modal detection method can be applied are first described. Specifically, the cross-modal detection method in the embodiment of the present application can be applied in the following scenarios A, B, and C, but the cross-modal detection method in the embodiment of the present application is not limited to these three application scenarios. The following is a brief introduction to scenario A, scenario B, and scenario C.

[0041] Scenario A:

[0042] Retrieval and matching for image data and text data. When a user views an image or picture and wants to describe the content or attributes of the image, the current user can only rely on his or her own cognitive ability and description ability to write out information such as the content or attributes of the image. However, through the cross-modal retrieval method of the embodiment of the present application, the target image can be used as a query input to automatically generate a text description that matches the target image and recommend it to the user. In addition to relying on his or her own cognitive ability and description ability, the user can also refer to the generated text description to obtain information such as the content or attributes of the image.

[0043] Scenario B:

[0044] Retrieve and match image data and voice data. For example, in the case of intelligent voice image search on mobile phones, current smartphones are equipped with voice assistants, but faced with the massive amount of pictures or videos in the mobile phones, these voice assistants cannot search for the corresponding images and videos based on the user's voice. This is itself a cross-modal retrieval problem, which involves the recognition and understanding of the user's voice, and then matching it with the image content in the mobile phone album. Through the cross-modal retrieval method of the embodiment of the present application, the user's voice can be used as a query input, and the image content in the mobile phone album is retrieved and matched to automatically generate an image that matches the user's voice.

[0045] Scenario C:

[0046] Retrieval and matching of video data and music data. For example, composing music for entertainment short videos / advertising videos. When a user records an entertainment short video such as a TikTok video, it is particularly important to compose music for it, but current software requires users to manually find music that is suitable for the video content. Through the cross-modal retrieval method of the embodiment of the present application, the cross-modal retrieval technology can be used to use the video as a query to automatically find the music that best matches it, and make personalized recommendations, for example, mainly considering the video's actions, shot transitions, and the matching of the music beats.

[0047] The following describes the method provided by this application from the perspectives of model training and model application:

[0048] The training method involved in the cross-modal detection method provided in the embodiment of the present application involves computer vision processing or natural language processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, to symbolize and formalize intelligent information modeling, extraction, preprocessing, and training of the training data in the data set, and finally obtain a trained cross-modal retrieval network or cross-modal retrieval model; and the cross-modal detection method provided in the embodiment of the present application can use the generator and discriminator trained in the generative adversarial network to input the input data into the trained generator to obtain generated data. After the cross-modal retrieval network is trained, the cross-modal detection method provided in the embodiment of the present application can use the cross-modal retrieval network to retrieve and match the input data of different modalities to obtain retrieval results.

[0049] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0050] (1) Image recognition

[0051] In the embodiment of the present application, image recognition is to use image processing and related methods such as machine learning and computer graphics to identify the category to which the image belongs or the attributes of the image according to the image. For example, in scene A and scene B, the content, attributes or category of the image are identified.

[0052] (2) Text Recognition

[0053] In the embodiments of the present application, text recognition is also called natural language recognition, which utilizes relevant methods such as linguistics, computer science, artificial intelligence, etc. to identify the intention, emotion or other attributes expressed by the text based on the text.

[0054] (3) Neural Network

[0055] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0056]

[0057] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0058] (4) Deep Neural Networks

[0059] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, b is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the number of coefficients W and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0060] (5) Convolutional Neural Networks

[0061] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0062] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0063] (6) Recurrent neural networks (RNN) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this common neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network of the current step, but also on the state of the network of the previous steps. This learning algorithm is called Back Propagation Through Time (BPTT).

[0064] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0065] (7) Loss Function

[0066] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0067] For example, in the present application, in the process of optimizing the initial sample generator, it is hoped that the first generated sample output by the initial sample generator is as close to the real sample as possible. Since the preset discriminator is a trained neural network, the preset discriminator can accurately identify the category to which the real sample belongs. When the preset discriminator can accurately identify the category to which the first generated sample belongs, it can be considered that the first generated sample has similar characteristics to the real sample, that is, close to the real sample. Therefore, by comparing the judgment result of the preset discriminator on the first generated sample with the real result that is really wanted, the weight vector of each layer of the neural network in the initial sample generator is updated according to the difference between the two (of course, there is usually a process before the first update, that is, pre-configuring parameters for each layer in the initial sample generator). For example, if the value of the judgment result of the preset discriminator is high, the weight vector is adjusted to make its value lower, and it is continuously adjusted until the preset discriminator can predict a value that is very close to the real result. Therefore, it is necessary to predefine "how to compare the difference between the judgment result and the real result", which is the loss function or objective function, which are important equations for measuring the difference between the judgment result and the real result. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the initial sample generator becomes a process of minimizing the loss as much as possible.

[0068] (8) Back propagation algorithm

[0069] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial sample generator during the training process, so that the reconstruction error loss of the initial sample generator becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial sample generator, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the target sample generator, such as the weight matrix.

[0070] (9) Generative Adversarial Networks

[0071] Generative adversarial networks (GAN) are a deep learning model. The model includes at least two modules: one module is a generative model (also called a generative network in the embodiment of the present application), and the other module is a discriminative model (also called a discriminative network in the embodiment of the present application). These two modules learn from each other through game, thereby generating better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, assume that there are two networks, G (generator) and D (discriminator), where G is a network that generates pictures, it receives a random noise z, and generates pictures through this noise, denoted as G (z); D is a discriminative network, which is used to determine whether a picture is "real". Its input parameter is x, x represents a picture, and the output D (x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0072] The following introduces the system architecture provided by the embodiments of the present application.

[0073] See attached Figure 1, the embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: data of multiple modalities, such as image or picture data, text data, video data, music data, etc.; and the training data is stored in the database 130, and the training device 120 is trained based on the training data maintained in the database 130 and the preset discriminator 121 to obtain the trained target generator 101. The training process may include that the training device 120 inputs the first training data into the initial generator 122 to obtain the first generated sample, and the initial generator 122 may be a deep neural network; further, the first generated sample is input into the preset discriminator 121 to obtain the discrimination result, wherein the preset discriminator 121 is obtained by training the first training data, and the first training data includes the real sample and the classification corresponding to the real sample; further, the real result of the first generated sample is determined according to the discrimination result, and the initial generator 122 is updated according to the difference between the real result of the first real sample and the discrimination result, so as to obtain the target generator 101. The detailed description of the training device 120 obtaining the target generator 101 based on the training data can be found in the relevant description of the following embodiment, which is not expanded here. While training the generator, the cross-modal retrieval method of the embodiment of the present application also trains the preset discriminator 121 based on the training data to obtain the trained target discriminator 102. The detailed description can be found in the relevant description of the following embodiment, which is not expanded here.

[0074] Since the standard training data set for cross-modal retrieval is multimodal data with a pre-existing matching relationship, the standard training data set is relatively small. In the embodiment of the present application, the target generator 101 obtained through training generates generated data based on the data in the database 130, and then adds the generated data to the database 130, thereby effectively expanding the training data set.

[0075] The training device 120 trains the cross-modal retrieval network based on the original training data (i.e., original data) and the added generated data in the database 130 to obtain the trained cross-modal retrieval network 103. The execution device 110 can use the trained cross-modal retrieval network 103 to search and match the input cross-modal data and output the retrieval results. For a detailed description, please refer to the relevant description of the following embodiment, which will not be expanded here.

[0076] It should be noted that, in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the target sample generator 101 entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or generate training data for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0077] The target sample generator 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle-mounted terminal, etc., or may be a server or a cloud, etc.

[0078] It is worth noting that Figure 1 It is only a schematic diagram of a system architecture provided in an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.

[0079] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0080] Figure 2 A chip hardware structure provided in an embodiment of the present application includes a neural network processor 20. The chip can be arranged as follows: Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 171. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. The algorithms of each layer in the convolutional neural network can be as follows: Figure 2 is implemented in the chip shown.

[0081] The neural network processor 20 can be any processor suitable for large-scale XOR operation processing, such as NPU, TPU, or GPU. Take NPU as an example: NPU can be mounted on the host CPU as a coprocessor, and the host CPU assigns tasks to it. The core part of NPU is the operation circuit 203, which is controlled by the controller 204 to extract the matrix data in the memory (201 and 202) and perform multiplication and addition operations.

[0082] In some implementations, the operation circuit 203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 203 is a two-dimensional systolic array. The operation circuit 203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 203 is a general-purpose matrix processor.

[0083] For example, assume there are input matrix A, weight matrix B, and output matrix C. Operation circuit 203 takes weight data of matrix B from weight memory 202 and caches it on each PE in operation circuit 203. Operation circuit 203 takes input data of matrix A from input memory 201, performs matrix operation based on input data of matrix A and weight data of matrix B, and stores partial results or final results of the obtained matrix in accumulator 208.

[0084] The unified memory 206 is used to store input data and output data. The weight data is directly transferred to the weight memory 202 through the memory unit access controller (DMAC, Direct Memory Access Controller) 205. The input data is also transferred to the unified memory 206 through the DMAC.

[0085] The bus interface unit (BIU) 210 is used for the interaction between the DMAC and the instruction fetch buffer 209; the bus interface unit 201 is also used for the instruction fetch buffer 209 to obtain instructions from the external memory; the bus interface unit 201 is also used for the storage unit access controller 205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0086] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 206, or to transfer weight data to the weight memory 202, or to transfer input data to the input memory 201.

[0087] The vector calculation unit 207 includes multiple operation processing units, and when necessary, further processes the output of the operation circuit 203, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 207 is mainly used for the calculation of non-convolutional layers or fully connected layers (FC) in neural networks, and can specifically process the calculation of Pooling, Normalization, etc. For example, the vector calculation unit 207 can apply a nonlinear function to the output of the operation circuit 203, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 207 generates a normalized value, a merged value, or both.

[0088] In some implementations, the vector calculation unit 207 stores the processed vectors in the unified memory 206. In some implementations, the vectors processed by the vector calculation unit 207 can be used as activation inputs of the operation circuit 203, for example, for use in subsequent layers in a neural network, such as Figure 2 As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 207 can also be used in the calculation of hidden layer 2 (232).

[0089] An instruction fetch buffer 209 connected to the controller 204, for storing instructions used by the controller 204;

[0090] The unified memory 206, the input memory 201, the weight memory 202 and the instruction fetch memory 209 are all on-chip memories. The external memory is independent of the NPU hardware architecture.

[0091] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 203 or the vector calculation unit 207.

[0092] The method involved in the embodiments of the present application is described in detail below.

[0093] Figure 3 A schematic diagram of a cross-modal retrieval method provided in an embodiment of the present application is provided. The method may be specifically performed as follows: Figure 1 The training device 120 is shown to perform.

[0094] Optionally, the method can be processed by the CPU, or by the CPU and GPU together, or by using other processors suitable for neural network computing instead of the GPU, such as Figure 2 The neural network processor 20 shown in the figure is not limited in this application. The method may include some or all of the following steps:

[0095] Step S300: training the generator and the discriminator in the generative adversarial network based on the original data in the data set;

[0096] Specifically, the generator in the generative adversarial network can be trained based on the original data in the data set using the maximum likelihood criterion, and the discriminator in the generative adversarial network can be trained based on the data generated by the generator using a three-category discriminant loss function; the trained generator is trained using a reinforcement learning strategy, and the three-category discriminant loss function continues to be used to train the discriminator in the generative adversarial network.

[0097] In one embodiment, the training of the generator in the generative adversarial network based on the original data in the data set using the maximum likelihood criterion may specifically include: initializing the training based on the original data in the data set using the maximum likelihood criterion loss function and using the Adam optimization algorithm. The training of the trained generator through a reinforcement learning strategy may specifically include: introducing a reinforcement learning strategy and a Monte Carlo search technique, and using the Adam optimization algorithm to train the generator after the initial training until convergence.

[0098] In one implementation, three types of discriminant loss functions can be used to train the discriminator in the generative adversarial network, so that the labels of samples generated by the generator are false and the labels of real samples are true. At the same time, it is determined whether the image content and the text description content match. At the same time, the "Adam optimization algorithm" is used for initialization training.

[0099] Step S302: generating data based on the original data by a trained generator;

[0100] Specifically, the generator is used to generate data belonging to the first modality for data belonging to the second modality in the original data. That is to say, the cross-modal retrieval method of the embodiment of the present application does not generate features of two modal data in an adversarial manner from two directions, but generates from one modality to another in a single direction and end-to-end; the embodiment of the present application is not mainly to use adversarial learning to narrow the semantic gap of cross-modal data, that is, it is not mainly to force the speech gap between cross-modal data to be narrowed, but mainly to synthesize more data to strengthen model training.

[0101] For example, taking image-text cross-modal retrieval as an example, the embodiment of the present application does not generate the features of the image-text in an adversarial manner from two directions, but generates the text from the original image in a single direction and end-to-end. The embodiment of the present application is more for generating text data from image data to synthesize more data to strengthen model training.

[0102] Step S304: using a neural network to train a cross-modal retrieval network based on the original data and the generated data.

[0103] Specifically, based on the original data and the generated data, a long short-term memory network is used to extract a first feature of the data belonging to the first modality, a deep neural network is used to extract a second feature of the data belonging to the second modality, and a linear transformation is used to map them to an embedding subspace; in the embedding subspace, the cosine similarity of each pair of data belonging to the first modality and the data belonging to the second modality is calculated, and a triplet loss function is used to train each input batch, and the Adam optimization algorithm is used for training until the model converges to obtain a cross-modal retrieval network.

[0104] In implementing the embodiment of the present application, a cross-modal retrieval network is trained based on the original data and generated data in the data set using a neural network; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from the data belonging to the second modality in the original data. The generated data can effectively expand the data set and can be combined with many model structures, with high universality; by using more training data, a more discriminative embedding feature subspace can be learned, and fine-grained matching relationships between cross-modalities can be mined, the discrimination between samples can be enhanced, and the heterogeneity between modalities can be reduced, and better discriminative features between cross-modalities in the embedding space can be obtained, thereby reducing the difficulty of similarity judgment between different modalities, thereby effectively improving the accuracy of cross-modal retrieval.

[0105] The cross-modal retrieval network in the embodiment of the present application is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result; the retrieval result is used to characterize the data retrieved from the first modality data that matches the second modality data.

[0106] Combine the following Figure 4 The flowchart of another embodiment of the cross-modal retrieval method provided by the present application is shown in FIG. Figure 1 The execution device 110 is shown to execute.

[0107] Optionally, the method can be processed by the CPU, or by the CPU and GPU together, or by using other processors suitable for neural network computing instead of the GPU, such as Figure 2 The neural network processor 20 shown in the figure is not limited in this application. The method may include some or all of the following steps:

[0108] Step S400: Acquire first modal data and second modal data to be retrieved;

[0109] Specifically, the first modality and the second modality of the embodiment of the present application are two different modalities, for example, the first modality is text and the second modality is image; or the first modality is music and the second modality is video, and so on.

[0110] Step S402: inputting the first modality data and the second modality data into a trained cross-modality retrieval network;

[0111] Specifically, the cross-modal retrieval network includes a network trained by a neural network based on the original data and generated data in the data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from the data belonging to the second modality in the original data. The cross-modal retrieval network can be obtained by the above Figure 3 The retrieval network trained in the embodiment will not be described in detail here.

[0112] Step S404: extracting a first feature of the first modal data and a second feature of the second modal data through the cross-modal retrieval network, and outputting a retrieval result according to a matching degree between the first feature and the second feature.

[0113] The following takes the case where the first modal data is text data and the second modal data is image data as an example. Figure 5 The flowchart of another embodiment of the cross-modal retrieval method provided by the present application is shown for explanation. Before training the generator and the discriminator in the generative adversarial network, the image data may also be enhanced, which may specifically include the following steps:

[0114] Step S500: normalizing the image data in the training data set to a uniform size, and then randomly taking a mirroring approach to the normalized data to perform image data enhancement;

[0115] Step S502: sending the enhanced image data into a pre-trained deep backbone convolutional neural network;

[0116] Specifically, the output feature map of a certain intermediate convolutional layer of the deep backbone convolutional neural network can be selected, and the image features can be further extracted using a fully connected layer; the obtained image feature vector is used as the initial global image feature vector after linear mapping.

[0117] In one implementation, when feature extraction is performed on text data, the embodiment of the present application may use a bidirectional recurrent neural network framework to extract text description features, and the specific steps may be:

[0118] Using a natural language syntactic analysis tool, m noun phrases are extracted from the sentence. For each word in the phrase, word vectors are first embedded, and then these word vectors are sequentially input into the bidirectional recurrent neural network. The outputs of the final bidirectional hidden state are concatenated and linearly transformed to obtain the feature vector corresponding to each phrase. For text description, word vectors are first embedded for each word in the sentence, and then these word vectors are sequentially input into the long short-term memory network to obtain the initial text description feature vector.

[0119] Step S504: adopting the maximum likelihood criterion to train the generator in the generative adversarial network; adopting the three-category discriminant loss function to train the discriminator in the generative adversarial network;

[0120] Specifically, the embodiment of the present application can respectively make the labels of the samples generated by the generator false and the labels of the real samples true, and at the same time determine whether the image content and the text description content match.

[0121] Step S506: The generator in the pre-trained generative adversarial network is trained using a reinforcement learning strategy; meanwhile, the discriminator in the generative adversarial network is trained using the three-category discriminant loss function;

[0122] Specifically, please refer to the above Figure 3 Step S300 in the embodiment will not be described in detail here.

[0123] Step S508: using the trained generator of the generative adversarial network to generate image text for the images in the data set;

[0124] Specifically, the embodiment of the present application does not perform adversarial generation of image text features from two directions, but generates text from the original image in a single direction and end-to-end. The embodiment of the present application is more for generating text data from image data to synthesize more data to strengthen model training.

[0125] Step S510: using a neural network to train a cross-modal retrieval network based on the original data and the generated data;

[0126] Specifically, the retrieval network uses deep convolutional neural networks and long short-term memory networks to extract image and text features respectively. The feature vectors are mapped to the embedding space after linear transformation. The cosine similarity of each group of images and texts is calculated for each batch of data. Finally, the retrieval network is trained using the triplet loss function.

[0127] That is to say, the embodiment of the present application includes two stages: image text generation and image text cross-modal retrieval. For the image text generation stage, it includes the use of a generative adversarial network combined with a reinforcement learning strategy for image text generation, which can effectively expand the training data set; for image text cross-modal retrieval, a deep convolutional neural network and a long short-term memory network are used to extract features of image and text descriptions, respectively, and the features are mapped to the embedding subspace through linear transformation, and finally the feature similarity between each group of image and text feature vectors is calculated; finally, a two-stage training scheme is used, and the maximum likelihood criterion and reinforcement learning strategy are used to train the generative adversarial network in two stages, and then the generator is used to generate more image text descriptions about a given image, thereby expanding the original training set, and finally all the data are used to train the retrieval network.

[0128] Step S512: During the test, the image and text description are respectively subjected to feature extraction using the trained cross-modal retrieval network to obtain image features and text features, and then sorted according to the cosine similarity between different samples to obtain the final retrieval result sequence.

[0129] Through the cross-modal retrieval method of the embodiment of the present application, the enhanced embedding space model introduced can, on the one hand, mine the fine-grained matching relationship between images and text descriptions, enhance the discrimination between samples, and reduce the heterogeneity between modalities, and obtain better discriminant features between images and text descriptions in the embedding space, so that the model can pay more attention to the matching relationship between text with low frequency in the training data set and given pictures; on the other hand, the training strategy of generative adversarial network combined with reinforcement learning can effectively expand the data set. Taking a large image-text cross-modal retrieval database as an example, it contains 82,783 images, each image corresponds to 5 text descriptions, and a total of 413,915 text descriptions can be generated. This is a general method that can be combined with many model structures and has high universality. This method can effectively improve the accuracy of image-text cross-modal retrieval.

[0130] In addition, the existing technology focuses on generating cross-modal data from the feature level, which is actually very difficult because features are usually high-dimensional (for example, images are 4096 dimensions). The existing technology usually makes a rough generation based on the assumption that the feature dimensions are independent of each other, so the generated results contain a lot of noise, which will affect the model's association with cross-modal data. The method of the present application is end-to-end image-to-text generation, which is very easy to operate, and can be replaced and optimized with many efficient models in the current image description field. Therefore, the data actually generated by the present application is very real and can be directly used for model training.

[0131] In addition, existing solutions use feature generation between cross-modalities as a constraint to force the semantic gap between cross-modal data to be narrowed, but the improvement in actual results is generally extremely limited, and a large number of hyperparameters are required for debugging. The method of this application uses data-level generation to generate a large amount of additional training data, which can help the model better fit the existing data and learn more discriminative cross-modal association representations. In addition, our model has great potential. When given large-scale training data, it can generate as much data as possible almost without limit, which can be conducive to learning more complex and deeper models.

[0132] In one embodiment, the convolutional neural network in step S502 is a deep neural network with a convolutional structure, which is a deep learning architecture. The deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to an image input therein.

[0133] like Figure 6 As shown, the convolutional neural network (CNN) 600 may include an input layer 610 , a convolutional layer / pooling layer 620 (wherein the pooling layer is optional), and a neural network layer 630 .

[0134] Convolutional layer / pooling layer 620:

[0135] Convolutional Layer:

[0136] like Figure 6 The convolution layer / pooling layer 620 shown may include layers 621-626 as shown in the examples. For example, in one implementation, layer 621 is a convolution layer, layer 622 is a pooling layer, layer 623 is a convolution layer, layer 624 is a pooling layer, layer 625 is a convolution layer, and layer 626 is a pooling layer. In another implementation, layers 621 and 622 are convolution layers, layer 623 is a pooling layer, layer 624 and 625 are convolution layers, and layer 626 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0137] The following will take convolutional layer 621 as an example to introduce the internal working principle of a convolutional layer.

[0138] The convolution layer 621 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator may be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the stride) along the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation.

[0139] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 600 can make correct predictions.

[0140] When the convolutional neural network 600 has multiple convolutional layers, the initial convolutional layer (for example, 621) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 600 increases, the features extracted by the later convolutional layers (for example, 622) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0141] Pooling layer:

[0142] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 6The layers 621-626 illustrated in 620 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0143] Neural network layer 630:

[0144] After being processed by the convolution layer / pooling layer 620, the convolution neural network 600 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 620 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 600 needs to use the neural network layer 630 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 630 may include multiple hidden layers (such as Figure 6 631, 632 to 63n) and output layer 640 shown in the figure, the parameters contained in the multi-layer hidden layer can be pre-trained according to the relevant training data of the specific task type, for example, the task type can include image generation, text generation, sample generation, etc. ...

[0145] After the multiple hidden layers in the neural network layer 630, that is, the last layer of the entire convolutional neural network 600 is the output layer 640, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 600 (such as Figure 6 The propagation from 610 to 640 is forward propagation), and the reverse propagation (such as Figure 6 The propagation from 640 to 610 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 600 and the error between the result output by the convolutional neural network 600 through the output layer and the ideal result.

[0146] It should be noted that if Figure 6The convolutional neural network 600 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0147] Through the cross-modal retrieval method of the embodiment of the present application, an enhanced embedding space is used to perform a more discriminative match between image regions and text description noun phrases. This method can simultaneously enhance the feature expression of both image and text modalities, and alleviate the semantic gap between different modal expressions to a certain extent. Using a generative adversarial network to generate samples that approximate the distribution of real samples can effectively expand the data set, while making the model more focused on words with less frequency in the training data, which helps to alleviate the semantic gap problem between images and texts. Based on the feature expression obtained in the enhanced embedding subspace, including text noun phrases and image region corresponding features, the difference between images and texts can be better distinguished, and the discriminant features related to image and text semantics can be more comprehensively learned, so that images and texts related to semantic content can be better retrieved successfully.

[0148] In order to better implement the above solution of the embodiment of the present application, the present application also provides a cross-modal retrieval device, such as Figure 7 The schematic diagram of the structure of the cross-modal retrieval device provided in the embodiment of the present application is shown (the device may specifically be Figure 1 Execution device 110), the cross-modal retrieval device 70 includes: a data acquisition unit 700, a data input unit 702 and a retrieval unit 704, wherein:

[0149] The data acquisition unit 700 is used to acquire the first modality data and the second modality data to be retrieved;

[0150] The data input unit 702 is used to input the first modality data and the second modality data into the trained cross-modality retrieval network;

[0151] The retrieval unit 704 is used to extract the first feature of the first modal data and the second feature of the second modal data through the cross-modal retrieval network, and output a retrieval result according to the matching degree of the first feature and the second feature; the retrieval result is used to represent the data retrieved from the first modal data and matching the second modal data;

[0152] Among them, the cross-modal retrieval network includes a network trained by a neural network based on original data and generated data in a data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data.

[0153] The units of the cross-modal retrieval device 70 in the embodiment of the present application are used to execute the above-mentioned method embodiments. Figures 1 to 6 The steps performed by the execution device in the cross-modal retrieval method in the embodiment are not repeated here.

[0154] like Figure 8 The schematic diagram of the structure of another embodiment of the cross-modal retrieval device provided by the present application is shown (the device may specifically be Figure 1 The cross-modal retrieval device 80 includes a first training unit 800, a data generating unit 802 and a second training unit 804, wherein:

[0155] The first training unit 800 is used to train the generator and the discriminator in the generative adversarial network based on the original data in the data set;

[0156] The data generating unit 802 is used to generate data based on the original data by using a trained generator; the generator is used to generate data belonging to the first modality from data belonging to the second modality in the original data;

[0157] The second training unit 804 is used to use a neural network to train a cross-modal retrieval network based on the original data and the generated data; wherein the cross-modal retrieval network is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result; the retrieval result is used to represent the data retrieved from the first modality data that matches the second modality data.

[0158] In one implementation, the first training unit 800 may be used to:

[0159] Based on the original data in the data set, the generator in the generative adversarial network is trained using the maximum likelihood criterion, and based on the data generated by the generator, the discriminator in the generative adversarial network is trained using three types of discriminant loss functions;

[0160] The trained generator is trained using a reinforcement learning strategy, while the three-category discriminant loss function is continued to be used to train the discriminator in the generative adversarial network.

[0161] In one embodiment, the training of the generator in the generative adversarial network based on the original data in the data set using the maximum likelihood criterion includes: initializing the training based on the original data in the data set using the maximum likelihood criterion loss function and the Adam optimization algorithm;

[0162] The training of the trained generator through a reinforcement learning strategy includes: introducing a reinforcement learning strategy and a Monte Carlo search technique, and using an Adam optimization algorithm to train the initialized trained generator until convergence.

[0163] In one implementation manner, the second training unit 804 may be used to:

[0164] Based on the original data and the generated data, extract a first feature of the data belonging to the first modality using a long short-term memory network, extract a second feature of the data belonging to the second modality using a deep neural network, and map them to an embedding subspace using a linear transformation;

[0165] The cosine similarity of each pair of data belonging to the first modality and the data belonging to the second modality is calculated in the embedding subspace, and each input batch is trained using a triplet loss function and an Adam optimization algorithm until the model converges to obtain a cross-modal retrieval network.

[0166] The units of the cross-modal retrieval device 80 in the embodiment of the present application are used to execute the above-mentioned method embodiments. Figures 1 to 6 The steps performed by the training device in the cross-modal retrieval method in the embodiment are not repeated here.

[0167] Fig. 9 It is a structural diagram of a cross-modal retrieval device provided in an embodiment of the present application. Fig. 9 The cross-modal retrieval device 900 (which may be a computer device) includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, the processor 902, and the communication interface 903 are connected to each other through the bus 904.

[0168] The memory 901 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 901 may store a program. When the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute each step of the cross-modal retrieval method of the embodiment of the present application.

[0169] The processor 902 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to execute the cross-modal retrieval method of the method embodiment of the present application.

[0170] The processor 902 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the sample generator of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 902. The above-mentioned processor 902 may also be a general-purpose processor, a digital signal processor (DigitalSignal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field ProgrammableGate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 901, and the processor 902 reads the information in the memory 901 and completes the cross-modal retrieval method of the embodiment of the method of the present application in combination with its hardware.

[0171] The communication interface 903 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 900 and other devices or a communication network. For example, training data can be obtained through the communication interface 903 .

[0172] The bus 904 may include a path for transmitting information between various components of the device 900 (eg, the memory 901 , the processor 902 , and the communication interface 903 ).

[0173] The specific implementation of each of the above-mentioned functional devices can refer to the relevant description in the above-mentioned method embodiments, and will not be repeated in the embodiments of the present application.

[0174] Fig.10 is a schematic block diagram of another cross-modal retrieval device in an embodiment of the present application; Fig.10 The cross-modal retrieval device 1000 shown (the device 1000 may specifically be a terminal) includes a memory 1001, a baseband chip 1002, a radio frequency module 1003, a peripheral system 1004, and a sensor 1005. The baseband chip 1002 includes at least one processor 10021, such as a CPU, a clock module 10022, and a power management module 10023; the peripheral system 1004 includes a camera 10041, an audio module 10042, a touch display screen 10043, etc. Further, the sensor 1005 may include a light sensor 10051, an acceleration sensor 10052, a fingerprint sensor 10053, etc.; the modules included in the peripheral system 1004 and the sensor 1005 may be increased or decreased according to actual needs. Any two of the above-mentioned connected modules can be specifically connected through a bus, which can be an industrial standard architecture (English: industry standard architecture, abbreviated as: ISA) bus, a peripheral component interconnect (English: peripheral component interconnect, abbreviated as: PCI) bus or an extended industry standard architecture (English: extended industry standard architecture, abbreviated as: EISA) bus, etc.

[0175] The radio frequency module 1003 may include an antenna and a transceiver (including a modem), the transceiver is used to convert the electromagnetic waves received by the antenna into electric current and finally into digital signals, and accordingly, the transceiver is also used to convert the digital signal to be output by the mobile phone into electric current and then into electromagnetic waves, and finally transmit the electromagnetic waves into free space through the antenna. The radio frequency module 1003 may also include at least one amplifier for amplifying the signal. Generally, wireless transmission can be performed through the radio frequency module 1003, such as Bluetooth (English: Bluetooth) transmission, wireless guarantee (English: Wireless-Fidelity, referred to as: WI-FI) transmission, third-generation mobile communication technology (English: 3rd-Generation, referred to as: 3G) transmission, fourth-generation mobile communication technology (English: the 4th Generation mobile communication, referred to as: 4G) transmission, etc.

[0176] The touch display screen 10043 can be used to display information input by the user or to display information to the user. The touch display screen 10043 may include a touch panel and a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 10021 to determine the type of touch event, and then the processor 10021 provides a corresponding visual output on the display panel according to the type of touch event. The touch panel and the display panel are two independent components to realize the input and output functions of the cross-modal retrieval device 1000, but in some embodiments, the touch panel and the display panel can be integrated to realize the input and output functions of the cross-modal retrieval device 1000.

[0177] Camera 10041 is used to acquire images and can be input into the trained cross-modal retrieval network.

[0178] The audio input module 10042 can specifically be a microphone, which can acquire speech and input it into the trained cross-modal retrieval network.

[0179] Sensor 1005 may include a light sensor 10051, an acceleration sensor 10052, and a fingerprint sensor 10053, wherein the light sensor 10051 is used to obtain the light intensity of the environment, the acceleration sensor 10052 (such as a gyroscope, etc.) can obtain the motion state of the cross-modal retrieval device 1000, and the fingerprint sensor 10053 can input fingerprint information; after the sensor 1005 senses the relevant signal, the signal is quantized into a digital signal and transmitted to the processor 10021 for further processing.

[0180] The memory 1001 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 1001 may optionally include at least one storage device located away from the aforementioned processor 10021. The memory 1001 may specifically include a storage instruction area and a storage data area, wherein the storage instruction area may store programs such as an operating system, a user interface program, and a communication interface program, and the storage data area may store data required by the processor to perform related operations, or data generated by performing related operations.

[0181] The processor 10021 is the control center of the cross-modal retrieval device 1000, and uses various interfaces and lines to connect various parts of the entire mobile phone. By running the program stored in the memory 1001 and calling the data stored in the memory 1001, the cross-modal retrieval device 1000 performs various functions. Optionally, the processor 10021 may include one or more application processors, which mainly process the operating system, user interface, and application programs. In the embodiment of the present application, the processor 10021 reads the information in the memory 1001 and completes the cross-modal retrieval method of the embodiment of the present application in combination with its hardware.

[0182] The radio frequency module 1003 enables the user to realize the communication function of the cross-modal retrieval device 1000 .

[0183] It should be noted that the specific implementation of each of the above functional units can refer to the relevant descriptions in the above method embodiments, and the embodiments of this application will not be repeated. Figures 1 to 6 The embodiments provided in and the corresponding description of their possible embodiments.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0186] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0187] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

[0188] The techniques of the present application may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0189] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cross-modal retrieval method, characterized in that: include: Acquire first modal data and second modal data to be retrieved, wherein the first modal data includes text data and the second modal data includes image data; Inputting the first modality data and the second modality data into a trained cross-modality retrieval network; Extracting a first feature of the first modality data and a second feature of the second modality data through the cross-modality retrieval network, and outputting a retrieval result according to a matching degree between the first feature and the second feature; the retrieval result is used to represent data retrieved from the first modality data and matching the second modality data; The cross-modal retrieval network includes a network trained by a neural network based on original data and generated data in a data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data; The generator and the discriminator in the generative adversarial network are trained simultaneously, including: based on the original data in the data set, using the maximum likelihood criterion loss function and the Adam optimization algorithm to initialize the training of the generator in the generative adversarial network; based on the data generated by the generator, using the three-category discriminant loss function to train the discriminator in the generative adversarial network; introducing the reinforcement learning strategy and the Monte Carlo search technology, using the Adam optimization algorithm to train the generator after the initialization training until convergence, and continuing to use the three-category discriminant loss function to train the discriminator in the generative adversarial network; The neural network is used for training based on the original data and the generated data in the data set, including: based on the original data and the generated data, a long short-term memory network is used to extract a first feature of the data belonging to the first modality, a deep neural network is used to extract a second feature of the data belonging to the second modality, and a linear transformation is used to map them to an embedding subspace; in the embedding subspace, the cosine similarity of each pair of data belonging to the first modality and the data belonging to the second modality is calculated, and a triplet loss function is used to train each input batch, and the Adam optimization algorithm is used for training until the model converges to obtain a cross-modal retrieval network.

2. A cross-modal retrieval method, characterized in that: include: The generator and the discriminator in the generative adversarial network are trained based on the original data in the data set, including: based on the original data in the data set, the maximum likelihood criterion loss function is used and the Adam optimization algorithm is used to initialize the training of the generator in the generative adversarial network; based on the data generated by the generator, the discriminator in the generative adversarial network is trained using a three-category discriminant loss function; the reinforcement learning strategy and the Monte Carlo search technology are introduced, and the Adam optimization algorithm is used to train the generator after the initialization training until convergence, and the three-category discriminant loss function is continued to be used to train the discriminator in the generative adversarial network; Generating data based on the original data by a trained generator; the generator is used to generate data belonging to the first modality from data belonging to the second modality in the original data; A cross-modal retrieval network is obtained by training a neural network based on the original data and the generated data, including: based on the original data and the generated data, using a long short-term memory network to extract a first feature of the data belonging to the first modality, using a deep neural network to extract a second feature of the data belonging to the second modality, and mapping them to an embedding subspace using a linear transformation; calculating the cosine similarity of each pair of data belonging to the first modality and data belonging to the second modality in the embedding subspace, and using a triplet loss function to train each input batch, using an Adam optimization algorithm for training until the model converges, and obtaining a cross-modal retrieval network; wherein the cross-modal retrieval network is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result, the first modality data includes text data, and the second modality data includes image data; the retrieval result is used to characterize the data retrieved from the first modality data that matches the second modality data.

3. A cross-modal retrieval device, characterized in that: include: A data acquisition unit, configured to acquire first modality data and second modality data to be retrieved, wherein the first modality data includes text data and the second modality data includes image data; A data input unit, used for inputting the first modality data and the second modality data into a trained cross-modality retrieval network; a retrieval unit, configured to extract a first feature of the first modal data and a second feature of the second modal data through the cross-modal retrieval network, and output a retrieval result according to a matching degree between the first feature and the second feature; the retrieval result is used to represent data retrieved from the first modal data and matching the second modal data; The cross-modal retrieval network includes a network trained by a neural network based on original data and generated data in a data set; the generated data includes data generated by a generator trained in a generative adversarial network based on the original data; the generator and the discriminator in the generative adversarial network are trained simultaneously to generate data belonging to the first modality from data belonging to the second modality in the original data; The generator and the discriminator in the generative adversarial network are trained simultaneously, including: based on the original data in the data set, using the maximum likelihood criterion loss function and the Adam optimization algorithm to initialize the training of the generator in the generative adversarial network; based on the data generated by the generator, using the three-category discriminant loss function to train the discriminator in the generative adversarial network; introducing the reinforcement learning strategy and the Monte Carlo search technology, using the Adam optimization algorithm to train the generator after the initialization training until convergence, and continuing to use the three-category discriminant loss function to train the discriminator in the generative adversarial network; The neural network is used for training based on the original data and the generated data in the data set, including: based on the original data and the generated data, a long short-term memory network is used to extract a first feature of the data belonging to the first modality, a deep neural network is used to extract a second feature of the data belonging to the second modality, and a linear transformation is used to map them to an embedding subspace; in the embedding subspace, the cosine similarity of each pair of data belonging to the first modality and the data belonging to the second modality is calculated, and a triplet loss function is used to train each input batch, and the Adam optimization algorithm is used for training until the model converges to obtain a cross-modal retrieval network.

4. A cross-modal retrieval device, characterized in that: include: The first training unit is used to train the generator and the discriminator in the generative adversarial network based on the original data in the data set, including: based on the original data in the data set, using the maximum likelihood criterion loss function and the Adam optimization algorithm to initialize the training of the generator in the generative adversarial network; based on the data generated by the generator, using the three-category discriminant loss function to train the discriminator in the generative adversarial network; introducing the reinforcement learning strategy and the Monte Carlo search technology, using the Adam optimization algorithm to train the generator after the initialization training until convergence, and continuing to use the three-category discriminant loss function to train the discriminator in the generative adversarial network; A data generating unit, configured to generate generated data based on the original data by a trained generator; the generator is configured to generate data belonging to the first modality from data belonging to the second modality in the original data; The second training unit is used to use a neural network to train a cross-modal retrieval network based on the original data and the generated data, including: based on the original data and the generated data, using a long short-term memory network to extract a first feature of the data belonging to the first modality, using a deep neural network to extract a second feature of the data belonging to the second modality, and mapping them to an embedding subspace using a linear transformation; calculating the cosine similarity of each pair of data belonging to the first modality and data belonging to the second modality in the embedding subspace, and using a triplet loss function to train each input batch, using an Adam optimization algorithm for training until the model converges to obtain a cross-modal retrieval network; wherein the cross-modal retrieval network is used to extract features from the input first modality data and the second modality data to be retrieved, and output a retrieval result, the first modality data includes text data, and the second modality data includes image data; the retrieval result is used to characterize the data retrieved from the first modality data that matches the second modality data.

5. A cross-modal retrieval device, characterized in that: The system comprises a processor, wherein the processor is configured to call a stored cross-modal retrieval program to execute the method according to claim 1 or 2.

6. A computer-readable storage medium, characterized in that: The computer storage medium stores program instructions, which, when executed by a processor, cause the processor to perform the method of claim 1 or 2.

Images