Coding device and data processing method and device

Abstract

The embodiment of the invention provides an encoding device, a data processing method and a data processing device.The encoding device comprises an encoding layer and at least one joint encoding unit, the encoding layer encodes a received first modal initial feature vector and a received second modal initial feature vector, and the at least one joint encoding unit is used for encoding the first modal initial feature vector and the second modal initial feature vector. The at least one joint coding unit generates a first modal feature vector and a second modal feature vector, the at least one joint coding unit performs joint coding on the first modal feature vector and the second modal feature vector, and the at least one joint coding unit comprises a coding module and a modal input switching module. The modal input switching module is configured to process the first modal feature vector and the second modal feature vector to obtain a first modal switching coding vector and a second modal switching coding vector; the coding module is configured to process the received first modal switching coding vector and the received second modal switching coding vector to generate a first target modal fusion vector and a second target modal fusion vector.

Description

[0001] This application is a divisional application with application number 202210789237.7, application date July 6, 2022, and invention name “Encoding device, data processing method and device”. Technical Field

[0002] The embodiments of the present specification relate to the field of computer technology, and in particular to a coding device. Background Art

[0003] With the development of computer technology and artificial intelligence, people's demand for machine perception and reasoning is increasing day by day, especially the understanding and matching of image and text data. Image-text matching and image-text prediction have attracted the interest of many researchers at home and abroad. As a basic and key task in the field of information retrieval and multimedia computing, it can not only effectively break the semantic gap and distribution barriers between vision and language, but also promote the development of upper-level applications (such as cross-modal retrieval, image annotation, and visual question answering). However, deep learning models with fixed structures are usually used at present, which can only achieve excellent performance in specific tasks, and for more complex tasks, it will affect the use effect of the model. Summary of the invention

[0004] In view of this, the present specification provides an encoding device. One or more embodiments of the present specification simultaneously involve two data processing methods and devices, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a coding device is provided, comprising: a multi-layer joint coding unit, each layer of the joint coding unit comprising a coding module and a modal input switching module, The modal input switching module is configured as follows: Select one of the first modal feature vector input to the current modal input switching module and the first modal switching encoding vector output by the modal input switching module of the previous layer as the first modal switching encoding vector output by the current modal input switching module, and Select one of the second modal feature vector input to the current modal input switching module and the second modal switching coding vector output by the modal input switching module of the previous layer as the second modal switching coding vector output by the current modal input switching module; The encoding module is configured as follows: Generate a first target modality fusion vector by learning and encoding the received first modality switching coding vector and fusing the received second modality switching coding vector, and The second target modality fusion vector is generated by learning and encoding the received second modality switching coding vector and fusing it with the received first modality switching coding vector.

[0006] According to a second aspect of an embodiment of this specification, a first data processing method is provided, including: Receiving a multimodal data matching request, inputting the first modality initial data and the second modality initial data carried in the multimodal data matching request into an encoding device, and obtaining a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device in the above embodiment; A pre-trained data matching network is determined based on the multimodal data matching request, and the first target modality fusion vector and the second target modality fusion vector are input into the data matching network to generate a data matching result.

[0007] According to a third aspect of the embodiments of this specification, a second data processing method is provided, including: Receiving a multimodal data prediction request, inputting the first modality initial data and the second modality initial data carried in the multimodal data prediction request into an encoding device, and obtaining a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device in the above embodiment, and the first modality initial data or the second modality initial data carries at least one mask data; A pre-trained data prediction network is determined based on the multimodal data prediction request, and the first target modality fusion vector and the second target modality fusion vector are input into the data prediction network to generate a data prediction result.

[0008] According to a fourth aspect of an embodiment of this specification, a first data processing device is provided, including: The first fusion vector obtaining module is configured to: receive a multimodal data matching request, input the first modality initial data and the second modality initial data carried in the multimodal data matching request into an encoding device, and obtain a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device in the above embodiment; The data matching module is configured to: determine a pre-trained data matching network based on the multimodal data matching request, input the first target modality fusion vector and the second target modality fusion vector into the data matching network, and generate a data matching result.

[0009] According to a fifth aspect of the embodiments of this specification, a second data processing device is provided, including: The second fusion vector obtaining module is configured to: receive a multimodal data prediction request, input the first modality initial data and the second modality initial data carried in the multimodal data prediction request into an encoding device, and obtain a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device in the above embodiment, and the first modality initial data or the second modality initial data carries at least one mask data; The data prediction module is configured to: determine a pre-trained data prediction network based on the multimodal data prediction request, input the first target modality fusion vector and the second target modality fusion vector into the data prediction network, and generate a data prediction result.

[0010] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above two data processing methods are implemented.

[0011] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above two data processing methods are implemented.

[0012] According to an eighth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above two data processing methods.

[0013] An embodiment of the present specification provides a coding device, which includes a coding layer and at least one joint coding unit, wherein the coding layer is trained to encode a received first modal initial feature vector and a second modal initial feature vector to generate a first modal feature vector and a second modal feature vector, and the at least one joint coding unit is trained to jointly encode the first modal feature vector and the second modal feature vector, and the at least one joint coding unit includes a coding module and a modal input switching module, and the modal input switching module is configured to process the first modal feature vector and the second modal feature vector to obtain a first modal switching coding vector and a second modal switching coding vector; the coding module is configured to process the received first modal switching coding vector and the second modal switching coding vector to generate a first target modal fusion vector and a second target modal fusion vector.

[0014] Specifically, the first modal initial feature vector and the second modal initial feature vector are encoded through the encoding layer and at least one joint encoding unit in the encoding device. Not only can the modal encoding vector be switched through the modal input switching module in the joint encoding unit to obtain the first modal switching encoding vector and the second modal switching encoding vector, but the modalities are then fused through the encoding module to obtain the first target modal fusion vector and the second target modal fusion vector, so that fusion vector representations can be obtained between different modalities, making the interaction mode between modalities more flexible, so as to enhance the fusion effect of the encoding device on the fusion of different modalities and enhance the subsequent use effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural diagram comparing the interaction mode between the existing model structure and the Switch-BERT model structure (encoding device) provided in one embodiment of this specification; Figure 2 is a schematic diagram of the structure of an encoding device provided by an embodiment of this specification; Figure 3 An embodiment of this specification provides a schematic diagram of selecting an attention mechanism in an encoding device; Figure 4 An embodiment of the present specification provides a schematic diagram of selecting a modal input feature vector in an encoding device; Figure 5 A schematic diagram of generating an initial feature vector by a visual embedding layer and a text embedding layer in an encoding device provided by an embodiment of the present specification; Figure 6 is a flow chart of a first data processing method provided by an embodiment of this specification; Figure 7 is a flow chart of a second data processing method provided by an embodiment of this specification; Figure 8 It is a schematic diagram of the training process of an image-text matching network, a covered language modeling network, and a covered region classification network in a data processing method provided in one embodiment of this specification; Fig. 9 It is a schematic diagram of the processing process of model application in the data processing method provided in one embodiment of this specification; Fig.10 is a structural schematic diagram of a first data processing device provided by an embodiment of this specification; Fig.11 is a structural schematic diagram of a second data processing device provided by an embodiment of this specification; Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0016] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0017] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0018] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0019] First, the terms involved in one or more embodiments of this specification are explained.

[0020] Multimodal learning: The development of mobile Internet is accompanied by the generation of hundreds of millions of texts, pictures, videos and voice data every day. Each source or form of information can be called a modality. Multimodal machine learning makes full use of the complementarity and redundancy between multi-source heterogeneous data of different modalities to achieve cross-modal content understanding and cognition.

[0021] Multimodal fusion: It is a very critical research point in multimodal research. It integrates information extracted from different modalities into a stable multimodal representation.

[0022] The development of mobile Internet is accompanied by the generation of hundreds of millions of texts, pictures, videos and voice data every day. Each source or form of information can be called a modality. Multimodal machine learning makes full use of the complementarity and redundancy between multi-source heterogeneous data of different modalities to achieve cross-modal content understanding and cognition. At present, the fusion of data forms from single to multiple modalities is becoming more and more important. Among them, the ability to model the interaction between modalities within the modality is the basis of multimodal machine learning. At present, a deep learning model with a fixed structure is used, which affects the model effect. The encoding device provided in the embodiment of this specification uses a dynamic network to learn the interaction between multimodalities. Switch-BERT for joint multimodal data learning solves this problem. Switch-BERT extends the BERT architecture by introducing learnable layer-by-layer and cross-layer interactions, and can learn to optimize the interaction mode between modalities end-to-end. One feature of this model is that it can adaptively select outputs from different layers, thereby alleviating the problem of modal mismatch.

[0023] See also Figure 1 , Figure 1 A structural diagram showing a comparison of the interaction between the existing model structure provided in the embodiments of this specification and the Switch-BERT model structure (encoding device) is shown.

[0024] Figure 1 The upper part is the existing model structure, and the lower part is the Switch-BERT model structure. When fusing multimodal data, such as vision and text, in the existing model structure, the visual and text data in the L layer are based on a fixed interaction method for cross-modal modeling; while in the Switch-BERT model structure, Figure 1 The lower part of the figure indicates that the visual and textual data can also go through a selection process and then perform cross-modal modeling to achieve a learnable interaction method, in which the visual data of the L-1 layer and the L layer are selected to be input into the model, and the textual data of the L-1 layer and the L layer are selected to be input into the model to complete the learning interaction method. Switch-BERT can use a dynamic structure to adaptively select the interaction method between inputs of different layers and learning modalities.

[0025] In this specification, a coding device is provided. This specification also involves two data processing methods and devices, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0026] It should be noted that the encoding device provided in the embodiments of this specification can be understood as an adjusted Switch-BERT model. By improving the structure of the original Switch-BERT model, the encoding device can select inputs of different layers for feature vectors between different modalities, and select different interaction methods between learning modalities to fuse modal vectors.

[0027] Figure 2 A schematic diagram of the structure of an encoding device provided according to an embodiment of the present specification is shown.

[0028] Figure 2 The system includes a coding device 200, a coding layer 202, and at least one joint coding unit 204, wherein the joint coding unit 204 includes a modal input switching module 2042 and a coding module 2044.

[0029] It should be noted that the coding layer 202 can be understood as a layer in the Switch-BERT model. The number of joint coding units 204 and the total number of coding layers 202 are the number of layers in the entire Switch-BERT model. The embodiment of this specification does not impose any limitation on the number of layers in the Switch-BERT model. For example, if there are 12 layers in the Switch-BERT model, then the joint coding unit 204 has 11 layers, and each layer has a modal input switching module 2042 and a coding module 2044. It should be noted that the modal input switching module of each joint coding unit in the Switch-BERT model is not limited in this embodiment, but each joint coding unit in this embodiment is configured with a modal input switching module 2042 and a coding module 2044.

[0030] In a specific implementation, the encoding device 200 includes an encoding layer 202 and at least one joint encoding unit 204, wherein the encoding layer 202 is trained to encode the received first modal initial feature vector and the second modal initial feature vector to generate a first modal feature vector and a second modal feature vector, and the at least one joint encoding unit 204 is trained to jointly encode the first modal feature vector and the second modal feature vector, and the at least one joint encoding unit includes an encoding module and a modal input switching module; The modal input switching module 2042 is configured to process the first modal feature vector and the second modal feature vector to obtain a first modal switching code vector and a second modal switching code vector; The encoding module 2044 is configured to process the received first modality switching encoding vector and second modality switching encoding vector to generate a first target modality fusion vector and a second target modality fusion vector.

[0031] Among them, the first modal initial feature vector can be understood as the feature data for vectorizing the initial data, such as the image initial feature vector; the second modal initial feature vector can be understood as the feature data for vectorizing the initial data, such as the text initial feature vector; the first modal feature vector can be understood as the feature vector after the encoding layer processes the modal data, such as the image feature vector, and the second modal feature vector can be understood as the feature vector after the encoding layer processes the modal data, such as the text feature vector; the first modal switching coding vector can be understood as the characteristic vector after selecting the modal feature vectors of different layers in the input coding module, such as the image switching coding vector, and the second modal switching coding vector can be understood as the characteristic vector after selecting the modal feature vectors of different layers in the input coding module, such as the text switching coding vector; the first target modal fusion vector can be understood as the vector that integrates the features of other modal data after learning and encoding the switching coding vector based on the self-attention mechanism, such as the image fusion vector, and the second target modal fusion vector can be understood as the vector that integrates the features of other modal data after learning and encoding the switching coding vector based on the self-attention mechanism, such as the text fusion vector.

[0032] It should be noted that the encoding device provided in the embodiments of this specification can be understood as a training process after a structural change to the Switch-BERT model. The input of the training process is a feature vector of two modal data, and the output of the training process is a feature vector after the fusion of the two modal features. By training the encoding device, the output data of the encoding device becomes more and more accurate, which facilitates the subsequent application of the model in application fields such as image-text matching and image-text prediction to achieve better and better application effects.

[0033] The following description of the model training process uses image data as the first modality data and text data as the second modality data as an example, but the embodiment of this specification does not specifically limit the specific modality type and may also include voice modality data.

[0034] In actual applications, image modality data and text modality data are first input into the encoding layer for encoding, and then the encoded image feature data and text feature data are input into at least one joint encoding unit to continue learning and fusing the feature vectors between the modalities to obtain a vector fused from the image and text feature data, and then the fused vector is further applied.

[0035] In addition, when it is determined that the number of joint coding units is two or more, the two or more joint coding units are used as a joint coding unit group; specifically, the number of the at least one joint coding unit is two or more and constitutes a joint coding unit group, and the joint coding unit group is configured to jointly encode the first modal feature vector and the second modal feature vector to generate a first target modal fusion vector and a second target modal fusion vector; Output information of a first joint coding unit in any two adjacent joint coding units in the joint coding unit group is input to an input end of a second joint coding unit.

[0036] In practical applications, the joint coding unit group uses the output of the coding layer of the first layer as the input of the joint coding unit group, and the output of the joint coding unit group is the final output data of the model, wherein the output information of the first joint coding unit in any two adjacent joint coding units between the joint coding unit groups is input to the input end of the second joint coding unit. For example, the output data of the joint coding unit of the L layer is the input data of the joint coding unit of the L+1 layer, so that the joint coding unit group has a repeated structure of multiple joint coding units, which will not be elaborated here.

[0037] Specifically, the encoding layer 202 is further configured to determine a first modal initial feature global vector of the first modal initial feature vector and a second modal initial feature global vector of the second modal initial feature vector; Calculating an initial feature matching value between the first modality initial feature global vector and the second modality initial feature global vector, and determining an initial feature classification probability value of the initial feature matching value based on a pre-trained initial feature selection classifier; Determine a simulated initial feature classification probability value based on the initial feature classification probability value, and select an initial attention mechanism according to the simulated initial feature classification probability value, wherein the initial attention mechanism includes one of a self-attention mechanism, a first cross-self-attention mechanism, a second cross-self-attention mechanism, and a joint self-attention mechanism; Based on the initial attention mechanism, self-attention calculation is performed on the first modal initial feature vector and the second modal initial feature vector respectively to generate a first modal feature vector and a second modal feature vector.

[0038] The initial feature matching value can be understood as the matching degree between the two modal feature vectors of the current input, for example, the matching degree between an image and a text.

[0039] In practical applications, the encoding layer can calculate the global vector of the initial image features (which can be obtained by weighted summation) based on the input image initial feature vector, and calculate the global vector of the initial text features (which can be obtained by weighted summation) based on the input text initial feature vector. For details, please refer to the following formula (1): Formula (1) in, It can be understood as the initial feature variable of the image. It can be understood as the initial feature variable of the text. It can be understood as the initial feature global vector of the image. It can be understood as the global vector of initial text features.

[0040] Furthermore, after determining the image initial feature global vector and the text initial feature global vector, the encoding layer can also calculate the initial feature matching value between the image initial feature global vector and the text initial feature global vector. For details, refer to the following formula (2): Formula (2) in, It can be understood as the matching degree between the initial feature global vector of the image and the initial feature global vector of the text, and the cosine similarity between the two global vectors is calculated using the dot multiplication algorithm.

[0041] Furthermore, the encoding layer can be applied to a pre-trained initial feature selection classifier to determine the initial feature matching value for the initial feature classification probability value. For details, refer to the following formula (3): Formula (3) in, It can be understood as matching The probability value in the interaction mode of different mapping spaces; MLP (Multilayer Perceptron) can be understood as a multi-layer perceptron. In addition to the input and output layers, the multi-layer perceptron can have multiple hidden layers in the middle, and the layers of the multi-layer perceptron are fully connected.

[0042] In the encoding layer, a pre-trained multi-layer perceptron is used to map out the classifications corresponding to different matching degrees. It should be noted that the specific number of classifications is a hyperparameter, which can be pre-set in different application scenarios. In the embodiment of the present specification, since images and texts are two modalities, the attention mechanism of the self-attention mechanism layer in the encoding layer can have four combination types, including self-attention mechanism, first cross-self-attention mechanism, second cross-self-attention mechanism, and joint self-attention mechanism; wherein the self-attention mechanism can be understood as a self-attention-self-attention mechanism, for example, image data only pays attention to image data, and text data only pays attention to text data; the first cross-self-attention mechanism can be understood as a self-attention-cross-self-attention mechanism, for example, image data pays attention to image data, and text data also pays attention to image data; the second cross-self-attention mechanism can be understood as a cross-self-attention-self-attention mechanism, for example, image data pays attention to text data, and text data also pays attention to text data; the joint self-attention mechanism can be understood as a joint attention mechanism, for example, image data pays attention to image data and text data, and text data pays attention to image data and text data; therefore, It can be understood as the probability values ​​of selecting different attention mechanisms. For example, the probability of selecting the self-attention mechanism is 0.3, the probability of the first cross-self-attention mechanism is 0.1, the probability of the second cross-self-attention mechanism is 0.2, and the probability of the joint self-attention mechanism is 0.4.

[0043] Then, how to select the attention mechanism during the training process is the main calculation work of the encoding layer. In order to select diversified attention mechanisms for learning in the encoding layer for different training samples during the training process, the current initial feature matching value needs to be disturbed and sampling noise added during the training process to change the selection probability value of each attention mechanism. For details, please refer to the following formula (4): Formula (4) in, It can be understood as mapping different types of attention mechanisms according to the initial feature matching value, which can be expressed as M 1 、M 2 、M 3 、M 4 , It can be understood as sampling noise. It can be understood as a probability distribution mediation factor.

[0044] After determining the probability value of each attention mechanism interaction space according to the initial feature matching value, the simulated initial feature classification probability value can be determined by formula (4), for example, the probability of selecting the self-attention mechanism is 0.2, the probability of the first cross-self-attention mechanism is 0.4, the probability of the second cross-self-attention mechanism is 0.1, and the probability of the joint self-attention mechanism is 0.3.

[0045] Furthermore, the final initial attention mechanism can be selected according to the simulated initial feature classification probability value. The initial attention mechanism is one of the self-attention mechanism, the first cross self-attention mechanism, the second cross self-attention mechanism, and the joint self-attention mechanism. The selection method can refer to formula (5) and formula (6): Formula (5) Formula (6) The input parameters can be obtained by referring to formula (7): Formula (7) It should be noted that during the model training process, formula (5) can be selected to determine the initial attention mechanism, and during the model application process, formula (6) can be selected to determine the initial attention mechanism.

[0046] Furthermore, after determining the initial attention mechanism, multi-head self-attention calculation is performed on the image initial feature vector and the text initial feature vector to generate an image feature vector and a text feature vector.

[0047] See also Figure 3 , Figure 3 A schematic diagram of selecting the attention mechanism in the encoding device provided in an embodiment of this specification is shown.

[0048] Figure 3 The attention mechanism switching module in the method selects the attention mechanism according to the input modality representation. After the self-attention-cross self-attention mechanism is selected according to the method described in the above embodiment, the image initial feature vector and the text initial feature vector are summed and normalized based on the self-attention-cross self-attention mechanism, and then input into the feedforward network for feedforward calculation, and then summed and normalized again, and then output.

[0049] In the encoding device provided in the embodiment of this specification, the selection of the self-attention mechanism in the encoding layer makes the learning method between different modalities more conducive to the fusion of features between modalities.

[0050] The modal input switching module 2042 is further configured to determine a first modal feature global vector of the first modal feature vector and a second modal feature global vector of the second modal feature vector; Calculating a feature matching value between the first modal feature global vector and the second modal feature global vector, and determining an initial classification probability value of the feature matching value based on a pre-trained feature selection classifier; Determine a simulated classification probability value based on the initial classification probability value, and select a target modal feature vector type according to the simulated classification probability value, wherein the target modal feature vector type includes one of a first modal feature vector and a second modal feature vector of a current modal input switching module, and one of a first modal switching coding vector and a second modal switching coding vector of a previous layer coding module connected to the current modal input switching module; A first modality switching coding vector and a second modality switching coding vector are determined based on the target modality feature vector type.

[0051] Among them, the feature matching value can be understood as the matching degree between the global vectors of the two modalities, which can reflect the matching degree between the image and the text.

[0052] In practical applications, the modal input switching module is used to select the two modal data input to the encoding module, and the selected objects are the image feature vector and text feature vector of the current encoding unit layer input to the modal input switching module, and the output data of the encoding module in the previous encoding unit layer, namely the image switching encoding vector and the text switching encoding vector. One of the four vectors is selected from each group and input to the encoding module in the joint encoding unit.

[0053] The specific selection mechanism can refer to the selection method for the self-attention mechanism in the above embodiment. Specifically, refer to the above formula (1) to calculate the image feature global vector and the text feature global vector, and then refer to formula (2) to calculate the feature matching value between the image feature global vector and the text feature global vector, and then refer to formula (3) to determine the initial classification probability value of the feature matching value, and then determine the simulation classification probability value based on the initial classification probability value, which can refer to the above formula (4). The pre-trained feature selection classifier has the same functionality as the pre-trained initial feature selection classifier in the above embodiment, and only the specific content parameters are different. The mapped four modal selection methods may include four types, namely, selecting the image feature vector and text feature vector of the current modal input switching module, the image switching encoding vector of the previous layer encoding module connected to the current modal input switching module and the text feature vector of the current modal input switching module, the image feature vector of the current modal input switching module and the text switching encoding vector of the previous layer encoding module connected to the current modal input switching module, and the image switching encoding vector and text switching encoding vector of the previous layer encoding module connected to the current modal input switching module.

[0054] Furthermore, the target modal feature vector type is selected according to the simulated classification probability value. For details, please refer to the following formula (8) and formula (9): Formula (8) Formula (9) The input parameters can refer to formula (10): Formula (10) It should be noted that, during the model training process, formula (8) can be selected to determine the target modal feature vector type, and during the model application process, formula (10) can be selected to determine the target modal feature vector type.

[0055] Furthermore, a first modal switching coding vector and a second modal switching coding vector are determined based on the target modal feature vector type.

[0056] See also Figure 4 , Figure 4 A schematic diagram of selecting a modal input feature vector in an encoding device provided in an embodiment of this specification is shown.

[0057] Figure 4 The output of the Switch-BERT layer at the bottom is the L-layer image representation and the L-layer text representation, and the L-1-layer image representation and the L-1-layer text representation are the output data of the modal input switching module of the previous layer. Then, after the modal input switching module receives the L-layer image representation and the L-layer text representation, it continues to calculate the global vectors of the L-layer image representation and the L-layer text representation respectively, and then determines the feature matching value between the L-layer image global representation and the L-layer text global representation, and then determines the probability value of each selection method, such as Figure 4 As shown in , the determined modality input is combined into L-1 layer image representation and L layer text representation, which are input to the next Switch-BERT layer.

[0058] The encoding device provided in the embodiments of this specification can achieve better modal feature fusion by selecting modal feature vectors in different layers to match vectors between different modalities.

[0059] Furthermore, the encoding module 2044 is further configured to determine a first modal switching encoding global vector of the first modal switching encoding vector and a second modal switching encoding global vector of the second modal switching encoding vector; Calculating a modality matching value between the first modality switching encoding global vector and the second modality switching encoding global vector, and determining an initial matching probability value of the modality matching value based on a pre-trained mode selection classifier; Determine a simulated matching probability value based on the initial matching probability value, and select a target attention mechanism according to the simulated matching probability value, wherein the target attention mechanism includes one of a self-attention mechanism, a first cross-self-attention mechanism, a second cross-self-attention mechanism, and a joint self-attention mechanism; Based on the target attention mechanism, self-attention calculation is performed on the first modal switching encoding vector and the second modal switching encoding vector respectively to generate a first target modal fusion vector and a second target modal fusion vector.

[0060] In practical applications, the selection of the attention mechanism in the encoding module can refer to the above-mentioned process of selecting the initial attention mechanism for the encoding layer 202. The difference from the selection process of the initial attention mechanism of the encoding layer 202 is that the input parameters are different, that is, the encoding module in this embodiment determines the global vector corresponding to the first mode switching coding vector and the second mode switching coding vector, which is different from the meaning of the global vector representation in the above-mentioned encoding layer, and the specific execution steps are not described in detail here.

[0061] The encoding device provided in the embodiment of this specification can also select the attention mechanism in the encoding module, which can evaluate the degree of semantic matching between modalities in each layer of the network and define the interaction mode space of the modalities to improve the modal fusion effect of the model.

[0062] In addition, the encoding device also includes a preprocessing module, The preprocessing module is configured to receive the multimodal initial data and determine the data type of the multimodal initial data; Determining the position features of the multimodal data, and extracting the data features of the multimodal initial data; A multimodal initial feature vector is generated based on the data type, the location feature, and the data feature, wherein the multimodal initial feature vector includes a first modality initial feature vector and a second modality initial feature vector.

[0063] Among them, the data type can be understood as the type of modal data, for example, the data type of image data can be identified as type 0, and the data type of text data can be identified as type 1; the position feature can be understood as the position information of the data feature, for example, the position of each area in the image data is the area position, and the sequential position of the text in the text data is the word position; the data feature can be understood as the feature of the corresponding data in each position feature, for example, the feature of the data at each area position in the image data is the area feature, and the feature of the data at each word position in the text data is the word feature.

[0064] In practical applications, the preprocessing module in the encoding device can determine the data type of the received multimodal initial data, then determine the multimodal position features, and then extract the corresponding data features, and then generate the multimodal initial feature vector by the embedding layer. For details, please refer to Figure 5 , Figure 5 A schematic diagram showing the generation of initial feature vectors by the visual embedding layer and the text embedding layer in the encoding device according to an embodiment of the present specification.

[0065] Figure 5 (a) is a schematic diagram of the processing of the visual embedding layer. Figure 5 (b) is a processing diagram of the text embedding layer; Specifically, Figure 5 The visual embedding layer in (a) first determines the modality as a visual modality from the multimodal initial data, that is, inputs the representation of the visual type into the visual embedding layer, then determines the regional position and regional features of each region in the image from the multimodal initial data, and then outputs the initial feature vector (visual representation) of the image through the visual embedding layer. Figure 5 The text embedding layer in (b) determines that the modality is a text-type modality from the multimodal initial data, that is, determines the representation of the text type and inputs it into the text embedding layer, then determines the word position and word features of the text initial data, then inputs the text type, word features and word position into the text embedding layer, and finally outputs the text initial feature vector (text representation). It should be noted that the specific implementation process of the embedding layer is not specifically limited in the embodiments of this specification.

[0066] The encoding device provided in the embodiments of this specification generates initial modal feature vectors between different modalities, thereby facilitating subsequent input to the encoding layer and at least one joint encoding unit to perform feature fusion between modalities.

[0067] Another embodiment of this specification provides a first data processing method, see Figure 6 , Figure 6 A flow chart of a first data processing method provided in an embodiment of the present specification is shown, which specifically includes the following steps.

[0068] Step 602: Receive a multimodal data matching request, input the first modal initial data and the second modal initial data carried in the multimodal data matching request into a preprocessing module in an encoding device to obtain a first multimodal initial feature vector and a second multimodal initial feature vector, wherein the encoding device is the encoding device in the above embodiment.

[0069] Among them, a multimodal data matching request can be understood as a data matching request between different modalities, such as a matching request for images and texts. In one embodiment of this specification, the application scenario of image and text matching is taken as an example to illustrate the data processing method, but it does not limit the specific multimodal data type.

[0070] In actual applications, after receiving the image-text matching request, the server first inputs the image data and text data into the preprocessing module in the encoding device respectively, and extracts features of the image data and text data based on the visual embedding layer and the text embedding layer in the preprocessing module to generate visual representation and text representation; it should be noted that the encoding device and the preprocessing module in the encoding device mentioned in this embodiment can refer to the data processing process of the encoding device and the preprocessing module described in the above embodiments, and this is not specifically limited in this embodiment.

[0071] Step 604: Input the first multimodal initial feature vector and the second multimodal initial feature vector into the encoding layer and at least one joint encoding unit in the encoding device to perform multimodal feature fusion to obtain a first target modality fusion vector and a second target modality fusion vector.

[0072] Furthermore, the server may also input the visual representation and the text representation into the encoding layer in the encoding device for encoding, and then input the output data in the encoding layer into at least one joint encoding unit, wherein the joint encoding unit includes a modal input switching module and an encoding module. It should be noted that both the encoding layer and the encoding module need to select an attention mechanism suitable for the current image, text and table data fusion learning based on the similarity between the image and text representations. In addition, the modal input switching module selects between image representations and text representations at different layers for the input data input into the encoding module. The specific process of determining the attention mechanism and the selection process of the modal input switching module can refer to the description of the above embodiment, and no further details will be given here.

[0073] Step 606: Determine a pre-trained data matching network based on the multimodal data matching request, input the first target modality fusion vector and the second target modality fusion vector into the data matching network, and generate a data matching result.

[0074] Among them, the pre-trained data matching network can be understood as the initial network model of image-text matching, which has the function of image-text matching. However, due to the low performance of the initial network model of image-text matching and the low training procedure of image-text matching, the effect of the pre-trained data matching network is poor. Therefore, the data processing method provided in the embodiment of this specification first fuses the visual features and text features through an encoding device, and then uses the fused visual features and text features to achieve the image-text matching task using a pre-trained data matching network, thereby improving the accuracy of the matching results of image-text matching.

[0075] Furthermore, the server can also determine a pre-trained data matching network based on the image-text matching request, input the image fusion vector and the text fusion vector fused by the encoding module into the data matching network, and then generate a data matching result, and the implementation effect of the data matching result is better than the implementation effect of using the pre-trained data matching network alone.

[0076] In addition, another application scenario that can be provided in another embodiment of the present specification is a picture retrieval scenario; specifically, before receiving the multimodal data matching request, it also includes: Receive an image data retrieval request, and determine at least two texts to be matched based on the image to be retrieved carried in the image data retrieval request; Generate a multimodal data matching request based on the image to be retrieved and the at least two texts to be matched; Accordingly, after generating the data matching result, the step further includes: Constructing a matching text corresponding to the image to be retrieved from the at least two texts to be matched based on the data matching result; or Before receiving the multimodal data matching request, the method further includes: Receive a text data search request, and determine at least two images to be matched based on the text to be searched carried in the text data search request; Generate a multimodal data matching request based on the text to be retrieved and the at least two pictures to be matched; Accordingly, after generating the data matching result, the step further includes: A matching picture corresponding to the text to be retrieved is constructed from the at least two pictures to be matched based on the data matching result.

[0077] It should be noted that the image and text retrieval scenario includes determining whether at least two retrieved text data match the target image, and determining whether at least two retrieved image data match the target text. Therefore, in actual application, the example of the server receiving a picture data retrieval request is used for explanation.

[0078] Specifically, after receiving the image data retrieval request, the server can filter out text data that may match the image to be retrieved in the retrieval engine based on the image to be retrieved. For example, if there are 50 results retrieved in the retrieval engine, 10 text data can be filtered out based on the precise sorting in the retrieval engine; then the image to be retrieved and the 10 text data are respectively input into the encoding device to fuse the image and text features, and the fused vector is input into the pre-trained image-text matching network to obtain the data matching result, and the matching text corresponding to the image to be retrieved is determined from the 10 text data based on the data matching result, thereby completing the image-text retrieval task.

[0079] In addition, the processing process of text data retrieval is consistent with the processing process of image data retrieval, so it will not be described in detail here.

[0080] In summary, the data processing method provided in the embodiments of this specification utilizes an encoding device to perform modal fusion on image feature data and text feature data, and when input into a pre-trained image-text matching network, it can more accurately achieve the image-text matching results of the model.

[0081] Another embodiment of this specification provides a second data processing method, see Figure 7 , Figure 7 A flow chart of a second data processing method provided in an embodiment of the present specification is shown, which specifically includes the following steps.

[0082] Step 702: Receive a multimodal data prediction request, input the first modality initial data and the second modality initial data carried in the multimodal data prediction request into a preprocessing module in an encoding device, and obtain a first modality initial feature vector and a second modality initial feature vector, wherein the encoding device is the encoding device in the above-mentioned embodiment, and the first modality initial data or the second modality initial data carries at least one mask data.

[0083] Among them, the multimodal data prediction request can be understood as a request between different modalities to predict the masked modal data. For example, if the features of the input image data are complete, but part of the text data corresponding to the image data is masked, the masked text data can be predicted; if the features of the input text data are complete, but part of the image data corresponding to the text data is masked, the image data of the masked area can be predicted. It should be noted that there is no limitation on the data content of the masked part in the multimodal data prediction request. Different network models can be trained according to different application scenarios to meet the corresponding application requirements. The following is an explanation using the text data prediction request as an example.

[0084] In actual applications, the server inputs the image initial data and text initial data carried in the text data prediction request into the preprocessing module in the encoding device for processing, wherein the text initial data is randomly masked. For example, if the text initial data is "I love China", after random masking, the word features of the word "love" can be masked and then participate in subsequent text data processing. The masked words are predicted through the above encoding device and the masked language modeling network.

[0085] Step 704: Input the first modal initial feature vector and the second modal initial feature vector into the encoding layer and at least one joint encoding unit in the encoding device to perform multimodal feature fusion to obtain a first target modal fusion vector and a second target modal fusion vector.

[0086] Furthermore, the server inputs the image initial feature vector and the text initial characteristic vector output by the preprocessing module in the encoding device into the encoding device and at least one joint encoding unit for multimodal fusion to obtain a fused image fusion vector and a text fusion vector, wherein the encoding device and at least one joint encoding unit can refer to the description in the above embodiments and will not be elaborated herein.

[0087] Step 706: Determine a pre-trained data prediction network based on the multimodal data prediction request, input the first target modality fusion vector and the second target modality fusion vector into the data prediction network, and generate a data prediction result.

[0088] Among them, the pre-trained data prediction network can be understood as a pre-trained covered area classification network or a covered language modeling network. In the application of predicting the covered area of ​​the image, the pre-trained covered area classification network can be used; in the application of predicting the covered content of the text, the pre-trained covered language modeling network can be used.

[0089] In practical applications, the fused image fusion vector and text fusion vector are processed according to different data prediction networks to generate corresponding data prediction results.

[0090] In summary, the data processing method provided in the embodiments of this specification utilizes an encoding device to perform modal fusion on image feature data and text feature data, and when input into a pre-trained data prediction network, it can more accurately achieve the prediction results of the model's image or text.

[0091] refer to Figure 8 , Figure 8 A schematic diagram of the training process of an image-text matching network, a covered language modeling network, and a covered region classification network in a data processing method provided in one embodiment of the present specification is shown.

[0092] It should be noted that the image-text matching network, the covered language modeling network, and the covered area classification network have been pre-trained during the application process. If the modal features are not fused through the encoding device, the application effect of the image-text matching network, the covered language modeling network, and the covered area classification network will not be good. However, after using the encoding device to fuse the features between the modalities, the effect of the model image-text matching task and the image or text prediction task will be improved.

[0093] See also Fig. 9 , Fig. 9 A schematic diagram of the processing process of the model application in the data processing method provided in the embodiment of this specification is shown.

[0094] It should be noted that Fig. 9 The upper middle part is a schematic diagram of the pre-training task, the middle is the joint switching input-attention mechanism encoder, and the lower part is the input data of the model; Fig. 9 The application process of the model is taken as an example to illustrate the distance, and this process can also be understood as the training process of the joint switching input-attention mechanism encoder with a large amount of training sample data. The combination of the image-text matching network, the masked area classification network, and the masked language modeling network in the above-mentioned pre-training tasks can be understood as the pre-training model required in the processing task after the application of the joint switching input-attention mechanism encoder.

[0095] In practical applications, Fig. 9 The data in the dotted box area in the lower middle part is the initial training data. Taking the initial training data as image data and text data as an example, after the image data is input into the visual embedding layer in the joint switching input-attention mechanism encoder, a visual representation is generated. Specifically, the image data includes visual type 0, region position information, and region features, wherein the region position information can be understood as the position information in each key region, and the region feature is the region feature in the region position information. At the same time, the first feature data in the region feature can be represented as the global feature data of the entire image, and the other feature data is the region feature data; after the text data is input into the text embedding layer in the joint switching input-attention mechanism encoder, a text representation is generated. Specifically, the text data includes text type 1, word position information, and word features, wherein the word position information can be determined according to the expression order of the text data, and at the same time, the first feature data of the word feature includes the full semantic information of CLS.

[0096] Furthermore, the visual representation and text representation are input into the first layer of the Switch-BERT layer in the joint switching input-attention mechanism encoder, and the output data is input into Fig. 9The modal input switching module in the dotted box in the joint switching input-attention mechanism encoder, and then the data output by the modal input switching module is input into the next Switch-BERT layer again. It should be noted that the modal input switching module and other Switch-BERT layers except the first Switch-BERT layer together constitute a repeatable unit. After determining that the total number of layers of the joint switching input-attention mechanism encoder is L layers, the repeatable unit is L-1 layers. Therefore, the final output from the joint switching input-attention mechanism encoder is the data of image fusion text features and the data of text fusion image features.

[0097] Furthermore, according to different processing tasks, the fused data is input into any pre-trained model of the image-text matching network, the covered area classification network or the covered language modeling network, so as to complete the corresponding image-text matching task, the covered area classification task and the covered word language modeling task.

[0098] It should be noted that during the training and application process, in the image-text matching task, it is not necessary to perform mask processing on the initial training data that is image data and text data; in the image-text prediction task, mask processing can be performed on the initial training data that is image data or text data accordingly (for example, Fig. 9 The affected part on the image data is the content of the random mask, and the text of the Mask on the text data represents the part of the random mask).

[0099] In summary, the data processing method provided in the embodiments of this specification utilizes a joint switching input-attention mechanism encoder to strengthen the modal fusion of image data and text data, and then uses the modal fusion data to perform specific execution tasks, thereby improving the task processing effect.

[0100] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Fig.10 FIG. 1 shows a schematic diagram of the structure of a first data processing device provided by an embodiment of the present specification. Fig.10 As shown, the device comprises: The first feature vector obtaining module 1002 is configured to receive a multimodal data matching request, input the first modal initial data and the second modal initial data carried in the multimodal data matching request into a preprocessing module in an encoding device, and obtain a first multimodal initial feature vector and a second multimodal initial feature vector, wherein the encoding device is the encoding device in the above embodiment; A first fusion vector obtaining module 1004 is configured to input the first multimodal initial feature vector and the second multimodal initial feature vector into the encoding layer and at least one joint encoding unit in the encoding device to perform multimodal feature fusion, so as to obtain a first target modality fusion vector and a second target modality fusion vector; The data matching module 1006 is configured to determine a pre-trained data matching network based on the multimodal data matching request, input the first target modality fusion vector and the second target modality fusion vector into the data matching network, and generate a data matching result.

[0101] Optionally, the device further comprises: A first receiving module is configured to receive an image data retrieval request, and determine at least two to-be-matched texts based on the to-be-retrieved image carried in the image data retrieval request; Generate a multimodal data matching request based on the image to be retrieved and the at least two texts to be matched; A first matching module is configured to construct a matching text corresponding to the image to be retrieved from the at least two texts to be matched based on the data matching result; or A second receiving module is configured to receive a text data retrieval request, and determine at least two to-be-matched images based on the to-be-retrieved text carried in the text data retrieval request; Generate a multimodal data matching request based on the text to be retrieved and the at least two pictures to be matched; The second matching module is configured to construct a matching picture corresponding to the text to be retrieved from the at least two pictures to be matched based on the data matching result.

[0102] The data processing device provided in the embodiments of this specification utilizes an encoding device to perform modal fusion on image feature data and text feature data, and when input into a pre-trained image-text matching network, it can more accurately achieve the image or text matching results of the model.

[0103] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0104] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Fig.11 FIG. 2 shows a schematic diagram of the structure of a second data processing device provided by an embodiment of the present specification. Fig.11 As shown, the device comprises: The second feature vector obtaining module 1102 is configured to receive a multimodal data prediction request, input the first modal initial data and the second modal initial data carried in the multimodal data prediction request into a preprocessing module in an encoding device, and obtain a first modal initial feature vector and a second modal initial feature vector, wherein the encoding device is the encoding device in the above embodiment, and the first modal initial data or the second modal initial data carries at least one mask data; A second fusion vector obtaining module 1104 is configured to input the first modal initial feature vector and the second modal initial feature vector into the encoding layer and at least one joint encoding unit in the encoding device to perform multimodal feature fusion to obtain a first target modal fusion vector and a second target modal fusion vector; The data prediction module 1106 is configured to determine a pre-trained data prediction network based on the multimodal data prediction request, input the first target modality fusion vector and the second target modality fusion vector into the data prediction network, and generate a data prediction result.

[0105] The data processing device provided in the embodiments of this specification utilizes an encoding device to perform modal fusion on image feature data and text feature data, and when input into a pre-trained data prediction network, it can more accurately achieve the prediction results of the model's image or text.

[0106] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0107] Fig.12 The block diagram of a computing device 1200 according to one embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0108] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0109] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0110] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 1200 may also be a mobile or stationary server.

[0111] The processor 1220 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0112] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0113] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0114] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0115] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0116] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0117] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0119] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A coding device, comprising a plurality of layers of joint coding units, each layer of the joint coding units comprising a coding module and a modal input switching module; The modal input switching module is configured as follows: Select one of the first modal feature vector input to the current modal input switching module and the first modal switching encoding vector output by the modal input switching module of the previous layer as the first modal switching encoding vector output by the current modal input switching module, and Select one of the second modal feature vector input to the current modal input switching module and the second modal switching coding vector output by the modal input switching module of the previous layer as the second modal switching coding vector output by the current modal input switching module; The encoding module is configured as follows: Generate a first target modality fusion vector by learning and encoding the received first modality switching coding vector and fusing the received second modality switching coding vector, and The second target modality fusion vector is generated by learning and encoding the received second modality switching coding vector and fusing it with the received first modality switching coding vector.

2. The encoding device according to claim 1, wherein the number of the multi-layer joint coding units is two or more and they constitute a joint coding unit group; The joint coding unit group is configured as: Jointly encoding the first modal feature vector and the second modal feature vector to generate a first target modal fusion vector and a second target modal fusion vector; In the joint coding unit group, output information of the joint coding unit of each layer is used as input information of the joint coding unit of the next layer.

3. The encoding device according to claim 1, wherein the modal input switching module is further configured to: Determining a first modal feature global vector of the first modal feature vector and a second modal feature global vector of the second modal feature vector; Calculating a feature matching value between the first modal feature global vector and the second modal feature global vector, and determining an initial classification probability value of the feature matching value based on a pre-trained feature selection classifier; A simulated classification probability value is determined based on the initial classification probability value, and a target modal feature vector type is selected according to the simulated classification probability value, wherein: The target modal feature vector type includes selecting one from a first modal feature vector and a second modal feature vector input to the current modal input switching module, and selecting one from a first modal switching coding vector and a second modal switching coding vector output by the modal input switching module of the previous layer; A first modality switching coding vector and a second modality switching coding vector are determined based on the target modality feature vector type.

4. The encoding device according to any one of claims 1 to 3, wherein the encoding module is further configured to: Determining a first modal switching coding global vector of the first modal switching coding vector and a second modal switching coding global vector of the second modal switching coding vector; Calculating a modality matching value between the first modality switching encoding global vector and the second modality switching encoding global vector, and determining an initial matching probability value of the modality matching value based on a pre-trained mode selection classifier; Determining a simulated match probability value based on the initial match probability value, and selecting a target attention mechanism according to the simulated match probability value; Based on the target attention mechanism, performing self-attention calculation on the first modality switching encoding vector to generate a first target modality fusion vector, and Based on the target attention mechanism, self-attention calculation is performed on the second modality switching encoding vector to generate a second target modality fusion vector.

5. The encoding device according to claim 1, further comprising an encoding layer; The encoding layer is configured as follows: Determine a first modal initial feature global vector of the first modal initial feature vector and a second modal initial feature global vector of the second modal initial feature vector; Calculating an initial feature matching value between the first modality initial feature global vector and the second modality initial feature global vector, and determining an initial feature classification probability value of the initial feature matching value based on a pre-trained initial feature selection classifier; Determining a simulated initial feature classification probability value based on the initial feature classification probability value, and selecting an initial attention mechanism according to the simulated initial feature classification probability value; Based on the initial attention mechanism, a self-attention calculation is performed on the first modal initial feature vector to generate a first modal feature vector, and Based on the initial attention mechanism, a self-attention calculation is performed on the second modality initial feature vector to generate a second modality feature vector.

6. The encoding device according to claim 1, further comprising a preprocessing module; The pre-processing module is configured to: receiving multimodal initial data, and determining a data type of the multimodal initial data; Determining position features of the multimodal initial data, and extracting data features of the multimodal initial data; A multimodal initial feature vector is generated based on the data type, the location feature, and the data feature, wherein the multimodal initial feature vector includes a first modality initial feature vector and a second modality initial feature vector.

7. A data processing method, comprising: Receiving a multimodal data matching request, inputting the first modality initial data and the second modality initial data carried in the multimodal data matching request into an encoding device, and obtaining a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device according to any one of claims 1 to 6; A pre-trained data matching network is determined based on the multimodal data matching request, and the first target modality fusion vector and the second target modality fusion vector are input into the data matching network to generate a data matching result.

8. The data processing method according to claim 7, before receiving the multimodal data matching request, further comprising: Receive an image data retrieval request, and determine at least two texts to be matched based on the image to be retrieved carried in the image data retrieval request; Generate a multimodal data matching request based on the image to be retrieved and the at least two texts to be matched; Accordingly, after generating the data matching result, the step further includes: Constructing a matching text corresponding to the image to be retrieved from the at least two texts to be matched based on the data matching result; or Before receiving the multimodal data matching request, the method further includes: Receive a text data search request, and determine at least two images to be matched based on the text to be searched carried in the text data search request; Generate a multimodal data matching request based on the text to be retrieved and the at least two pictures to be matched; Accordingly, after generating the data matching result, the step further includes: A matching picture corresponding to the text to be retrieved is constructed from the at least two pictures to be matched based on the data matching result.

9. A data processing method, comprising: Receiving a multimodal data prediction request, inputting the first modality initial data and the second modality initial data carried in the multimodal data prediction request into an encoding device, and obtaining a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device according to any one of claims 1 to 6, and the first modality initial data or the second modality initial data carries at least one mask data; A pre-trained data prediction network is determined based on the multimodal data prediction request, and the first target modality fusion vector and the second target modality fusion vector are input into the data prediction network to generate a data prediction result.

10. A data processing device, comprising: The first fusion vector obtaining module is configured to: receive a multimodal data matching request, input the first modality initial data and the second modality initial data carried in the multimodal data matching request into an encoding device, and obtain a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device according to any one of claims 1 to 6; The data matching module is configured to: determine a pre-trained data matching network based on the multimodal data matching request, input the first target modality fusion vector and the second target modality fusion vector into the data matching network, and generate a data matching result.

11. A data processing device, comprising: The second fusion vector obtaining module is configured to: receive a multimodal data prediction request, input the first modality initial data and the second modality initial data carried in the multimodal data prediction request into an encoding device, and obtain a first target modality fusion vector and a second target modality fusion vector, wherein the encoding device is the encoding device according to any one of claims 1 to 6, and the first modality initial data or the second modality initial data carries at least one mask data; The data prediction module is configured to: determine a pre-trained data prediction network based on the multimodal data prediction request, input the first target modality fusion vector and the second target modality fusion vector into the data prediction network, and generate a data prediction result.

12. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method described in any one of claims 7 to 9 are implemented.

13. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 7 to 9.

Images