Multi-modal data processing method and device

Abstract

The embodiment of the invention provides a multi-modal data processing method and device.The method comprises the steps that input multi-modal data are obtained, and the multi-modal data comprise at least two pieces of modal type data; acquiring a plurality of initial data features corresponding to each piece of modal type data in the at least two pieces of modal type data; performing feature screening on the plurality of initial data features to obtain at least one key data feature, the key data feature being a feature capable of representing preset information of the multi-modal data in the initial data features; and outputting cross-modal joint features corresponding to the multi-modal data by using the key data features. Through the multi-modal data processing method and device, the problem that the multi-modal data processing efficiency is low is solved, and then the effect of improving the multi-modal data processing efficiency is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a method and device for processing multimodal data. Background Art

[0002] In the related art, when processing multimodal data, redundant features may exist in the local features extracted from the multimodal data, thereby affecting the processing accuracy of the multimodal data, and further resulting in the problem of low processing efficiency of the multimodal data. Therefore, there is the problem of low processing efficiency of the multimodal data.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing multimodal data to at least solve the problem of low efficiency in processing multimodal data in the related art.

[0005] According to an embodiment of the present application, a method for processing multimodal data is provided, comprising:

[0006] Acquire input multimodal data, wherein the multimodal data includes at least two modal type data; acquire multiple initial data features corresponding to each modal type data in the at least two modal type data; perform feature screening on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature in the initial data features that can characterize preset information of the multimodal data; and utilize the key data feature to output a cross-modal joint feature corresponding to the multimodal data.

[0007] According to another embodiment of the present application, a multimodal data processing device is provided, including:

[0008] A first acquisition unit is used to acquire input multimodal data, wherein the multimodal data includes at least two modal type data; a second acquisition unit is used to acquire multiple initial data features corresponding to each modal type data in the at least two modal type data; a screening unit is used to perform feature screening on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature in the initial data feature that can characterize the preset information of the multimodal data; an output unit is used to output a cross-modal joint feature corresponding to the multimodal data using the key data feature.

[0009] As an optional solution, the above-mentioned screening unit includes: a construction module, which is used to construct a corresponding plurality of key weights for each of the above-mentioned multiple initial data features, wherein the above-mentioned key weights are used to indicate the degree of retention of the data features; an integration module, which is used to integrate the above-mentioned each initial data feature with the corresponding above-mentioned key weights to obtain the integration results corresponding to the above-mentioned each initial data feature; and a screening module, which is used to use the above-mentioned integration results to perform feature screening on the above-mentioned multiple initial data features to obtain the above-mentioned at least one key data feature.

[0010] As an optional solution, the above-mentioned construction module includes: an input submodule, used to input the above-mentioned multiple initial data features into the gated recurrent unit; a capture submodule, used to capture the long-term dependencies in the feature sequence through the gating mechanism of the above-mentioned gated recurrent unit; a first acquisition submodule, used to use the above-mentioned long-term dependencies to obtain the hidden state output by the above-mentioned gated recurrent unit at each time step, wherein the above-mentioned hidden state contains all historical information up to the current time step; a mapping submodule, used to map the above-mentioned hidden state into a weight vector, wherein each element in the above-mentioned weight vector corresponds to each key weight among the above-mentioned multiple key weights.

[0011] As an optional solution, the above-mentioned construction module includes: an adjustment submodule, which is used to use the weight coefficient of L1 regularization to adjust the sparse regularization strength of the above-mentioned multiple key weights, wherein the above-mentioned weight coefficient is obtained by iteratively updating using a gradient descent algorithm.

[0012] As an optional solution, the above-mentioned output unit includes: an extraction module, which is used to perform multi-level feature extraction on the above-mentioned key data features to obtain multi-layer features; a decoupling module, which is used to decouple each layer of features in the above-mentioned multi-layer features using different decoupling modules to obtain independent semantic features of each layer; and an output module, which is used to use the above-mentioned independent semantic features to output the above-mentioned cross-modal joint features.

[0013] As an optional solution, the above-mentioned extraction module includes: a first extraction sub-module, which is used to perform multi-level encoding on the text key features in the above-mentioned key data features to extract the context information of each word; and an encoding sub-module, which is used to encode the above-mentioned text key features layer by layer in combination with the above-mentioned context information to obtain text features at different semantic levels.

[0014] As an optional solution, the above-mentioned extraction module includes: a second extraction sub-module, which is used to extract low-level features and high-level features of the image after multi-layer convolution from the image key features in the above-mentioned key data features, wherein the features extracted after the above-mentioned multi-layer convolution respectively represent feature representations of different detail granularities.

[0015] As an optional scheme, the above-mentioned output unit includes: a first fusion module, used to perform feature fusion on multiple first features belonging to the same modality type among the above-mentioned key data features to obtain N first cross-modal features, where N is a positive integer; a second fusion module, used to perform feature fusion on multiple second features belonging to different modality types among the above-mentioned key data features to obtain M second cross-modal features, where M is a positive integer; a first acquisition module, used to acquire the above-mentioned cross-modal joint features based on the above-mentioned N first cross-modal features and the above-mentioned M second cross-modal features.

[0016] As an optional scheme, the above-mentioned first fusion module includes: a determination submodule, used to determine the first current feature and the second current feature from the above-mentioned multiple first features; a calculation submodule, used to calculate the dot product of the query matrix and the key matrix to obtain a dot product result, wherein the above-mentioned query matrix contains the above-mentioned first current feature and the above-mentioned second current feature, and the above-mentioned key matrix is ​​used to calculate the similarity with the above-mentioned query matrix; a conversion submodule, used to convert the above-mentioned dot product result into an attention weight, wherein the above-mentioned attention weight is used to represent the feature similarity between the above-mentioned first current feature and the above-mentioned second current feature; a summation submodule, used to perform weighted summation of the above-mentioned attention weight and the value matrix to obtain a weighted summation result, wherein the above-mentioned value matrix contains specific information of the above-mentioned first current feature and the above-mentioned second current feature; a first fusion submodule, used to use the above-mentioned weighted summation result to fuse the above-mentioned first current feature and the above-mentioned second current feature to obtain the above-mentioned first cross-modal feature.

[0017] As an optional scheme, the above-mentioned second fusion module includes: a second acquisition sub-module, used to obtain a similarity matrix between the above-mentioned multiple second features, wherein the above-mentioned similarity matrix is ​​used to reflect the similarity between the above-mentioned multiple second features; a second fusion sub-module, used to fuse features with high similarity among the above-mentioned multiple second features based on the above-mentioned similarity matrix to obtain the above-mentioned M second cross-modal features.

[0018] As an optional solution, the above-mentioned first acquisition unit includes: a second acquisition module, used to obtain multimodal sample data input into a cross-modal model, wherein the above-mentioned cross-modal model is used to output corresponding description text from image input, or to generate a corresponding description image from text input; an iteration module, used to iteratively update model parameters of the above-mentioned cross-modal model until the model convergence conditions are met, thereby obtaining the trained above-mentioned cross-modal model.

[0019] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0020] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0021] According to another embodiment of the present application, a computer program product is provided, including a computer program, wherein when the computer program is configured to be executed by a processor, the steps in any one of the above method embodiments are performed.

[0022] Through the present application, input multimodal data is obtained, wherein the multimodal data includes at least two modal type data; multiple initial data features corresponding to each modal type data in the at least two modal type data are obtained; feature screening is performed on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature in the initial data feature that can characterize preset information of the multimodal data; and using the key data feature, a cross-modal joint feature corresponding to the multimodal data is output.

[0023] Specifically, after obtaining the input multimodal data, the initial data features in the multimodal data are extracted, and then the initial features are screened to remove redundant features, and then the key data features that can characterize the multimodal data are obtained, and then the cross-modal joint features are obtained based on the key data features. Since the redundant features are effectively removed through feature screening, the processing accuracy of the multimodal data is improved, which can solve the problem of low processing efficiency of the multimodal data, thereby achieving the technical effect of improving the processing efficiency of the multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of an application environment of a method for processing multimodal data according to an embodiment of the present application;

[0025] Figure 2 is a flowchart of a method for processing multimodal data according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a method for processing multimodal data according to an embodiment of the present application;

[0032] Fig. 9 It is a structural block diagram of a multimodal data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a multimodal data processing method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing multimodal data in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] In this embodiment, a method for processing multimodal data is provided. Figure 2 is a flowchart of a method for processing multimodal data according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0039] Step S202, obtaining input multimodal data, wherein the multimodal data includes at least two modal type data;

[0040] In an optional embodiment, multimodal data may, but is not limited to, refer to data containing at least two modal types, and the modal type data may, but is not limited to, data of different modal types such as text, image, audio, video, etc.

[0041] Step S204, obtaining a plurality of initial data features corresponding to each of the at least two modal type data;

[0042] In an optional embodiment, the initial data features may be understood, but are not limited to, as data features directly extracted from original modality type data.

[0043] Step S206, performing feature screening on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature in the initial data features that can characterize preset information of the multimodal data;

[0044] In an optional embodiment, the key data features may be, but are not limited to, data features obtained after feature screening of initial data features, and may be, but are not limited to, features that can characterize preset information of multimodal data.

[0045] It should be noted that when processing multimodal data, the initial data features of the multimodal data obtained often contain a lot of redundant information. If the initial data features are not screened, the redundant information will affect the processing of the multimodal data, which will lead to low efficiency and accuracy in multimodal data processing.

[0046] Step S208: using key data features, output cross-modal joint features corresponding to the multimodal data.

[0047] In an optional embodiment, the cross-modal joint feature may be understood, but is not limited to, as fusing or combining key data features of different modalities to generate a joint feature that can simultaneously characterize different types of modal data.

[0048] It should be noted that, based on the initial data features corresponding to each modal data from at least two modal types of data, the system further performs feature screening, retaining those features that can reflect the preset information during the screening process, thereby obtaining key data features. Through screening, the most relevant and representative features in at least two modal types of data can be obtained, thereby removing redundant features that may have a negative impact on the processing of multimodal data, reducing the impact of redundant features, and improving the model's ability to capture key information of multimodal data, thereby improving the accuracy and efficiency of multimodal data processing.

[0049] Through the above steps, the input multimodal data is obtained, wherein the multimodal data includes at least two modal type data; a plurality of initial data features corresponding to each modal type data in the at least two modal type data are obtained; feature screening is performed on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature in the initial data features that can characterize the preset information of the multimodal data; and the cross-modal joint features corresponding to the multimodal data are output using the key data features.

[0050] Specifically, after obtaining the input multimodal data, the initial data features in the multimodal data are extracted, and then the initial features are screened to remove redundant features, and then the key data features that can characterize the multimodal data are obtained, and then the cross-modal joint features are obtained based on the key data features. Since the redundant features are effectively removed through feature screening, the technical purpose of improving the processing accuracy of the multimodal data is achieved, and then the technical effect of improving the processing efficiency of the multimodal data is achieved.

[0051] The execution subject of the above steps may be a server, a terminal, etc., but is not limited thereto.

[0052] Before screening the multiple initial data features to obtain at least one key data feature, the method further includes:

[0053] S1-1, constructing a corresponding plurality of key weights for each of the plurality of initial data features, wherein the key weights are used to indicate a degree of retention of the data features;

[0054] S1-2, integrating each initial data feature with the corresponding key weight to obtain the integration result corresponding to each initial data feature;

[0055] Perform feature screening on multiple initial data features to obtain at least one key data feature, including:

[0056] S1-3, using the integration result, perform feature screening on multiple initial data features to obtain at least one key data feature.

[0057] In an optional embodiment, the key weight may be, but is not limited to, a weight value assigned to each initial data feature during the feature screening process, and may be, but is not limited to, used to represent the importance or retention degree of the data feature.

[0058] In an optional embodiment, the integration processing may be, but is not limited to, combining the initial data features with the corresponding key weights before feature screening, and may also be, but is not limited to, adjusting the key weights corresponding to the initial data features.

[0059] It should be noted that before feature screening, it is necessary to assign key weights to each initial data feature to characterize the degree of retention of the feature, and then obtain the integration result through integration processing, and finally perform feature screening based on the integration result. Through the introduction and integration of key weights, it is possible to more finely control which features should be retained and which features are redundant features during the screening process, so as to more accurately generate key features that can characterize multimodal data, thereby ensuring the effectiveness and accuracy of feature screening.

[0060] Through the embodiment of the present application, a plurality of corresponding key weights are constructed for each of the plurality of initial data features, wherein the key weight is used to indicate the degree of retention of the data feature; each initial data feature is integrated with the corresponding key weight to obtain the integration result corresponding to each initial data feature; and the integration result is used to perform feature screening on the plurality of initial data features to obtain at least one key data feature. Thus, the key features capable of characterizing multimodal data are generated more accurately, thereby achieving the technical effect of ensuring the effectiveness and accuracy of feature screening.

[0061] As an optional solution, a plurality of corresponding key weights are constructed for each of the plurality of initial data features, including:

[0062] S2-1, inputting multiple initial data features into the gated recurrent unit;

[0063] S2-2, capturing long-term dependencies in feature sequences through the gating mechanism of gated recurrent units;

[0064] S2-3, using long-term dependencies, obtain the hidden state output by the gated recurrent unit at each time step, where the hidden state contains all historical information up to the current time step;

[0065] S2-4, mapping the hidden state to a weight vector, wherein each element in the weight vector corresponds to each key weight in the plurality of key weights.

[0066] In an optional embodiment, the gated recurrent unit (GRU) may include, but is not limited to, a gating mechanism, may be used to control the flow and update of information, and may also be used to capture long-term dependencies. The gating mechanism may include, but is not limited to, an update gate and a reset gate, the update gate may be used to determine how to update the unit state at the current moment, and the reset gate may be used to determine how to forget or reset the old state information.

[0067] In an optional embodiment, the long-term dependency may refer to, but is not limited to, the relationship between a current state and an earlier state as time steps pass.

[0068] In an optional embodiment, the hidden state may include, but is not limited to, all historical information or encoding of historical information up to the current time step, and may be updated as the sequence progresses, but is not limited to.

[0069] It should be noted that when the gated recurrent unit processes a feature sequence, it updates its hidden state at each time step based on the long-term dependency and the current input. The hidden state contains the encoding of the historical information of the sequence at each time point in the sequence, which is the key for the gated recurrent unit to capture the sequence dependency.

[0070] In an optional embodiment, the weight vector may be, but is not limited to, a vector generated by mapping the hidden state of a gated recurrent unit, and the elements of the weight vector may be, but are not limited to, key weights corresponding to initial data features, i.e., corresponding to each of multiple key weights.

[0071] It should be noted that inputting the feature sequence into the gated recurrent unit and using its gating mechanism to process the sequence features can effectively capture the contextual relationship and long-term dependency between features. These hidden states processed by the gated recurrent unit are then mapped to a weight vector, each element of which represents the key weight of the corresponding initial feature. By using the gated recurrent unit and its gating mechanism, the feature screening process can be more finely controlled to ensure that the retained features are more important features for the processing of multimodal data, thereby improving the effectiveness of cross-modal joint features in characterizing multimodal data. At the same time, through the generation of weight vectors, quantitative control of the degree of feature retention can be achieved, further optimizing the process of feature screening and cross-modal fusion, thereby improving the processing efficiency of multimodal data.

[0072] Through the embodiment of the present application, multiple initial data features are input into the gated recurrent unit; the long-term dependencies in the feature sequence are captured through the gating mechanism of the gated recurrent unit; the hidden state output by the gated recurrent unit at each time step is obtained by using the long-term dependencies, wherein the hidden state contains all historical information up to the current time step; the hidden state is mapped to a weight vector, wherein each element in the weight vector corresponds to each key weight in multiple key weights. Thus, the technical purpose of realizing quantitative control of the degree of feature retention is achieved, and the technical effect of improving the processing efficiency of multimodal data is achieved.

[0073] As an optional solution, in the process of constructing corresponding multiple key weights for each of the multiple initial data features, the method further includes:

[0074] The weight coefficient of L1 regularization is used to adjust the sparse regularization strength of multiple key weights, wherein the weight coefficient is iteratively updated by applying the gradient descent algorithm.

[0075] In an optional embodiment, L1 regularization can be understood as, but not limited to, a regularization technique, can be, but not limited to, ensuring the sparsity of model weights by adding the sum of the absolute values ​​of weights (L1 norm) to the loss function, and can be, but not limited to, used to reduce redundant features for feature selection.

[0076] In an optional embodiment, the gradient descent algorithm may be, but is not limited to, an optimization algorithm, may be, but is not limited to, used to minimize the loss function of a machine learning model, may be, but is not limited to, calculating the gradient of the loss function with respect to the model weights, and then updating the weights at a certain learning rate to move in the direction of reducing the loss. In this embodiment, the gradient descent algorithm may be, but is not limited to, used to iteratively update the weight coefficients used for L1 regularization to achieve the optimal sparse regularization strength.

[0077] In an optional embodiment, the sparsity regularization strength may refer to, but is not limited to, the degree of sparsification effect of L1 regularization on model weights.

[0078] It should be noted that the feature screening process is optimized by using L1 regularization and iterative updates of weight coefficients to achieve more accurate feature selection and model sparsification. In cross-modal learning tasks, the model usually needs to process a large number of initial data features, which contain a lot of redundant and irrelevant information. Through L1 regularization, the weights can be selectively compressed so that the weights of unimportant features are close to zero, thereby achieving feature selection. The weight coefficient plays the role of adjusting the intensity of sparse regularization. By iteratively updating this coefficient through the gradient descent algorithm, a balance point can be found that can remove redundant features without over-sparse important features. This optimization process helps to improve the model's ability to capture key information, reduce the waste of computing resources, and improve the efficiency and accuracy of cross-modal learning tasks when training or optimizing multimodal data processing models.

[0079] Through the embodiment of the present application, the sparse regularization strength of multiple key weights is adjusted by using the weight coefficient of L1 regularization, wherein the weight coefficient is obtained by iteratively updating by applying the gradient descent algorithm. Thus, the technical purpose of improving the ability to capture key information is achieved, and the technical effect of improving the efficiency and accuracy of cross-modal learning tasks is achieved.

[0080] As an optional solution, key data features are used to output cross-modal joint features corresponding to multimodal data, including:

[0081] S3-1, perform multi-layer feature extraction on key data features to obtain multi-layer features;

[0082] S3-2, use different decoupling modules to decouple the features of each layer in the multi-layer features to obtain independent semantic features at each level;

[0083] S3-3, utilizes independent semantic features to output cross-modal joint features.

[0084] In an optional embodiment, multi-level feature extraction may, but is not limited to, refer to using different levels of a deep neural network to capture multi-level information of features during the feature extraction stage.

[0085] In an optional embodiment, in a deep learning model, a decoupling module may be used, but is not limited to, to separate different components of different modal features so that each component can be independently represented and optimized, thereby helping the model to more accurately understand the relationship between different modal features, avoid cross-interference of information, and improve model performance.

[0086] In an optional embodiment, the independent semantic features may be, but are not limited to, understood as features processed by the decoupling module. The independent semantic features obtained at each layer may be, but are not limited to, feature representations that can clearly reflect the specific semantic information of the layer.

[0087] In an optional embodiment, multi-layer feature extraction is performed on key data features to obtain multi-layer features, which can be, but not limited to, multi-layer deep feature extraction of key data features to capture semantic information at different levels. By using different layers of a deep neural network, low-level local features can be gradually increased to high-level global features.

[0088] To further illustrate, taking image features as an example, the shallow layers of the convolutional neural network can be used to capture edge and texture information, while the deep layers can be used to extract semantic objects and scene information. For text features, the bottom layers of the gated recurrent unit or transformer model can be used to encode vocabulary information, while the high layers are used to capture the contextual relationships of sentences and paragraphs.

[0089] In an optional embodiment, different decoupling modules are used to decouple the features of each layer in the multi-layer features to obtain independent semantic features of each layer. It can be understood, but not limited to, that after obtaining the multi-layer features, it is necessary to further use the decoupling module to decouple the features of each layer to separate independent semantic features. The purpose of decoupling is to ensure that the features of each layer can be independently represented and optimized to avoid cross-information interference between features of different layers, which is very important for cross-modal understanding tasks.

[0090] In an optional embodiment, independent semantic features are used to output cross-modal joint features, which can be understood, but not limited to, as fusing the decoupled independent semantic features to generate the final cross-modal joint features.

[0091] It should be noted that the representation of multimodal data is optimized through multi-level feature extraction and decoupling, as well as the fusion of independent semantic features, to improve the performance of cross-modal learning tasks. First, through different layers of the deep neural network, the key data features are extracted layer by layer, from low-level local details to high-level global semantics, to capture the multi-level information of the features. Then, the decoupling module is used to decouple the features of each layer, separate the independent semantic features, avoid information confusion between layers, and enable each layer of features to independently represent specific semantic information. Finally, the independent multi-layer features are integrated into cross-modal joint features through a fusion strategy. This joint feature can fully reflect the multi-layer semantic information of images and texts, and provide a more accurate and comprehensive representation for cross-modal tasks. The technical solution of the present invention significantly improves the depth and accuracy of cross-modal understanding through hierarchical feature extraction and decoupling, laying the foundation for building a more intelligent multimodal learning system.

[0092] Through the embodiments of the present application, multi-layer feature extraction is performed on key data features to obtain multi-layer features; different decoupling modules are used to decouple the features of each layer in the multi-layer features to obtain independent semantic features of each layer; and the independent semantic features are used to output cross-modal joint features. Thus, the technical purpose of obtaining cross-modal joint features is achieved, and the technical effect of providing more accurate and comprehensive representation for cross-modal tasks is achieved.

[0093] As an optional solution, multi-layer feature extraction is performed on key data features to obtain multi-layer features, including:

[0094] S4-1, multi-level encoding of the key features of the text in the key data features to extract the context information of each word;

[0095] S4-2, combined with context information, encodes the key features of the text layer by layer to obtain text features at different semantic levels.

[0096] In an optional embodiment, the key feature of the file may be, but is not limited to, a screened key feature containing text information.

[0097] In an optional embodiment, multi-level coding can be understood, but is not limited to, as, in the field of natural language processing, multi-level coding refers to using the multi-layer capabilities of deep neural networks to encode text data layer by layer to capture semantic information at different levels.

[0098] In an optional embodiment, the semantic level may refer to, but is not limited to, the semantic structure from words, phrases to sentences and paragraphs.

[0099] It should be noted that contextual information is crucial to understanding the meaning of words. By encoding the key features of the text layer by layer, each layer of encoding considers the position of the word in the text, the semantics of the surrounding words, and their contribution to the overall text meaning, so as to extract deeper contextual information.

[0100] Furthermore, by combining context information, the key features of the text are encoded layer by layer to obtain text features at different semantic levels. By combining context information with multi-level encoding technology, the model can mine multi-level semantic structures from the text, from vocabulary to phrases, sentences and even paragraphs, and generate text features that contain both the surface meaning of vocabulary and reflect its deep role, significantly improving the performance of cross-modal learning tasks.

[0101] Through the embodiment of the present application, the key text features in the key data features are encoded at multiple levels to extract the context information of each word; combined with the context information, the key text features are encoded layer by layer to obtain text features at different semantic levels. Thus, the technical purpose of generating text features that contain both the surface meaning of the vocabulary and reflect its deep function is achieved, thereby achieving the technical effect of improving the performance of cross-modal learning tasks.

[0102] As an optional solution, multi-layer feature extraction is performed on key data features to obtain multi-layer features, including:

[0103] From the key features of the image in the key data features, low-level features and high-level features are extracted after multi-layer convolution to obtain the image, wherein the features extracted after multi-layer convolution represent feature representations of different detail granularities respectively.

[0104] In an optional embodiment, the key features of the image may be, but are not limited to, features screened out from the key data features for the image modality, and may be, but are not limited to, include the main content and features of the image.

[0105] In an optional embodiment, in a convolutional neural network, multi-layer convolution may, but is not limited to, refer to a network comprising multiple convolution layers, each of which is responsible for capturing different features in an image. Shallow convolution may, but is not limited to, be used to capture low-level features, while deep convolution may, but is not limited to, be used to extract high-level features.

[0106] In an optional embodiment, low-level features may be, but are not limited to, features extracted from shallow layers of a multi-layer convolutional neural network, and may be, but are not limited to, used to represent local details of an image. High-level features may be, but are not limited to, features extracted from key image features in deep layers of a multi-layer convolutional neural network, and may be, but are not limited to, used to represent the global structure of an image.

[0107] It should be noted that low-level features and high-level features are extracted from the key features of the image through multi-layer convolutional neural networks. In the image feature extraction stage, the shallow convolutional layers of the multi-layer convolutional neural network are usually responsible for capturing the low-level features of the image, which is the basis of image understanding. The deep convolutional layers further process these low-level features and extract high-level features. The feature representation of each convolutional output reflects the image features of different detail granularity, from local details to global semantics, to construct a hierarchical feature space, which provides a richer and more comprehensive image feature representation for cross-modal learning tasks.

[0108] Through the embodiments of the present application, low-level features and high-level features of the image are extracted from the key features of the image in the key data features after multi-layer convolution, wherein the features extracted after multi-layer convolution respectively represent feature representations of different detail granularities. Thus, the technical purpose of constructing a hierarchical feature space is achieved, and further, the technical effect of providing a richer and more comprehensive image feature representation for cross-modal learning tasks is achieved.

[0109] As an optional solution, key data features are used to output cross-modal joint features corresponding to multimodal data, including:

[0110] S5-1, performing feature fusion on multiple first features of the same modality type in the key data features to obtain N first cross-modal features, where N is a positive integer;

[0111] S5-2, performing feature fusion on multiple second features belonging to different modal types in the key data features to obtain M second cross-modal features, where M is a positive integer;

[0112] S5-3, obtaining a cross-modal joint feature based on the N first cross-modal features and the M second cross-modal features.

[0113] In an optional embodiment, multiple first features of the same modality type may, but are not limited to, refer to key data features belonging to the same modality.

[0114] In an optional embodiment, the second features of different modal types may refer to, but are not limited to, key data features belonging to different modalities, and may represent, but are not limited to, the main content of data of different modal types.

[0115] In an optional embodiment, feature fusion may refer to, but is not limited to, a process of integrating multiple feature vectors into a comprehensive feature representation in a certain manner, and may refer to, but is not limited to, feature fusion by concatenation, weighted summation, and the like.

[0116] It should be noted that for key data features belonging to the same modality type, feature fusion processing is performed to obtain N first cross-modal features, which helps to integrate the information within the same modality and construct a more comprehensive and integrated modality internal feature representation. Next, cross-modal fusion is performed on key data features from different modal types to obtain M second cross-modal features. These features fuse information from different modalities and construct a cross-modal comprehensive representation space. Finally, cross-modal joint features are generated by further integrating the N first cross-modal features and the M second cross-modal features. This feature representation not only contains rich information of multimodal data, but also ensures the diversity and depth of features through a hierarchical fusion strategy, providing a more accurate and comprehensive feature basis for cross-modal tasks, significantly improving the model's cross-modal understanding and matching capabilities, and thus improving the processing efficiency of multimodal data.

[0117] Through the embodiment of the present application, multiple first features of the same modality type in the key data features are feature fused to obtain N first cross-modal features, where N is a positive integer; multiple second features of different modality types in the key data features are feature fused to obtain M second cross-modal features, where M is a positive integer; based on the N first cross-modal features and the M second cross-modal features, a cross-modal joint feature is obtained. Thus, the technical purpose of generating a cross-modal joint feature is achieved, and the technical effect of improving the processing efficiency of multimodal data is achieved.

[0118] As an optional solution, multiple first features of the same modality type in the key data features are fused to obtain N first cross-modal features, including:

[0119] Perform the following steps until N first cross-modal features are obtained:

[0120] S6-1, determining a first current feature and a second current feature from a plurality of first features;

[0121] S6-2, calculating a dot product of a query matrix and a key matrix to obtain a dot product result, wherein the query matrix includes the first current feature and the second current feature, and the key matrix is ​​used to calculate similarity with the query matrix;

[0122] S6-3, converting the dot product result into an attention weight, wherein the attention weight is used to represent the feature similarity between the first current feature and the second current feature;

[0123] S6-4, performing weighted summation on the attention weight and the value matrix to obtain a weighted summation result, wherein the value matrix includes specific information of the first current feature and the second current feature;

[0124] S6-5 uses the weighted summation result to fuse the first current feature and the second current feature to obtain a first cross-modal feature.

[0125] In an optional embodiment, the query matrix may be, but is not limited to, a matrix used to represent the current focus. The key matrix may be, but is not limited to, used to calculate similarity with the query matrix to help determine the allocation of attention. The value matrix may be, but is not limited to, containing specific information of the features, and may be, but is not limited to, used for weighted summation according to the attention weight.

[0126] In an optional embodiment, the dot product may be understood, but is not limited to, as a process of multiplying the rows of the first matrix by the elements corresponding to the columns of the second matrix, and then adding the results to obtain a new matrix or scalar value.

[0127] In an optional embodiment, the attention weight can be but is not limited to being understood as a weight converted through a dot product result, representing the relative importance between different features, and can be but is not limited to being used to represent the feature similarity between a first current feature and a second current feature.

[0128] In an optional embodiment, determining the first current feature and the second current feature from the multiple first features can be understood as, but not limited to, in the input stage of the self-attention mechanism, the model needs to select two features from multiple first features of the same modality: the first current feature and the second current feature. These two features will be used to calculate the attention weights to determine the relative importance and similarity between them, which is the basis for feature fusion.

[0129] In an optional embodiment, the dot product of a query matrix and a key matrix is ​​calculated to obtain a dot product result, wherein the query matrix includes the first current feature and the second current feature, and the key matrix is ​​used to calculate the similarity with the query matrix, which can be understood as but not limited to determining the similarity between the first current feature and the second current feature by calculating the dot product of the query matrix and the key matrix.

[0130] In an optional embodiment, the dot product result is converted into an attention weight, wherein the attention weight is used to represent the feature similarity between the first current feature and the second current feature, which can be understood, but not limited to, as converting the dot product result into an attention weight to represent the similarity between the first current feature and the second current feature.

[0131] In an optional embodiment, the attention weight and the value matrix are weighted and summed to obtain a weighted sum result, wherein the value matrix contains specific information of the first current feature and the second current feature, which can be understood, but not limited to, as weighted summing the value matrix with the attention weight to generate a fused feature representation. The value matrix contains specific information of the first and second current features, while the attention weight guides the fusion of these features, and the weighted sum result represents the fused first cross-modal feature, which integrates multiple information of the current features.

[0132] In an optional embodiment, the weighted summation result is used to fuse the first current feature and the second current feature to obtain the first cross-modal feature, which can be understood as, but not limited to, using the weighted summation result to fuse the first current feature and the second current feature to generate the first cross-modal feature. The weighted summation result contains the comprehensive information of the two features, and through further fusion processing, a more comprehensive and integrated feature representation can be obtained, which helps the model to more deeply understand the structure and content of the same modality data.

[0133] It should be noted that through the calculation of the query matrix, key matrix and value matrix of the self-attention mechanism, the model can intelligently determine the relative importance of different features, and perform feature fusion based on these importances, reduce information redundancy, and improve the expressiveness of features and the generalization ability of the model. In multimodal tasks such as image description generation and cross-modal retrieval, the model's deep understanding of the same modality data can be significantly improved, laying a solid foundation for generating higher-quality cross-modal joint features, and further improving the accuracy and robustness of multimodal data processing.

[0134] Through the embodiment of the present application, the first current feature and the second current feature are determined from multiple first features; the dot product of the query matrix and the key matrix is ​​calculated to obtain the dot product result, wherein the query matrix contains the first current feature and the second current feature, and the key matrix is ​​used to calculate the similarity with the query matrix; the dot product result is converted into an attention weight, wherein the attention weight is used to represent the feature similarity between the first current feature and the second current feature; the attention weight and the value matrix are weighted and summed to obtain a weighted sum result, wherein the value matrix contains specific information of the first current feature and the second current feature; the weighted sum result is used to fuse the first current feature and the second current feature to obtain the first cross-modal feature. Thus, the technical purpose of obtaining the first cross-modal feature is achieved, and the technical effect of improving the accuracy and robustness of multimodal data processing is achieved.

[0135] As an optional solution, multiple second features belonging to different modal types in the key data features are fused to obtain M second cross-modal features, including:

[0136] S7-1, obtaining a similarity matrix between the plurality of second features, wherein the similarity matrix is ​​used to reflect the similarity between the plurality of second features;

[0137] S7-2, based on the similarity matrix, the features with high similarity among the multiple second features are fused to obtain M second cross-modal features.

[0138] In an optional embodiment, the similarity matrix may be, but is not limited to, a matrix used to quantify and describe the similarities between different features.

[0139] It should be noted that in cross-modal feature fusion, the construction of the similarity matrix is ​​based on the dot product or other similarity measurement methods between features, which is used to quantify the similarity between the second features. By calculating the similarity scores between features, a matrix that quantifies the relationship between features can be obtained. This matrix will be used to guide the subsequent feature fusion process to ensure that the model can identify and integrate features with high correlation.

[0140] Furthermore, the second features with high similarity are fused to generate the second cross-modal features. The fusion process usually includes weighted summation, concatenation, or the use of graph convolutional neural networks to integrate the feature pairs with higher scores to generate M second cross-modal features. These features integrate the key information between different modalities, improve the accuracy and robustness of cross-modal understanding and matching, and thus improve the processing accuracy of multimodal data.

[0141] Through the embodiment of the present application, a similarity matrix between multiple second features is obtained, wherein the similarity matrix is ​​used to reflect the similarity between multiple second features; based on the similarity matrix, the features with high similarity among the multiple second features are fused to obtain M second cross-modal features. Thus, the technical purpose of fusing high-similarity features based on the similarity matrix is ​​achieved, and the technical effect of improving the processing accuracy of multimodal data is achieved.

[0142] As an optional solution, the above multimodal data processing method further includes:

[0143] S8-1, obtaining input multimodal data, including: obtaining multimodal sample data for input into a cross-modal model, wherein the cross-modal model is used to output corresponding description text from image input, or to generate corresponding description images from text input;

[0144] In the process of using the key data features to output the cross-modal joint features corresponding to the multi-modal data, the method further includes:

[0145] S8-2, iteratively update the model parameters of the cross-modal model until the model convergence conditions are met to obtain a trained cross-modal model.

[0146] In an optional embodiment, the cross-modal model may be, but is not limited to, a neural network model capable of processing and understanding data of different modalities.

[0147] In an optional embodiment, the modal sample data may refer to, but is not limited to, a sample data set containing information of multiple modalities, and may be used to train a cross-modal model so that it can understand and generate association information between different modalities, but is not limited to.

[0148] In an optional embodiment, model parameters may refer to, but are not limited to, learnable variables such as weights and biases in a neural network, and may be used to determine, but are not limited to, the behavior and performance of the model.

[0149] In an optional embodiment, the model convergence condition may be, but is not limited to, a standard or threshold. When the performance indicator of the model reaches this condition, it is considered that the model has learned the characteristics of the sample data, and the iterative update can be stopped to avoid overfitting or waste of resources.

[0150] It should be noted that by obtaining multimodal sample data containing images and texts, or texts and images, these data are the basis for training cross-modal models. Then, the model uses the key data features in these data to learn and generate cross-modal joint features. This process involves multiple steps such as feature extraction, feature fusion, and feature optimization. During the model training process, it is necessary to continuously iterate and update the model parameters, and gradually improve the performance of the model through optimization algorithms such as back propagation and gradient descent until the model convergence conditions are met. When the performance indicators of the model on the training set no longer change significantly, it is considered that the model has learned the characteristics of the sample data. At this time, training can be stopped to obtain a trained cross-modal model. Through this iterative update and convergence check strategy, it can be ensured that the model can be continuously optimized during the training process, improve its ability to understand and generate cross-modal features, and thus improve the processing efficiency of multimodal data.

[0151] Through the embodiment of the present application, the input multimodal data is obtained, including: obtaining multimodal sample data for input into a cross-modal model, wherein the cross-modal model is used to output corresponding description text from image input, or to generate corresponding description images from text input; in the process of using key data features to output cross-modal joint features corresponding to the multimodal data, the method also includes: iteratively updating the model parameters of the cross-modal model until the model convergence conditions are met to obtain a trained cross-modal model. Thus, the technical purpose of training a cross-modal model through multimodal data is achieved, and the technical effect of improving the processing efficiency of multimodal data is achieved.

[0152] As an optional solution, for ease of understanding, the multimodal data processing method is applied to the cross-modal learning scenario.

[0153] In an optional embodiment, a method for joint understanding of text and images based on cross-modal learning aims at the problem of information redundancy in the local features of extracted images and texts during feature decoupling and reconstruction. A feature selection module and sparse regularization are used: the feature selection module selects important features during the decoupling process, and sparse regularization further weakens redundant information in feature distribution, making feature representation more concise and effective; multi-layer decoupling and multi-modal contrast attention are used: multi-layer decoupling can capture semantics at different levels, and combined with the cross-modal contrast attention mechanism, it helps to achieve more detailed feature alignment and fusion, and comprehensively optimize the image-text relationship structure; in cross-modal retrieval tasks, the use of feature selection modules and sparse regularization can effectively reduce redundant features, thereby optimizing feature decoupling and reconstruction effects. The following are specific implementation steps:

[0154] Step S9-1, feature extraction

[0155] The forward and backward features of each word are extracted using bidirectional gated recurrent units, recurrent neural networks, and Transformer models. The mean is used as the word feature, and then combined to obtain the local features of the text. The target detection model FasterR-CNN is used to extract the target area features of the image. The target area features are mapped to the local features of the image through a fully connected network. Transformer can be, but is not limited to, an architecture based on the attention mechanism.

[0156] Step S9-2, constructing a feature selection module

[0157] Introducing a gating mechanism: Using the gating mechanism in the gated recurrent unit to dynamically control the flow of features, filter out features that are helpful for cross-modal retrieval, and build a weight (or gate) for each image and text feature. These weights can be generated by a separate network layer, and the sigmoid activation function is used to generate weights between 0 and 1, indicating the degree of feature retention;

[0158] Apply feature selection weights: Multiply the local features of the image and text by the generated weights to screen out more important features and filter out irrelevant or unimportant features. Assuming there are image features V and text features T, after using the feature selection weights, the features obtained are V'=V·G(V) and T'=T·G(T), where G represents the weight generated by the feature selection module, thereby further removing redundant image and text features and retaining more discernible features;

[0159] To further illustrate, the above steps S9-1 and S9-2 are shown in FIG3. First, text feature extraction is performed. After the text feature extraction, the local features of the text are generated through a bidirectional gated recurrent unit or a Transformer model. While the text feature is extracted, image feature extraction is performed, and then the local features of the image are obtained through the target extraction model Faster R-CNN. Then, the local features of the text and the local features of the image are input into the feature selection module to generate feature selection weights, and then the weighted features are obtained, followed by sparse regularization processing, and finally the features are fused to generate joint features.

[0160] Step S9-3, sparse regularization processing

[0161] Apply L1 regularization: Add L1 regularization to the feature selection module to sparse the distribution of weights, causing some weight values ​​to approach zero, further reducing redundant information. By introducing the L1 regularization term in the loss function, the generated feature selection weight matrix is ​​made sparse, thereby controlling the number of selected features.

[0162] The goal is to minimize the loss function L=L task +λ∑|G|, where L task is the task loss (such as classification loss or retrieval loss), λ is the weight coefficient of L1 regularization, which is used to control the strength of sparse regularization;

[0163] Gradient descent update: In each training iteration, the gradient descent algorithm is applied to update the model parameters and feature selection weights at the same time. L1 regularization will gradually make the weights of irrelevant features tend to zero, thereby achieving a sparse effect;

[0164] To further illustrate, the above step S9-3 is as follows Figure 4 As shown in the figure, first input the features, then generate feature selection weights through the feature selection module, then perform feature weighting, add the weighted features to L1 regularization, calculate the regularization loss, then optimize the target loss function, and then update through gradient descent to obtain sparse features, and finally output streamlined features.

[0165] Step S9-4, reconstructing features and generating joint features

[0166] Combine the local features of the image and text after feature selection and sparse regularization to form a more streamlined feature vector for cross-modal feature fusion;

[0167] Use cross-modal fusion methods such as graph convolutional neural networks (GCNs) to model the relationship between image and text features and generate joint features with contextual and semantic information;

[0168] This joint feature contains the key relationship structure between images and texts and removes redundant information, which helps improve the accuracy of cross-modal retrieval;

[0169] To further illustrate, the above step S9-3 is as follows Figure 5 As shown in the figure, the streamlined image and text features are input into the graph convolutional neural network to generate joint features, which are then generated through cross-modal relationship modeling, and finally cross-modal retrieval optimization is performed.

[0170] Step S9-5, training and optimization

[0171] Combined with the sparse regularized objective loss function for joint training, the feature selection module and feature fusion model are optimized using back propagation and gradient descent algorithms, and the L1 regularization weight coefficient is adjusted during the optimization process. To ensure a balance between the effect of feature sparsification and model performance.

[0172] In an optional embodiment, the image-text relationship structure is optimized by multi-layer decoupling and multi-modal contrast attention, and the specific steps include:

[0173] Step S10-1, multi-layer decoupling module

[0174] Multi-layer decoupling is to capture semantic information in multi-modal (such as images and text) at different levels, so that the model can perform more detailed decoupling at the feature level to improve the understanding of multi-level semantics. The following is a specific implementation method:

[0175] 1) Hierarchical feature extraction: For text, use bidirectional LSTM, GRU or Transformer encoders to perform multi-level encoding on the text to extract the context information of each word, and then encode layer by layer to generate features at different semantic levels. For images, convolutional neural networks can be used to extract low-level and high-level features of the image. The features extracted after each layer of convolution represent the feature representation of different detail granularity (such as edges, textures, semantic objects, etc.);

[0176] 2) Selection of decoupling layers: For the features extracted at each layer, different decoupling modules (such as self-attention or residual connection) are applied to decouple independent feature components to ensure that the semantic features of each layer can be represented and optimized separately. Feature decoupling loss is used to further decouple the feature layers and avoid information redundancy between features at different levels.

[0177] To further illustrate, the above step S10-1 is as follows Figure 6As shown in the figure, when text feature decoupling output is required, text feature extraction is performed first, and text-level features are obtained through bidirectional LSTM / GTU / Transformer. Then, the text features are optimized through the decoupling module, and finally the text features are decoupled and output. When image feature decoupling output is required, image feature extraction is performed first, and image-level features are obtained through the convolutional neural network. Then, the image features are optimized through the decoupling module, and finally the image features are decoupled and output.

[0178] Step S10-2, cross-modal contrastive attention mechanism

[0179] The cross-modal contrastive attention mechanism aims to achieve more fine-grained feature alignment and information fusion between different modalities to comprehensively optimize the relationship structure between images and texts. The implementation methods include:

[0180] 1) Intra-modal contrastive attention: In text features and image features, self-attention calculation is performed on the features of each modality respectively, and the similarity between features is calculated by dot multiplication to capture the mutual relationship of features within the modality, calculate the weight of each feature vector within the modality, and obtain the enhanced feature representation within the modality. The self-attention formula is used:

[0181]

[0182] Q: Query matrix, which is the input features to be aligned;

[0183] K: key matrix (Key), used to calculate similarity;

[0184] V: Value matrix, containing information about input features;

[0185] d k : The dimension of the key vector, used for scaling to avoid gradients that are too large or too small;

[0186] For example, suppose we are processing a text sequence and inputting the feature representation of 4 words. Each feature is a 3-dimensional vector. Suppose these 4 words become the following matrix X after the embedding layer:

[0187]

[0188] In order to generate the query, key, and value matrix, we need to define the query weight matrix W Q , key weight matrix W K Sum value weight matrix W V , assuming these weight matrices are:

[0189]

[0190] Calculate the query matrix Q:

[0191]

[0192] Calculate the key matrix K:

[0193]

[0194] Calculate the value matrix V:

[0195]

[0196] Calculate the attention score, calculate the dot product of Q and K, then scale it, and then use the softmax function to get the attention score:

[0197]

[0198] Assumption d k =3, divide the dot product result ≈1.732, where the square root of 3 is approximately equal to 1.732, and then apply softmax to each row:

[0199]

[0200] After softmax conversion, we get the attention score matrix, which is multiplied by the value matrix V to get the self-attention output:

[0201]

[0202] Through this process, the self-attention mechanism can fuse information according to the similarity weight, focus related features together, and achieve feature alignment and optimization.

[0203] 2) Inter-modal contrastive attention, using the interactive attention layer to establish correspondence between features of different modalities. Contrastive attention first calculates the similarity matrix of image features and text features, reflecting the similarity between image regions and text words. Based on the similarity matrix, the features with high similarity are aligned and fused to generate cross-modal features. In the implementation, the "similarity allocation" method is used to strengthen high-similar regions and weaken low-similar regions. Cross-modal contrastive loss (such as contrastive learning loss) is introduced to ensure that the aligned image and text features are more consistent in similarity.

[0204] To further illustrate, the above step S10-2 is as follows: Figure 7As shown in the figure, self-attention is calculated for text features and image features at the same time to obtain intra-text contrast attention and intra-image contrast attention respectively, and then enhanced text features are obtained by intra-text contrast attention, and enhanced image features are obtained by intra-image contrast attention. Then, interactive attention is obtained based on the enhanced text features by calculating the similarity matrix, and interactive attention is obtained based on the enhanced image features. Then, aligned image and text features are obtained based on the interactive attention, and finally cross-modal features are output.

[0205] Step S10-3, fusion and feature alignment

[0206] The cross-modal alignment features at each level are fused to further optimize the relationship structure between images and texts. The alignment features at each level are aggregated (such as weighted summation and concatenation) to integrate semantic information at different levels. Residual connections or skip connections are used for each layer of features to retain the original low-level and high-level semantic details.

[0207] The aligned features are restructured using a graph convolutional neural network (GCN) to better capture the relationship between images and text. The reconstructed features are processed with nonlinear activation to improve the expressiveness of the features.

[0208] To further illustrate, the above step S10-3 is as follows Figure 8 As shown in the figure, weighted summation / concatenation is performed on the aligned text features and the aligned image features to obtain the fused features, which are then reconstructed by the graph convolutional network to obtain the structured feature output, and then feature activation processing is performed to finally obtain the final joint features.

[0209] Step S10-4, training and optimization

[0210] The comprehensive use of multiple loss functions such as cross-modal contrastive loss, feature alignment loss, and reconstruction loss enables the model to simultaneously optimize the decoupling and alignment effects during training. During back propagation, the weights in the multi-layer decoupling and contrastive attention modules are updated layer by layer to ensure that the decoupling and contrastive attention of each layer can be fully trained, thereby achieving high-quality image-text feature alignment at all levels.

[0211] Through the above embodiments, multi-layer decoupling combined with multimodal contrastive attention can capture fine-grained semantic information at different levels, optimize the relational structure of image and text modalities, and make the cross-modal retrieval results more accurate.

[0212] Through the embodiments of the present application, first, through the combination of feature selection module and sparse regularization, the scheme can effectively screen out the most representative features in the decoupling process, and further weaken the redundant information through the sparse regularization mechanism, so that the feature representation is more concise and efficient. The feature selection module can automatically identify and retain important features related to the task, thereby reducing the interference of irrelevant or redundant features and ensuring that the model focuses on the features that contribute the most to the task. Sparse regularization introduces the L1 regularization term in the loss function, so that the weights of unimportant features gradually tend to zero, further compresses the feature space, reduces the computational complexity, and improves the interpretability of the model. In this way, the feature representation becomes more concise and sparse, avoids the accumulation of redundant information, promotes more accurate decision-making and reasoning of the model in high-dimensional feature space, and enhances the generalization ability of the model, reduces the risk of overfitting, thereby improving the effect and accuracy of cross-modal retrieval.

[0213] Furthermore, through the multi-layer decoupling and multimodal contrastive attention scheme, the different levels of semantic information of the image and text modalities can be effectively captured and fused, thereby improving the accuracy and robustness of cross-modal retrieval. Multi-layer decoupling captures the multi-level and fine-grained semantics in the features, allowing the model to identify the hierarchical relationship within the modality, avoiding the information loss and redundancy caused by simple feature fusion. Combined with the cross-modal contrastive attention mechanism, the model's ability to recognize the similarities and differences between modalities is enhanced through contrastive learning, allowing the model to more accurately align similar content in images and texts, while distinguishing irrelevant features in different modalities and effectively reducing noise. This scheme makes full use of the relational structure information of each modality to form a more expressive joint feature representation in the semantic space, which helps to improve the refinement of feature alignment and the understanding of complex image-text relationships, thereby improving the accuracy and generalization of cross-modal retrieval systems.

[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0215] In this embodiment, a multimodal data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0216] Fig. 9 is a structural block diagram of a multimodal data processing device according to an embodiment of the present application, such as Fig. 9 As shown, the device comprises:

[0217] A first acquisition unit 902 is used to acquire input multimodal data, wherein the multimodal data includes at least two modal type data;

[0218] A second acquisition unit 904 is used to acquire a plurality of initial data features corresponding to each of the at least two modality type data;

[0219] A screening unit 906 is used to screen the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature of the initial data features that can characterize the preset information of the multimodal data;

[0220] The output unit 908 is used to output the cross-modal joint features corresponding to the multimodal data using the key data features.

[0221] As an optional solution, the screening unit 906 includes: a construction module, which is used to construct a corresponding plurality of key weights for each of the plurality of initial data features, wherein the key weights are used to indicate the degree of retention of the data features; an integration module, which is used to integrate each of the initial data features with the corresponding key weights to obtain an integration result corresponding to each of the initial data features; and a screening module, which is used to perform feature screening on the plurality of initial data features using the integration result to obtain at least one key data feature.

[0222] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0223] As an optional solution, a construction module includes: an input submodule for inputting multiple initial data features into a gated recurrent unit; a capture submodule for capturing long-term dependencies in a feature sequence through a gating mechanism of the gated recurrent unit; a first acquisition submodule for utilizing the long-term dependencies to acquire a hidden state output by the gated recurrent unit at each time step, wherein the hidden state includes all historical information up to the current time step; and a mapping submodule for mapping the hidden state into a weight vector, wherein each element in the weight vector corresponds to each key weight among multiple key weights.

[0224] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0225] As an optional solution, the construction module includes: a regulating submodule for adjusting the sparse regularization strength of multiple key weights using the weight coefficient of L1 regularization, wherein the weight coefficient is obtained by iteratively updating by applying a gradient descent algorithm.

[0226] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0227] As an optional solution, the output unit 908 includes: an extraction module, which is used to perform multi-level feature extraction on key data features to obtain multi-layer features; a decoupling module, which is used to decouple each layer of features in the multi-layer features using different decoupling modules to obtain independent semantic features of each layer; and an output module, which is used to use independent semantic features to output cross-modal joint features.

[0228] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0229] As an optional solution, the extraction module includes: a first extraction submodule, which is used to perform multi-level encoding on the text key features in the key data features to extract the context information of each word; and an encoding submodule, which is used to encode the text key features layer by layer in combination with the context information to obtain text features at different semantic levels.

[0230] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0231] As an optional solution, the extraction module includes: a second extraction submodule, which is used to extract low-level features and high-level features of the image after multi-layer convolution from the image key features in the key data features, wherein the features extracted after multi-layer convolution respectively represent feature representations of different detail granularities.

[0232] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0233] As an optional scheme, the output unit 908 includes: a first fusion module, used to perform feature fusion on multiple first features belonging to the same modality type in the key data features to obtain N first cross-modal features, where N is a positive integer; a second fusion module, used to perform feature fusion on multiple second features belonging to different modality types in the key data features to obtain M second cross-modal features, where M is a positive integer; a first acquisition module, used to acquire cross-modal joint features based on the N first cross-modal features and the M second cross-modal features.

[0234] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0235] As an optional scheme, the first fusion module includes: a determination submodule, used to determine the first current feature and the second current feature from multiple first features; a calculation submodule, used to calculate the dot product of the query matrix and the key matrix to obtain a dot product result, wherein the query matrix contains the first current feature and the second current feature, and the key matrix is ​​used to calculate the similarity with the query matrix; a conversion submodule, used to convert the dot product result into an attention weight, wherein the attention weight is used to represent the feature similarity between the first current feature and the second current feature; a summation submodule, used to perform weighted summation of the attention weight and the value matrix to obtain a weighted summation result, wherein the value matrix contains specific information of the first current feature and the second current feature; a first fusion submodule, used to use the weighted summation result to fuse the first current feature and the second current feature to obtain a first cross-modal feature.

[0236] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0237] As an optional solution, the second fusion module includes: a second acquisition submodule, used to obtain a similarity matrix between multiple second features, wherein the similarity matrix is ​​used to reflect the similarity between the multiple second features; a second fusion submodule, used to fuse features with high similarity among the multiple second features based on the similarity matrix to obtain M second cross-modal features.

[0238] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0239] As an optional solution, the first acquisition unit 902 includes: a second acquisition module, used to acquire multimodal sample data input into a cross-modal model, wherein the cross-modal model is used to output corresponding description text from an image input, or to generate a corresponding description image from a text input; an iteration module, used to iteratively update model parameters of the cross-modal model until the model convergence conditions are met, thereby obtaining a trained cross-modal model.

[0240] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for processing multimodal data, which will not be described in detail in this example.

[0241] It should be noted that the above-mentioned virtual devices (modules, units, sub-modules, sub-units, components, etc.) can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned virtual devices are all located in the same processor; or, the above-mentioned virtual devices are located in different processors in any combination.

[0242] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0243] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0244] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0245] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0246] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0247] An embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is configured to be executed by a processor, the computer program performs the steps in any of the above method embodiments.

[0248] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0249] Obviously, those skilled in the art should understand that the above-mentioned virtual devices or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0250] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing multimodal data, characterized in that: include: Acquire input multimodal data, wherein the multimodal data includes at least two modal type data; Acquire a plurality of initial data features corresponding to each of the at least two modality type data; Performing feature screening on the multiple initial data features to obtain at least one key data feature, wherein the key data feature is a feature of the initial data features that can characterize preset information of the multimodal data; Utilizing the key data features, output cross-modal joint features corresponding to the multimodal data.

2. The method according to claim 1, characterized in that Before screening the multiple initial data features to obtain at least one key data feature, the method further includes: Constructing a corresponding plurality of key weights for each of the plurality of initial data features, wherein the key weights are used to indicate a degree of retention of the data features; Integrate the initial data features with the corresponding key weights to obtain integration results corresponding to the initial data features; The performing feature screening on the multiple initial data features to obtain at least one key data feature includes: utilizing the integration result to perform feature screening on the multiple initial data features to obtain the at least one key data feature.

3. The method according to claim 2, characterized in that The step of constructing a corresponding plurality of key weights for each of the plurality of initial data features comprises: Inputting the plurality of initial data features into a gated recurrent unit; Capturing long-term dependencies in feature sequences through the gating mechanism of the gated recurrent unit; Using the long-term dependency, obtaining the hidden state output by the gated recurrent unit at each time step, wherein the hidden state includes all historical information up to the current time step; The hidden state is mapped into a weight vector, wherein each element in the weight vector corresponds to a respective key weight of the plurality of key weights.

4. The method according to claim 2, characterized in that: In the process of constructing a corresponding plurality of key weights for each of the plurality of initial data features, the method further comprises: The sparse regularization strength of the multiple key weights is adjusted by using the weight coefficient of L1 regularization, wherein the weight coefficient is obtained by iteratively updating by applying a gradient descent algorithm.

5. The method according to claim 1, characterized in that The utilizing the key data features to output cross-modal joint features corresponding to the multimodal data includes: Performing multi-layer feature extraction on the key data features to obtain multi-layer features; Decoupling each layer of features in the multi-layer features using different decoupling modules to obtain independent semantic features at each level; The cross-modal joint feature is outputted by utilizing the independent semantic feature.

6. The method according to claim 5, characterized in that The multi-layered feature extraction of the key data features to obtain multi-layer features includes: Performing multi-level encoding on the text key features in the key data features to extract context information of each word; Combined with the context information, the key features of the text are encoded layer by layer to obtain text features at different semantic levels.

7. The method according to claim 5, characterized in that The multi-layered feature extraction of the key data features to obtain multi-layer features includes: From the key features of the image in the key data features, low-level features and high-level features obtained after multi-layer convolution of the image are extracted, wherein the features extracted after the multi-layer convolution respectively represent feature representations of different detail granularities.

8. The method according to claim 1, characterized in that The utilizing the key data features to output cross-modal joint features corresponding to the multimodal data includes: Performing feature fusion on multiple first features of the same modality type in the key data features to obtain N first cross-modal features, where N is a positive integer; Performing feature fusion on multiple second features belonging to different modal types in the key data features to obtain M second cross-modal features, where M is a positive integer; The cross-modal joint feature is obtained based on the N first cross-modal features and the M second cross-modal features.

9. The method according to claim 8, characterized in that The step of fusing multiple first features of the same modality type in the key data features to obtain N first cross-modal features includes: Perform the following steps until the N first cross-modal features are obtained: Determining a first current feature and a second current feature from the plurality of first features; Calculating a dot product of a query matrix and a key matrix to obtain a dot product result, wherein the query matrix includes the first current feature and the second current feature, and the key matrix is ​​used to calculate similarity with the query matrix; Converting the dot product result into an attention weight, wherein the attention weight is used to represent the feature similarity between the first current feature and the second current feature; Performing weighted summation on the attention weight and the value matrix to obtain a weighted summation result, wherein the value matrix includes specific information of the first current feature and the second current feature; The first current feature and the second current feature are fused using the weighted sum result to obtain the first cross-modal feature.

10. The method according to claim 9, characterized in that The step of fusing multiple second features of different modal types in the key data features to obtain M second cross-modal features includes: Acquire a similarity matrix between the plurality of second features, wherein the similarity matrix is ​​used to reflect the similarity between the plurality of second features; Based on the similarity matrix, features with high similarity among the multiple second features are fused to obtain the M second cross-modal features.

11. The method according to any one of claims 1 to 10, characterized in that The obtaining of input multimodal data includes: obtaining multimodal sample data for input into a cross-modal model, wherein the cross-modal model is used to output corresponding description text from image input, or to generate corresponding description images from text input; In the process of using the key data features to output the cross-modal joint features corresponding to the multimodal data, the method also includes: iteratively updating the model parameters of the cross-modal model until the model convergence conditions are met to obtain the trained cross-modal model.

12. A multimodal data processing device, characterized in that: include: A first acquisition unit, configured to acquire input multimodal data, wherein the multimodal data includes at least two modal type data; A second acquisition unit, configured to acquire a plurality of initial data features corresponding to each of the at least two modality type data; A screening unit, configured to perform feature screening on the plurality of initial data features to obtain at least one key data feature, wherein the key data feature is a feature of the initial data features that can characterize preset information of the multimodal data; An output unit is used to output the cross-modal joint features corresponding to the multimodal data using the key data features.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 11 are implemented.

Images