Multi-modal feature fusion method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of machine vision, and discloses a multi-modal feature fusion method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: extracting feature vectors of preprocessed target data, the feature vectors comprising visual features and text feature vectors; calling the initialized convolution operator to carry out dimension reduction operation on the visual features to obtain a one-dimensional feature tensor; and splicing and fusing the one-dimensional feature tensor and the text feature vector to obtain a visual text fusion feature. Through a central diffusion mode, a two-dimensional feature vector is converted into a one-dimensional feature vector, more visual information is reserved for fusion, and a large language model is helped to better understand the visual information, so that the performance of the visual language model is improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a multimodal feature fusion method, device, equipment, storage medium and computer program product. Background Art

[0002] The current visual language model adapter solution aligns visual features with sequence features through simple and rough flattening, which loses a lot of visual information and causes the performance of the visual language model to deteriorate. Summary of the invention

[0003] The main purpose of this application is to provide a multimodal feature fusion method, apparatus, device, storage medium and computer program product, aiming to solve the technical problem that the current visual language model adapter solution aligns visual features with sequence features through simple and rough flattening, loses a large amount of visual information, and causes the performance of the visual language model to decline.

[0004] To achieve the above objectives, the present application proposes a multimodal feature fusion method, which includes:

[0005] Extracting a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector;

[0006] Calling the initialized convolution operator to perform dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor;

[0007] The one-dimensional feature tensor is concatenated and fused with the text feature vector to obtain a visual-text fusion feature.

[0008] Optionally, the step of calling the initialized convolution operator to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor includes:

[0009] The central parts of the preset initial operator are set as the preset initial threshold to obtain the initialized convolution operator;

[0010] Determine the convolution kernel of each dimension in the convolution operator;

[0011] The convolution kernel is convolved with a feature region of a corresponding dimension in the visual feature to obtain each visual feature value, and the visual feature values ​​are concatenated in sequence to obtain a one-dimensional feature tensor.

[0012] Optionally, after the step of concatenating and fusing the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature, the step further includes:

[0013] Generate a first text sequence according to the visual-text fusion feature;

[0014] Calculating the probability distribution of each text answer in the first text sequence, and determining the first text answer with the highest probability in the first text sequence;

[0015] Comparing the probability of the first text answer with a preset probability threshold to obtain a comparison result;

[0016] If the comparison result is that the probability of the first text answer is greater than a preset probability threshold, the first text answer is post-processed and delivered to the target user.

[0017] Optionally, after the step of comparing the probability of the first text answer with a preset probability threshold to obtain a comparison result, the method further includes:

[0018] If the comparison result is that the probability of the first text answer is less than a preset probability threshold, the first text answer is concatenated with the visual text fusion feature to obtain a target fusion feature;

[0019] Generate a second text sequence according to the target fusion feature;

[0020] Calculating the probability distribution of each text answer in the second text sequence, and determining the second text answer with the highest probability in the second text sequence;

[0021] When the probability of the second text answer is greater than a preset probability threshold, the second text answer is post-processed and delivered to the target user.

[0022] Optionally, the step of extracting the feature vector of the preprocessed target data includes:

[0023] Acquire initial data input by a target user, wherein the initial data includes image data and text data;

[0024] Adjusting the image data in the initial data based on a preset image input threshold, and inputting the adjusted image data into a convolutional neural network to obtain a multi-dimensional feature tensor;

[0025] Processing the text data according to the Transformer model to generate a text feature vector;

[0026] The multidimensional feature tensor and the text feature vector are combined to obtain a feature vector of the target data.

[0027] Optionally, before the step of processing the text data according to the Transformer model to generate a text feature vector, the step further includes:

[0028] Performing word segmentation processing on the sentences in the text data to obtain text segmentation;

[0029] The text segmentation words are replaced with target index words according to a preset vocabulary, and a text feature vector is generated according to the target index words and the Transformer model.

[0030] In addition, to achieve the above purpose, the present application also proposes a multimodal feature fusion device, the multimodal feature fusion device comprising:

[0031] A feature extraction module, used to extract a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector;

[0032] A dimensionality reduction operation module, used for calling the initialized convolution operator to perform dimensionality reduction operation on the visual features to obtain a one-dimensional feature tensor;

[0033] The feature fusion module is used to concatenate and fuse the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a multimodal feature fusion device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the multimodal feature fusion method described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal feature fusion method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal feature fusion method described above.

[0037] The present application discloses extracting feature vectors of preprocessed target data, wherein the feature vectors include visual features and text feature vectors; calling the initialized convolution operator to perform dimensionality reduction operations on the visual features to obtain a one-dimensional feature tensor; splicing and fusing the one-dimensional feature tensor with the text feature vector to obtain visual text fusion features. Through a central diffusion method, the two-dimensional feature vector is converted to one-dimensional, retaining more visual information for fusion, helping the large language model to better understand the visual information, thereby improving the effect of the visual language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flow chart of the first embodiment of the multimodal feature fusion method of the present application;

[0041] Figure 2 This is a demonstration diagram of the traditional dimensionality reduction method;

[0042] Figure 3 An example diagram of the convolution operator used in this application;

[0043] Figure 4 A schematic diagram of a dimensionality reduction method in the multimodal feature fusion method of this application;

[0044] Figure 5 Flowchart of VLM algorithm reasoning for this application;

[0045] Figure 6 This is a flow chart of the second embodiment of the multimodal feature fusion method of the present application;

[0046] Figure 7 This is a flow chart of the third embodiment of the multimodal feature fusion method of the present application;

[0047] Figure 8 This is a schematic diagram of the module structure of the multimodal feature fusion device according to an embodiment of the present application;

[0048] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the multimodal feature fusion method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: extracting the feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector; calling the initialized convolution operator to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor; splicing and fusing the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature.

[0053] Since the current multimodal large model is frequently used in the field of vision, the basic process of the visual multimodal large model (VLM, Vision-Language Model) is as follows: first, the basic visual features are extracted from the video or picture through VIT (Vision Transformer) and the like, and then the two-dimensional image features are converted into one-dimensional sequence features through a modality adapter to achieve the same purpose as the one-dimensional Prompt (feature) dimension, which is convenient for subsequent splicing operations. After completion, the new Prompt obtained by splicing and fusion of language Prompt and visual feature Prompt is input into the LLM (LargeLanguage Model) large model to complete the visual tasks such as recognition and positioning through the large model. In the above process, most of the current VLM adapters use MLP (Multilayer Perceptron) or other simple network structures, and directly stretch the two-dimensional features into one dimension through simple feature processing. However, it can be found that in this stretching process, a lot of spatial information, feature dependency information, transition information and other visual-related information are lost, resulting in the problem of low generation effect after the visual features are forcibly aligned with the sequence information.

[0054] Therefore, this application provides a new adapter network structure, which converts the two-dimensional feature matrix into one dimension through a central diffusion method, retains more visual information, helps LLM better understand visual information, and thus improves the effect of the VLM model.

[0055] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a large language model as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a multimodal feature fusion method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the multimodal feature fusion method of the present application.

[0057] In this embodiment, the multimodal feature fusion method includes:

[0058] Step S10, extracting a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector.

[0059] It should be noted that target data refers to multimodal data including image data and text data. Data is the object of processing and analysis by the model, aiming to extract valuable feature information from it to achieve vision-related tasks such as visual question answering and image description generation. Feature vectors are used to describe certain features or attributes of data. In the context of multimodal feature fusion, feature vectors are divided into visual feature vectors and text feature vectors. Visual feature vectors are an abstract representation of visual information in image data, which summarizes various visual attributes of the image such as color, shape, texture, etc.; text feature vectors are quantitative expressions of semantic information of text data, which can capture key information such as vocabulary, grammar, and semantic relationships in the text.

[0060] It can be understood that the feature vector of the target data after preprocessing can be the feature vector of the image data and the text data. At the same time, before the extraction, the target data is preprocessed, and the image data is preprocessed, including operations such as scaling, cropping and normalization. After preprocessing, a convolutional neural network (such as VIT) is used to extract visual features from the image. The VIT model divides the image into multiple regions and generates corresponding feature representations for each region, and finally outputs a tensor feature representing different regions or objects of the image, whose size is (1024, 21, 21), which is the visual feature vector. The text data is preprocessed, including word segmentation, encoding and padding. After these processes, the text input is processed by the Transformer model to generate a corresponding text feature vector (Text Prompt). The Transformer model uses its self-attention mechanism and other technologies to capture the semantic relationship between words in the text, thereby generating a feature vector that can accurately express the text semantic information, and provide feature information of the text modality for subsequent feature fusion.

[0061] Step S20, calling the initialized convolution operator to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor.

[0062] It can be understood that the convolution operator is a tool for processing visual features, which is composed of multiple convolution kernels of different sizes, such as 9*9, 11*11 and 21*21. The one-dimensional feature tensor is a data structure obtained after dimensionality reduction operation. The two-dimensional visual feature tensor is converted into a one-dimensional form, which is more suitable for fusion with the text feature vector.

[0063] It should be understood that by setting each central part in the preset initial operator to the preset initial threshold, the initialized convolution operator can ensure that each operator only needs to operate on a specific part of the input when the convolution operator based on the central diffusion idea is operated. For example, initializing the central part to 0 can ensure that in subsequent operations, the operator will not process values ​​that are not within its own control range, thereby ensuring the accuracy and independence of the feature extraction process that spreads from the center to the surroundings.

[0064] In one example, reference Figure 2 and Figure 3 , Figure 2 This is a demonstration diagram of the traditional dimensionality reduction method. Figure 3 An example diagram of the convolution operator used in this application. Figure 1 The dimensionality reduction method used by the current mainstream VLM algorithm is shown. Its core operation is to directly flatten the two-dimensional feature matrix, with the aim of reducing the two-dimensional visual features to one dimension so as to align them with the sequence features. In this process, no other special processing is performed on the features, but the dimensional form of the features is simply changed. When the model obtains a two-dimensional visual feature matrix, such as a visual feature size of (1024, 21, 21), the elements in the matrix can be directly arranged in a certain order (usually row by row or column by column) to form a one-dimensional vector. For example, for a simple 2x2 two-dimensional matrix [[a, b], [c, d]], after flattening, a one-dimensional vector [a, b, c, d] will be obtained. Although this method is low-cost, it has obvious defects. First, the simple flattening operation may lead to the loss of important spatial information and local features, thereby affecting the model's understanding of the image content. Second, the high-dimensional sparsity after flattening the features reduces the model's expressiveness in the subsequent processing process, resulting in a decrease in the performance of visual tasks (such as recognition and positioning).

[0065] Figure 2 The structure of a convolution operator used in this application is shown. In the figure, rdm indicates random initialization of the position. The core idea of ​​the operator is to extract image features through a central diffusion idea. This process implies a cognitive process through time sequence, which is consistent with the thinking process of the human body when recognizing images, so as to be consistent with the text prompt in time sequence. There are 11 convolution kernels in the algorithm, including 1*1, 3*3, 5*5, 7*7, 9*9, 11*11 to 21*21 convolution kernels. 16 groups of these 11 convolution kernels are initialized in the algorithm, and the value obtained by each convolution operator is located in the 16 positions where the convolution kernel is located. For example, the value obtained by the 1*1 convolution kernel is filled in the first 16 positions, and the value of the 3*3 operator is located in the second group of 16 positions, and so on. A feature sequence of length 11*16=176 is obtained in total.

[0066] Of course, in order to use the convolution operator based on the center diffusion idea to perform dimensionality reduction operation, the step S20 may include:

[0067] The central parts of the preset initial operator are set as the preset initial threshold to obtain the initialized convolution operator; the convolution kernels of each dimension in the convolution operator are determined; the convolution kernel is convolved with the feature area of ​​the corresponding dimension in the visual feature to obtain each visual feature value, and the visual feature values ​​are concatenated in sequence to obtain a one-dimensional feature tensor.

[0068] It can be understood that in order to enable the convolution operator to accurately extract features based on the idea of ​​central diffusion when subsequently operating with visual features. By setting the central part to a preset initial threshold, it is ensured that each convolution kernel only operates on the input of its corresponding specific area to avoid interference with other areas. For example, for a 1*1 convolution kernel, when its center position (idx=11, idy=11) operates with the visual feature tensor, it only focuses on the features of the center position and will not be affected by other positions, where idx and idy are used to specify the position index in the visual feature tensor, and "idx=11, idy=11" indicates the position with a horizontal coordinate (or row index) of 11 in the two-dimensional visual feature tensor. For a 3*3 convolution kernel, when its central area (idx=[10, 12], idy=[10, 12]) is operated, the initialization method of the central part to 0 makes it mainly extract the features of a circle of areas around the center, without involving the features of the outer areas. At the same time, when processing an image containing a person and a background, the 1*1 convolution kernel can extract the local feature details of the person's face, the 3*3 convolution kernel can further obtain the feature relationship between the person's face and its surrounding hair, clothing and other adjacent parts, and larger convolution kernels such as 11*11 or 21*21 can capture the spatial relationship and overall features between the person as a whole and the background. Through their collaborative work, the visual features of the image can be extracted more comprehensively and accurately.

[0069] In one example, reference Figure 4 , Figure 4 This is a schematic diagram of the dimensionality reduction method in the multimodal feature fusion method of this application. First, the adapter obtains the tensor of size (1024, 21, 21) passed by VIT in the previous step, and the convolution kernel calculation begins. The 1*1 convolution kernel and the center white area are calculated to obtain V1, and the 3*3 operator and the center red area are calculated to obtain V3, and so on. Finally, a total of 11 calculation results from V1 to V21 are obtained. This group of results is spliced ​​to obtain a one-dimensional visual feature. The calculation process is as follows:

[0070] Kernel(1*1)*Tensor(idx=11, idy=11)=V1;

[0071] Kernel(3*3)*Tensor(idx=[10,12], idy=[10,12])=V3;

[0072] Kernel(5*5)*Tensor(idx=[9,14], idy=[9,14])=V5;

[0073] …

[0074] Kernel(21*21)*Tensor(idx=[1,21], idy=[1,21])=V21;

[0075] Among them, Kernel represents the size of the convolution kernel, Tensor represents a specific position, and Kernel(1*1)*Tensor(idx=11, idy=11)=V1 can be the multiplication and summation of the 1*1 convolution kernel and the elements at the center position of the visual feature tensor (idx=11, idy=11) to obtain the visual feature value V1.

[0076] Through the above operations, we obtain a one-dimensional tensor Vision Prompt = [V1, V3, V5, V7, V9, V11, V13, V15, V17, V19, V21], completing the dimensionality reduction operation of the visual features.

[0077] Step S30, concatenating and fusing the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature.

[0078] It is understandable that the one-dimensional feature tensor contains the visual feature information extracted from the image. After a series of operations such as dimensionality reduction by the convolution operator based on the center diffusion idea, important visual content such as the image's spatial information and feature dependency information is retained. The text feature vector is a quantitative expression of the semantic information of the text data. After being processed by the Transformer model, it can capture key information such as vocabulary, grammar, and semantic relationships in the text. The purpose of splicing and fusing the two is to integrate the information of the two different modalities of vision and text into a unified feature representation, so that the model can use both types of information to understand and process tasks at the same time. For example, in visual question-answering tasks, visual features can provide information such as objects and scenes in the image, and text features (questions) can guide the model to focus on specific aspects of the image, thereby answering questions more accurately. For example, if Vision Prompt is represented as [V1, V3, V5, …, V21] (where V1, V3, etc. are visual feature values ​​obtained by operating the convolution kernel with the visual feature tensor), and Text Prompt is represented as [T1, T2, T3, …, Tn] (where T1, T2, etc. are elements in the text feature vector), then the concatenated visual-text fusion features can be represented as [V1, V3, V5, …, V21, T1, T2, T3, …, Tn].

[0079] In one example, reference Figure 5 , Figure 5 This is a flowchart of the VLM algorithm reasoning for this application. The shape of the input image data IMG in the figure is (1024, 1024, 3), which means that the image is 1024 pixels wide, 1024 pixels high, and has 3 color channels. The image data is processed by the Visual Transformer (VIT), and the output feature shape becomes (1024, 21, 21). After the adapter module, the feature shape becomes (1024, 176) and is converted into a one-dimensional feature tensor. After completing the dimensionality reduction operation, Vision Prompt can be directly spliced ​​and fused with the one-dimensional Text Prompt (text feature vector) to complete the visual-text bimodal feature fusion. After feature fusion, the model enters the reasoning stage, and gradually generates descriptions for the fused features based on the large language model reasoning.

[0080] In this embodiment, the feature vector of the preprocessed target data is extracted, wherein the feature vector includes a visual feature and a text feature vector; the initialized convolution operator is called to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor; the one-dimensional feature tensor is concatenated and fused with the text feature vector to obtain a visual text fusion feature. Through a central diffusion method, the two-dimensional feature vector is converted to one dimension, retaining more visual information for fusion, helping the large language model to better understand the visual information, thereby improving the effect of the visual language model.

[0081] Reference Figure 6 , Figure 6 This is a flow chart of the second embodiment of the multimodal feature fusion method of the present application. Based on the above-mentioned first embodiment, the second embodiment of the multimodal feature fusion method of the present application is proposed.

[0082] In the second embodiment, after step S30, the method further includes:

[0083] Step S301: Generate a first text sequence according to the visual-text fusion feature.

[0084] It should be noted that the first text sequence is text content generated based on visual-text fusion features through a specific generation algorithm (such as a large language model or other text generation model implementation), which can be an ordered sequence of characters or words, which can describe scenes, objects, events, etc. related to the visual content to a certain extent, or perform some operations on the visual content (such as answering questions about the visual content).

[0085] It is understandable that after decoding and obtaining the semantic representation, the model begins to generate the first text sequence and selects appropriate words one by one to construct the text sequence based on the information in the semantic representation. This selection process is usually based on a probability model, that is, the model calculates the probability of each word appearing in the current semantic environment, and then selects the word with the highest probability as the next element in the text sequence. For example, if the fused features contain the visual information of "red flowers" in the image and the text prompt of "describing flowers", the model may first select "this flower" as the beginning of the text sequence, and then continue to select words such as "with round petals" to complete the text sequence based on other visual features of the flower (such as petal shape, etc.) and further requirements of the text prompt.

[0086] Step S302: Calculate the probability distribution of each text answer in the first text sequence, and determine the first text answer with the highest probability in the first text sequence.

[0087] It is understandable that the probability distribution of each text answer in the first text sequence is usually calculated based on the probability calculation mechanism of the language model. In deep learning, the large language model learns the semantic relationship between words, grammatical structure, and common language patterns during the training process. With this learned knowledge, the model can estimate the probability of each possible text answer according to the given context (visual text fusion features and the generated first text sequence).

[0088] In one example, assuming that there are specific words in the vocabulary of the model, for each position in the first text sequence, the model will calculate the probability of the specific word appearing at that position. For example, for a simple question "What animals are in the picture?", the first text sequence generated may be "There is one in the picture", and the model will then calculate the probability of each word in the vocabulary (such as "cat", "dog", "bird", etc.) appearing at this position. If the model calculates that the probability of "cat" after "There is one in the picture" is 0.4, the probability of "dog" is 0.3, the probability of "bird" is 0.2, and the probabilities of other words are lower, then "cat" is selected as the first text answer because it has the highest probability under the current model calculation and is considered to be the most likely word to correctly describe the animal in the image.

[0089] Step S303: compare the probability of the first text answer with a preset probability threshold to obtain a comparison result.

[0090] It should be noted that by comparing the probability of the first text answer with the preset probability threshold, the reliability of the answer can be determined. By setting a probability threshold, the answers generated by the model can be preliminarily screened and evaluated to determine whether they meet a certain credibility standard.

[0091] It is understandable that the setting of the preset probability threshold needs to take into account the specific task requirements and data characteristics. If the task has high accuracy requirements, for example, in some key visual question-answering applications (such as visual question-answering in medical image diagnosis, image understanding and decision-related questions and answers in autonomous driving scenarios, etc.), a higher probability threshold needs to be set to ensure that the answers given by the model have high credibility and avoid serious consequences due to wrong answers. In some tasks that have relatively low accuracy requirements and focus more on diversity or exploration (such as creative image description generation, etc.), the probability threshold will be set relatively low to allow the model to generate some answers that are not extremely probable but have a certain degree of novelty or creativity.

[0092] Step S304: If the comparison result is that the probability of the first text answer is greater than a preset probability threshold, the first text answer is post-processed and delivered to the target user.

[0093] It is understandable that even if the probability of the first text answer is greater than the preset probability threshold, indicating that it has a high reliability, there may still be some problems with direct output. Post-processing can further optimize the answer to make it more in line with the user's needs and expectations and enhance the user experience. For example, in formal report generation, the answer needs to be converted into a specific format (such as a table, paragraph structure, etc.); in the interface for interaction with the user, the answer needs to be simplified or explanatory notes added for better understanding by the user.

[0094] Of course, in order to enhance the robustness and adaptability of the model, after step S303, the following steps may also be included:

[0095] If the comparison result is that the probability of the first text answer is less than a preset probability threshold, the first text answer is concatenated with the visual text fusion feature to obtain a target fusion feature; a second text sequence is generated according to the target fusion feature; the probability distribution of each text answer in the second text sequence is calculated, and the second text answer with the highest probability in the second text sequence is determined; when the probability of the second text answer is greater than the preset probability threshold, the second text answer is post-processed and delivered to the target user.

[0096] It should be noted that the target fusion feature is a new feature vector obtained by splicing the first text answer with the visual text fusion feature when the probability of the first text answer is less than a preset probability threshold. By combining the less reliable answer generated previously with the original fusion feature, the model can better understand the previous attempt during the re-reasoning process, so that it is possible to adjust the direction and content of the generated answer and increase the probability of generating a more accurate and reasonable answer. The second text sequence is the result of the model's re-reasoning and generation using the target fusion feature containing the first text answer information after the first generated first text answer is not ideal. It is the text content generated by the model's second attempt to obtain a more appropriate answer. The second text answer is the text content with the highest probability determined from the second text sequence. It is a new answer attempt generated by the model after iteration after considering the first text answer information (through the target fusion feature).

[0097] It is understandable that after obtaining the target fusion feature, the model regenerates the second text sequence based on this feature that contains more information (original fusion feature + first text answer). This process is similar to generating the first text sequence, but because the input fusion feature has changed, the model will re-evaluate and analyze the information in the feature and combine words in different ways to generate a new text sequence. Through such an iterative process (a new round of fusion feature construction and answer generation when the first text answer is not ideal), the model has the opportunity to continuously optimize the answer and improve the accuracy and reliability of the answer.

[0098] In one example, during the reasoning process based on the autoregressive paradigm, the model gradually calculates the probability distribution of the answer through the Prompt and the current image features, selects the answer with the highest probability in the current step, and then splices the generated answer into the Prompt, and generates the answer for the next step based on the new Prompt. This cycle continues until the model generates the answer with the highest probability as the end mark, representing the result of the reasoning process, and the LLM model outputs the generated result.

[0099] In this embodiment, a first text sequence is generated based on the visual text fusion feature; the probability distribution of each text answer in the first text sequence is calculated, and the first text answer with the highest probability in the first text sequence is determined; the probability of the first text answer is compared with a preset probability threshold to obtain a comparison result; if the comparison result is that the probability of the first text answer is greater than the preset probability threshold, the first text answer is post-processed and delivered to the target user. By generating a text sequence based on the visual text fusion feature and calculating the probability distribution of each text answer to determine the most likely answer, the result that best matches the visual and text information can be selected from a large number of candidate answers.

[0100] Reference Figure 7 , Figure 7 This is a flow chart of the third embodiment of the multimodal feature fusion method of the present application. Based on the above-mentioned second embodiment, the third embodiment of the multimodal feature fusion method of the present application is proposed.

[0101] In this embodiment, the step S10 includes:

[0102] Step S101, obtaining initial data input by a target user, wherein the initial data includes image data and text data.

[0103] It should be noted that the initial data is the original data input into the system or model by the target user, which includes image data and text data. Image data comes from various image resources owned by the target user, such as photos taken by the user, pictures downloaded from the Internet, image files stored in the local device, etc. Text data can be questions input by the user. If it is used for visual question answering tasks, the text data is the questions asked by the user, such as "What is the color of the animal in the picture?" "Who is the author of this painting?", etc. These questions guide the model to focus on specific aspects of the image and provide corresponding answers. If it is used for image description generation tasks, the text data may be the user's requirements for the description style or theme, such as "Please describe this picture in vivid language" "Describe the activities of the people in the picture", etc., or some initial prompt information.

[0104] Step S102, adjusting the image data in the initial data based on a preset image input threshold, and inputting the adjusted image data into a convolutional neural network to obtain a multi-dimensional feature tensor.

[0105] It should be noted that the preset image input threshold is one or a set of standard values ​​(or conditions) pre-set before processing the image data, which is used to determine whether the image data meets the requirements of subsequent processing (such as input to the convolutional neural network). It can involve multiple aspects of the image, such as size, pixel value range, resolution, etc. For example, the preset image input threshold may stipulate that the width and height of the image must be within a certain range (such as the width is not less than 500 pixels and not more than 2000 pixels, the height is not less than 500 pixels and not more than 2000 pixels), or the pixel value should be in a specific interval (such as [0, 255]). Ensure that the image data input to the convolutional neural network has appropriate specifications and quality so that the network can effectively perform operations such as feature extraction.

[0106] It is understandable that when the image data that has been adjusted to meet the preset image input threshold requirements is input into the convolutional neural network, the network will process the image in sequence according to its internal structure and algorithm. First, the convolution layer will perform multiple convolution operations on the image to extract local features of different levels and scales. These features will exist in the form of tensors (such as a two-dimensional feature matrix, etc.). As the network goes deeper, after processing by the pooling layer, etc., the features will be continuously integrated and transformed, and finally a multi-dimensional feature tensor will be obtained. This multi-dimensional feature tensor contains rich feature information extracted from the image, and its dimensions may be three-dimensional, four-dimensional, etc., such as (1024, 21, 21). Different network structures and processing flows may produce feature tensors of different dimensions, which need to be further processed as a follow-up (such as fusion with text features, etc.).

[0107] Step S103, processing the text data according to the Transformer model to generate a text feature vector.

[0108] It should be noted that Transformer is a deep learning architecture based on the attention mechanism, which uses a new multi-head attention mechanism and fully connected feedforward network components to process input data. It mainly consists of two parts: encoder and decoder (in some application scenarios, encoder or decoder can also be used separately). The encoder is used to extract and encode the input sequence, converting the input text data into a feature representation that is more convenient for subsequent processing and understanding; the decoder generates the final text output based on the features of the encoder output and additional input (such as part of the target sequence) in tasks that require the generation of text output. Text feature vectors play a vital role in multimodal processing (such as visual-language multimodal tasks). As a feature representation of text data, it can be fused with the visual feature vector extracted from the image data (such as by concatenation, weighted summation, etc.), so that the model can use information from both visual and textual modalities to complete various tasks, such as accurately answering questions about the content of the image in the visual question-answering task, or generating image descriptions that are more in line with the actual image and meet user requirements in the image description generation task.

[0109] Of course, in order to analyze the text semantics more accurately and reduce the influence of lexical ambiguity on text feature extraction, before step S103, the following is also included:

[0110] Sentences in the text data are segmented to obtain text segmentations; the text segmentations are replaced with target index words according to a preset vocabulary, and a text feature vector is generated according to the target index words and a Transformer model.

[0111] It should be understood that word segmentation is a basic operation in natural language processing. It refers to splitting sentences in text data into relatively independent and meaningful vocabulary or sub-word units according to certain rules. For the text input by the user, it is difficult to directly process such a continuous sequence. By breaking it down into meaningful units through word segmentation, the model can analyze and process these units separately, and then understand the semantics, grammar and other information of the entire sentence.

[0112] It is understandable that after completing the word segmentation processing of the text data to obtain the text segmentation, each text segmentation will be compared with the preset vocabulary one by one. If there is a corresponding vocabulary in the vocabulary for the text segmentation, then the text segmentation is replaced with its target index word in the vocabulary. Through this replacement operation, the text segmentation originally in the form of natural language is converted into a sequence of target index words in digital form. This makes the text data more in line with the processing requirements of computers and deep learning models in form, making it easier to provide it as input to the Transformer model for further processing.

[0113] Step S104, combining the multidimensional feature tensor and the text feature vector to obtain a feature vector of the target data.

[0114] It is understandable that by combining the multidimensional feature tensor and the text feature vector, the model can make full use of the visual and text information in the fused feature vector to accurately judge the relationship between the image content and the text description, and then give an accurate answer in the visual question answering task. For example, if the dimension of the multidimensional feature tensor is (1024, 21, 21) and the dimension of the text feature vector is (768, 1), a feature vector of the target data with a dimension of (1024, 768, 21, 21, 1) may be obtained by concatenation.

[0115] In this embodiment, the initial data input by the target user is obtained, wherein the initial data includes image data and text data; the image data in the initial data is adjusted based on a preset image input threshold, and the adjusted image data is input into a convolutional neural network to obtain a multidimensional feature tensor; the text data is processed according to a Transformer model to generate a text feature vector; the multidimensional feature tensor and the text feature vector are combined to obtain a feature vector of the target data. Adjusting the input image data based on the preset image input threshold and then inputting it into the convolutional neural network can make the image data more suitable for model processing and improve the quality and efficiency of image feature extraction.

[0116] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multimodal feature fusion method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0117] This application also provides a multi-modal feature fusion device, please refer to Figure 8 , the multimodal feature fusion device comprises:

[0118] A feature extraction module 10 is used to extract a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector;

[0119] A dimensionality reduction operation module 20 is used to call the initialized convolution operator to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor;

[0120] The feature fusion module 30 is used to concatenate and fuse the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature.

[0121] The multimodal feature fusion device provided by the present application adopts the multimodal feature fusion method in the above-mentioned embodiment, which can solve the technical problem that the current visual language model adapter solution completes the alignment of visual features and sequence features by simple and rough flattening, loses a lot of visual information, and causes the performance of the visual language model to decline. Compared with the prior art, the beneficial effects of the multimodal feature fusion device provided by the present application are the same as the beneficial effects of the multimodal feature fusion method provided by the above-mentioned embodiment, and the other technical features in the multimodal feature fusion device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0122] The present application provides a multimodal feature fusion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal feature fusion method in the above-mentioned embodiment one.

[0123] Reference below Fig. 9 , which shows a schematic diagram of the structure of a multimodal feature fusion device suitable for implementing the embodiment of the present application. The multimodal feature fusion device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The multimodal feature fusion device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Fig. 9As shown, the multimodal feature fusion device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multimodal feature fusion device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the multimodal feature fusion device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a multimodal feature fusion device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0125] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0126] The multimodal feature fusion device provided by the present application adopts the multimodal feature fusion method in the above embodiment, which can solve the technical problem that the current visual language model adapter solution completes the alignment of visual features and sequence features by simple and rough flattening, loses a lot of visual information, and causes the performance of the visual language model to decline. Compared with the prior art, the beneficial effects of the multimodal feature fusion device provided by the present application are the same as the beneficial effects of the multimodal feature fusion method provided by the above embodiment, and the other technical features in the multimodal feature fusion device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the multimodal feature fusion method in the above-mentioned embodiment.

[0130] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0131] The above-mentioned computer-readable storage medium may be included in the multimodal feature fusion device; or it may exist independently without being assembled into the multimodal feature fusion device.

[0132] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the multimodal feature fusion device, the multimodal feature fusion device executes the multimodal feature fusion method described above.

[0133] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0136] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal feature fusion method, and can solve the technical problem that the current visual language model adapter solution completes the alignment of visual features and sequence features through simple and rough flattening, loses a large amount of visual information, and causes the performance of the visual language model to decline. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multimodal feature fusion method provided by the above-mentioned embodiment, and will not be repeated here.

[0137] The present application also provides a computer program product, including a computer program, which implements the steps of the multimodal feature fusion method as described above when executed by a processor.

[0138] The computer program product provided by the present application can solve the technical problem that the current visual language model adapter solution aligns visual features with sequence features by simple and rough flattening, which loses a lot of visual information and causes the performance of the visual language model to decline. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multimodal feature fusion method provided by the above embodiment, which will not be repeated here.

[0139] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multimodal feature fusion method, characterized in that: The multimodal feature fusion method comprises: Extracting a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector; Calling the initialized convolution operator to perform dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor; The one-dimensional feature tensor is concatenated and fused with the text feature vector to obtain a visual-text fusion feature.

2. The multimodal feature fusion method according to claim 1, characterized in that: The step of calling the initialized convolution operator to perform a dimensionality reduction operation on the visual feature to obtain a one-dimensional feature tensor includes: The central parts of the preset initial operator are set as the preset initial threshold to obtain the initialized convolution operator; Determine the convolution kernel of each dimension in the convolution operator; The convolution kernel is convolved with a feature region of a corresponding dimension in the visual feature to obtain each visual feature value, and the visual feature values ​​are concatenated in sequence to obtain a one-dimensional feature tensor.

3. The multimodal feature fusion method according to claim 1, characterized in that: After the step of concatenating and fusing the one-dimensional feature tensor with the text feature vector to obtain the visual text fusion feature, the method further includes: Generate a first text sequence according to the visual-text fusion feature; Calculating the probability distribution of each text answer in the first text sequence, and determining the first text answer with the highest probability in the first text sequence; Comparing the probability of the first text answer with a preset probability threshold to obtain a comparison result; If the comparison result is that the probability of the first text answer is greater than a preset probability threshold, the first text answer is post-processed and delivered to the target user.

4. The multimodal feature fusion method according to claim 3, characterized in that: After the step of comparing the probability of the first text answer with a preset probability threshold to obtain a comparison result, the method further includes: If the comparison result is that the probability of the first text answer is less than a preset probability threshold, the first text answer is concatenated with the visual text fusion feature to obtain a target fusion feature; Generate a second text sequence according to the target fusion feature; Calculating the probability distribution of each text answer in the second text sequence, and determining the second text answer with the highest probability in the second text sequence; When the probability of the second text answer is greater than a preset probability threshold, the second text answer is post-processed and delivered to the target user.

5. The multimodal feature fusion method according to claim 1, characterized in that: The step of extracting the feature vector of the preprocessed target data includes: Acquire initial data input by a target user, wherein the initial data includes image data and text data; Adjusting the image data in the initial data based on a preset image input threshold, and inputting the adjusted image data into a convolutional neural network to obtain a multi-dimensional feature tensor; Processing the text data according to the Transformer model to generate a text feature vector; The multidimensional feature tensor and the text feature vector are combined to obtain a feature vector of the target data.

6. The multimodal feature fusion method as claimed in claim 5, characterized in that Before the step of processing the text data according to the Transformer model to generate a text feature vector, the step further includes: Performing word segmentation processing on the sentences in the text data to obtain text segmentation; The text segmentation words are replaced with target index words according to a preset vocabulary, and a text feature vector is generated according to the target index words and the Transformer model.

7. A multimodal feature fusion device, characterized in that: The device comprises: A feature extraction module, used to extract a feature vector of the preprocessed target data, wherein the feature vector includes a visual feature and a text feature vector; A dimensionality reduction operation module, used for calling the initialized convolution operator to perform dimensionality reduction operation on the visual features to obtain a one-dimensional feature tensor; The feature fusion module is used to concatenate and fuse the one-dimensional feature tensor with the text feature vector to obtain a visual-text fusion feature.

8. A multimodal feature fusion device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal feature fusion method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal feature fusion method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the multimodal feature fusion method according to any one of claims 1 to 6 are implemented.

Images