Multi-round image and text understanding and positioning method and device based on multi-modal and multi-format unification

By dividing the image into image blocks and using the state space model to perform 2D spatial sequence scanning and sequence discretization processing in the direction of the meter-shaped structure, the problem of high computational complexity in multimodal data processing is solved, and the neural network model is realized Efficient processing of multimodal data and strong support for graphic and text tasks.

CN118797562BActive Publication Date: 2024-12-06INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411282777.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-12-06
Estimated Expiration
2044-09-13

AI Technical Summary

Technical Problem

The existing neural network model has high computational complexity when processing multimodal data, which is difficult to meet the actual application requirements. The state space model has problems with disagreement when processing two-dimensional images and multimodal data, which limits its inability to visual tasks. Application.

Method used

By dividing the image to be processed into image blocks, the state space model is used to perform 2D spatial sequence scanning and sequence discretization processing in the direction of the quad structure, fuse the image and text features, and generate multimodal fusion features to perform graphic and text understanding or positioning Task.

Benefits of technology

The complexity of model calculation is reduced, the processing efficiency of multimodal data is improved, and the efficient processing of long sequences and high-dimensional data is realized, providing strong support for multimodal graphic and text tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118797562B_ABST
    Figure CN118797562B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-round image-text understanding and positioning method and device based on multi-modal and multi-form unification, which relates to the field of image data processing technology. The method includes: obtaining multi-modal data for performing image-text understanding or positioning tasks, and dividing the image to be processed of the multi-modal data into multiple image blocks; determining the text features of the text to be processed of the multi-modal data and the image sequence features of the image blocks; performing a 2D space sequence scan based on the direction of the cross-shaped structure on the image sequence features through a state-space model feature extraction module to obtain a scan sequence feature; performing a linear mapping based on sequence discretization through a state-space model feature processing module to obtain discrete sequence features, which are then scanned and merged, and then fused with text features to obtain multi-modal fusion features for performing image-text understanding or positioning tasks. Through this application, the defect of high model calculation complexity of the neural network model when processing long input sequences or high-dimensional data is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a multi-round image and text understanding and positioning method and device based on multi-modal and multi-format unification. Background Art

[0002] In some computer vision scenarios, such as image target recognition, image semantic segmentation, visual question answering, etc., the computational complexity of neural network models (such as convolutional neural networks and recurrent neural networks) is usually very high when processing long sequences or high-dimensional input data. Especially when it comes to the fusion of multimodal data (such as images and text), the computational cost is higher and the efficiency is lower, which makes it difficult to meet the needs of actual applications.

[0003] The state-space model (SSM) is a discrete version of a continuous system. It discretizes continuous time parameters so that it can be used in an efficient deep learning framework, which can reduce the complexity of model calculations. However, SSM is not suitable for processing two-dimensional images and multimodal data, which limits its application in visual tasks. Summary of the invention

[0004] The present invention provides a multi-round image and text understanding and positioning method and device based on multi-modal and multi-format unification, so that the state space model can adapt to multi-modal data, so as to solve the defect of high model calculation complexity when the neural network model in the prior art processes long sequence or high-dimensional data as input.

[0005] The present invention provides a multi-round image and text understanding and positioning method based on multi-modal and multi-format unification, comprising the following steps:

[0006] Acquire multimodal data for performing image and text understanding or positioning tasks, and divide the image to be processed included in the multimodal data into multiple image blocks;

[0007] Determining text features of the to-be-processed text included in the multimodal data, and determining an image code and a position code of the image block, and fusing the image code with the position code to obtain an image sequence feature of the image block;

[0008] The image sequence features of the image block are scanned in 2D space sequence based on the direction of the cross-shaped structure by a feature extraction module of the state space model to obtain a scanning sequence feature;

[0009] Through the feature processing module of the state space model, the linear mapping process based on sequence discretization is performed on the scanning sequence features to obtain the discrete sequence features of the image block;

[0010] The discrete sequence features of the image block are scanned and merged to obtain the fused sequence features of the image to be processed, and the fused sequence features are fused with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task.

[0011] In some embodiments, an image start token is inserted into the head of the image sequence feature of the image block, an image end token is inserted into the end of the image sequence feature of the image block, and the image sequence feature of the image block is subjected to a 2D spatial sequence scan based on the direction of the cross-shaped structure to obtain a scan sequence feature;

[0012] When the feature extraction module based on the state space model scans the image start token, the image sequence features of the image block are sequentially scanned in 2D space sequence according to the direction of the cross-shaped structure;

[0013] When the feature extraction module based on the state space model scans the image end token, the 2D space sequence scanning is ended to obtain the scanning sequence features.

[0014] In some embodiments, the direction of the cross-shaped structure includes at least one of the following sequential scanning directions:

[0015] From left to right; from top to bottom; from right to left; from bottom to top;

[0016] From top left to bottom right; from top right to bottom left; from bottom right to top left; from bottom left to top right.

[0017] In some embodiments, performing linear mapping processing based on sequence discretization on the scanning sequence features to obtain discrete sequence features of the image block includes:

[0018] Discretize the scanning sequence features to obtain hidden state sequence features;

[0019] Based on the hidden state sequence characteristics, determining a structured convolution kernel for performing a global convolution operation;

[0020] A global convolution operation is performed on the scanning sequence features through the structured convolution kernel to obtain discrete sequence features of the image block.

[0021] In some embodiments, the discretization sequence processing is performed according to a preset discrete time step, and the discretization sequence processing of the scanning sequence features to obtain the hidden state sequence features includes:

[0022] The first continuous time parameter and the second continuous time parameter in the state space model are discretely converted through a preset time scale parameter to obtain a state transfer matrix and a discrete input matrix;

[0023] In a discrete time step, the scan sequence features are discretized by using the state transfer matrix and the discrete input matrix to obtain hidden state sequence features.

[0024] In some embodiments, determining a structured convolution kernel for performing a global convolution operation based on the hidden state sequence feature includes:

[0025] Get the discrete output matrix in the state-space model;

[0026] Performing feature vector conversion on the hidden state sequence features through the discrete output matrix to obtain discrete output features;

[0027] A structured convolution kernel for performing a global convolution operation is determined based on the discrete output characteristics, the discrete output matrix, the state transfer matrix, and the discrete input matrix.

[0028] The present invention also provides a multi-round image and text understanding and positioning device based on multi-modal and multi-format unification, characterized in that the device includes the following modules:

[0029] An image processing module, used to obtain multimodal data for performing image and text understanding or positioning tasks, and to divide the image to be processed included in the multimodal data into multiple image blocks;

[0030] A feature coding module, used to determine text features of the text to be processed included in the multimodal data, and determine the image coding and position coding of the image block, and fuse the image coding and the position coding to obtain image sequence features of the image block;

[0031] A sequence scanning module, used for performing a 2D space sequence scanning based on the direction of the cross-shaped structure on the image sequence features of the image block through a feature extraction module of a state space model to obtain a scanning sequence feature;

[0032] A sequence discretization module, used for performing linear mapping processing based on sequence discretization on the scanning sequence features through a feature processing module of a state space model to obtain discrete sequence features of the image block;

[0033] The sequence merging module is used to scan and merge the discrete sequence features of the image block to obtain the fused sequence features of the image to be processed, and to fuse the fused sequence features with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multi-round image and text understanding and positioning method based on multi-modal and multi-format unification as described in any one of the above is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification as described in any one of the above is implemented.

[0036] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned multi-round image-text understanding and positioning methods based on multi-modal and multi-format unification.

[0037] The multi-round image-text understanding and positioning method and device based on multi-modal and multi-format unification provided by the present invention, when processing the image to be processed in the multi-modal data, converts the image to be processed into an image block and performs position embedding to adapt it to the needs of an efficient deep learning model. The image sequence features are expanded based on a cross-shaped sequence scanning through the state space model, which can globally perceive the image sequence features, has the advantages of efficient feature fusion, strong adaptability and high computational efficiency, enhances the processing ability of multi-modal data, and realizes the efficient processing of long sequences and high-dimensional data, providing strong support for multi-modal image-text tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification provided by the present invention.

[0040] Figure 2 It is a schematic diagram of the principle of the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification provided by the present invention.

[0041] Figure 3 This is an example diagram of 2D spatial sequence scanning based on the "M"-shaped structure direction provided by the present invention.

[0042] Figure 4 It is a schematic diagram of processing image data using the state space model provided by the present invention.

[0043] Figure 5 It is a structural schematic diagram of a multi-round image-text understanding and positioning device based on multi-modal and multi-format unification provided by the present invention.

[0044] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Combine the following Figure 1 Describe the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification of the present invention, Figure 1 is a flow chart of a multi-round image-text understanding and positioning method based on multi-modal and multi-format unification provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 105.

[0047] Step 101: Acquire multimodal data for performing image and text understanding or positioning tasks, and divide the image to be processed included in the multimodal data into multiple image blocks.

[0048] In image-text comprehension or positioning tasks, the input data is generally multimodal data, including images and text. In image-text comprehension tasks, it is necessary to identify some targets in the image based on text prompts (for example, please find the red object in the image) or questions (for example, please answer what the object is in the image). In image-text positioning tasks, it is necessary to locate some targets in the image based on text prompts or questions, and the text content of these prompts or questions is generally used to describe the targets that need to be identified or located.

[0049] In the image and text understanding or positioning task of the embodiment of the present invention, the data used is multimodal data, wherein the multimodal data includes the image to be processed and the text to be processed. Figure 2 As shown, Figure 2 is a schematic diagram of a multi-round image-text understanding and positioning method based on multi-modal and multi-format unification provided by the present invention, Figure 2 It describes a picture-text comprehension task. The image to be processed is an image of an elephant, and the text to be processed describes the question, which is "What kind of animal is in the picture?".

[0050] When performing image-text understanding or positioning tasks, it is necessary to divide the image to be processed included in the multimodal data into multiple image blocks and determine the text features of the text to be processed included in the multimodal data. To split, , H and W represent the height and width of the image to be processed, respectively, and C represents the number of image channels. First, the image to be processed Convert to image blocks ,in, , is the size of the image block. The conversion process here is the process of segmenting the image to be processed. The segmentation process can be implemented using a convolutional neural network. The sizes of the multiple image blocks obtained by segmentation are the same, for example Figure 2 As shown, the elephant image is divided into 9 image blocks of the same size.

[0051] Step 102: determine the text features of the text to be processed included in the multimodal data, and determine the image coding and position coding of the image block, and fuse the image coding and the position coding to obtain the image sequence features of the image block.

[0052] like Figure 2 As shown in the figure, for the text to be processed included in the multimodal data, "What kind of animal is in the picture?", the embedding layer neural network can be used to embed the text to be processed (i.e., Embedding) to obtain the text features of the text to be processed. The text features are essentially text embedding sequences. , then the image encoding and position encoding of the image block must also be determined. Image encoding can be performed by embedding a layer of neural network on the image block. Perform embedding processing (Embedding), the image block Linearly map to a vector with feature dimension D to obtain an image block Here, the embedding layer neural network can be a convolutional neural network with a convolution kernel size of 16×16, and as a projection layer, the image block Linearly maps to a vector of length D.

[0053] The position information of each image block in the image to be processed is also embedded to obtain the corresponding position code. , the image coding and position coding are fused to obtain the image sequence features of the image block.

[0054] Step 103 : Using the feature extraction module of the state space model, perform a 2D space sequence scan on the image sequence features of the image block based on the direction of the cross-shaped structure to obtain a scan sequence feature.

[0055] After determining the text features of the text to be processed and the image sequence features of the image block through step 102, the text features and the image sequence features are input into the state space model for processing. The state space model is divided into two modules, one is a feature processing module for state space modeling, and the other is a feature processing module for linear mapping. The state space model in the embodiment of the present invention is formed by stacking N layers of neural networks. In each layer of the neural network, not only state space modeling but also linear mapping is required. Each layer of the neural network is specifically a multi-layer perceptron module (also known as a feedforward neural network), which is composed of two linear layers (Linear layer) and a nonlinear layer (such as SiLU) separating the two linear layers. Both state space modeling and feedforward neural networks use residual connections, and layer normalization operations are also used before them.

[0056] Here, text features and image sequence features are taken as the feature sequence of mixed images and texts. Input to the state-space model, where n represents the nth layer of the state-space model (n≤N), It represents the feature sequence of the initial input image and text mixture, and finally the feature sequence Input to the nth layer of the state space model neural network, output As the sequence mapping result corresponding to the multimodal data.

[0057] In each layer of the neural network, two model structures are used. In the first model structure, the feature dimension D of the hidden state sequence feature is set to 192, and the extended state dimension (that is, the feature dimension of the discrete sequence feature) E is set to 384. In the second model structure, the feature dimension D of the hidden state sequence feature is set to 384, and the extended state dimension (that is, the feature dimension of the discrete sequence feature) E is set to 768. These two model structures can be reasonably selected according to the size of the segmented image block size P.

[0058] The specific processing process of the state space model is described below.

[0059] like Figure 2 As shown, firstly, the text features and image sequence features are input as a mixed feature sequence of text and images into the N-layer neural network of the state space model, and the image sequence features of the image block are subjected to a 2D space sequence scan based on the cross-shaped structure direction through the feature extraction module of the neural network layer of the state space model to obtain the scanned sequence features. However, before the sequence scanning, the embodiment of the present invention inserts an image start token at the head of the image sequence features of the image block, which is expressed as [IMG_ST], and inserts an image end token at the end of the image sequence features of the image block, which is expressed as [IMG_ED], as the start mark and end mark of the visual feature SSM processing.

[0060] When performing sequence scanning, the feature extraction module of the state space model performs SSM processing (also known as sequence scanning) on ​​the feature sequence of the mixed image and text. When the sequence scans the text feature, the text feature SSM processing is directly performed. Since the text feature is essentially a text embedding sequence, the text feature can be scanned sequentially in a single direction. When the feature extraction module based on the state space model scans the image start token [IMG_ST], the visual feature SSM processing is performed, and the image sequence features of the image block are sequentially scanned in 2D space sequence according to the direction of the cross-shaped structure. Here, the cross-shaped structure direction is also the traditional eight directions, and the traditional eight directions are used as the eight sequence scanning directions, specifically including at least one of from left to right; from top to bottom; from right to left; from bottom to top; from top left to bottom right; from top right to bottom left; from bottom right to top left; from bottom left to top right. Since the image block corresponding to the image to be processed is two-dimensional, this sequence scanning is also called 2D space sequence scanning.

[0061] For 2D spatial sequence scanning of the cross-shaped structure direction, please refer to Figure 3 , Figure 3 This is an example diagram of performing 2D spatial sequence scanning based on the "M"-shaped structure direction provided by the present invention. Among them, the sequence scanning directions from top to bottom and from bottom to top can be referred to Figure 3 As shown in (a), the sequence scanning direction from left to right and from right to left can be seen in Figure 3 As shown in (b), the sequential scanning direction from lower left to upper right and from upper right to lower left can be seen in Figure 3 As shown in (c), the sequential scanning direction from upper left to lower right and from lower right to upper left can be seen in Figure 3 As shown in (d) in .

[0062] See also Figure 4 , Figure 4 Schematic diagram of the state space model provided by the present invention for processing image data. Figure 4 As shown, when performing 2D space sequence scanning, you can select any one of the eight sequence scanning directions to start sequentially scanning the image sequence features of the image block. This process is called scan expansion. Scan expansion can integrate any pixel information with all other pixel information scanned from different directions. When the feature extraction module based on the state space model scans the image end token [IMG_ED], the 2D space sequence scanning is ended and the scanning sequence features are obtained. Thus, by performing sequence scanning in one sequence scanning direction, a set of scanning sequence features of the image block can be obtained. And according to the eight sequence scanning directions, eight groups of scanning sequence features of the image block can be obtained.

[0063] The embodiment of the present invention performs cross-shaped sequence scanning expansion on the image sequence features through the state space model, can globally perceive the image sequence features, has the advantages of efficient feature fusion, strong adaptability and high computational efficiency, enhances the processing capability of multimodal data, realizes efficient processing of long sequences and high-dimensional data, and provides strong support for multimodal tasks.

[0064] Step 104: Perform linear mapping processing based on sequence discretization on the scan sequence features through the feature processing module of the state space model to obtain discrete sequence features of the image block.

[0065] like Figure 4 As shown, while the state space modeling is performed based on the feature extraction module of the state space model neural network layer, it is also necessary to use the feature processing module of the state space model neural network layer to perform linear mapping based on sequence discretization on the scanning sequence features of each image block obtained by scanning the 2D space sequence, so as to obtain the discrete sequence features of the image block.

[0066] In the feature processing module, the linear mapping based on sequence discretization is divided into two processes, namely the discretization sequence processing process and the global convolution operation process. First, the discretization sequence processing process is introduced. This process is performed according to the preset discrete time step t. Since the state-space model is a continuous-time model, how to integrate it into the existing efficient deep learning algorithm framework is a huge challenge. In order to overcome this obstacle, the discretization process becomes crucial, and the main goal of discretization is to convert ordinary differential equations into discrete functions. This conversion is crucial to align the model with the sampling rate of the signal contained in the input data in order to achieve computationally efficient operations. In the process of discrete serialization processing, the scanning sequence features are first discretized to obtain the hidden state sequence features. Specifically, through the preset time scale parameters in the state-space model , the first continuous time parameter A and the second continuous time parameter B in the state space model are discretely transformed to obtain the state transfer matrix and the discrete input matrix , as a discrete parameter.

[0067] The discrete conversion method adopted by the present invention is a zero-order hold method, as shown in the following formula (1) and formula (2):

[0068] (1)

[0069] (2)

[0070] In the above formulas (1) and (2), exp represents the natural exponential function. represents a preset time scale parameter, A represents a preset first continuous time parameter in the state space model, B represents a preset second continuous time parameter in the state space model, Represents the discrete parameter vector of the computation process, satisfying .

[0071] Scan sequence features (denoted as , represents the scanning sequence at discrete time step t), and after the scanning sequence features are input into the feature processing module, within the discrete time step t, the state transfer matrix and the discrete input matrix Scan sequence characteristics After discretization, we get the hidden state sequence characteristics, recorded as .

[0072] Specifically, here at the first discrete time step (t=0), the input scan sequence features , only by discrete input matrix Scan sequence characteristics Discretize and obtain the hidden state sequence characteristics , entering the second discrete time step (t=1).

[0073] At each discrete time step t after the second discrete time step (t=1), the state transfer matrix The hidden state sequence features obtained for the last discrete time step t-1 Perform eigenvector transformation and then pass the discretized input matrix The scan sequence characteristics of the current discrete time step t input Perform feature vector transformation, and finally add the results of the two feature vector transformations to obtain the hidden state sequence feature obtained by the current discrete time step t , so each discrete time step t corresponds to the hidden state sequence feature The calculation formula can be expressed as formula (3):

[0074] (3)

[0075] In the above formula (3), t represents the current discrete time step, t-1 represents the previous discrete time step of the current discrete time step, represents the hidden state sequence characteristics obtained at the last discrete time step t-1, represents the state transfer matrix, Represents the discrete input matrix.

[0076] The above formula (3) can be used to calculate the hidden state sequence characteristics of each discrete time step t (t>0), until the last discrete time step is calculated and output to obtain the hidden state sequence characteristics .

[0077] Next is the global convolution operation process. First, based on the hidden state sequence characteristics, the structured convolution kernel used for global convolution operation is determined. Specifically, the discrete output matrix C in the state space model is obtained, and the hidden state sequence characteristics are quantized by the discrete output matrix C. Perform feature vector conversion and output the discrete output features of discrete time step t , expressed as the following formula (4):

[0078] (4)

[0079] In the above formula (4), C represents the discrete output matrix in the state space model, Represents the discretized sequence characteristics of the current discrete time step.

[0080] Then according to the discrete output features , discrete output matrix C, state transfer matrix and the discrete input matrix , determine the structured convolution kernel for global convolution operation The determination process here is a calculation process of formula derivation, which is explained in detail below.

[0081] As mentioned before, at the first discrete time step (t=0), the input scan sequence features , only by discrete input matrix Scan sequence characteristics Discretize and obtain the hidden state sequence characteristics , that is , Combining formula (3) and formula (4) to deduce, we know that , similarly, we can deduce , and then we can deduce that , where M is the input scan sequence feature The sequence length is the same as the discrete time step t. Each of The parameters of the global convolution operation can determine the structured convolution kernel , expressed as the following formula (5):

[0082] (5)

[0083] In the above formula (5), represents the state transfer matrix, represents the discrete input matrix, C represents the discrete output matrix in the state space model, and M represents the sequence length of the input scanning sequence features.

[0084] Determine the structured convolution kernel Afterwards, through the structured convolution kernel Scan sequence characteristics (i.e. ) performs a global convolution operation to obtain the discrete sequence feature y of the image block, which is calculated as follows (6):

[0085] (6)

[0086] In the above formula (6), Represents the structured convolution kernel, x represents the input scanning sequence features, where .

[0087] like Figure 4 As shown, after a group of scanning sequence features of image blocks obtained by each sequence scanning method are processed by the feature processing module, a group of discrete sequence features of corresponding image blocks are still obtained for subsequent scanning and merging.

[0088] Step 105: Scan and merge the discrete sequence features of the image block to obtain the fused sequence features of the image to be processed, and fuse the fused sequence features with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform image and text understanding or positioning tasks.

[0089] like Figure 4 As shown, after the feature processing process of the state space model is completed, the discrete sequence features of the image block are scanned and merged in each sequence scanning mode to obtain the fused sequence features of the image to be processed.

[0090] Here, for each sequence scanning method, the discrete sequence features of each image block are fused (for example, if there are 25 image blocks, the discrete sequence features of the 25 image blocks are fused). Since the size of each image block is the same, the dimension of the discrete sequence features is also the same. Therefore, the fusion method can be direct vector concatenation or vector addition. Then the sequence features fused by each sequence scanning method are merged again (for example, if there are 8 sequence scanning methods, the sequence features fused by the 8 sequence scanning methods are merged) to obtain the image to be processed (for example Figure 2 The fusion sequence features of the elephant image shown in Figure 1 are shown in Figure 2. The merging process is similar to the fusion process, which can be direct vector concatenation or vector addition. The feature fusion process is also the scanning merging process.

[0091] like Figure 2 As shown in the figure, after the scanning and merging is completed, that is, the visual feature SSM processing is completed, if the feature sequence of the mixed image and text has not been scanned, the text feature SSM processing is continued, that is, the one-way scanning of the text embedding sequence is resumed until the scanning of the feature sequence of the mixed image and text is completed. After the scanning process is completed, it indicates that the processing process of the state space model is completed. According to the processing method of multimodal data, the fused sequence features of the image to be processed are fused with the text features after the text feature SSM processing of the text to be processed, and the multimodal fusion features are obtained. The multimodal fusion features can be used to perform image and text understanding or positioning tasks. For example, a classifier can be used to map the multimodal fusion features to obtain the recognition probability of the animal category in the image to be processed (elephant image), where the probability of identifying the elephant is the largest, and the text to be processed "What kind of animal is in the picture" can be answered to complete the image and text understanding or positioning tasks.

[0092] In the embodiment of the present invention, an image start token and an image end token are added before and after the image sequence features, and scanning and expansion are performed from the image start token, and the 2D space scanning module of the "M"-shaped structure is entered. When the image end token is encountered, scanning and merging are performed to restore the unidirectional scanning of the text features. The 2D space scanning module of the "M"-shaped structure ensures the robustness of the output image sequence features by utilizing the complementarity of the receptive fields of each sequence scanning direction. It has the advantages of strong global perception ability, efficient feature fusion, strong adaptability and high computational efficiency. Through multi-directional scanning and feature fusion methods, it provides an efficient and robust solution for multimodal data processing.

[0093] The multi-round image and text understanding and positioning device based on multi-modal and multi-format unification provided by the present invention is described below. The multi-round image and text understanding and positioning device based on multi-modal and multi-format unification described below and the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification described above can be referenced to each other.

[0094] See also Figure 5 , Figure 5 is a structural schematic diagram of a multi-round image-text understanding and positioning device based on multi-modal and multi-format unification provided by the present invention, such as Figure 5As shown, the multi-round image and text understanding and positioning device based on multi-modal and multi-format unification specifically includes an image processing module 501, a feature encoding module 502, a sequence scanning module 503, a sequence discretization module 504, and a sequence merging module 505. Among them, the image processing module 501 is used to obtain multimodal data for performing image and text understanding or positioning tasks, and divide the image to be processed included in the multimodal data into multiple image blocks; the feature encoding module 502 is used to determine the text features of the text to be processed included in the multimodal data, and determine the image encoding and position encoding of the image block, and fuse the image encoding with the position encoding to obtain the image sequence features of the image block; the sequence scanning module 503 is used to perform a 2D space sequence scanning based on the cross-shaped structure direction on the image sequence features of the image block through the feature extraction module of the state space model to obtain the scanning sequence features; the sequence discretization module 504 is used to perform a linear mapping processing based on sequence discretization on the scanning sequence features through the feature processing module of the state space model to obtain the discrete sequence features of the image block; the sequence merging module 505 is used to scan and merge the discrete sequence features of the image block to obtain the fused sequence features of the image to be processed, and fuse the fused sequence features with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image and text understanding or positioning tasks.

[0095] It can be seen from this that the beneficial effects produced by the multi-round image and text understanding and positioning device based on multi-modal and multi-form unification provided by the present invention should be the same as the multi-round image and text understanding and positioning method based on multi-modal and multi-form unification described above. Therefore, the beneficial effects produced by the multi-round image and text understanding and positioning device based on multi-modal and multi-form unification will not be repeated here.

[0096] Figure 6 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a multi-round image-text understanding and positioning method based on multi-modal and multi-form unification, the method comprising: obtaining multi-modal data for performing image-text understanding or positioning tasks, dividing the image to be processed included in the multi-modal data into multiple image blocks; determining the text features of the text to be processed included in the multi-modal data, and determining the image code and position code of the image block, fusing the image code with the position code to obtain the image sequence features of the image block; performing a 2D space sequence scan based on the cross-shaped structure direction on the image sequence features of the image block through a feature extraction module of a state-space model to obtain a scan sequence feature; performing a linear mapping process based on sequence discretization on the scan sequence features through a feature processing module of a state-space model to obtain a discrete sequence feature of the image block; scanning and merging the discrete sequence features of the image block to obtain a fused sequence feature of the image to be processed, and fusing the fused sequence feature with the text feature to obtain a multi-modal fusion feature, wherein the multi-modal fusion feature is used to perform the image-text understanding or positioning task.

[0097] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-round image-text understanding and positioning method based on multi-modal and multi-format unification provided by the above-mentioned methods. The method includes: obtaining multimodal data for performing image-text understanding or positioning tasks, dividing the image to be processed included in the multimodal data into multiple image blocks; determining the text features of the text to be processed included in the multimodal data, and determining the image coding and position coding of the image block, and fusing the image coding with the position coding to obtain to the image sequence features of the image block; through the feature extraction module of the state-space model, the image sequence features of the image block are scanned in 2D space sequence based on the direction of the cross-shaped structure to obtain the scan sequence features; through the feature processing module of the state-space model, the scan sequence features are processed by linear mapping based on sequence discretization to obtain the discrete sequence features of the image block; the discrete sequence features of the image block are scanned and merged to obtain the fused sequence features of the image to be processed, and the fused sequence features are fused with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task.

[0099] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-round image-text understanding and positioning method based on multi-modal and multi-format unification provided by the above-mentioned methods, the method comprising: acquiring multi-modal data for performing image-text understanding or positioning tasks, dividing the image to be processed included in the multi-modal data into a plurality of image blocks; determining text features of the text to be processed included in the multi-modal data, and determining the image coding and position coding of the image block, fusing the image coding with the position coding to obtain image sequence features of the image block; The feature extraction module of the state-space model performs a 2D space sequence scanning based on the cross-shaped structure direction on the image sequence features of the image block to obtain the scanning sequence features; the feature processing module of the state-space model performs a linear mapping processing based on sequence discretization on the scanning sequence features to obtain the discrete sequence features of the image block; the discrete sequence features of the image block are scanned and merged to obtain the fused sequence features of the image to be processed, and the fused sequence features are fused with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-round image and text understanding and positioning method based on multi-modal and multi-format unification, characterized in that: The method comprises: Acquire multimodal data for performing image and text understanding or positioning tasks, and divide the image to be processed included in the multimodal data into multiple image blocks; Determining text features of the to-be-processed text included in the multimodal data, and determining an image code and a position code of the image block, and fusing the image code with the position code to obtain an image sequence feature of the image block; The image sequence features of the image block are subjected to 2D spatial sequence scanning based on the cross-shaped structure direction through the feature extraction module of the state space model to obtain the scanning sequence features, wherein the cross-shaped structure direction includes at least one of the following sequence scanning directions: from left to right; from top to bottom; from right to left; from bottom to top; from top left to bottom right; from top right to bottom left; from bottom right to top left; from bottom left to top right; Through the feature processing module of the state space model, the linear mapping process based on sequence discretization is performed on the scanning sequence features to obtain the discrete sequence features of the image block; Scanning and merging the discrete sequence features of the image block to obtain the fused sequence features of the image to be processed, and fusing the fused sequence features with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task, and scanning and merging the discrete sequence features of the image block to obtain the fused sequence features of the image to be processed, including: For each sequence scanning mode, the discrete sequence features of each image block are fused, and the sequence features fused by each sequence scanning mode are merged again to obtain the fused sequence features of the image to be processed.

2. The multi-round image-text understanding and positioning method based on multi-modal and multi-format unification according to claim 1 is characterized in that: An image start token is inserted into the head of the image sequence feature of the image block, an image end token is inserted into the end of the image sequence feature of the image block, and the image sequence feature of the image block is subjected to a 2D spatial sequence scan based on the direction of the cross-shaped structure to obtain the scan sequence feature, including: When the feature extraction module based on the state space model scans the image start token, the image sequence features of the image block are sequentially scanned in 2D space sequence according to the direction of the cross-shaped structure; When the feature extraction module based on the state space model scans the image end token, the 2D space sequence scanning is ended to obtain the scanning sequence features.

3. The multi-round image-text understanding and positioning method based on multi-modal and multi-format unification according to claim 1 is characterized in that: The performing linear mapping processing based on sequence discretization on the scanning sequence features to obtain discrete sequence features of the image block includes: Discretize the scanning sequence features to obtain hidden state sequence features; Based on the hidden state sequence characteristics, determining a structured convolution kernel for performing a global convolution operation; A global convolution operation is performed on the scanning sequence features through the structured convolution kernel to obtain discrete sequence features of the image block.

4. The multi-round image-text understanding and positioning method based on multi-modal and multi-format unification according to claim 3 is characterized in that: The discretization sequence processing is performed according to a preset discrete time step, and the scanning sequence features are discretized to obtain hidden state sequence features, including: The first continuous time parameter and the second continuous time parameter in the state space model are discretely converted through a preset time scale parameter to obtain a state transfer matrix and a discrete input matrix; In a discrete time step, the scan sequence features are discretized by using the state transfer matrix and the discrete input matrix to obtain hidden state sequence features.

5. The multi-round image-text understanding and positioning method based on multi-modal and multi-format unification according to claim 3 is characterized in that: The step of determining a structured convolution kernel for performing a global convolution operation based on the hidden state sequence feature includes: Get the discrete output matrix in the state-space model; Performing feature vector conversion on the hidden state sequence features through the discrete output matrix to obtain discrete output features; A structured convolution kernel for performing a global convolution operation is determined based on the discrete output characteristics, the discrete output matrix, the state transfer matrix, and the discrete input matrix.

6. A multi-round image and text understanding and positioning device based on multi-modal and multi-format unification, characterized in that: The device comprises: An image processing module, used to obtain multimodal data for performing image and text understanding or positioning tasks, and to divide the image to be processed included in the multimodal data into multiple image blocks; A feature coding module, used to determine text features of the text to be processed included in the multimodal data, and determine the image coding and position coding of the image block, and fuse the image coding and the position coding to obtain image sequence features of the image block; A sequence scanning module is used to perform a 2D space sequence scanning based on a cross-shaped structure direction on the image sequence features of the image block through a feature extraction module of a state space model to obtain a scanning sequence feature, wherein the cross-shaped structure direction includes at least one of the following sequence scanning directions: from left to right; from top to bottom; from right to left; from bottom to top; from top left to bottom right; from top right to bottom left; from bottom right to top left; from bottom left to top right; A sequence discretization module, used for performing linear mapping processing based on sequence discretization on the scanning sequence features through a feature processing module of a state space model to obtain discrete sequence features of the image block; A sequence merging module is used to scan and merge the discrete sequence features of the image blocks to obtain fused sequence features of the image to be processed, and to fuse the fused sequence features with the text features to obtain multimodal fusion features, wherein the multimodal fusion features are used to perform the image-text understanding or positioning task, and the discrete sequence features of the image blocks are scanned and merged to obtain the fused sequence features of the image to be processed, including: for each sequence scanning method, the discrete sequence features of each image block are fused, and the sequence features fused by each sequence scanning method are merged again to obtain the fused sequence features of the image to be processed.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-round image and text understanding and positioning method based on multi-modal and multi-format unification as described in any one of claims 1 to 5 is implemented.