Multi-modal-based data analysis method, device and equipment and storage medium

Abstract

The invention provides a data analysis method and device based on multiple modes, equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the field of large models and image processing. The specific implementation scheme is as follows: acquiring a to-be-analyzed image; wherein the image to be analyzed contains structured data; performing feature extraction processing on the to-be-analyzed image to obtain an image feature vector of the to-be-analyzed image; wherein the image feature vector represents structured data in the to-be-analyzed image; determining an analysis result of the to-be-analyzed image according to the image feature vector and preset cue word information; wherein the preset cue word information is text data representing a preset grammar structure, and the analysis result represents structured data expressed by the preset grammar structure. By combining the image modal data and the text modal data related to the structured data, the analysis accuracy of the structured data is improved.

Description

Technical Field

[0001] The present disclosure relates to large models in the field of artificial intelligence and the field of image processing, and particularly relates to a multi-modal based data parsing method, apparatus, device, and storage medium. Background Art

[0002] The parsing and conversion of mathematical formulas are key tasks in fields such as academic papers, formula editors, and optical character recognition, and usually require parsing mathematical formulas into structured data in the form of LaTeX code.

[0003] However, in the process of LaTeX structured parsing, there are problems such as high complexity of grammar parsing, chaotic symbol hierarchical relationships, and high misrecognition rates. This results in inaccurate LaTeX code, affecting subsequent formula rendering, calculation, or conversion. Summary of the Invention

[0004] The present disclosure provides a multi-modal based data parsing method, apparatus, device, and storage medium.

[0005] According to a first aspect of the present disclosure, there is provided a multi-modal based data parsing method, including:

[0006] Obtaining an image to be parsed; wherein, the image to be parsed contains structured data;

[0007] Performing feature extraction processing on the image to be parsed to obtain an image feature vector of the image to be parsed; wherein, the image feature vector characterizes the structured data in the image to be parsed;

[0008] Determining a parsing result of the image to be parsed according to the image feature vector and preset prompt word information; wherein, the preset prompt word information is text data characterizing a preset grammar structure, and the parsing result characterizes structured data expressed in the preset grammar structure.

[0009] According to a second aspect of the present disclosure, there is provided a multi-modal based data parsing apparatus, including:

[0010] An obtaining unit, configured to obtain an image to be parsed; wherein, the image to be parsed contains structured data;

[0011] An extraction unit, configured to perform feature extraction processing on the image to be parsed to obtain an image feature vector of the image to be parsed; wherein, the image feature vector characterizes the structured data in the image to be parsed;

[0012] An analysis unit, configured to determine an analysis result of the image to be analyzed according to the image feature vector and preset prompt word information; wherein, the preset prompt word information is text data representing a preset grammar structure, and the analysis result represents structured data expressed in the preset grammar structure.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor;

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the method described in the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect of the present disclosure are implemented.

[0019] According to the technology of the present disclosure, the accuracy of structured data analysis is improved.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0022] Figure 1 is a schematic flowchart of a multi-modal data analysis method provided according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of an image to be analyzed provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic flowchart of a multi-modal data analysis method provided according to an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of the training process of a first dictionary network provided according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic flowchart of a multi-modal data parsing method provided according to an embodiment of the present disclosure;

[0027] Figure 6 is a flowchart for parsing multi-modal formulas provided according to an embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of a multi-modal data parsing device provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of a multi-modal data parsing device provided according to an embodiment of the present disclosure;

[0030] Figure 9 is a block diagram of an electronic device for implementing the multi-modal data parsing method according to an embodiment of the present disclosure;

[0031] Figure 10 is a block diagram of an electronic device for implementing the multi-modal data parsing method according to an embodiment of the present disclosure. Detailed implementation manners

[0032] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] LaTeX is a form of expression for mathematical formulas, and structured LaTeX code can be used to represent mathematical formulas. For example, the mathematical formula is The structured data of LaTeX is sum_{i=1}^n i=\frac{n(n + 1)}{2}.

[0034] The parsing and conversion of mathematical formulas are key tasks in fields such as academic papers, formula editors, and optical character recognition. Usually, mathematical formulas need to be converted into the form of structured data in LaTeX for computer calculation and processing of formulas. However, in the process of LaTeX structured parsing of mathematical formulas, there are problems such as high difficulty in parsing complex syntax, chaotic symbol hierarchy relationships, and high misrecognition rates. Currently, the parsing of mathematical formulas mainly relies on preset parsing rules or deep learning methods. However, when faced with complex mathematical expressions such as long formulas, nested operators, matrices, and integrals, existing solutions often have difficulty accurately parsing the hierarchical structure of the formulas, resulting in inaccurate parsing of LaTeX code and affecting subsequent formula rendering, calculation, or conversion, etc.

[0035] The present disclosure provides a multi-modal based data parsing method, apparatus, device, and storage medium, which are applied to large models in the field of artificial intelligence and the field of image processing to improve the accuracy of structured data parsing.

[0036] It should be noted that the model in this embodiment is not a model for a specific user and does not reflect the personal information of a specific user. It should be noted that the images in this embodiment come from a public data set.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0038] To enable readers to more deeply understand the implementation principle of the present disclosure, the following Figures 1 - 10 is used to further refine the embodiments.

[0039] Figure 1 As shown in the flowchart of a multi-modal based data parsing method provided by an embodiment of the present disclosure, this method can be executed by a multi-modal based data parsing apparatus. As Figure 1 shown, this method includes the following steps:

[0040] S101. Obtain an image to be parsed; wherein, the image to be parsed contains structured data.

[0041] Exemplarily, the structured data can be data in the form of a formula, and the image to be parsed is an image containing structured data, that is, an image containing a formula. The image to be parsed can be obtained by taking pictures, screenshots, uploading, downloading, etc. The image to be parsed can contain one or more mathematical formulas and can also contain content other than mathematical formulas. For example, it can contain other patterns, texts, etc.

[0042] Figure 2 is a schematic diagram of the image to be parsed. Figure 2It contains a mathematical formula, which is y = sinx. That is, the LaTeX syntax structure data of this mathematical formula can be analyzed and obtained.

[0043] S102. Perform feature extraction processing on the image to be analyzed to obtain the image feature vector of the image to be analyzed; wherein, the image feature vector represents the structured data in the image to be analyzed.

[0044] Exemplarily, a feature extraction network can be pre-constructed to perform feature extraction processing on the image to be analyzed and obtain the image feature vector from the image to be analyzed. The image feature vector can represent the structured data in the image to be analyzed. The feature extraction network can be a convolutional neural network. In this embodiment, the network structure of the feature extraction network is not specifically limited. For example, it can include a convolutional layer, a pooling layer, etc.

[0045] The image to be analyzed is data in the image modality, and the feature extraction network is a visual network. Therefore, the image feature vector obtained by performing feature extraction processing on the image to be analyzed is feature data related to the image modality of the mathematical formula.

[0046] S103. Determine the analysis result of the image to be analyzed according to the image feature vector and the preset prompt word information; wherein, the preset prompt word information is text data representing the preset syntax structure, and the analysis result represents the structured data expressed in the preset syntax structure.

[0047] Exemplarily, prompt word information is preset, and the prompt word information is text modality data of preset content. The content of the prompt word information can represent the preset syntax structure. Through the prompt word information, the mathematical formula in the image to be analyzed can be converted into the structured data of this syntax structure, that is, the expression form of the mathematical formula is changed. For example, the content of the prompt word information can be "You are an expert in structured data analysis. Please analyze the formula into LaTeX structured data. The syntax structure requirements of the LaTeX structured data are XXX".

[0048] The image feature vector is feature data related to the image modality, and the prompt word information is the quantity in the text modality. The two modalities of image and text can be combined to analyze the mathematical formula in the image to be analyzed to obtain the analysis result. The analysis result is the structured data expressed in the preset syntax structure, that is, the expression form of the mathematical formula in the image is changed. For example, the mathematical formula in the image to be analyzed is expressed as The analysis result is sum_{i=1}^n i=\frac{n(n + 1)}{2}.

[0049] When combining the image feature vector and the prompt information, the prompt information can be first subjected to feature extraction processing to obtain feature data related to the text modality. Then, by combining the feature data related to the image modality and the feature data related to the text modality, the parsing result of the image to be parsed is obtained. For example, the image feature vector and the feature data related to the text modality can be input into a preset parsing model, and the parsing model fuses these two types of feature data, so that the image feature vector is converted into a parsing result according to the feature data related to the text modality. It is also possible to directly input the image feature vector and the preset prompt information into the preset parsing model to obtain the parsing result. In this embodiment, the parsing model can be a pre-trained neural network model for outputting the parsing result. The model structure of the parsing model is not specifically limited in this embodiment. When fusing the two types of feature data, alignment, splicing, etc. can also be performed to enhance the correlation between the mathematical formula and the LaTeX syntax structure.

[0050] In the embodiments of the present disclosure, first, feature extraction is performed on the image to be parsed in the image modality to obtain an image feature vector of the mathematical formula in the image. Then, by combining the prompt information in the text modality, the final parsing result is obtained. It realizes the multi-modal data fusion, enhances the corresponding relationship between the mathematical formula and the LaTeX syntax structure, and improves the accuracy, readability, and structuring degree of LaTeX parsing.

[0051] Figure 3 It is a schematic flowchart of a multi-modal data parsing method provided by the embodiments of the present disclosure, and this embodiment is an optional embodiment based on the above embodiments.

[0052] In this embodiment, feature extraction processing is performed on the image to be parsed to obtain an image feature vector of the image to be parsed, including: inputting the image to be parsed into a preset visual dictionary network to obtain the visual features of the image to be parsed; wherein, the preset visual dictionary network represents a convolutional network for extracting visual features, and the visual features represent the visual information in the image to be parsed; performing dimensionality reduction processing on the visual features of the image to be parsed to obtain an image feature vector of the image to be parsed.

[0053] As Figure 3 shown, the method includes the following steps:

[0054] S301. Obtain the image to be parsed; wherein, the image to be parsed contains structured data.

[0055] Exemplarily, this step can refer to the above step S101 and will not be elaborated here.

[0056] S302. Input the image to be parsed into a preset visual dictionary network to obtain the visual features of the image to be parsed. Among them, the preset visual dictionary network represents a convolutional network for extracting visual features, and the visual features represent the visual information in the image to be parsed.

[0057] Exemplarily, a visual dictionary network is pre-constructed. The visual dictionary network is a convolutional network for visual feature extraction. That is, the visual dictionary network can be a kind of feature extraction network. The visual dictionary is a technology used in computer vision and image processing. In practical applications, the visual dictionary network can help improve the efficiency and accuracy of image processing and analysis. In this embodiment, the network structure of the visual dictionary network is not specifically limited.

[0058] Input the image to be parsed into the visual dictionary network, and the visual dictionary network outputs the visual features of the image to be parsed. The visual features can represent the visual information in the image to be parsed, that is, the visual-related information of the structured data can be extracted. For example, information such as the position and contour of symbols in the structured data can be obtained.

[0059] In this embodiment, the preset visual dictionary network includes a first dictionary network and a second dictionary network. The first dictionary network represents a convolutional network for extracting visual features at a first resolution, and the second dictionary network represents a convolutional network for extracting visual features at a second resolution. The first resolution is greater than the second resolution. Inputting the image to be parsed into the preset visual dictionary network to obtain the visual features of the image to be parsed includes: inputting the image to be parsed into the first dictionary network to obtain the first features of the image to be parsed. Among them, the first features represent the visual features output by the first dictionary network; and inputting the image to be parsed into the second dictionary network to obtain the second features of the image to be parsed. Among them, the second features represent the visual features output by the second dictionary network.

[0060] Specifically, two visual dictionary networks can be pre-constructed, namely the first dictionary network and the second dictionary network. The network architectures of the two visual dictionary networks can be the same, but the network parameters are different. For example, the resolution parameters of the two visual dictionary networks are different, that is, the features of mathematical formulas at different resolutions can be extracted.

[0061] The first dictionary network extracts visual features at the first resolution, and the second dictionary network extracts visual features at the second resolution. The first resolution is greater than the second resolution. For example, the first resolution is 1024×1024, and the second resolution is 224×224. That is to say, the first dictionary network can extract high-resolution and high-precision visual features to understand the structured data in more fine-grained detail, and the second dictionary network can extract low-resolution and low-precision visual features to globally understand the structured data.

[0062] Input the image to be parsed into the first dictionary network and the second dictionary network respectively. Output the high-precision structured data visual features, i.e., the first features, through the first dictionary network, and output the low-precision general visual features as the second features through the second dictionary network. Both the first features and the second features are used as the visual features of the image to be parsed. That is, the visual features of the image to be parsed can include the first features and the second features.

[0063] The beneficial effect of such a setting is that different-resolution visual features are extracted through two visual dictionary networks, realizing the combination of multi-scale features, avoiding feature omission, enhancing the robustness of structured data parsing, significantly reducing the error rate of parsing, and improving the parsing accuracy.

[0064] S303. Perform dimensionality reduction on the visual features of the image to be parsed to obtain the image feature vector of the image to be parsed.

[0065] Exemplarily, after obtaining the visual features, dimensionality reduction can be performed on the visual features to uniformly process the visual features into a preset vector dimension, generating a vectorized representation of the mathematical formula for subsequent calculations. For example, dimensionality reduction can be performed through a preset Embedding layer. That is, an Embedding layer can be connected after the visual dictionary network, and the Embedding layer can receive the visual features output by the visual dictionary network and perform dimensionality reduction on the visual features. The data after dimensionality reduction is determined as the image feature vector of the image to be parsed.

[0066] In this embodiment, the visual features of the mathematical formula are extracted through the formula visual dictionary network, that is, the feature data in the image modality is obtained, which is convenient for subsequent fusion of multi-modal features. Then dimensionality reduction is performed to standardize the dimension of the feature data and improve the subsequent calculation efficiency.

[0067] In this embodiment, performing dimensionality reduction on the visual features of the image to be parsed to obtain the image feature vector of the image to be parsed includes: inputting the first features and the second features into a preset embedding layer respectively for dimensionality reduction to obtain the vector after dimensionality reduction corresponding to the first features and the vector after dimensionality reduction corresponding to the second features; determining the vector after dimensionality reduction corresponding to the first features as the first dimensionality reduction vector, and determining the vector after dimensionality reduction corresponding to the second features as the second dimensionality reduction vector; and performing fusion processing on the first dimensionality reduction vector and the second dimensionality reduction vector to obtain the image feature vector of the image to be parsed.

[0068] Specifically, if the visual dictionary network includes a first dictionary network and a second dictionary network, an Embedding layer can be connected after each of the first dictionary network and the second dictionary network. The first feature is input to the Embedding layer after the first dictionary network, and the second feature is input to the Embedding layer after the second dictionary network. The Embedding layer performs dimensionality reduction processing on the received visual features respectively, and obtains the dimensionality-reduced vector corresponding to the first feature and the dimensionality-reduced vector corresponding to the second feature. That is to say, the Embedding layer after the first dictionary network outputs the dimensionality-reduced vector corresponding to the first feature, and the Embedding layer after the second dictionary network outputs the dimensionality-reduced vector corresponding to the second feature.

[0069] The dimensionality-reduced vector corresponding to the first feature is determined as the first dimensionality-reduced vector. The dimensionality-reduced vector corresponding to the second feature is determined as the second dimensionality-reduced vector. Then, the first dimensionality-reduced vector and the second dimensionality-reduced vector are fused, and the fusion result is the image feature vector of the image to be parsed. The vector dimensions of the first dimensionality-reduced vector and the second dimensionality-reduced vector are the same, and they can be directly fused in matrix-vector form. For example, addition processing can be performed to obtain the image feature vector.

[0070] The beneficial effect of such a setting is that the two visual features are respectively subjected to dimensionality reduction processing to obtain two matrix vectors with the same dimensionality size, and then fused to achieve multi-scale visual feature fusion, enhance robustness, and improve the parsing accuracy of structured data.

[0071] In this embodiment, the method further includes: obtaining a pre-collected training dataset; wherein, the training dataset includes a first image and a second image, the first image contains structured data, and the second image does not contain structured data; inputting the first image and the second image into the initial model to obtain a model output result; wherein, the initial model includes a first dictionary network to be trained, and the model output result represents whether the image contains structured data; training the initial model according to the model output result to obtain a trained first dictionary network.

[0072] Specifically, the second dictionary network can extract visual features with low resolution and is a currently common visual dictionary network. The first dictionary network extracts high-precision visual features and needs to be pre-trained.

[0073] A training dataset is pre-collected, and the training dataset can include a large number of first images and second images. The first image refers to an image containing structured data, and the second image refers to an image not containing structured data. For example, the first image is a formula image, and the second image is a non-formula image. A first image and a second image can form a sample pair. The training dataset is a sample pair set, including multiple sample pairs, and the second image is a negative sample.

[0074] An initial model is pre-constructed, and the initial model can be used to determine whether the image contains structured data. The first image and the second image are both input into the initial model to obtain a model output result, which characterizes whether the image contains structured data, that is, it outputs whether the first image contains structured data and whether the second image contains structured data. The initial model is trained according to the model output result. For example, if the output result indicates that the first image does not contain structured data or the second image contains structured data, it means that the model is not trained yet, and it can be trained according to the backpropagation algorithm and the preset loss function until the model training is completed. In this embodiment, the training process of the model is not specifically limited. By using negative sample contrast learning, the ability of the model to distinguish between mathematical formulas and ordinary text images can be improved, thereby enhancing the ability to extract formula features.

[0075] The architecture of the first dictionary network is pre-constructed as the first dictionary network to be trained. The initial model may include the first dictionary network to be trained. The first dictionary network can be used to extract features from the first image and the second image respectively, and determine whether the image contains structured data according to the extracted visual features. If the initial model training is completed, the first dictionary network is also trained, that is, the trained first dictionary network is obtained.

[0076] The beneficial effect of such a setting is that the second dictionary network is a general visual dictionary network, and the first dictionary network is a trained visual dictionary network. The first dictionary network can be in the initial model to train the initial model, and after training, the first dictionary network in the model is obtained, so as to use the first dictionary network to extract the fine-grained visual features of mathematical formulas and improve the accuracy of formula recognition. By training the first dictionary network, the understanding of the fine-grained formula structures such as complex symbols, superscripts, subscripts, and matrices is enhanced, and the parsing error rate is significantly reduced.

[0077] In this embodiment, the initial model includes the first dictionary network to be trained and the pre-trained generative model; the first image and the second image are input into the initial model to obtain a model output result, including: inputting the first image and the second image into the first dictionary network to be trained to obtain the visual feature corresponding to the first image and the visual feature corresponding to the second image; determining the visual feature corresponding to the first image as the first training feature, and determining the visual feature corresponding to the second image as the second training feature; inputting the first training feature and the first training feature into the pre-trained generative model to obtain the model output result corresponding to the first image and the model output result corresponding to the second image.

[0078] Specifically, the initial model includes a first dictionary network to be trained and a pre-trained generative model. The first dictionary network can be located before the pre-trained generative model. In this embodiment, no specific limitation is imposed on the pre-trained generative model. For example, the pre-trained generative model can be the OPT 1.2B pre-trained model.

[0079] Input the sample pair into the initial model, that is, input the first image and the second image into the first dictionary network first. The first dictionary network extracts features from the first image and the second image respectively to obtain the visual features of the first image and the visual features of the second image. The visual features can represent the visual information in the image. Determine the visual features of the first image as the first training feature, and determine the visual features of the second image as the second training feature.

[0080] Both the first training feature and the second training feature output by the first dictionary network are input into the pre-trained generative model, and the output model outputs the result. The pre-trained generative model determines the image category of the image corresponding to the first training feature and determines the image category of the image corresponding to the second training feature. That is, the output result of the model can represent the image category of the image, that is, whether there is structured data in the image. The image category can be one of the two categories: the first image or the second image. The pre-trained generative model outputs the judgment result of the image category. If the predicted image category of the model is the same as the actual image category, it is determined that the training of the initial model is completed, and the trained first dictionary network is obtained.

[0081] The beneficial effect of such a setting is that the first dictionary network can extract fine-grained features of structured data, and training the initial model is to train the first dictionary network. By predicting the image category, it can be determined whether the first dictionary network correctly extracts the visual features. By training the first dictionary network, the understanding of fine-grained formula structures such as complex symbols, subscripts, and matrices is enhanced, and the error rate of subsequent parsing is significantly reduced.

[0082] In this embodiment, the initial model includes an embedding layer; inputting both the first training feature and the second training feature into the pre-trained generative model to obtain the model output results of the first image and the second image includes: inputting both the first training feature and the second training feature into the embedding layer for dimensionality reduction processing to obtain the dimensionality-reduced vector corresponding to the first training feature and the dimensionality-reduced vector corresponding to the second training feature; determining the dimensionality-reduced vector corresponding to the first training feature as the third dimensionality-reduced vector, and determining the dimensionality-reduced vector corresponding to the second training feature as the fourth dimensionality-reduced vector; inputting both the third dimensionality-reduced vector and the fourth dimensionality-reduced vector into the pre-trained generative model to obtain the model output results of the first image and the second image.

[0083] Specifically, the initial model further includes an embedding layer, which is located between the first dictionary network and the pre-trained generative model and can be used to reduce the dimensionality of visual features. After obtaining the first training feature and the second training feature, the first dictionary network can input the first training feature and the second training feature into the embedding layer, and the embedding layer respectively reduces the dimensionality of the first training feature and the second training feature to obtain a reduced-dimensional vector corresponding to the first training feature and a reduced-dimensional vector corresponding to the second training feature. The reduced-dimensional vector corresponding to the first training feature is determined as the third reduced-dimensional vector, and the reduced-dimensional vector corresponding to the second training feature is determined as the fourth reduced-dimensional vector.

[0084] Then, the third reduced-dimensional vector and the fourth reduced-dimensional vector are input into the pre-trained generative model. The pre-trained generative model determines the image category of the image corresponding to the third reduced-dimensional vector and determines the image category of the image corresponding to the fourth reduced-dimensional vector, thereby outputting the model output result. The result output by the model can include the model output result of the first image and the model output result of the second image. The model output result of the first image is the image category of the image corresponding to the third reduced-dimensional vector, and the model output result of the second image is the image category of the image corresponding to the fourth reduced-dimensional vector.

[0085] The beneficial effect of such a setting is that by setting the embedding layer, the dimensionality of visual features can be reduced, the vector dimensions of visual features can be unified, which is convenient for the calculation of the pre-trained generative model, and the determination efficiency and accuracy of the model output result can be improved.

[0086] Figure 4 It is a schematic diagram of the training process of the first dictionary network. Figure 4 The upper image in does not contain formulas, and the lower image contains formulas. These two images are used as a pair of sample pairs and input into the first dictionary network. The dashed arrow represents the data processing flow of the upper image, and the solid arrow represents the data processing flow of the lower image. The first dictionary network outputs the visual features of the two images and inputs them into the embedding layer. The embedding layer then inputs the two reduced-dimensional vector data into the pre-trained generative model. The model output result corresponding to the upper image is "This is an unstructured data image", and the model output result corresponding to the lower image is "This is a structured data image".

[0087] S304. Determine the parsing result of the image to be parsed according to the image feature vector and the preset prompt word information; wherein, the preset prompt word information is text data representing a preset grammar structure, and the parsing result represents structured data expressed in the preset grammar structure.

[0088] Exemplarily, this step can refer to the above step S103 and will not be elaborated here.

[0089] In the embodiments of the present disclosure, first, feature extraction is performed on the image to be parsed in the image modality to obtain an image feature vector of a mathematical formula in the image. Then, in combination with the prompt information in the text modality, a final parsing result is obtained. This realizes multi-modal data fusion, enhances the correspondence between the mathematical formula and the LaTeX syntax structure, and improves the accuracy, readability, and structuring degree of LaTeX parsing.

[0090] Figure 5 FIG. is a schematic flowchart of a data parsing method based on multi-modalities provided by an embodiment of the present disclosure. This embodiment is an optional embodiment based on the above embodiment.

[0091] In this embodiment, according to the image feature vector and the preset prompt information, the parsing result of the image to be parsed is determined, including: determining the text vector data corresponding to the preset prompt information; where the text vector data is the prompt information in vector form; and determining the parsing result of the image to be parsed according to the image feature vector and the text vector data.

[0092] As Figure 5 shown, the method includes the following steps:

[0093] S501. Obtain the image to be parsed; where the image to be parsed contains structured data.

[0094] Exemplarily, this step can refer to the above step S101 and will not be elaborated herein.

[0095] S502. Perform feature extraction processing on the image to be parsed to obtain an image feature vector of the image to be parsed; where the image feature vector characterizes the structured data in the image to be parsed.

[0096] Exemplarily, this step can refer to the above step S102 and will not be elaborated herein.

[0097] S503. Determine the text vector data corresponding to the preset prompt information; where the text vector data is the prompt information in vector form.

[0098] Exemplarily, there is preset prompt information, and the prompt information is text data of preset content. The content of the prompt information can be a preset syntax structure, for example, it can be the syntax structure of LaTeX formula structured data.

[0099] Convert the prompt information into vector-form data. For example, a preset feature extraction network can be used to perform feature extraction processing on the prompt information, and the obtained vector is determined as the text vector data. In this embodiment, the extraction method of the text vector data is not specifically limited.

[0100] S504. Determine the parsing result of the image to be parsed based on the image feature vector and the text vector data.

[0101] Exemplarily, combine the feature vector of the image modality and the feature vector of the text modality, that is, combine the image feature vector and the text vector data, to obtain the parsing result of the image to be parsed. For example, the image feature vector and the text vector data can be jointly input into a preset large model, and the large model performs operations such as splicing or fusion on these two vectors to obtain the parsing result.

[0102] In this embodiment, by combining the prompt information, the LaTeX parsing ability is enhanced, the joint modeling of vision and language is realized, and the accuracy of structured data parsing is improved.

[0103] In this embodiment, determining the parsing result of the image to be parsed according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data includes: performing alignment processing on the image feature vector and the text vector data to obtain the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data; determining the parsing result of the image to be parsed according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data.

[0104] Specifically, after converting the prompt information into a vector form, the vector dimensions of the image feature vector and the text vector data may be different, which is not conducive to the subsequent splicing of the two vectors. Therefore, the text vector data and the image feature vector can be aligned. Alignment means unifying the vector dimensions, that is, the matrix width. For example, the aligned vector corresponding to the image feature vector has a dimension of 256×768, and the aligned vector corresponding to the text vector data has a dimension of 5×768.

[0105] Input the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data into a preset large model, and the large model outputs the parsing result of the image to be parsed.

[0106] The beneficial effect of such a setting is that through the multi-modal alignment of the image feature vector and the text vector data, the association between the structured data and the LaTeX syntax structure is enhanced, and the accuracy and readability of LaTeX parsing are improved.

[0107] In this embodiment, determining the parsing result of the image to be parsed according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data includes: performing splicing processing on the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data to obtain a spliced vector; wherein, the spliced vector represents the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data; input the spliced vector into a preset large model to obtain the parsing result of the image to be parsed output by the preset large model.

[0108] Specifically, the aligned vector corresponding to the text vector data is determined as the Text Token, and the aligned vector corresponding to the image feature vector is determined as the Image Token. The Text Token and the Image Token are subjected to a concat (concatenation) process, that is, the two vectors are combined into one vector to obtain the concatenated vector. For example, if the Text Token is 5×768 and the Image Token is 256×768, then the concatenated vector is (5 + 256)×768.

[0109] Then, the concatenated vector is input into a preset large model to obtain the parsing result of the image to be parsed output by the large model. For example, the large model can be a 7B-scale multimodal large model. Using a 7B-scale multimodal large model for LaTeX formula parsing can improve the parsing accuracy of complex mathematical formulas.

[0110] The beneficial effect of such a setting is that through the concatenation of the Image Token and the Text Token, the combination of vision and text is realized, and the parsing accuracy of complex mathematical formulas and the structural degree of the parsing result are improved.

[0111] Figure 6 It is a flowchart of multimodal formula parsing. The image to be parsed is input to the first dictionary network and the second dictionary network at the same time. The first dictionary network outputs the first feature to the embedding layer to obtain the first reduced-dimensional vector, and the second dictionary network outputs the second feature to the embedding layer to obtain the second reduced-dimensional vector. The first reduced-dimensional vector and the second reduced-dimensional vector are fused through an ADD (fusion) layer to obtain the image feature vector. That is, the ADD layer can fuse the feature information from different visual networks. There is a preset prompt word information, and the vectorized representation of the prompt word information is obtained as the text vector data. The image feature vector and the text vector data are subjected to multimodal alignment processing to obtain the Image Token and the Text Token. The Image Token and the Text Token are concatenated by concat to obtain the concatenated vector, and after large model inference, the LaTeX formula structured parsing result is finally output.

[0112] The large model can be pre-trained. If training is required, the image to be trained is also input into two visual dictionary networks to obtain two visual features. After passing through the embedding layer and the ADD layer, the fused image feature vector to be trained is obtained. Determine the text vector data of the prompt information, align and splice the image feature vector to be trained and the text vector data, and input them into the initial large model to obtain the parsing result of the large model. The preset gradient descent method is used for training. For example, the preset loss function and backpropagation algorithm can be used until the large model meets the preset training completion conditions. In this embodiment, the training process of the large model is not specifically limited.

[0113] In the embodiments of the present disclosure, the image to be parsed in the image modality is first subjected to feature extraction to obtain the image feature vector of the mathematical formula in the image. Then, combined with the prompt information in the text modality, the final parsing result is obtained. It realizes the multi-modal data fusion, enhances the correspondence between the mathematical formula and the LaTeX syntax structure, and improves the accuracy, readability, and structuring degree of LaTeX parsing.

[0114] Figure 7 It is a structural block diagram of a multi-modal data parsing device provided by the embodiments of the present disclosure. For ease of explanation, only the parts related to the embodiments of the present disclosure are shown. Refer to Figure 7 , the multi-modal data parsing device 700 includes: an acquisition unit 701, an extraction unit 702, and an analysis unit 703.

[0115] The acquisition unit 701 is configured to acquire the image to be parsed; wherein, the image to be parsed contains structured data;

[0116] The extraction unit 702 is configured to perform feature extraction processing on the image to be parsed to obtain the image feature vector of the image to be parsed; wherein, the image feature vector represents the structured data in the image to be parsed;

[0117] The analysis unit 703 is configured to determine the parsing result of the image to be parsed according to the image feature vector and the preset prompt information; wherein, the preset prompt information is text data representing a preset syntax structure, and the parsing result represents the structured data expressed in the preset syntax structure.

[0118] Figure 8 It is a structural block diagram of a multi-modal data parsing device provided by the embodiments of the present disclosure, as Figure 8 shown, the multi-modal data parsing device 800 includes an acquisition unit 801, an extraction unit 802, and an analysis unit 803, wherein the extraction unit 802 includes a feature extraction module 8021 and a dimensionality reduction module 8022.

[0119] A feature extraction module 8021, configured to input the image to be parsed into a preset visual dictionary network to obtain visual features of the image to be parsed; wherein, the preset visual dictionary network represents a convolutional network for extracting visual features, and the visual features represent visual information in the image to be parsed.

[0120] A dimensionality reduction module 8022, configured to perform dimensionality reduction processing on the visual features of the image to be parsed to obtain an image feature vector of the image to be parsed.

[0121] In one example, the preset visual dictionary network includes a first dictionary network and a second dictionary network. The first dictionary network represents a convolutional network for extracting visual features at a first resolution, and the second dictionary network represents a convolutional network for extracting visual features at a second resolution. The first resolution is greater than the second resolution. The feature extraction module 8021 includes:

[0122] A first extraction sub-module, configured to input the image to be parsed into the first dictionary network to obtain a first feature of the image to be parsed; wherein, the first feature represents visual features output by the first dictionary network; and,

[0123] A second extraction sub-module, configured to input the image to be parsed into the second dictionary network to obtain a second feature of the image to be parsed; wherein, the second feature represents visual features output by the second dictionary network.

[0124] In one example, the dimensionality reduction module 8022 includes:

[0125] A first dimensionality reduction sub-module, configured to input the first feature and the second feature into a preset embedding layer respectively for dimensionality reduction processing to obtain a dimensionality-reduced vector corresponding to the first feature and a dimensionality-reduced vector corresponding to the second feature;

[0126] A second dimensionality reduction sub-module, configured to determine the dimensionality-reduced vector corresponding to the first feature as a first dimensionality-reduced vector, and determine the dimensionality-reduced vector corresponding to the second feature as a second dimensionality-reduced vector;

[0127] A fusion module, configured to perform fusion processing on the first dimensionality-reduced vector and the second dimensionality-reduced vector to obtain an image feature vector of the image to be parsed.

[0128] In one example, it further includes:

[0129] A data acquisition unit, configured to acquire a pre-collected dataset to be trained; wherein, the dataset to be trained includes a first image and a second image. The first image contains structured data, and the second image does not contain structured data.

[0130] An image input unit for inputting the first image and the second image into an initial model to obtain a model output result; wherein, the initial model includes a first dictionary network to be trained, and the model output result represents whether the image contains structured data;

[0131] A model training unit for training the initial model according to the model output result to obtain a trained first dictionary network.

[0132] In one example, the initial model includes a first dictionary network to be trained and a pre-trained generative model; the image input unit includes:

[0133] An image input module for inputting both the first image and the second image into the first dictionary network to be trained to obtain the visual feature corresponding to the first image and the visual feature corresponding to the second image;

[0134] A feature determination module for determining the visual feature corresponding to the first image as the first training feature and the visual feature corresponding to the second image as the second training feature;

[0135] A result output module for inputting both the first training feature and the second training feature into the pre-trained generative model to obtain the model output result corresponding to the first image and the model output result corresponding to the second image.

[0136] In one example, the initial model includes an embedding layer; the result output module is specifically configured to:

[0137] Input both the first training feature and the second training feature into the embedding layer for dimensionality reduction processing to obtain the dimensionality-reduced vector corresponding to the first training feature and the dimensionality-reduced vector corresponding to the second training feature;

[0138] Determine the dimensionality-reduced vector corresponding to the first training feature as the third dimensionality-reduced vector and the dimensionality-reduced vector corresponding to the second training feature as the fourth dimensionality-reduced vector;

[0139] Input both the third dimensionality-reduced vector and the fourth dimensionality-reduced vector into the pre-trained generative model to obtain the model output result of the first image and the model output result of the second image.

[0140] In one example, the parsing unit 803 includes:

[0141] A text determination module for determining the text vector data corresponding to the preset prompt word information; wherein, the text vector data is the prompt word information in vector form;

[0142] An image parsing module, configured to determine a parsing result of the image to be parsed according to the image feature vector and the text vector data.

[0143] In one example, the image parsing module includes:

[0144] An alignment sub-module, configured to perform alignment processing on the image feature vector and the text vector data to obtain an aligned vector corresponding to the image feature vector and an aligned vector corresponding to the text vector data;

[0145] A result determination sub-module, configured to determine a parsing result of the image to be parsed according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data.

[0146] In one example, the result determination sub-module is specifically configured to:

[0147] Perform splicing processing on the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data to obtain a spliced vector; wherein, the spliced vector represents the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data;

[0148] Input the spliced vector into a preset large model to obtain a parsing result of the image to be parsed output by the preset large model.

[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0150] Figure 9 A structural block diagram of an electronic device provided for an embodiment of the present disclosure is shown in Figure 9 As shown, the electronic device 900 includes: at least one processor 902; and a memory 901 communicatively connected to the at least one processor 902; wherein, the memory stores instructions executable by the at least one processor 902, and the instructions are executed by the at least one processor 902 so that the at least one processor 902 can execute the multi-modal data parsing method of the present disclosure.

[0151] The electronic device 900 further includes a receiver 903 and a transmitter 904. The receiver 903 is configured to receive instructions and data sent by other devices, and the transmitter 904 is configured to send instructions and data to external devices.

[0152] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0153] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which includes: a computer program stored in a readable storage medium, and at least one processor of the electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to execute the solution provided in any of the above embodiments.

[0154] Figure 10 FIG. shows a schematic block diagram of an exemplary electronic device 1000 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] As Figure 10 shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0156] A plurality of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0157] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as the multi-modal data parsing method. For example, in some embodiments, the multi-modal data parsing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the multi-modal data parsing method described above can be executed. Alternatively, in other embodiments, the computing unit 1001 can be configured to execute the multi-modal data parsing method by any other suitable means (e.g., by means of firmware).

[0158] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0162] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0163] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system or a server combined with a blockchain.

[0164] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and no limitation is imposed herein.

[0165] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A multimodal data analysis method, comprising: Acquire an image to be analyzed; wherein the image to be analyzed contains structured data; Performing feature extraction processing on the image to be analyzed to obtain an image feature vector of the image to be analyzed; wherein the image feature vector represents structured data in the image to be analyzed; The parsing result of the image to be parsed is determined according to the image feature vector and preset prompt word information; wherein the preset prompt word information is text data representing a preset grammatical structure, and the parsing result represents structured data expressed by the preset grammatical structure.

2. The method according to claim 1, wherein: The step of performing feature extraction processing on the image to be analyzed to obtain an image feature vector of the image to be analyzed includes: Inputting the image to be analyzed into a preset visual dictionary network to obtain visual features of the image to be analyzed; wherein the preset visual dictionary network represents a convolutional network for extracting visual features, and the visual features represent visual information in the image to be analyzed; Perform dimensionality reduction processing on the visual features of the image to be analyzed to obtain an image feature vector of the image to be analyzed.

3. The method according to claim 2, wherein: The preset visual dictionary network includes a first dictionary network and a second dictionary network, the first dictionary network represents a convolutional network for extracting visual features at a first resolution, the second dictionary network represents a convolutional network for extracting visual features at a second resolution, and the first resolution is greater than the second resolution; The step of inputting the image to be analyzed into a preset visual dictionary network to obtain visual features of the image to be analyzed includes: Inputting the image to be parsed into the first dictionary network to obtain a first feature of the image to be parsed; wherein the first feature represents a visual feature output by the first dictionary network; and, The image to be parsed is input into the second dictionary network to obtain a second feature of the image to be parsed; wherein the second feature represents a visual feature output by the second dictionary network.

4. The method according to claim 3, wherein: The step of performing dimensionality reduction processing on the visual features of the image to be analyzed to obtain an image feature vector of the image to be analyzed includes: Inputting the first feature and the second feature into a preset embedding layer for dimensionality reduction processing respectively, to obtain a dimensionality reduction vector corresponding to the first feature and a dimensionality reduction vector corresponding to the second feature; Determine the reduced-dimensionality vector corresponding to the first feature as a first reduced-dimensionality vector, and determine the reduced-dimensionality vector corresponding to the second feature as a second reduced-dimensionality vector; The first dimensionality reduction vector and the second dimensionality reduction vector are fused to obtain an image feature vector of the image to be analyzed.

5. The method according to claim 3 or 4, further comprising: Acquire a pre-collected data set to be trained; wherein the data set to be trained includes a first image and a second image, the first image contains structured data, and the second image does not contain structured data; Inputting the first image and the second image into an initial model to obtain a model output result; wherein the initial model includes a first dictionary network to be trained, and the model output result represents whether the image contains structured data; The initial model is trained according to the model output result to obtain a trained first dictionary network.

6. The method according to claim 5, wherein: The initial model includes a first dictionary network to be trained and a pre-trained generative model; the first image and the second image are input into the initial model to obtain a model output result, including: Inputting the first image and the second image into a first dictionary network to be trained to obtain visual features corresponding to the first image and visual features corresponding to the second image; Determining the visual feature corresponding to the first image as a first training feature, and determining the visual feature corresponding to the second image as a second training feature; The first training feature and the second training feature are both input into the pre-trained generative model to obtain a model output result corresponding to the first image and a model output result corresponding to the second image.

7. The method according to claim 6, wherein: The initial model includes an embedding layer; the first training feature and the second training feature are input into the pre-trained generative model to obtain a model output result corresponding to the first image and a model output result corresponding to the second image, including: Inputting the first training feature and the second training feature into the embedding layer for dimensionality reduction processing to obtain a dimensionality reduction vector corresponding to the first training feature and a dimensionality reduction vector corresponding to the second training feature; Determine the dimension-reduced vector corresponding to the first training feature as a third dimension-reduced vector, and determine the dimension-reduced vector corresponding to the second training feature as a fourth dimension-reduced vector; The third dimensionality reduction vector and the fourth dimensionality reduction vector are both input into the pre-trained generative model to obtain a model output result corresponding to the first image and a model output result corresponding to the second image.

8. The method according to any one of claims 1 to 7, wherein: Determining the analysis result of the image to be analyzed according to the image feature vector and preset prompt word information includes: Determine the text vector data corresponding to the preset prompt word information; wherein the text vector data is the prompt word information in vector form; The parsing result of the image to be parsed is determined according to the image feature vector and the text vector data.

9. The method according to claim 8, wherein: Determining the parsing result of the image to be parsed according to the image feature vector and the text vector data includes: Performing alignment processing on the image feature vector and the text vector data to obtain an aligned vector corresponding to the image feature vector and an aligned vector corresponding to the text vector data; The parsing result of the image to be parsed is determined according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data.

10. The method according to claim 9, wherein: The step of determining the parsing result of the image to be parsed according to the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data comprises: Performing splicing processing on the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data to obtain a spliced ​​vector; wherein the spliced ​​vector represents the aligned vector corresponding to the image feature vector and the aligned vector corresponding to the text vector data; The spliced ​​vector is input into a preset large model to obtain a parsing result of the image to be parsed output by the preset large model.

11. A multimodal data analysis device, comprising: An acquisition unit, used for acquiring an image to be analyzed; wherein the image to be analyzed contains structured data; An extraction unit, configured to perform feature extraction processing on the image to be analyzed to obtain an image feature vector of the image to be analyzed; wherein the image feature vector represents structured data in the image to be analyzed; The parsing unit is used to determine the parsing result of the image to be parsed according to the image feature vector and preset prompt word information; wherein the preset prompt word information is text data representing a preset grammatical structure, and the parsing result represents structured data expressed by the preset grammatical structure.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product, wherein: The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 10 when being executed by a processor.

Images