MTU data decoupling formation and model training method, system and device and medium

Abstract

The invention discloses an MTU data decoupling and model training method, system and device and a medium, which are mutually corresponding schemes, in the scheme, an MTU data synthesis process is decoupled into two independent steps of table image rendering and table question and answer pair generation, accurate MTU data can be synthesized by combining collected table data, and the accuracy of the MTU data is improved. The method has the advantages of low cost, high efficiency and high robustness, the generated question and answer pairs are more accurate and more coherent, and illusion is obviously reduced; on the basis, the method is used for training the MTU model, the performance of multi-modal table understanding can be greatly improved, experiments show that the synthesized MTU data are used for training the MTU model, the most advanced performance is achieved on multiple test sets, and the effectiveness and generalization of the method are proved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal table understanding, and in particular to a method, system, device and medium for decoupling MTU data and model training. Background Art

[0002] Multimodal Table Understanding (MTU) is a model that retrieves and infers the table image based on the user's question after the user inputs a table image and the corresponding question, and finally outputs the correct answer. Due to the limited scale of multimodal table understanding data, the performance of existing models is limited.

[0003] Although a large number of methods have emerged in the multimodal community to improve the performance of MTU by designing different network structures, such methods cannot fundamentally solve the problem of missing data, thus affecting the optimal performance of the model. Another method is to use a multimodal large language model (MLLM) to synthesize more samples, but this will lead to hallucinations and generate incorrect question-answer sample pairs, affecting the parameter optimization of the subsequent training of the MTU model, and the cost of annotating data is high.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, device and medium for decoupling MTU data into a model training method, system, device and medium, which can synthesize the MTU data required for MTU model training and train an MTU model with greatly improved performance.

[0006] The objective of the present invention is achieved through the following technical solutions: A method for decoupling MTU data, comprising: Collect tabular data; By using data enhancement technology, different table data are processed in a diversified manner to obtain visually diverse table images, wherein each table data corresponds to a table image; Generate prompt words corresponding to each table data based on the set question category, and guide the first language model to output questions and answers; The table image, question and answer corresponding to each table data are integrated to form MTU data, where the MTU is multimodal table understanding.

[0007] An MTU model training method, comprising: Build the MTU model, including: hybrid multi-resolution visual encoder, projection layer and second largest language model; The MTU data is generated by the aforementioned MTU data decoupling method, and the MTU model is trained, and the training steps include: for each MTU data, the resolution of the table image is respectively increased and decreased, and the obtained images are called high-resolution images and low-resolution images; the hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial context relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; the projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model, and outputs the converted visual features; the second largest language model outputs the answer according to the converted visual features and the corresponding question; a loss function is constructed by combining the answer output by the second largest language model and the answer contained in the MTU data, and the MTU model is optimized by using the loss function.

[0008] An MTU data decoupling system is used to implement the above-mentioned MTU data decoupling method, and the system includes: Data collection unit, used to collect tabular data; A table image rendering unit, used for performing diversified processing on different table data through data enhancement technology to obtain visually diversified table images, wherein each table data corresponds to a table image; A table question-answer pair generating unit, used to generate prompt words corresponding to each table data according to the set question category, and guide the first language model to output questions and answers; The MTU data generating unit is used to integrate the table image, question and answer corresponding to each table data to form MTU data, wherein the MTU is multimodal table understanding.

[0009] An MTU model training system, comprising: Model building unit, used to build the MTU model, including: hybrid multi-resolution visual encoder, projection layer and second largest language model; A model training unit is used to generate MTU data using the above-mentioned MTU data decoupling method and train the MTU model. The training steps include: for each MTU data, the resolution of the table image is respectively increased and decreased, and the images obtained are called high-resolution images and low-resolution images; the hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial context relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; the projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model and outputs the converted visual features; the second largest language model outputs the answer according to the converted visual features and the corresponding question; a loss function is constructed by combining the answer output by the second largest language model with the answer contained in the MTU data, and the MTU model is optimized using the loss function.

[0010] A processing device, comprising: one or more processors; a memory for storing one or more programs; Among them, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned MTU data decoupling method and MTU model training method.

[0011] A readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned MTU data decoupling method and MTU model training method.

[0012] It can be seen from the technical solution provided by the present invention that the MTU data synthesis process is decoupled into two independent steps: table image rendering and table question and answer pair generation. The collected table data can be combined to synthesize accurate MTU data, avoiding the problem of hallucinations that are easy to occur when using MLLM annotations. In addition, an MTU model can be trained in combination with MTU data and a significant performance improvement can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0014] Figure 1 A flowchart of a method for decoupling MTU data provided by an embodiment of the present invention; Figure 2 A schematic diagram of a method for decoupling MTU data provided by an embodiment of the present invention; Figure 3 A schematic diagram of an MTU model training method provided by an embodiment of the present invention; Figure 4 A schematic diagram of an MTU data decoupling system provided by an embodiment of the present invention; Figure 5 A schematic diagram of an MTU model training system provided by an embodiment of the present invention; Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0016] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0017] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0018] The following is a detailed description of a method, system, device and medium for decoupling MTU data and training a model provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed. The instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0019] Embodiment 1 The embodiment of the present invention provides a method for decoupling MTU data. Figure 1 As shown, it mainly includes the following steps: Step 1: Collect tabular data.

[0020] In the embodiment of the present invention, tabular data may be collected from open source data sets.

[0021] Preferably, the collected table data may be preprocessed, and in subsequent steps the preprocessed table data may be used to render table images and generate table question-answer pairs.

[0022] Specifically, the preprocessing step may include: converting the table data into table codes containing the table content and structure; filtering the table codes to retain the table codes whose number of rows and columns meets the set requirements; and then using the table codes obtained by preprocessing to generate corresponding table images, questions and answers.

[0023] Step 2: Rendering of table image.

[0024] In the embodiment of the present invention, different table data are processed in a diversified manner through data enhancement technology to obtain visually diversified table images, wherein each table data corresponds to a table image.

[0025] Preferably, the table attributes and table layouts in different table data can be processed in a diversified manner.

[0026] Step 3: Generate table question and answer pairs.

[0027] Generate prompt words corresponding to each table data in combination with the set question category, and guide the first language model to output questions and answers.

[0028] Preferably, the set question category is the MTU question type, and the question category is subdivided into specific subcategories; for each table data, a prompt word generator randomly selects several subcategories to generate corresponding prompt words.

[0029] Preferably, the prompt word includes a specified output format, guiding the first largest language model to output questions and answers in a corresponding format, and the answer output by the first largest language model includes two answers, one is an answer including a reasoning process, and the other is a direct result.

[0030] Step 4: Combine the table image, questions and answers corresponding to each table data to form MTU data.

[0031] It should be noted that the above-mentioned step numbers are mainly used to identify different steps and do not represent the execution order of the steps. The specific execution order of the steps can be determined by technical personnel in this field in combination with the content of the steps; for example, there is no strict execution order for step 2 and step 3. The two can be executed in parallel or in sequence in any order.

[0032] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0033] like Figure 2 As shown, it is a schematic diagram of the MTU data decoupling method provided by an embodiment of the present invention, which mainly includes two independent parts: table image rendering and table question-answer pair generation. The two parts are independent of each other and can work simultaneously. Before executing the above two parts, it is necessary to collect the necessary table data as a medium to synthesize the required MTU data. The following is a detailed introduction to each part of the work. In addition, the present invention can support multiple types of languages, for example, Figure 2 Examples are provided for English and Chinese language types.

[0034] 1. Data collection.

[0035] In an embodiment of the present invention, tabular data may be collected from an open source data set, and the tabular data may be in a text sequence format of a table, such as HTML (Hypertext Markup Language), LaTeX (a high-quality typesetting system format), and Markdown (a lightweight markup language).

[0036] Exemplarily, for example, 113K (K means thousand) and 652K text sequence tables can be collected from the open source datasets FinTabNet and TableLLama, respectively, mainly including tabular data in HTML and Markdown formats.

[0037] After that, preprocessing is required: (1) Convert the collected table data into table codes that only contain table content and structure. (2) Filter the table codes and retain those whose number of rows and columns meets the set requirements. For example, retain table codes with a row number between 3 and 60 and a column number of at least 3.

[0038] Exemplarily, 636K tables were finally screened out for subsequent synthesis.

[0039] 2. Table image rendering.

[0040] In the embodiments of the present invention, a series of data enhancement techniques are used to enhance the visual diversity of table images. These enhancement techniques are applied to various table attributes, such as background color, font size, and font style, to generate a series of visually distinct table images. In addition, different table layouts are randomly selected, including pure rows, pure columns, or fully gridded separators, to achieve more diverse visual effects. Through these enhancement techniques, visually diverse table images are finally synthesized, which are very similar to the table images in the existing MTU samples.

[0041] Exemplarily, through the above process, 636K <code, image> samples are synthesized.

[0042] 3. Generate table question and answer pairs.

[0043] Compared with directly using a large language model for table question-answer pair generation, this paper defines 6 main question categories and 11 more detailed subcategories to ensure that the synthesized data can cover a wide range of MTU question types. The question types in MTU are mainly divided into 6 categories, including 11 subcategories: retrieval (table retrieval), data operation (counting, sorting, determining ranges, filtering), numerical calculation (simple numerical calculation, complex calculation), free question answering (free answer questions), selection (multiple choice, table fact verification) and summary (table summary).

[0044] Retrieval involves identifying and extracting specific information or multiple table cells directly from a table. Data manipulation processes table data to retrieve answers based on questions. These operations include: counting, sorting, determining ranges, and filtering. Numerical calculations manipulate quantitative data in a table for analysis. This includes simple operations of adding, subtracting, multiplying, and dividing numbers (i.e. simple numerical calculations) as well as complex mixed operations (i.e. complex calculations). Free-response integrates facts and reasoning into a coherent sentence when answering questions. Selection involves choosing 1 correct option from N answers (i.e. multiple choice), usually N is 4. Table fact verification is a special case where N is 2 and requires a choice between a yes or no answer. Summarizing involves giving a concise and coherent description or title of the key information in the table.

[0045] In the embodiment of the present invention, a prompt word generator is designed to generate a specific prompt word template for standardizing the question type and output format generated by the first language model. Figure 2 As shown in the prompt word generator in , we first randomly select M different subcategories from the given task pool, where . The task pool represents the 6 question types defined above. In addition, the output format is specified in the prompt word, and a structured template is provided for the generated question-answer pair <question, answer>. Each question corresponds to a detailed answer and a short answer to reflect the problem-solving process. Through the above process, a variety of prompt templates are generated, which significantly enriches the diversity of question types generated by the first large language model. In addition, introducing detailed steps to reason and calculate the answers to questions in the reasoning process is crucial for the MTU large model to implement MTU tasks. Like humans solving problems, large language models are good at solving problems step by step instead of directly generating answers to questions, which increases the learning difficulty of the MTU model.

[0046] Exemplarily, the first language model can be selected as the Doubao-Pro model, which is a commercial Doubao-Pro model.

[0047] For example, 636K code is used to generate 1.8M (M stands for million) question-answer sample pairs <code, question, answer>. The generated image is then combined with the question-answer pairs using the code as an intermediate to create MTU data <image, question, answer>.

[0048] Compared with the existing multimodal table understanding synthesis method, the present invention has the following advantages: 1) Lower cost. Since the large language model is cheap and the number of input tokens of the table code is less than that of the image, the present invention achieves lower cost in the synthesis process. 2) Higher efficiency. The commercial large language model Doubao-Pro can handle parallel requests, which greatly improves the efficiency of data annotation. 3) Stronger robustness. The present invention directly processes structured table codes, so that the large language model can achieve higher accuracy and reliability in row and column logic, contextual reasoning and content consistency. In addition, the present invention can also reduce the visual noise and information loss of the table code. Compared with the existing MLLM synthesis method, the question and answer pairs generated by the large language model are more accurate and coherent, and the hallucinations are significantly reduced.

[0049] Embodiment 2 The embodiment of the present invention provides an MTU model training method, which mainly includes: (1) Construct the MTU model, including a hybrid multi-resolution visual encoder, a projection layer, and a second language model.

[0050] (2) Generate MTU data using the method provided in the preceding embodiment, and train the MTU model.

[0051] The training steps include: for each MTU data, the resolution of the table image is increased or decreased respectively, and the obtained images are called high-resolution images and low-resolution images; the hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial context relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; the projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model, and outputs the converted visual features; the second largest language model outputs the answer (the reasoning process and the final result) according to the converted visual features and the corresponding questions; a loss function is constructed by combining the answer output by the second largest language model with the answer contained in the MTU data, and the MTU model is optimized using the loss function.

[0052] Exemplarily, the second largest language model may be a Vicuna-1.5 7B model, which is an open source large language model based on a self-attention mechanism, and 7B (B stands for billion) is the number of parameters of the model.

[0053] like Figure 3 As shown in the figure, it is a schematic diagram of the MTU model training method. The hybrid multi-resolution visual encoder receives high-resolution and low-resolution table images as input, and uses the corresponding visual encoder to extract relevant local and global information, and finally concatenates them in the feature dimension. Among them, the high-resolution image allows the second largest language model to obtain more visual information, which is also important for table images. For example, a five-stage ConvNeXt can be used to encode the high-resolution image input and extract local features. Even though ConvNeXt can extract local features from the text area of ​​the table image, it cannot capture the spatial contextual relationship between cells. Therefore, low-resolution images are also used to model the global spatial relationship in the table image. For example, the ViT-L / 14 model can be used to model the global spatial relationship in the table image with a low-resolution image as input. Here, ConvNeXt is a convolutional neural network, and ViT-L / 14 is a self-attention neural network.

[0054] Specifically, given an input table image, its resolution is increased and adjusted to a fixed resolution size H×H to obtain a high-resolution image , H is the resolution height and width, for example, you can set H to 1536. Input ConvNeXt to obtain feature map , ConvNeXt converts the high-resolution image The image is downsampled by 64 times, resulting in 24×24 visual features, each with a dimension of 3072. At the same time, the resolution of the input table image is reduced to obtain a low-resolution image. For example, the resolution can be 336×336. After ViT-L / 14 processing, 576 visual features are output. , the size of each visual feature is 1024. Finally, the feature dimensions are merged and To form mixed-resolution visual features.

[0055] In the embodiment of the present invention, a two-layer multi-layer perceptron can be used as a projection layer to convert visual features into the embedding space of the second largest language model. Finally, the converted visual features are merged with the embedded text features and input into the second largest language model to generate answers. The embedded text features here refer to the features obtained after embedding the questions; similarly, the answers here include: answers including reasoning processes and direct results.

[0056] For each MTU data, the answer contained therein and the answer generated by the second largest language model can be used to construct a loss function, thereby optimizing the MTU model. The optimization process involved in this part can refer to conventional technology and will not be elaborated in the present invention.

[0057] Experiments show that the present invention uses synthetic MTU data to train the MTU model. In 24 in-domain and out-of-domain tests, the present invention achieves state-of-the-art performance on 21 test sets, proving the effectiveness and generalizability of the present invention.

[0058] The above solution provided by the embodiment of the present invention can answer the table images and questions input by the user, and finally the trained MTU model can be used for tasks such as table data analysis and understanding in daily work. In practice, it can be installed in a computer, mobile phone and other devices in the form of software to communicate with the user.

[0059] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0060] Embodiment 3 The present invention also provides an MTU data decoupling system, which is mainly used to implement the method provided in the above-mentioned embodiment 1, such as Figure 4 As shown, the system mainly includes: Data collection unit, used to collect tabular data; A table image rendering unit, used for performing diversified processing on different table data through data enhancement technology to obtain visually diversified table images, wherein each table data corresponds to a table image; A table question-answer pair generating unit is used to generate a question corresponding to each table data in combination with a set question category, and guide the first language model to output an answer to obtain a question and an answer corresponding to each table data; The MTU data generating unit is used to integrate the table image, question and answer corresponding to each table data to form MTU data, wherein the MTU is multimodal table understanding.

[0061] Considering that the MTU data decoupling process has been introduced in detail in the above embodiment, it will not be repeated here.

[0062] Embodiment 4 The present invention also provides an MTU model training system, which is mainly used to implement the method provided in the above-mentioned embodiment 2, such as Figure 5 As shown, the system mainly includes: Model building unit, used to build the MTU model, including: hybrid multi-resolution visual encoder, projection layer and second largest language model; A model training unit is used to generate MTU data using the aforementioned method and train the MTU model, wherein the training steps include: for each MTU data, the resolution of the table image is respectively increased and decreased, and the images obtained are called high-resolution images and low-resolution images; a hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial contextual relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; a projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model and outputs the converted visual features; the second largest language model outputs an answer based on the converted visual features and the corresponding question; a loss function is constructed by combining the answer output by the second largest language model with the answer contained in the MTU data, and the MTU model is optimized using the loss function.

[0063] Considering that the MTU model training process has been introduced in detail in the above embodiments, it will not be repeated here.

[0064] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the above embodiments only use the division of the above-mentioned functional modules as examples. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0065] Embodiment 5 The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments (i.e., MTU data decoupling method, MTU model training method).

[0066] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0067] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0068] Embodiment 6 The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the aforementioned embodiment (ie, the MTU data decoupling method and the MTU model training method).

[0069] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0070] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for decoupling MTU data, characterized in that: include: Collect tabular data; By using data enhancement technology, different table data are processed in a diversified manner to obtain visually diverse table images, wherein each table data corresponds to a table image; Generate prompt words corresponding to each table data based on the set question category, and guide the first language model to output questions and answers; The table image, question and answer corresponding to each table data are integrated to form MTU data, where the MTU is multimodal table understanding.

2. The MTU data decoupling method according to claim 1, characterized in that: Also includes: Preprocessing the collected table data includes: converting the table data into table codes containing table content and structure; filtering the table codes and retaining the table codes whose number of rows and columns meets the set requirements; Using the table codes obtained through preprocessing, the corresponding table images, questions and answers are generated.

3. The MTU data decoupling method according to claim 1, characterized in that: The diversified processing of different table data by using data enhancement technology includes: diversified processing of table attributes and table layouts in different table data.

4. The MTU data decoupling method according to claim 1, characterized in that: The generating of prompt words corresponding to each table data in combination with the set question category includes: The set problem category is the MTU problem type, and the problem category is subdivided into specific subcategories; For each table data, a prompt word generator randomly selects several subcategories to generate corresponding prompt words.

5. A method for decoupling MTU data according to claim 1 or 4, characterized in that: The prompt word includes a specified output format, guiding the first language model to output questions and answers in a corresponding format, and the answer output by the first language model includes two answers, one is an answer including a reasoning process, and the other is a direct result.

6. A MTU model training method, characterized in that: include: Build the MTU model, including: hybrid multi-resolution visual encoder, projection layer and second largest language model; The method according to any one of claims 1 to 5 is used to generate MTU data, and the MTU model is trained, wherein the training steps include: for each MTU data, the resolution of the table image is respectively increased and decreased, and the obtained images are called high-resolution images and low-resolution images; the hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial contextual relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; the projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model, and outputs the converted visual features; the second largest language model outputs an answer based on the converted visual features and the corresponding question; a loss function is constructed by combining the answer output by the second largest language model with the answer contained in the MTU data, and the MTU model is optimized using the loss function.

7. An MTU data decoupling system, characterized in that: For implementing the method described in any one of claims 1 to 5, the system comprises: Data collection unit, used to collect tabular data; A table image rendering unit, used for performing diversified processing on different table data through data enhancement technology to obtain visually diversified table images, wherein each table data corresponds to a table image; A table question-answer pair generating unit, used to generate prompt words corresponding to each table data according to the set question category, and guide the first language model to output questions and answers; The MTU data generating unit is used to integrate the table image, question and answer corresponding to each table data to form MTU data, wherein the MTU is multimodal table understanding.

8. An MTU model training system, characterized in that: include: Model building unit, used to build the MTU model, including: hybrid multi-resolution visual encoder, projection layer and second largest language model; A model training unit, used to generate MTU data using the method described in any one of claims 1 to 5, and train the MTU model, the training steps comprising: for each MTU data, performing resolution upscaling and downscaling operations on the table image, respectively, and the images obtained are called high-resolution images and low-resolution images; a hybrid multi-resolution visual encoder extracts local features from the text area in the high-resolution image, extracts the spatial contextual relationship between cells from the low-resolution image, obtains global features, and fuses the local features with the global features to obtain mixed-resolution visual features; a projection layer converts the mixed-resolution visual features to the embedding space of the second largest language model, and outputs the converted visual features; the second largest language model outputs an answer based on the converted visual features and the corresponding question; a loss function is constructed by combining the answer output by the second largest language model with the answer contained in the MTU data, and the MTU model is optimized using the loss function.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images