Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. Relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining problem information about table data in a table image library triggered by a target object; processing the problem information and the table image library through double encoders in a target model to obtain a plurality of table images related to the problem information; and obtaining target reply information based on the plurality of table images through a generation model in the target model, and returning the target reply information to the target object. Through the table data processing method and device, the problem that the accuracy of table data processing is low due to the fact that table data are queried and processed on the basis of a text mode in the related technology is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a data processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] In professional fields such as finance, medicine, and electricity, table data carries a large amount of structured and semi-structured information, including numbers, time series, classification data, and charts. Users often need to perform precise queries and complex logical operations on this data to support data analysis, report writing, and business decision-making. Existing technologies rely on text modalities for retrieval and analysis, such as using keyword matching to locate and extract information from tables. However, existing technologies have significant shortcomings when dealing with complex table data: text-based retrieval methods cannot fully understand the visual layout and structure of tables, such as merged cells, multi-level headers, and cross-page tables, leading to the omission or misidentification of key information, reducing the comprehensiveness and accuracy of query results.

[0003] To address the problem of low accuracy in processing table data based on text modalities in related technologies, no effective solutions have been proposed. SUMMARY

[0004] The main purpose of the present application is to provide a data processing method and device, a storage medium and an electronic device to solve the problem of low accuracy in processing table data based on text modalities in related technologies.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method includes: obtaining question information about table data in a table image library triggered by a target object; processing the question information and the table image library through a double encoder in a target model to obtain a plurality of table images related to the question information; obtaining target reply information based on the plurality of table images through a generation model in the target model, and returning the target reply information to the target object.

[0006] Further, the double encoder in the target model processes the question information and the table image library to obtain a plurality of table images related to the question information, including: encoding the table images in the table image library through a visual encoder in the double encoder to obtain first feature information; encoding the question information through a text encoder in the double encoder to obtain second feature information; and obtaining the plurality of table images from the table image library according to the first feature information and the second feature information.

[0007] Further, obtaining the plurality of table images from the table image library according to the first feature information and the second feature information comprises: performing similarity calculation on the first feature information and the second feature information to obtain a plurality of similarities; sorting table images in the table image library according to the plurality of similarities to obtain sorted table images; and obtaining the plurality of table images according to the sorted table images.

[0008] Further, obtaining target reply information based on the plurality of table images by a generative model in the target model comprises: performing structured analysis on each table image to obtain a semantic graph corresponding to each table image; and processing the plurality of table images, the semantic graphs, and the question information by the generative model to obtain the target reply information.

[0009] Further, performing structured analysis on each table image to obtain a semantic graph corresponding to each table image comprises: segmenting each table image to obtain a plurality of cell images; performing text recognition on the plurality of cell images to obtain a target string; positioning processing each table image to determine a row-column relationship between the plurality of cell images; and obtaining the semantic graph according to the target string and the row-column relationship.

[0010] Further, the target model is trained by the following steps: obtaining a training sample set, wherein the training sample set at least comprises a table image sample, structured data corresponding to the table image sample, a real question-answer pair corresponding to the table image sample, and an inference link corresponding to the real question-answer pair; processing the training sample set by an initial model to obtain a predicted question-answer pair; and iteratively processing the initial model according to the predicted question-answer pair to obtain the target model.

[0011] Further, iteratively processing the initial model according to the predicted question-answer pair to obtain the target model comprises: obtaining a target table image sample used by the initial model to output the predicted question-answer pair; obtaining a first reward value according to the association between the target table image sample and a question; obtaining a second reward value according to the association between the predicted question-answer pair and the real question-answer pair; and iteratively processing the initial model according to the first reward value and the second reward value to obtain the target model.

[0012] In order to achieve the above object, according to another aspect of the present application, a data processing apparatus is provided. The apparatus comprises: a first acquisition unit configured to acquire question information triggered by a target object about table data in a table image library; a processing unit configured to process the question information and the table image library by a dual encoder in a target model to obtain a plurality of table images related to the question information; and a determination unit configured to obtain target reply information based on the plurality of table images by a generation model in the target model, and return the target reply information to the target object.

[0013] Further, the processing unit comprises: a first encoding module configured to encode table images in the table image library by a visual encoder in the dual encoder to obtain first feature information; a second encoding module configured to encode the question information by a text encoder in the dual encoder to obtain second feature information; and a first determination module configured to obtain the plurality of table images from the table image library according to the first feature information and the second feature information.

[0014] Further, the first determination module comprises: a calculation submodule configured to calculate the similarity of the first feature information and the second feature information to obtain a plurality of similarities; a sorting submodule configured to sort table images in the table image library according to the plurality of similarities to obtain sorted table images; and a determination submodule configured to obtain the plurality of table images according to the sorted table images.

[0015] Further, the determination unit comprises: an analysis module configured to analyze each table image to obtain a semantic graph corresponding to each table image; and a processing module configured to process the plurality of table images, the semantic graphs and the question information by the generation model to obtain the target reply information.

[0016] Further, the analysis module comprises: a segmentation submodule configured to segment each table image to obtain a plurality of cell images; a recognition submodule configured to recognize text of the plurality of cell images to obtain target strings; a processing submodule configured to determine a row-column relationship between the plurality of cell images by positioning each table image; and a determination submodule configured to obtain the semantic graph according to the target strings and the row-column relationship.

[0017] Further, the target model is trained by using the following device: a second acquisition unit configured to acquire a training sample set, wherein the training sample set at least comprises a table image sample, structured data corresponding to the table image sample, a real question-answer pair corresponding to the table image sample, and an inference link corresponding to the real question-answer pair; a processing unit configured to process the training sample set by using an initial model to obtain a predicted question-answer pair; and an iteration unit configured to iterate the initial model according to the predicted question-answer pair to obtain the target model.

[0018] Further, the iteration unit comprises: an acquisition module configured to acquire a target table image sample used by the initial model to output the predicted question-answer pair; a second determination module configured to obtain a first reward value according to the association between the target table image sample and a question; a third determination module configured to obtain a second reward value according to the association between the predicted question-answer pair and the real question-answer pair; and an iteration module configured to iterate the initial model according to the first reward value and the second reward value to obtain the target model.

[0019] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a memory storing an executable program and a processor configured to run the program, wherein the program is configured to execute any of the above data processing methods when running.

[0020] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which stores a program, wherein the program is configured to control a device where the storage medium is located to execute any of the above data processing methods when running.

[0021] According to another aspect of the embodiments of the present application, a computer program product is also provided, which comprises a computer program or instructions, and the computer program or instructions are configured to implement any of the above data processing methods when executed by a processor.

[0022] In the embodiments of the present application, the following steps are adopted: acquiring question information about table data in a table image library triggered by a target object; processing the question information and the table image library by using a double encoder in a target model to obtain a plurality of table images related to the question information; and obtaining target reply information based on the plurality of table images by using a generation model in the target model, and returning the target reply information to the target object, thereby solving the technical problem in the related art that the accuracy of processing table data is relatively low because the table data is processed based on a text mode.

[0023] In this solution, a dual-encoder architecture performs deep learning processing on both the question information and the table image database. This not only understands the text-level query but also perceives the visual structure of the tables, enhancing retrieval accuracy. After processing, the dual encoder identifies multiple table images most relevant to the question. The generation model, based on these retrieved table images, integrates visual and textual information to generate a response. The dual encoder effectively integrates the visual features of table images with the semantic information of the text modality, enhancing the retrieval engine's capabilities and enabling it to handle more diverse and complex data structures. By fusing visual and textual modal information, the solution more accurately understands the question and locates the answer, avoiding information omissions or misunderstandings inherent in traditional text-based retrieval, thus achieving the technical effect of improving the accuracy of table data processing. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown.

[0026] Figure 2 This is a flowchart of a data processing method provided according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the target model provided according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of a data processing method provided according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application;

[0030] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the collected information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and related users or institutions are provided with an interface to provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0034] Embodiment 1

[0035] According to the embodiments of the present application, a method for data processing is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0036] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the data processing method is shown. As Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0040] The display can be a touch screen liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0041] In the above operating environment, the present application provides a data processing method as shown in Figure 2 Figure 2 is a flowchart of the data processing method according to an embodiment of the present application. The data processing method includes:

[0042] In step S201, question information about table data in the table image library triggered by a target object is obtained.

[0043] Optionally, the question input of the user (i.e., the target object) can be received through a user interface (such as a web form, a mobile application interface, or a voice interaction system). The question can be in natural language form, for example: “Please find out the revenue of all power companies in 2022”.

[0044] In step S202, the question information and the table image library are processed by a double encoder in the target model to obtain a plurality of table images related to the question information.

[0045] Optionally, the double encoder can be composed of a visual encoder and a text encoder. The visual encoder is configured to process image data in the table image library and extract visual features thereof, including table layout, cell structure, image elements, etc. The text encoder is responsible for processing the question information input by the user and converting it into text feature representation, including the theme, keywords, and context of the question.

[0046] ​Then, the table image can be analyzed by a visual encoder to extract key visual features that can reflect the structure information of the table and non-text information (such as charts, pictures, special symbols, etc.) contained in the image. The text encoder encodes the question information to generate text features that can express the intention of the question. After extracting the features of the question and the table, the similarity between the text features of the question information and the visual features of the table image can be calculated (for example, using cosine similarity, Euclidean distance or a double tower model) to find multiple table images that best match the question features. Through the processing of the question information and the table image library by the double encoder, the user demand can be effectively connected with the data resources, realizing the ability to accurately retrieve related tables from massive image data, and providing a solid foundation for subsequent data analysis and answer generation.

[0047] In step S203, the target reply information is obtained based on the plurality of table images by a generation model in the target model, and the target reply information is returned to the target object.

[0048] Optionally, after the retrieval phase ends, the generation model receives the multiple table images selected by the double encoder. For each table image, the generation model can convert the text content in the image into structured data and map the question information to the semantic level of the table data using natural language processing techniques. Based on the fused information, the generation model performs logical reasoning and data analysis to determine the most appropriate reply content. For example, the generation model calculates the table data (such as sum, average, percentage change), or analyzes the data distribution (such as finding the maximum value, minimum value or statistical data). In this process, the generation model understands the intention behind the question, such as whether the user is seeking a specific numerical value, comparing data changes between different time periods, etc. The generation model outputs the corresponding target reply information according to the question information and the content of the table data.

[0049] It should be noted that the target reply information can include direct answers, and can also include explanations, references to data sources or additional information to increase the explainability and completeness of the answers. For example, if the question asks "How much did the overall profit of the power company decrease in the first quarter of this year compared to last year?", the answer can include the specific number, the percentage decrease, the time period involved, and potential reasons for the decrease, etc.

[0050] In summary, the dual-encoder architecture separately processes the problem information and the table image library through deep learning, not only understanding the query at the text level, but also perceiving the visual structure of the table, enhancing the accuracy of the retrieval. After processing by the dual-encoder, the most relevant table images to the problem can be found, and the model generates reply information based on the retrieved table images, integrating visual and textual information. The dual-encoder effectively integrates the visual features of the table images and the semantic information of the text modal, enhancing the capabilities of the retrieval engine to handle more diverse and complex data structures. By fusing visual and textual modal information, the problem can be more accurately understood and the answer can be more accurately located, avoiding information omission or misunderstanding caused by traditional text-based retrieval, thereby achieving the technical effect of improving the accuracy of processing table data.

[0051] Optionally, in the data processing method provided by the embodiments of the present application, the processing of the problem information and the table image library by the dual-encoder in the target model to obtain a plurality of table images related to the problem information comprises: encoding the table images in the table image library by the visual encoder in the dual-encoder to obtain first feature information; encoding the problem information by the text encoder in the dual-encoder to obtain second feature information; and obtaining the plurality of table images from the table image library according to the first feature information and the second feature information.

[0052] In an optional embodiment, the visual encoder extracts deep features for each table image in the table image library, converting the table image into a visual feature vector (i.e., the first feature information described above), which contains multiple information such as visual elements, table structure, and graphic style in the image. The text encoder encodes the semantic information of the problem information submitted by the user, converting the natural language expression of the problem into second feature information.

[0053] According to the first feature information (visual features of the table image) and the second feature information (text features of the user's problem), cross-modal feature matching is performed. For example, by calculating similarity measures such as cosine similarity, Euclidean distance, and information entropy, those table images whose visual features match the text features, or in other words, those table images that best reflect the essence of the user's problem, are found. In this way, visual and semantic information can be considered comprehensively to achieve higher-dimensional data retrieval.

[0054] Finally, according to the results of the above feature matching, a plurality of table images related to the problem information are selected from the table image library.

[0055] By separating the visual encoder and the text encoder for processing and then performing feature fusion, key information can be extracted and matched from two different modalities, avoiding errors that may be caused by single-modal retrieval. For example, for data containing charts, special symbols or complex table structures, it is difficult for a text encoder alone to fully understand the content, while the dual-encoder framework can integrate visual and semantic information, significantly improving the accuracy of retrieval.

[0056] Optionally, in the data processing method provided by the embodiment of the application, obtaining a plurality of table images from the table image library according to the first feature information and the second feature information comprises: performing similarity calculation on the first feature information and the second feature information to obtain a plurality of similarities; sorting the table images in the table image library according to the plurality of similarities to obtain sorted table images; and obtaining the plurality of table images according to the sorted table images.

[0057] In an optional embodiment, the similarity calculation is performed on the first feature information (the table image feature extracted by the visual encoder) and the second feature information (the question information feature extracted by the text encoder), for example, cosine similarity, Jaccard similarity, Manhattan distance or Euclidean distance, etc.

[0058] According to the plurality of calculated similarities, the table images in the table image library are sorted to obtain an ordered list from the highest similarity to the lowest similarity. This sorting process ensures that the table images most closely related to the question information are placed in the front, facilitating subsequent retrieval and analysis work. Finally, according to the sorted results, the top several table images with the highest similarity are selected as the preliminary retrieval results related to the question information. The number of selections can be flexibly set according to actual conditions and requirements, which not only ensures the diversity of retrieval results, but also avoids the interference caused by too many irrelevant results.

[0059] Not only can the information related to the question be accurately retrieved from the table image library, but also the diversity and reliability of the retrieval results are increased by selecting multiple table images, providing high-quality input for subsequent data analysis and answer generation, and thus improving the technical effect of the accuracy of processing table data.

[0060] Optionally, in the data processing method provided by the embodiment of the application, the target reply information is obtained by the generation model in the target model based on the plurality of table images, comprising: performing structured analysis on each table image to obtain a semantic graph corresponding to each table image; and processing the plurality of table images, the semantic graphs and the question information by the generation model to obtain the target reply information.

[0061] In an optional embodiment, for each retrieved table image, a structured parsing operation is first performed. This process utilizes character recognition techniques and table structure parsing algorithms to convert the table image into readable structured data while constructing a semantic graph for each table image. For example, the semantic graph can represent cells data as nodes, logical relationships between cells such as adjacency, inclusion, equivalence, etc. as edges, and structural features of the table such as merged cells, associations between headers and data, etc. Through the above steps, visual information is transformed into machine-processable structured form, facilitating subsequent deep understanding and analysis.

[0062] Then, the parsed semantic graph, original table image, and user's question information are input into the generation model. The generation model can understand and integrate cross-modal information, capturing complex relationships between text and image, such as mapping text questions to specific cells or data items in the image. Utilizing the semantic graph to identify logical associations between data items, such as calculating growth rates, average values, etc. Combining visual information from the table image, such as charts, special symbols, or image annotations, to enhance understanding of the data. Based on the integration of multi-modal information, the generation model generates specific, accurate, and highly interpretable target answer information according to the user's question. For example, the target answer information can include the following content:

[0063] Direct numerical or statistical results, such as "The overall revenue of the power industry in 2022 is xxxxxx yuan". Logical processes of data analysis, such as "By comparing the data of 2022 and 2021, it is found that the revenue has increased by 5.3%". Additional information such as data sources, calculation methods or specific conditions to enhance the credibility and comprehensiveness of the answer.

[0064] Through the above steps, the table data can be deeply understood from multiple perspectives, not only dealing with complex multi-modal information, but also generating detailed, accurate and highly interpretable answers, greatly improving the quality of information retrieval and service.

[0065] Optionally, in the data processing method provided in the embodiments of the present application, the structured parsing of each table image to obtain the semantic graph corresponding to each table image comprises: segmenting each table image to obtain a plurality of cell images; performing text recognition on the plurality of cell images to obtain a target string; positioning each table image to determine the row-column relationship between the plurality of cell images; and obtaining the semantic graph according to the target string and the row-column relationship.

[0066] In an optional embodiment, the structured parsing of each table image includes the following steps: first, each retrieved table image needs to be segmented into multiple cell images. This process can use computer vision techniques such as edge detection, connected component analysis, template matching, etc. to identify and separate the cell regions in the table, creating separate image fragments for each cell.

[0067] Then, text recognition is performed on the segmented cell images to convert the text in the images into editable target strings. This step is crucial for extracting data from cells, whether it's numbers, units, dates, or other text descriptions, into a structured format for subsequent computer processing and semantic understanding.

[0068] After text recognition, the row and column relationships of each cell in the table image need to be located. For example, identify the location of the cell in the table, determine which cells are in the same row or column, and handle complex cases such as merged cells or cross-row / column data. This can be achieved by analyzing the layout structure of the table, such as using image analysis algorithms to identify cell boundaries and the spatial relationships between cells, thereby constructing a hierarchical and logical architecture between cells.

[0069] Finally, based on the obtained target strings and the row and column relationships between cells, a semantic graph of each table image is constructed. The semantic graph is a data structure that expresses the internal connections and logical relationships of table data. In the semantic graph, each node represents a cell data (target string), and the edge represents the relationship between cells (such as adjacent within a row, adjacent within a column, merged relationship, etc.). Constructing a semantic graph allows the generation model to better understand the structure and content of table data, thereby generating more accurate and logical answers.

[0070] Through fine table image segmentation and cell text recognition, the accuracy of data extraction is guaranteed, avoiding data processing errors caused by image quality problems or text recognition errors. The generation model based on the semantic graph can give the logical reasoning process of the answer, including the data source and calculation steps, increasing the transparency and verifiability of the answer, and improving the trust of users on the answer.

[0071] Optionally, in the data processing method provided in the embodiments of the present application, the following steps are used to train the target model: obtaining a training sample set, wherein the training sample set at least includes table image samples, structured data corresponding to the table image samples, real question and answer pairs corresponding to the table image samples, and inference links corresponding to the real question and answer pairs; processing the training sample set through an initial model to obtain predicted question and answer pairs; and iterating the initial model according to the predicted question and answer pairs to obtain the target model.

[0072] In an optional embodiment, the following steps can be used to train the target model: first, collect and prepare the training sample set. It should be noted that the training sample set can include table image samples: table images containing complex structures and multi-modal information can be obtained by scanning. Structured data: converting table images into structured data for model learning and understanding. Real question and answer pairs: typical query questions and corresponding correct answers annotated by business experts. Reasoning links: the thought process or calculation steps of each real question and answer pair, i.e. the process description of how to derive the answer from the table data.

[0073] Then, the initial model (untrained model) is used to process the table image samples and question information in the training sample set to generate predicted question and answer pairs. Finally, based on the differences between the predicted question and answer pairs and the real question and answer pairs, reinforcement learning, contrastive learning or supervised learning methods are used to iteratively optimize the initial model. After multiple rounds of iterative training, the model is optimized and converges to a stable state, at which point the model is the final target model. The target model can more accurately understand the semantics of table images and questions, generate predicted answers that closely match the real answers, and has the ability to perform logical reasoning from data.

[0074] For example, calculating the loss function: comparing the predicted answer with the real answer, calculating the loss as the direction of optimization. Parameter update: adjusting the model parameters according to the feedback of the loss function to reduce the difference between the predicted answer and the real answer.

[0075] It should be noted that the initial model can also output a predicted reasoning link, i.e. the model's step-by-step description of how to derive the predicted answer from the given table data, including but not limited to the cell position of data extraction, the calculation formula used, the data source cited, etc. Compare the predicted reasoning link with the reasoning link of the real question and answer pair to analyze the differences between the two, especially in terms of data extraction, logical reasoning and calculation steps. Then, based on the comparative analysis of the reasoning link, the specific direction of model parameter optimization can be determined. For example, if it is found that the model frequently extracts data from the wrong cell, the visual encoder's ability to recognize cell positions may need to be strengthened; if the model deviates in logical reasoning, the text encoder's understanding of question semantics needs to be optimized.

[0076] Since the training sample set contains table image samples and corresponding structured data, the target model learns to process visual information and semantic information simultaneously during the training process, thereby being able to more accurately locate key data when parsing complex tables and improving the accuracy of retrieval and understanding.

[0077] Optionally, in the data processing method provided in the embodiments of the present application, the iteration of the initial model according to the predicted question answer pair to obtain the target model comprises: obtaining a target table image sample used by the initial model when outputting the predicted question answer pair; obtaining a first reward value according to the relevance between the target table image sample and the question; obtaining a second reward value according to the relevance between the predicted question answer pair and the real question answer pair; and iterating the initial model according to the first reward value and the second reward value to obtain the target model.

[0078] In an optional embodiment, when outputting the predicted question answer pair by the initial model, the target table image sample related to the question is first searched, and then the predicted question answer pair is obtained according to the searched target table image sample.

[0079] In each iteration of the training, the initial model first needs to search for a target table image sample related to the input question. This searching process depends on the visual encoder of the model, which analyzes the keywords of the input question, matches with the visual features in the table image library, and thus locates the most relevant target table image sample.

[0080] The relevance between the searched target table image sample and the question is evaluated to obtain a first reward value. This reward value reflects the pros and cons of the model's retrieval ability, that is, whether the model can accurately find the table data highly related to the question. The calculation method can be based on multiple indicators such as the closeness of keyword matching, the relevance of table content, and the similarity of visual features.

[0081] Based on the searched target table image sample, the initial model uses the generative model to generate the predicted question answer pair. This generation process not only includes the answer itself, but also includes the reasoning link that the model considers, that is, the logical steps from data to answer. The generated predicted question answer pair is compared with the real question answer pair in the training sample set to obtain a second reward value. The second reward value measures the accuracy of the model in generating the answer, including the correctness of the answer content and the rationality of the reasoning logic. The second reward value can be obtained by calculating the similarity score of the answer and judging whether the calculation process of the answer conforms to the logical rules.

[0082] According to the first reward value and the second reward value, the initial model is iteratively optimized using a reinforcement learning algorithm (such as PPO, DPO, etc.). The model learns how to more accurately locate the relevant table data in the retrieval stage, and more accurately generate the answer pair matching the real answer in the generation stage. By adjusting the model parameters to maximize the reward value, the model gradually improves its ability to process complex table data.

[0083] By calculating two reward values, the retrieval and generation of two key links are optimized. The first reward value ensures that the model can find highly relevant table data for the question, while the second reward value ensures that the answer generated based on these data is accurate and reasonable. This double optimization mechanism significantly improves the overall performance of data processing and improves the accuracy of table analysis in professional fields such as power, finance, and medicine.

[0084] In an optional embodiment, the schematic diagram of the target model is as shown in Figure 3 The target model can be used to process queries of complex tables such as financial statements and market research data in an intelligent report analysis system. For example, the user asks the question "What is the trend of net profit growth of a certain company over the past five years?" The workflow of the target model is as follows: the visual encoder extracts key visual features from the report image, such as table title, data distribution, chart information, etc. The text encoder understands and encodes the key points of the question, such as "company name", "five years", "net profit", "growth trend", etc. The combination of visual features and semantic features provides a comprehensive modal representation for report retrieval, finding the most relevant report in the report library and sorting by matching degree. The generation model outputs "The company's net profit has shown a steady upward trend over the past five years, with an average annual growth rate of 10% and a maximum annual growth rate of 15%." The user can provide feedback through interaction with the system, such as confirming the accuracy of the results or asking more specific queries, to prompt the model to adapt and optimize based on the feedback, continuously improving its report analysis capabilities.

[0085] In an optional embodiment, data processing can be implemented as shown in the schematic diagram Figure 4 The data processing includes: modal training dataset construction: collecting table images and their corresponding structured data, annotating question-answer pairs and chain-of-thought by business experts, and aligning images, structured data, and annotated texts into cross-modal training samples. Multi-modal reinforcement learning training: using a dual encoder structure (visual encoder extracts table image features, and text encoder processes queries and table content), designing retrieval rewards (based on result relevance) and generation rewards (based on answer accuracy), and optimizing the policy model end-to-end through the DPO algorithm. Intelligent retrieval and generation: retrieval stage: input query text, calculate its similarity with the table image library through the multi-modal model, and return Top-K relevant table images and structured data; analysis stage: text recognition and structured analysis of the retrieved table to construct a table semantic graph (nodes are cell data, and edges are row-column / logical relationships); generation stage: input user questions, retrieved table data, and semantic graph into the generation model to output answers containing accurate numerical values and reasoning basis.

[0086] In an optional embodiment, the annual operating report intelligent analysis of provincial power companies can be realized by the present application:

[0087] Technical implementation: multi-modal training data: collect power revenue report scans (image modality) for nearly 5 years, and match Excel structured data, annotated by power economic experts Typical problem chain (such as "a regional annual electricity growth rate calculation: first locate the regional row → extract the electricity consumption of the current and last year → calculate the growth rate") Reinforcement learning training: visual encoder is particularly optimized for feature extraction of power report specific format (such as cross-page merged cells, numerical items with notes) Text encoder injects power industry terms (such as "line loss rate" "cross-region power transmission").

[0088] Actual application: input "2023 East China wind power consumption growth rate", the system automatically: (1) retrieve the report page containing "East China" and "wind power consumption"; (2) analyze the table to identify cross-page continuation notes; (3) generate the answer "23.7% (data source: 2023 report appendix 6, considering the statistical caliber adjustment in the notes)".

[0089] In an optional embodiment, the quarterly financial report intelligent interpretation can be realized by the present application:

[0090] Technical implementation: multi-modal training data: build a paired dataset containing 3000+ listed company financial report PDF and HTML table, and label typical analysis problems (such as "net profit change calculation: this period net profit → last period net profit → seasonal adjustment factor"); Special processing mechanism: for financial table characteristics: design currency unit automatic conversion module (such as "million US dollars → billion RMB"); Establish financial index correlation rule library (such as "net profit = total operating income - total operating cost + other income").

[0091] Actual application: when querying "A company's R&D expense ratio to operating income in 2024Q2": (1) accurately locate the "R&D expense" note table of the consolidated income statement; (2) automatically associate the operating income data of the main table; (3) generate "ratio 5.2%, 1.8pct higher than the industry average" and mark the data traceability path.

[0092] In an optional embodiment, the medical test report interpretation can be realized by the present application:

[0093] Technical implementation: Multimodal training data: Collect 100,000+ inspection report images (including blood routine, biochemistry, etc.), align with LIS system data, and label by chief physician: Abnormal value determination rules (such as "blood glucose > 6.1 mmol / L and fasting → label abnormal"); Multi-index association logic (such as "hemoglobin and red blood cell count need to be jointly determined"); Medical special optimization: Develop test table special recognition model (recognize handwritten numbers, special symbols ↑↓); Integrate medical reference value knowledge base (adjust threshold according to age / gender).

[0094] Actual application: Input "Is the patient ID12345's blood potassium a critical value?" (1) Retrieve the patient's latest biochemistry report; (2) Automatically match "Blood potassium 5.8 mmol / L (reference range 3.5-5.5)" when parsing; (3) Generate "High potassiumemia critical value, suggest immediate review" and highlight the relevant indicators.

[0095] In an optional embodiment, the present application can also realize scientific research literature Meta analysis:

[0096] Technical implementation: Multimodal training data: Collect 100,000+ experimental data tables in PDF papers, label: Discipline-specific table header semantics (such as "p<0.05*" indicates significance); Data correlation patterns (such as "Table 2 and Figure 3 need to be cross-verified"); Research-specific functions: Develop multi-table correlation analysis module; Integrate unit conversion (such as "ng / mL→μmol / L").

[0097] Actual application: Query "IL-6 levels in the experimental group in the paper [DOI:10.xxx]": (1) Accurately locate Table 3 in the Results section of the paper; (2) Analyze the significance markers marked with asterisks; (3) Generate "48.2±5.3 pg / mL (*p=0.032 vs control group)" and automatically convert to SI units.

[0098] In an optional embodiment, the present application can realize intelligent tracking of retail commodity prices:

[0099] Technical implementation: Multimodal training data: Collect price comparison screenshots from major e-commerce platforms, label: Promotion information recognition rules (such as "strikethrough price → original price"); Price fluctuation patterns (such as "historical lowest price judgment logic"); Business intelligence optimization: Develop dynamic table tracking module (handle AJAX loading data); Build price time series database.

[0100] Actual application: query "mobile phone X lowest price platform in the past 90 days": (1) retrieve price history snapshots of each platform; (2) analyze promotional notes (such as "limited-time discount"); (3) generate "lowest price ￥2999 (B platform 2024 / 6 / 18 during the big promotion, 15% lower than the average price)".

[0101] Through multi-modal training data, the model can understand both the visual structure and textual content of the table, reducing the problem of mis-matching or information omission caused by traditional text retrieval. Multi-modal reinforcement learning is used to optimize the model, enabling it to automatically learn the relevance of visual elements (such as row and column structure) and semantic content (such as numerical values and units) in table data, thereby improving the accuracy of complex queries.

[0102] The data processing method provided by the embodiments of the present application acquires question information about table data in the table image library triggered by a target object; a double encoder in a target model processes the question information and the table image library to obtain a plurality of table images related to the question information; and a generation model in the target model obtains target reply information based on the plurality of table images and returns the target reply information to the target object, thereby solving the technical problem of low accuracy of processing table data caused by querying and processing table data based on a text modality in related technologies.

[0103] In the present solution, the double encoder architecture performs deep learning processing on the question information and the table image library, not only understanding the text-level query but also perceiving the visual structure of the table, enhancing the accuracy of the retrieval. The double encoder can find the most relevant table images to the question after processing, and the generation model generates reply information based on the retrieved table images, integrating visual and textual information. The double encoder effectively integrates the visual features of the table images and the semantic information of the text modality, enhancing the capabilities of the retrieval engine to handle more diverse and complex data structures. By fusing visual and textual modalities, the question can be more accurately understood and the answer can be more accurately located, avoiding the information omission or misunderstanding caused by traditional text-based retrieval, thereby achieving the technical effect of improving the accuracy of processing table data.

[0104] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0105] Example 2

[0106] The embodiment of the present application also provides a data processing device. It should be noted that the data processing device of the embodiment of the present application can be used to execute the data processing method provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application is introduced as follows.

[0107] According to the embodiment of the present application, a device for implementing the above data processing method is also provided, as shown in the following formula (1). Figure 5 The device comprises a first acquisition unit 501, a processing unit 502 and a determination unit 503.

[0108] The first acquisition unit 501 is configured to acquire question information about table data in a table image library triggered by a target object.

[0109] The processing unit 502 is configured to process the question information and the table image library through a double encoder in a target model to obtain a plurality of table images related to the question information.

[0110] The determination unit 503 is configured to obtain target reply information based on the plurality of table images through a generation model in the target model, and return the target reply information to the target object.

[0111] The data processing device provided by the embodiment of the present application acquires question information about table data in a table image library triggered by a target object through the first acquisition unit 501. The processing unit 502 processes the question information and the table image library through a double encoder in a target model to obtain a plurality of table images related to the question information. The determination unit 503 obtains target reply information based on the plurality of table images through a generation model in the target model, and returns the target reply information to the target object. The technical problem that the accuracy of processing table data is relatively low in the related art because the table data is queried and processed based on a text mode is solved.

[0112] In the present scheme, the double encoder architecture respectively processes the question information and the table image library through deep learning, not only understands the query at the text level, but also perceives the visual structure of the table, thereby enhancing the accuracy of retrieval. The double encoder can find a plurality of table images most relevant to the question after processing. The generation model generates reply information based on the plurality of table images retrieved, and integrates visual and text information. The double encoder effectively integrates the visual features of the table image and the semantic information of the text mode, thereby enhancing the ability of the retrieval engine to cope with more diversified and complex data structures. By fusing the information of the visual and text modes, the question can be more accurately understood and the answer can be more accurately located, thereby avoiding information omission or misunderstanding caused by traditional text-based retrieval, and further achieving the technical effect of improving the accuracy of processing table data.

[0113] Optionally, in the data processing apparatus provided in the embodiments of the present application, the processing unit comprises: a first encoding module configured to encode the table image in the table image library by a visual encoder in the dual encoder to obtain first feature information; a second encoding module configured to encode the question information by a text encoder in the dual encoder to obtain second feature information; and a first determining module configured to obtain the plurality of table images from the table image library according to the first feature information and the second feature information.

[0114] Optionally, in the data processing apparatus provided in the embodiments of the present application, the first determining module comprises: a calculation sub-module configured to calculate the similarity of the first feature information and the second feature information to obtain a plurality of similarities; a sorting sub-module configured to sort the table images in the table image library according to the plurality of similarities to obtain sorted table images; and a determining sub-module configured to obtain the plurality of table images according to the sorted table images.

[0115] Optionally, in the data processing apparatus provided in the embodiments of the present application, the determining unit comprises: an analysis module configured to analyze each table image to obtain a semantic graph corresponding to each table image; and a processing module configured to process the plurality of table images, the semantic graphs and the question information by the generation model to obtain the target reply information.

[0116] Optionally, in the data processing apparatus provided in the embodiments of the present application, the analysis module comprises: a segmentation sub-module configured to segment each table image to obtain a plurality of cell images; a recognition sub-module configured to recognize text in the plurality of cell images to obtain a target string; a processing sub-module configured to position each table image to determine the row-column relationship between the plurality of cell images; and a determining sub-module configured to obtain the semantic graph according to the target string and the row-column relationship.

[0117] Optionally, in the data processing apparatus provided in the embodiments of the present application, the target model is trained by the following apparatus: a second obtaining unit configured to obtain a training sample set, wherein the training sample set comprises at least a table image sample, structured data corresponding to the table image sample, a real question answer pair corresponding to the table image sample, and an inference link corresponding to the real question answer pair; a processing unit configured to process the training sample set by an initial model to obtain a predicted question answer pair; and an iteration unit configured to iterate the initial model according to the predicted question answer pair to obtain the target model.

[0118] Optionally, in the data processing apparatus provided by the embodiment of the present application, the iteration unit comprises: an acquisition module, configured to acquire a target table image sample used when the initial model outputs a predicted question answer pair; a second determination module, configured to obtain a first reward value according to the relevance between the target table image sample and the question; a third determination module, configured to obtain a second reward value according to the relevance between the predicted question answer pair and a real question answer pair; and an iteration module, configured to iterate the initial model according to the first reward value and the second reward value to obtain a target model.

[0119] It should be noted that the first acquisition unit 501, the processing unit 502 and the determination unit 503 correspond to steps S201 to S203 in Embodiment One, and the three units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules or units can be hardware components or software components stored in the memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b, …, 102n), and the above units can also be run in the computer terminal 10 provided in Embodiment One as a part of the apparatus.

[0120] Embodiment 3

[0121] Embodiments of the present application can provide an electronic device, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. As Figure 6 shown, the electronic device can include one or more (only one is shown in the figure) processors 602, a memory 604, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, an audio module, and a display. Figure 6

[0122] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and apparatus in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned methods. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0123] ​The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining question information triggered by a target object about table data in a table image library; processing the question information and the table image library through a dual encoder in the target model to obtain a plurality of table images related to the question information; obtaining target reply information based on the plurality of table images through a generation model in the target model, and returning the target reply information to the target object.

[0124] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: processing the question information and the table image library through a dual encoder in the target model to obtain a plurality of table images related to the question information includes: encoding the table images in the table image library through a visual encoder in the dual encoder to obtain first feature information; encoding the question information through a text encoder in the dual encoder to obtain second feature information; obtaining the plurality of table images from the table image library according to the first feature information and the second feature information.

[0125] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining the plurality of table images from the table image library according to the first feature information and the second feature information includes: performing similarity calculation on the first feature information and the second feature information to obtain a plurality of similarities; sorting the table images in the table image library according to the plurality of similarities to obtain sorted table images; obtaining the plurality of table images according to the sorted table images.

[0126] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining target reply information based on the plurality of table images through a generation model in the target model includes: performing structured analysis on each table image to obtain a semantic graph corresponding to each table image; processing the plurality of table images, the semantic graphs and the question information through the generation model to obtain the target reply information.

[0127] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: performing structured analysis on each table image to obtain a semantic graph corresponding to each table image includes: segmenting each table image to obtain a plurality of cell images; performing text recognition on the plurality of cell images to obtain a target string; positioning each table image to determine a row-column relationship between the plurality of cell images; obtaining the semantic graph according to the target string and the row-column relationship.

[0128] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining a target model by training the following steps: obtaining a training sample set, wherein the training sample set comprises at least a table image sample, structured data corresponding to the table image sample, a real question answer pair corresponding to the table image sample, and an inference link corresponding to the real question answer pair; processing the training sample set by an initial model to obtain a predicted question answer pair; and iterating the initial model according to the predicted question answer pair to obtain the target model.

[0129] Further, the iterating the initial model according to the predicted question answer pair to obtain the target model comprises: obtaining a target table image sample used by the initial model to output the predicted question answer pair; obtaining a first reward value according to the relevance between the target table image sample and a question; obtaining a second reward value according to the relevance between the predicted question answer pair and the real question answer pair; and iterating the initial model according to the first reward value and the second reward value to obtain the target model.

[0130] Those skilled in the art can understand that, Figure 6 The structure shown is only schematic, and the electronic device can also be a smart phone, a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 6 This does not limit the structure of the electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 6 Figure 6 This does not limit the structure of the electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0131] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing the related hardware of the terminal device, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0132] Embodiment 4

[0133] The embodiments of the present application also provide a computer readable storage medium. Optionally, in the embodiment, the storage medium can be used to save the program code executed by the data processing method provided in Embodiment 1.

[0134] Optionally, in the embodiment, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.​

[0135] The application also provides a computer program product, which is suitable for executing the steps of the data processing method when executed on a data processing device.

[0136] The above-mentioned sequence numbers of the embodiments of the application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0137] In the above-described embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0138] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiments described above are only schematic. For example, the division of units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, and can be electrical or other forms.

[0139] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiments.

[0140] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0141] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0142] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining question information triggered by a target object about table data in a table image library; processing the question information and the table image library through a double encoder in a target model to obtain a plurality of table images related to the question information; obtaining target reply information based on the plurality of table images through a generation model in the target model, and returning the target reply information to the target object.

2. The method of claim 1, wherein, The processing of the question information and the table image library through the double encoder in the target model to obtain a plurality of table images related to the question information comprises: encoding table images in the table image library through a visual encoder in the double encoder to obtain first feature information; encoding the question information through a text encoder in the double encoder to obtain second feature information; obtaining the plurality of table images from the table image library according to the first feature information and the second feature information.

3. The method of claim 2, wherein, The obtaining of the plurality of table images from the table image library according to the first feature information and the second feature information comprises: performing similarity calculation on the first feature information and the second feature information to obtain a plurality of similarities; sorting table images in the table image library according to the plurality of similarities to obtain sorted table images; obtaining the plurality of table images according to the sorted table images.

4. The method of claim 1, wherein, The obtaining of target reply information based on the plurality of table images through the generation model in the target model comprises: performing structured analysis on each table image to obtain a semantic graph corresponding to each table image; processing the plurality of table images, the semantic graph and the question information through the generation model to obtain the target reply information.

5. The method of claim 4, wherein, The performing of structured analysis on each table image to obtain a semantic graph corresponding to each table image comprises: segmenting each table image to obtain a plurality of cell images; performing text recognition on the plurality of cell images to obtain a target string; positioning each table image to determine the row-column relationship between the plurality of cell images; obtaining the semantic graph according to the target string and the row-column relationship.

6. The method of claim 1, wherein, The target model is trained by the following steps: obtaining a training sample set, wherein the training sample set at least comprises a table image sample, structured data corresponding to the table image sample, a real question answer pair corresponding to the table image sample, and an inference link corresponding to the real question answer pair; processing the training sample set through an initial model to obtain a predicted question answer pair; iterating the initial model according to the predicted question answer pair to obtain the target model.

7. The method of claim 6, wherein, The iteration of the initial model according to the predicted question answer pair to obtain the target model comprises: obtaining a target table image sample used by the initial model when outputting the predicted question answer pair; obtaining a first reward value according to the relevance between the target table image sample and a question; obtaining a second reward value according to the relevance between the predicted question answer pair and the real question answer pair; Iterate the initial model according to the first reward value and the second reward value to obtain the target model.

8. A data processing apparatus, characterized by, The method comprises the steps of: A first acquisition unit is configured to acquire question information triggered by a target object and related to table data in a table image library. A processing unit is configured to process the question information and the table image library through a double encoder in a target model to obtain a plurality of table images related to the question information. A determination unit is configured to obtain target reply information based on the plurality of table images through a generation model in the target model, and return the target reply information to the target object.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored executable program, wherein the executable program controls a device in which the computer readable storage medium is located to execute the data processing method of any one of claims 1 to 7 when the executable program is running.

10. An electronic device, comprising: The method comprises the steps of: A memory stores an executable program; A processor is configured to run the program, wherein the program executes the data processing method of any one of claims 1 to 7 when the program is running.

Images