Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: analyzing a target document to obtain a structured text and a semantic processing result of the target document; and receiving question information for the target document, and generating at least two candidate reply results according to the question information, the structured text and the semantic processing result. And according to the at least two candidate reply results, determining a target reply result corresponding to the question information, and outputting the target reply result. According to the method, the question and answer accuracy of the target document is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In specialized technical fields such as materials science and chemical engineering, experimental reports, technical specifications, process flow documents, and patent documents contain a wealth of crucial information, which is of vital value for technological research and development, production practice, and academic research. In daily work and research scenarios, researchers, engineers, and other users have a need to quickly obtain specific information from these professionally formatted documents to assist in project advancement, solving practical problems, or conducting academic exploration. Currently, information is mainly extracted from these professional documents and related questions answered through traditional keyword-based retrieval methods and simple semantic analysis question-answering systems. However, current question-answering methods for professionally formatted documents suffer from low accuracy.

[0003] Therefore, improving the accuracy of question-and-answer responses for target documents is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, storage medium, and program product to solve the technical problem of low accuracy in answering questions about professionally formatted documents.

[0005] Firstly, this application provides a data processing method, including:

[0006] Parse the target document to obtain the structured text and semantic processing results of the target document;

[0007] Receive query information for the target document;

[0008] Based on the question information, the structured text, and the semantic processing results, at least two candidate response results are generated;

[0009] Based on at least two candidate response results, determine the target response result corresponding to the question information;

[0010] Output the target response result.

[0011] Optionally, determining the target response result corresponding to the question information based on at least two candidate response results includes:

[0012] Based on at least two candidate response results, a commonality index of at least two candidate response results and an accuracy score for each candidate response result are determined, wherein the accuracy score is calculated based on the candidate response results, the structured text, and the semantic processing results;

[0013] The target response result is determined based on the commonality index and the accuracy score.

[0014] Optionally, determining the target response result based on the commonality index and the accuracy score includes:

[0015] If the commonality index is greater than or equal to a preset threshold, the target response result is determined based on the candidate response results corresponding to the accuracy scores that are greater than or equal to a first preset accuracy threshold.

[0016] Optional, also includes:

[0017] If the commonality index is less than the preset threshold, at least two candidate response results are fused to obtain a fused response result.

[0018] Based on the fused response result, the structured text, and the semantic processing result, the fusion accuracy score of the fused response result is obtained;

[0019] If the fusion accuracy score is greater than or equal to the second preset accuracy threshold, the fusion response result is taken as the target response result;

[0020] If the fusion accuracy score is less than the second preset accuracy threshold, the target response result is determined based on the candidate response results whose accuracy scores are greater than or equal to the first preset accuracy threshold.

[0021] Optionally, the step of generating at least two candidate response results based on the question information, the structured text, and the semantic processing result includes:

[0022] Using at least two large language models, at least two candidate response results are output based on the question information, the structured text, and the semantic processing results.

[0023] Optionally, parsing the target document to obtain the structured text and semantic processing results of the target document includes:

[0024] The target document is parsed to obtain the structured text of the target document and the location information of the target object in the target document. The target object includes at least one of the following: an image, a table, and a special semantic element.

[0025] Extract the visual content of the target object based on the location information;

[0026] The structured text and the visual content are subjected to semantic analysis to obtain the semantic processing result.

[0027] Optionally, extracting the visual content of the target object based on the location information includes:

[0028] Based on the location information, determine the content in the target area of ​​the target document that corresponds to the location information;

[0029] The content is subjected to image recognition to obtain the visual content of the target object.

[0030] Optionally, the semantic analysis processing of the structured text and the visual content to obtain the semantic processing result includes:

[0031] A dynamic window mechanism is used to perform semantic understanding on the structured text to obtain semantic understanding results. The window length of the dynamic window mechanism is related to the text length of the structured text.

[0032] The semantic understanding results are then converted into text semantic vectors.

[0033] The semantic processing result is obtained based on the text semantic vector.

[0034] Secondly, this application provides a data processing apparatus, comprising:

[0035] The parsing module is used to parse the target document and obtain the structured text and semantic processing results of the target document;

[0036] A receiving module is used to receive query information for the target document;

[0037] The first processing module is used to generate at least two candidate response results based on the question information, the structured text, and the semantic processing results;

[0038] The second processing module is used to determine the target response result corresponding to the question information based on at least two candidate response results;

[0039] The output module is used to output the target response result.

[0040] Thirdly, this application provides an electronic device, including: a processor and a memory; the processor and the memory are communicatively connected.

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0043] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0044] Fifthly, this application provides a computer program product that, when executed by a processor, is used to implement the method as described in any one of the first aspects.

[0045] The data processing method, apparatus, equipment, storage medium, and program products provided in this application obtain structured text and semantic processing results by parsing the target document, and generate multiple candidate response results based on this and the question information. Then, the target response result is determined and output from the candidate responses. This fully utilizes the structured features and deep semantic information of the target document. The comprehensive evaluation and screening of multiple candidate results ensures the comprehensiveness and accuracy of the response generation, thereby effectively improving the accuracy and robustness of question answering for the target document and better meeting the needs of professional and technical fields for efficient and accurate information acquisition. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0048] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0049] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0053] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first device" and "second device" are merely used to distinguish different devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0056] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0058] Currently, traditional keyword-based retrieval methods can only mechanically search within documents based on preset keywords, making it difficult to accurately capture semantic-level connections. They often cannot provide accurate answers to complex semantic questions. Simple semantic analysis question-answering systems, while capable of some semantic understanding, lack in-depth mining of document structure features. When processing professionally formatted documents, their ability to extract and integrate key information is limited, resulting in unsatisfactory question-answering results.

[0059] These existing technologies are insufficient to fully extract the structured features and deep semantic information contained in professional documents with diverse formats and complex semantics in the field of professional technology, and therefore cannot provide accurate and comprehensive answers to questions.

[0060] In view of this, this application provides a data processing method that obtains structured text and semantic processing results by parsing the target document, generates multiple candidate response results based on these results and combined with question information, and then determines and outputs the target response result from the candidate responses. This method fully utilizes the structured features and deep semantic information of the target document, and ensures the comprehensiveness and accuracy of the response generation through comprehensive evaluation and screening of multiple candidate results. This effectively improves the accuracy and robustness of question answering for the target document and better meets the needs of professional and technical fields for efficient and accurate information acquisition.

[0061] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0063] S101. Parse the target document to obtain the structured text and semantic processing results of the target document.

[0064] The target document can be in formats such as literature, reports, and papers from various professional fields, including materials science and chemical engineering. The target document format can be, for example, PDF or Word.

[0065] In this step, the layout information of the target document can be parsed first. For example, the text, formulas, images, tables, references, etc. in the target document can be extracted. Then, the text and formulas are converted into structured text, making it have a clear hierarchy and logical structure and being able to retain the original content. For special types such as the converted structured text, images, tables, references, etc., semantic analysis can be performed by identifying their features to obtain the corresponding semantic processing results.

[0066] For example, the target document can be parsed by an existing document parsing engine to obtain the layout information of the target document. Taking the target document in PDF format as an example, for instance, the MinerU parsing engine can be used to extract the structured information of the target document (such as the text content and its page number, paragraph position, title hierarchy relationship, row and column structure and cell content of the table, position and size of the picture, LaTeX representation of the formula, chapter titles and corresponding page indexes of the directory, list items and format information of the references, etc.).

[0067] A possible implementation method is to obtain the positions of images, tables, references, etc. in the target document from the layout information, identify the regions corresponding to the positions based on a visual model to obtain the corresponding visual content. Then, the parsed text content, formulas, and the identified visual content are input into a semantic processing model (such as a large language model) to extract the semantic processing result of the target document.

[0068] S102. Receive the question information regarding the target document.

[0069] In this step, the question information regarding the target document can be obtained in response to the user's input operation. For example, the user can directly input the question information in the user interface displayed on the electronic device that executes the data processing method of this embodiment, or can send the question information to the electronic device through a third-party device, etc. The question information usually presents in the form of natural language, such as "What are the main research results mentioned in this report?". Optionally, to ensure the stability of the subsequent output content structure, the question information can also add structural requirements: for example, "1. It is necessary to quote the original content; 2. Classify and sort (technological innovation, experimental results, etc.)" and so on.

[0070] After receiving the question information, the input question information can be preprocessed, including operations such as removing stop words and stemming. Stop words refer to words that frequently appear in the text but have no important role in semantic understanding, such as "of", "is", "in", etc. Removing stop words can reduce the noise of the text and improve the accuracy of information extraction. Stemming refers to restoring a word to its stem form. For example, "running" is restored to "run". Through stemming, the variant forms of words can be reduced and the matching efficiency of words can be improved.

[0071] Then, natural language processing (NLP) techniques can be used to semantically understand the question information, identifying its intent and focus. For example, syntactic analysis algorithms can be used to analyze the sentence structure of the question information and determine its subject, predicate, object, and other components. Semantic role labeling algorithms can also be used to label the semantic roles of each component in the sentence, such as agent, patient, and instrument. Through these analyses, the meaning and focus of the question can be accurately understood, providing a basis for generating subsequent responses.

[0072] S103. Based on the question information, structured text, and semantic processing results, generate at least two candidate response results.

[0073] In this step, the question information, structured text, and semantic processing results can be input into a large language model for processing. The response output of the large language model is then used as the candidate response. In this process, the structured text represents the original content of the target document, while the semantic processing results represent the semantic relationships, implicit logical connections, and domain-specific knowledge content between the text elements in the target document. The large language model can then use the structured text and semantic processing results as reference content for the response, determining the appropriate response based on this reference content.

[0074] One possible approach is to input the question information, structured text, and semantic processing results into the same large language model for multiple processing steps to generate at least two candidate response results.

[0075] Another possible approach is to input the question information, structured text, and semantic processing results into at least two large language models for processing, generating at least two candidate response results, with each large language model generating at least one candidate response result.

[0076] S104. Based on at least two candidate response results, determine the target response result corresponding to the question information.

[0077] In this step, at least two candidate responses can be evaluated and filtered to determine the most suitable target response.

[0078] One possible implementation involves evaluating the accuracy of each candidate response based on the structured text and semantic processing results of the target document, and determining the candidate response with the highest accuracy as the target response. For example, the candidate response with the highest accuracy can be used as the target response, or multiple candidate responses with high accuracy can be merged to generate the target response. This evaluation operation can be determined using a pre-trained evaluation model, such as a large language model or a machine learning model.

[0079] Another possible approach is to determine a commonality index between at least two candidate responses and an accuracy score for each candidate response, based on the commonality index and accuracy score. Then, the target response is determined from the at least two candidate responses based on the commonality index and accuracy score.

[0080] S105, Output the target response result.

[0081] Once the target response is determined, it can be output to the user in an appropriate manner. The output method can be selected based on the user's needs and system settings; for example, the target response can be output in text, voice, or document format.

[0082] For example, the target response can be displayed as text on the screen of an electronic device. During display, the text can be formatted to make it clearer and easier to read. For instance, responses to questions about technical reports can be divided into paragraphs and headings, clearly presenting the different key points. Alternatively, the target response can be converted into speech and read aloud. During this conversion, speech synthesis technology can be used, selecting appropriate voice styles and intonations to improve the user's auditory experience. Or, the target response can be integrated into a document of a specific format and directly output as that document.

[0083] The method provided in this application provides a method that obtains structured text and semantic processing results by parsing a target document, generates multiple candidate response results based on these results and combined with question information, and then determines and outputs the target response result from the candidate responses. This method makes full use of the structured features and deep semantic information of the target document, and ensures the comprehensiveness and accuracy of the response generation through comprehensive evaluation and screening of multiple candidate results. This effectively improves the accuracy and robustness of question answering for the target document and better meets the needs of professional and technical fields for efficient and accurate information acquisition.

[0084] The following section provides a detailed explanation of how step S104 above determines the target response result corresponding to the question information based on at least two candidate response results. Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 2 As shown, the aforementioned step S104 may specifically include:

[0085] S201. Based on at least two candidate response results, determine the commonality index of at least two candidate response results and the accuracy score of each candidate response result.

[0086] The accuracy score is calculated based on the candidate response results, structured text, and semantic processing results.

[0087] In this step, when determining the commonality index, the similarity between at least two candidate responses can be compared. For example, text similarity algorithms such as cosine similarity and edit distance can be used to calculate the similarity score between candidate responses. Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them; the smaller the angle, the closer the cosine value is to 1, indicating higher similarity. Edit distance refers to the minimum number of editing operations required to convert one string into another, including insertion, deletion, and replacement. The smaller the edit distance, the more similar the two strings are. A higher similarity score indicates greater commonality between candidate responses, and thus a higher commonality index.

[0088] When calculating accuracy scores, candidate responses can be compared with structured text and semantic processing results to check whether the information in the candidate responses matches the structured text and semantic processing results, and whether it accurately answers the question. For example, information retrieval and matching algorithms can be used to evaluate and score candidate responses. For instance, taking a scientific report as the target document, it can be checked whether the research findings in the candidate responses match the content of the report and whether they accurately answer the question. During information retrieval and matching, methods such as keyword matching and semantic matching can be used to compare candidate responses with structured text and semantic processing results, calculating the degree and accuracy of the match.

[0089] S202. Determine the target response result based on the commonality index and accuracy score.

[0090] One possible implementation involves determining a comprehensive score for each candidate response based on preset weights, a commonality index, and an accuracy score. The target response is then selected from the candidate responses based on this comprehensive score. For example, the commonality index could be weighted at 0.6, and the accuracy score at 0.4. Then, for each candidate response, a comprehensive score is calculated by weighting its commonality index and accuracy score according to the set weights. The comprehensive scores of all candidate responses are compared, and the candidate response with the highest comprehensive score is selected as the target response.

[0091] Another possible implementation involves judging the commonality index and accuracy score according to preset rules, and then selecting the corresponding strategy to determine the target response based on the judgment result. For example, this implementation can be achieved through the following sub-steps:

[0092] S2021. Determine whether the commonality index of at least two candidate response results is greater than or equal to a preset threshold.

[0093] The preset threshold can be set according to actual needs, such as the complexity of the application scenario, the tolerance for response accuracy, and the statistical analysis results of historical data. For example, the preset threshold could be 0.7, 0.8, etc.

[0094] If the commonality index is greater than or equal to the preset threshold, it indicates that at least two candidate responses have a high degree of similarity and contain a lot of consistent information content. This means that these responses have reached a certain degree of consensus on the core viewpoints or key information, and the target response can be selected from them. Step S2022 is then executed. If the commonality index is less than the preset threshold, it indicates that the differences between the candidate responses are more obvious, each containing different information points or viewpoints. It is difficult to determine the appropriate response directly through simple screening. Step S2023 is then executed.

[0095] S2022. Determine the target response result based on the candidate response results corresponding to the accuracy scores that are greater than or equal to the first preset accuracy threshold.

[0096] Since at least two candidate responses exhibit high similarity, the accuracy of each candidate response can be further assessed using the structured text and semantic processing results of the target text. For example, the similarity between the candidate response and the semantic vector of the target text obtained through semantic processing can be calculated. The accuracy score is then used to determine the level of similarity.

[0097] Optionally, the target response result can be generated by fusing candidate response results with accuracy scores greater than or equal to a first preset accuracy threshold. Alternatively, the candidate response result with the highest accuracy score among those with accuracy scores greater than or equal to the first preset accuracy threshold can be used as the target response result.

[0098] S2023. Merge at least two candidate response results to obtain the merged response result.

[0099] When the commonality index is less than a preset threshold, it indicates a significant difference between candidate responses. In this case, at least two candidate responses can be merged to generate a more comprehensive target response. The merging method could include weighted concatenation, where weights are determined based on factors such as the accuracy score of each candidate response; higher accuracy scores result in higher weights. Then, each candidate response is combined according to its weight to form the target response. During the concatenation process, attention should be paid to the coherence and logic of the responses to avoid content conflicts and repetitions.

[0100] S2024. Based on the fusion response results, structured text, and semantic processing results, obtain the fusion accuracy score of the fusion response results.

[0101] After obtaining the fused response results, their accuracy can be evaluated. For example, the fused response results can be compared with the structured text and semantic processing results, and information retrieval and matching algorithms can be used to calculate the fusion accuracy score. Specifically, keyword matching algorithms can be used to determine the fusion accuracy score by counting the number and degree of keyword matches between the fused response results and the structured text and semantic processing results; semantic similarity algorithms can also be used to analyze the semantic similarity between the fused response results and the structured text and semantic processing results to determine the fusion accuracy score; factors affecting accuracy, such as grammatical errors and spelling errors, can also be considered in the fused response results to determine the fusion accuracy score. Alternatively, all of the above factors can be considered to determine the fusion accuracy score.

[0102] If the fusion accuracy score is greater than or equal to the second preset accuracy threshold, the fused response result will be used as the target response result. Specifically, if the fusion accuracy score of the fused response result is greater than or equal to the second preset accuracy threshold, it indicates that the fused response result accurately answers the question. In this case, the fused response result can be used as the target response result. The second preset accuracy threshold is a standard value pre-set according to specific application scenarios and requirements, used to determine whether the accuracy of the fused response result meets the requirements.

[0103] If the fusion accuracy score is less than the second preset accuracy threshold, the target response is determined based on candidate responses with accuracy scores greater than or equal to the first preset accuracy threshold. Specifically, if the fusion accuracy score of the fused response is less than the second preset accuracy threshold, it indicates that the fused response failed to accurately answer the question. In this case, it is necessary to further filter the candidate responses to select those that meet the criteria. For example, candidate responses with accuracy scores greater than or equal to the first preset accuracy threshold can be selected. Among these candidate responses that meet the accuracy requirements, the target response is further determined based on other factors. These other factors may include, for example, the completeness of the response (whether the response comprehensively covers the key points of the question); the conciseness of the response (whether the response expresses the content in a concise and clear manner, avoiding lengthy and complex expressions); and the logicality of the response (whether the organization of the response content follows a logical order and whether there is a reasonable connection between the various parts).

[0104] The method provided in this application determines the target response to a question by comprehensively considering the commonality index and accuracy score of at least two candidate responses and employing flexible and diverse strategies. On one hand, the commonality index and accuracy score are weighted according to preset weights, and the optimal candidate response is selected as the target response through a comprehensive scoring method. This method can quickly and effectively find relatively accurate and consistent responses in most cases. On the other hand, the commonality index and accuracy score are judged by preset rules, and different strategies are selected accordingly. When the commonality index is high, the target response is further filtered or merged based on the accuracy score to ensure the accuracy of the response. When the commonality index is low, the candidate response results are first merged, and then the accuracy of the merged response result is evaluated. Based on the evaluation result, it is decided whether to use it as the target response or to filter it again from the candidate responses to address situations where there are large differences in the candidate response results. By processing candidate response results in a comprehensive, detailed, and flexible manner, the target response result that meets the needs of the question can be determined more accurately, improving the quality and reliability of the response and providing users with more valuable information feedback.

[0105] For the aforementioned step S103, taking the generation of at least two candidate response results through at least two large language models as an example, at least two large language models can be used to output at least two candidate response results based on the question information, structured text, and semantic processing results.

[0106] Among them, the large language model is a language model trained based on deep learning techniques, possessing powerful language understanding and generation capabilities. Examples of such large language models include the GPT series and BERT. Question information, structured text, and semantic processing results can be input into at least two large language models.

[0107] When inputting information, it is first encoded and transformed to conform to the input format of the large language model. For example, text information can be converted into word vector sequences, and image information into feature vectors. These vectors are then input into the input layer of the large language model. The large language model processes and analyzes the input information, performing feature extraction and semantic understanding through multi-layered neural networks. During processing, an attention mechanism is used to focus on important parts of the input information, improving the model's understanding capabilities.

[0108] Each large language model processes and analyzes the input information based on its own training data and algorithms to generate corresponding response results. Different large language models may have different training data and algorithms, thus the generated response results will also differ. For example, one large language model may focus more on extracting directly relevant information from structured text, while another may combine semantic processing results for deeper reasoning and generation. In generating response results, the large language model predicts the probability of the next word based on the input information. Through continuous prediction and generation, a complete response text, i.e., candidate response results, is formed.

[0109] The following section provides a detailed explanation of how step S101 above specifically parses the target document to obtain its structured text and semantic processing results. Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 3 As shown, the aforementioned step S101 may specifically include:

[0110] S301. Parse the target document to obtain the structured text of the target document and the location information of the target object in the target document.

[0111] The target object includes at least one of the following: image, table, and special semantic element.

[0112] This step allows for in-depth analysis of the report's text content. For example, the text is scanned line by line, using syntactic analysis and semantic understanding algorithms to identify paragraph structure, heading levels, and other information. Paragraphs are divided based on logical relationships and semantic coherence between sentences, ensuring each paragraph has a clear theme and content. Headings are determined by font formatting features and semantic information to establish their level and importance. Furthermore, the text can be meticulously divided into chapters and paragraphs to create structured text, making the report clearer and easier to read.

[0113] For target objects, image processing and pattern recognition technologies can be used to detect and locate images, tables, and other elements within the document. For images, their page number, coordinates, and other positional information within the document can be recorded. When recording coordinates, a coordinate system can be used, with the top-left corner of the document page as the origin, recording the coordinates of the top-left and bottom-right corners of the image to ensure accurate positioning. For tables, features such as borders and separators can be identified to determine the table's scope and cell layout, and its position within the document can be recorded. Furthermore, for special semantic elements, such as references, semantic analysis and dictionary matching technologies can be used for identification and to mark their positions within the document.

[0114] S302. Extract the visual content of the target object based on the location information.

[0115] In this step, the visual content of target objects in the target document can be extracted using a visual model. This visual model can be, for example, AlexNet or YOLO series models in convolutional neural networks; this application does not impose any limitations on this. The visual model can perform visual recognition on the regions corresponding to the location information of each target object to obtain the visual content of the target object.

[0116] Specifically, the visual content of the target object can be extracted through the following sub-steps:

[0117] S3021. Based on the location information, determine the content in the target area of ​​the target document that corresponds to the location information.

[0118] In this step, the location information includes key data such as the page number and coordinates of the target object within the document. Based on this location information, the specific page containing the target object can be located. Then, based on the target object's coordinates, the specific area within that page is determined. The content within this specific area is the content corresponding to that target object, which is then identified and extracted to obtain the visual content of each target object.

[0119] S3022. Perform image recognition on the content to obtain the visual content of the target object.

[0120] For example, taking a target document containing images and tables as an example, the page and region where each image and table is located can be accurately found in the target document based on the recorded location information of each image and table. For tables, the range and content of each cell can be further determined. When determining the cell range, edge detection and contour extraction algorithms can be used to identify the table borders and separators, thereby accurately defining the boundary of each cell. For the content in the table, Optical Character Recognition (OCR) technology can be used to convert it into text. During OCR recognition, the table image can be preprocessed (such as grayscale conversion, binarization, and noise reduction) to improve the recognition accuracy. At the same time, character segmentation and feature matching algorithms can be used to perform character recognition on the preprocessed table image, converting the text in the table into editable text.

[0121] For text and patterns in images, OCR technology and image feature extraction algorithms can be used for recognition and extraction. For text in an image, the text region can be detected and located, and then OCR technology can be used to convert the text into actual text. For patterns in an image, their feature information, such as texture, shape, and color, can be extracted. Texture features can be extracted using algorithms such as gray-level co-occurrence matrix and local binary pattern; shape features can be extracted using algorithms such as contour analysis and shape descriptors; and color features can be extracted using algorithms such as color histogram and color moments. These feature information can be used to classify and recognize images, providing a foundation for subsequent analysis and understanding.

[0122] S303. Perform semantic analysis on structured text and visual content to obtain semantic processing results.

[0123] One possible approach is to perform semantic analysis on structured text and visual content separately, and then fuse the results. Specifically, for structured text, preprocessing can be performed first, including stop word removal and lemmatization. Then, dependency parsing (DP) from natural language processing is used to analyze the dependency relationships between words in the sentence, thereby understanding the grammatical and semantic structure of the text and obtaining preliminary semantic information. For visual content, object detection (OD) algorithms from computer vision can be used to identify various target objects in the image. Then, semantic segmentation (SS) technology is used to finely divide the region where each target object is located, clarifying its semantic category. Finally, the semantic information obtained from structured text analysis and the semantic information obtained from visual content analysis are correlated and matched, ultimately fusing them to form a complete semantic processing result.

[0124] Another possible approach is to first integrate structured text and visual content, and then perform unified semantic analysis. This approach can be implemented through the following sub-steps:

[0125] S3031. A dynamic window mechanism is used to perform semantic understanding on structured text to obtain semantic understanding results.

[0126] The window length of the dynamic window mechanism is related to the text length of the structured text.

[0127] The dynamic window mechanism is a method that adaptively adjusts the window size based on the text length. When performing semantic understanding on structured text, the text can be divided into multiple windows, each containing a certain amount of text content. The window length is dynamically adjusted according to the overall length of the structured text. For longer texts, a larger window length can be configured to improve processing efficiency; for shorter texts, a smaller window length can be configured to ensure the accuracy of semantic understanding.

[0128] Within each window, natural language processing algorithms can be used to perform semantic analysis on the text, identifying entities, relationships, events, and other information. For example, Named Entity Recognition (NER) algorithms can be used to identify entities such as material names, experimental methods, and performance indicators; Relation Extraction (RE) algorithms can be used to identify relationships between entities, such as the preparation relationship between materials and experimental methods, or the performance correlation between materials and performance indicators; Event Detection (ED) algorithms can be used to identify experimental events and performance test events described in the text. When performing named entity recognition, deep learning-based models, such as Bidirectional Long Short-Term Memory (BiLSTM) combined with Conditional Random Fields (CRF), can be used to perform sequence labeling on the text and identify different types of entities. When performing relation extraction, convolutional neural networks (CNNs) or graph neural networks (GNNs) can be used to model and predict the semantic relationships between entities.

[0129] S3032. Convert the semantic understanding results into text semantic vectors.

[0130] After obtaining the semantic understanding results, a word embedding model can be used to convert each word in the text into a vector representation. Then, these vectors are combined and weighted according to the semantic understanding results to obtain the text semantic vector.

[0131] Word embedding models map words into a high-dimensional vector space, ensuring that semantically similar words are close together in the vector space. Examples of such models include Word2Vec and GloVe. After converting words into vectors, each word is assigned a different weight based on the importance of information such as entities, relationships, and events in the semantic understanding results. Then, through weighted summation or other combinations, the vectors of each word are combined into a text semantic vector. During weighted summation, the vectors can be weighted according to the word weights, and then the weighted vectors are summed to obtain the text semantic vector.

[0132] S3033. Obtain the semantic processing result based on the text semantic vector.

[0133] In this step, association matching can be performed based on the obtained text semantic vectors to establish association relationships between text semantic vectors with high correlation. This will enable the large language model to quickly determine all semantic processing results related to the question information when generating response results based on the semantic processing results.

[0134] Specifically, clustering algorithms such as K-Means and DBSCAN can be used to cluster text semantic vectors, grouping similar texts into one category. During clustering, the similarity between text semantic vectors can be calculated, and the texts are divided into different clusters based on this similarity. Taking the K-Means algorithm as an example, K centroids can be randomly selected first, and then each text semantic vector can be assigned to the cluster containing the nearest centroid. Afterward, the centroids of each cluster are recalculated, and this process is iterated until the cluster division no longer changes. The semantic relationships are then determined based on the final clusters.

[0135] Alternatively, classification algorithms can be used to categorize text semantic vectors and determine the text's topic and category. Commonly used classification algorithms include Naive Bayes and Logistic Regression. When performing classification, a pre-trained classification model can be used to categorize text based on its semantic vector features.

[0136] Furthermore, the similarity between text semantic vectors can be calculated to assess the degree of semantic association between texts. For example, for different chapters in a scientific report, the similarity between their text semantic vectors can be calculated to determine their relevance. When calculating similarity, metrics such as cosine similarity and Euclidean distance can be used. If the semantic vectors of two texts have a high similarity, it indicates that they have a strong semantic association; if the similarity is low, it indicates that they have a large semantic difference.

[0137] Based on existing technical solutions, context-related word embedding models, such as BERT, Sentence-BERT, and ELMo, can be used to dynamically encode the input text through pre-trained deep bidirectional Transformer or bidirectional LSTM language models to obtain low-dimensional dense vector representations at the sentence level. Furthermore, cosine similarity or inner product metrics can be used to achieve end-to-end computation of semantic similarity between sentences.

[0138] The method provided in this application lays a solid foundation for subsequent processing by meticulously analyzing the target document, accurately extracting structured text, and clearly defining the location information of the target object. It extracts the visual content of the target object based on the location information using a scientifically sound visual model, and employs various advanced algorithms to perform semantic analysis on the visual content and structured text separately or uniformly, comprehensively and deeply mining the semantic information in the document. Through diverse semantic processing methods, such as clustering, classification, and similarity calculation, it accurately obtains semantic processing results, thereby enabling a more efficient and accurate understanding of the target document content. This provides a reliable basis for generating responses to questions based on the semantic processing results, improving the accuracy and effectiveness of information processing and response generation.

[0139] For example, Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. Figure 4 As shown, the system includes the MinerU parsing engine, an information extraction model, and a large language model.

[0140] The MinerU parsing engine is used to parse the target document, obtaining its structured text and the location information of the target object within the document. The structured text and the location information of the target object output by the MinerU parsing engine are then input into the information extraction model.

[0141] Information extraction models can collaboratively process visual and language models to obtain semantic processing results for the target document (including semantic results of structured text and semantic results of the target object). The visual model is used to extract the visual content of the target object based on its location information. The language model is used to process the structured text and visual content to obtain the semantic processing results of the target document.

[0142] After obtaining the structured text and semantic processing results of the target document, these can be used as reference content for the large language model to generate response results for the user-input question. This system uses at least two large language models to process and generate candidate response results based on the question information, structured text, and semantic processing results, with each large language model generating one candidate response result. Then, based on the generated multiple candidate response results, the target response result corresponding to the question information is determined, thereby improving the accuracy of question answering for the target document.

[0143] Optionally, this system can also configure a chunked loading mechanism for one or more of the MinerU parsing engine and information extraction model. When processing the target document, the memory usage of each page of the target document will be controlled within a preset threshold (such as controlling the memory usage of each page within 8GB) to improve resource utilization efficiency.

[0144] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 5 As shown, the data processing device may include: a parsing module 11, a receiving module 12, a first processing module 13, a second processing module 14, and an output module 15. In one possible implementation, it may also include an output module 13.

[0145] Parsing module 11 is used to parse the target document and obtain the structured text and semantic processing results of the target document.

[0146] The receiving module 12 is used to receive query information for the target document.

[0147] The first processing module 13 is used to generate at least two candidate response results based on the question information, structured text, and semantic processing results.

[0148] The second processing module 14 is used to determine the target response result corresponding to the question information based on at least two candidate response results.

[0149] Output module 15 is used to output the target response result.

[0150] Optionally, the second processing module 14 is specifically used to determine the commonality index of at least two candidate response results and the accuracy score of each candidate response result, based on the candidate response results, structured text, and semantic processing results. The target response result is then determined based on the commonality index and accuracy score.

[0151] Optionally, the second processing module 14 is specifically used to determine the target response result based on the candidate response results corresponding to the accuracy scores that are greater than or equal to the first preset accuracy threshold when the commonality index is greater than or equal to a preset threshold.

[0152] Optionally, the second processing module 14 is further configured to fuse at least two candidate response results to obtain a fused response result when the commonality index is less than a preset threshold. Based on the fused response result, structured text, and semantic processing results, a fusion accuracy score is obtained for the fused response result. If the fusion accuracy score is greater than or equal to a second preset accuracy threshold, the fused response result is used as the target response result. If the fusion accuracy score is less than the second preset accuracy threshold, the target response result is determined based on candidate response results whose accuracy scores are greater than or equal to a first preset accuracy threshold.

[0153] Optionally, the first processing module 13 is specifically used to output at least two candidate response results based on the question information, structured text, and semantic processing results using at least two large language models.

[0154] Optionally, the parsing module 11 is specifically used to parse the target document to obtain the structured text of the target document and the location information of the target object in the target document. The visual content of the target object is extracted based on the location information. Semantic analysis processing is performed on the structured text and visual content to obtain the semantic processing result. The target object includes at least one of the following: an image, a table, and a special semantic element.

[0155] Optionally, the parsing module 11 is specifically used to determine the content in the target region of the target document corresponding to the location information based on the location information. Image recognition is performed on the content to obtain the visual content of the target object.

[0156] Optionally, parsing module 11 is specifically used to perform semantic understanding on structured text using a dynamic window mechanism to obtain semantic understanding results. The window length of the dynamic window mechanism is related to the text length of the structured text. The semantic understanding results are then converted into text semantic vectors. Semantic processing results are obtained based on the text semantic vectors.

[0157] The data processing apparatus provided in this application embodiment can execute the data processing method in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0158] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device is used to execute the aforementioned data processing method. Figure 6 As shown, the electronic device 600 may include at least one processor 601, a memory 602, and a communication interface 603.

[0159] The memory 602 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0160] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0161] The processor 601 is used to execute computer execution instructions stored in the memory 602 to implement the methods described in the foregoing method embodiments. The processor 601 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0162] The processor 601 can communicate and interact with external devices through the communication interface 603. These external devices can be, for example, the user-used electronic devices mentioned above. In specific implementations, if the communication interface 603, memory 602, and processor 601 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0163] Optionally, in a specific implementation, if the communication interface 603, memory 602, and processor 601 are integrated on a single chip, then the communication interface 603, memory 602, and processor 601 can communicate through an internal interface.

[0164] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the methods described in the above embodiments.

[0165] This application also provides a program product including executable instructions stored in a readable storage medium. At least one processor of a computing device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the computing device to perform the methods described above.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Parse the target document to obtain the structured text and semantic processing results of the target document; Receive query information for the target document; Based on the question information, the structured text, and the semantic processing results, at least two candidate response results are generated; Based on at least two candidate response results, determine the target response result corresponding to the question information; Output the target response result.

2. The method according to claim 1, characterized in that, The step of determining the target response result corresponding to the question information based on at least two candidate response results includes: Based on at least two candidate response results, a commonality index of at least two candidate response results and an accuracy score for each candidate response result are determined, wherein the accuracy score is calculated based on the candidate response results, the structured text, and the semantic processing results; The target response result is determined based on the commonality index and the accuracy score.

3. The method according to claim 2, characterized in that, The step of determining the target response result based on the commonality index and the accuracy score includes: If the commonality index is greater than or equal to a preset threshold, the target response result is determined based on the candidate response results corresponding to the accuracy scores that are greater than or equal to a first preset accuracy threshold.

4. The method according to claim 3, characterized in that, Also includes: If the commonality index is less than the preset threshold, at least two candidate response results are fused to obtain a fused response result. Based on the fused response result, the structured text, and the semantic processing result, the fusion accuracy score of the fused response result is obtained; If the fusion accuracy score is greater than or equal to the second preset accuracy threshold, the fusion response result is taken as the target response result; If the fusion accuracy score is less than the second preset accuracy threshold, the target response result is determined based on the candidate response results whose accuracy scores are greater than or equal to the first preset accuracy threshold.

5. The method according to claim 1, characterized in that, The step of generating at least two candidate response results based on the question information, the structured text, and the semantic processing results includes: Using at least two large language models, at least two candidate response results are output based on the question information, the structured text, and the semantic processing results.

6. The method according to claim 1, characterized in that, The parsing of the target document yields the structured text and semantic processing results of the target document, including: The target document is parsed to obtain the structured text of the target document and the location information of the target object in the target document. The target object includes at least one of the following: an image, a table, and a special semantic element. Extract the visual content of the target object based on the location information; The structured text and the visual content are subjected to semantic analysis to obtain the semantic processing result.

7. The method according to claim 6, characterized in that, The step of extracting the visual content of the target object based on the location information includes: Based on the location information, determine the content in the target area of ​​the target document that corresponds to the location information; The content is subjected to image recognition to obtain the visual content of the target object.

8. The method according to claim 6, characterized in that, The semantic analysis processing of the structured text and the visual content to obtain the semantic processing result includes: A dynamic window mechanism is used to perform semantic understanding on the structured text to obtain semantic understanding results. The window length of the dynamic window mechanism is related to the text length of the structured text. The semantic understanding results are then converted into text semantic vectors. The semantic processing result is obtained based on the text semantic vector.

9. A data processing apparatus, characterized in that, include: The parsing module is used to parse the target document and obtain the structured text and semantic processing results of the target document; A receiving module is used to receive query information for the target document; The first processing module is used to generate at least two candidate response results based on the question information, the structured text, and the semantic processing results; The second processing module is used to determine the target response result corresponding to the question information based on at least two candidate response results; The output module is used to output the target response result.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images