Document data entry method, electronic device, storage medium and program product

Abstract

The invention discloses a document data entry method, electronic equipment, a storage medium and a program product, and relates to the technical field of document recognition, and the document data entry method comprises the following steps: obtaining a to-be-recognized document containing at least one to-be-recognized page; the document to be recognized is recognized through the optical character recognition technology, a first recognition result and a target page are obtained, and the target page is a page to be recognized containing non-text elements; identifying the target page through the target large language model to obtain a second identification result; and inputting the first recognition result and the second recognition result into the target form according to the similarity between the form field of the preset target form and the first recognition result and the second recognition result, and obtaining the input target form. Through cooperative work of the OCR and the large language model, synchronous and efficient recognition of text and non-text information is achieved, and the accuracy and the automation level of document recognition and document data entry are improved in combination with an intelligent matching entry mechanism.

Description

Technical Field

[0001] This application relates to the field of document recognition technology, and in particular to document data entry methods, electronic devices, storage media and program products. Background Technology

[0002] With the continuous development of computer technology, image processing technology, and artificial intelligence technology, the technical means of clinical research data collection are also constantly being updated and iterated, gradually moving from early manual data entry to automation and intelligence to meet the growing demand for clinical research data processing.

[0003] Currently, OCR (Optical Character Recognition) technology, commonly used in clinical research data acquisition systems, generally involves three basic steps: image preprocessing, character recognition, and manual data entry. However, it is highly dependent on document format, and its recognition accuracy is easily affected by factors such as document structure, font style, table layout, and background interference. When dealing with non-standardized or unstructured documents such as laboratory reports and examination reports, it often struggles to accurately segment and recognize the entries and related unit information within tables, leading to an increased error rate in document data entry.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a document data entry method, electronic device, storage medium, and program product, aiming to solve the technical problem of how to improve the accuracy of document recognition and data entry.

[0006] To achieve the above objectives, this application proposes a document data entry method, which includes: Obtain the document to be identified, wherein the document to be identified contains at least one page to be identified; The document to be identified is identified using optical character recognition technology to obtain a first identification result and a target page, wherein the target page is a page to be identified that contains non-text elements; The target page is identified using the target large language model to obtain a second identification result; Based on the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, the first recognition result and the second recognition result are entered into the target form to obtain the entered target form.

[0007] In one embodiment, before the step of identifying the target page through the target large language model and obtaining the second identification result, the method further includes: Based on the first identification result corresponding to the target page, determine the keyword list of the target page; The target large language model is determined based on the keyword list and the preset mapping relationship between keywords and large language models.

[0008] In one embodiment, the non-text element includes a bar chart, the large language model includes a visual large language model, and the step of identifying the target page through the target large language model to obtain a second identification result includes: Using the visual big language model, the target bar chart is determined from the target page based on the form fields of the target form; The visual big language model is used to identify the column structure of the target histogram and extract the label information of each column structure. The visual big language model is used to extract numerical information of each columnar structure from the preset surrounding area of ​​each columnar structure. The second identification result is obtained by associating the label information and numerical information of each columnar structure.

[0009] In one embodiment, the non-text element includes a table, and the step of identifying the target page using a target large language model to obtain a second identification result includes: Using the target large language model, the target table is determined from the target page based on the form fields of the target form, and the arrangement structure of the target table is identified; According to the arrangement structure, the data of each row / column in the first recognition result corresponding to the target table are concatenated into a string to obtain the table string data corresponding to each row / column; By using the target large language model, the table string data is parsed to determine the data item name and result value of each row / column, thus obtaining the second recognition result.

[0010] In one embodiment, the step of inputting the first identification result and the second identification result into the target form based on the similarity between the form fields of the preset target form and the first identification result and the second identification result, to obtain the input target form, includes: For any form field, calculate the similarity between the form field and each data field in the first recognition result and the second recognition result; In the case where there are data fields with a similarity greater than a preset similarity threshold, the data field with the highest similarity is determined as the target field, and the value of the target field is mapped to the value of the form field in the target form to obtain the target form after data entry.

[0011] In one embodiment, after the step of obtaining the second identification result, the method further includes: The semantic entropy of the second recognition result is determined using the target large language model; Based on the target large language model, it is determined whether the second recognition result conforms to the preset data logic; If the semantic entropy is greater than a preset entropy threshold or the second recognition result does not conform to the data logic, the second recognition result is adjusted according to the document to be recognized.

[0012] In one embodiment, after the step of obtaining the entered target form, the method further includes: The database write request for the target form after it has been entered is intercepted through a preset real-time verification interface. The target form associated with the database write request is verified using preset data verification rules within the electronic data capture system. If the verification is successful, the entered target form is written into the preset database; If verification fails, an alarm message will be output.

[0013] Furthermore, to achieve the above objectives, this application also proposes a document data entry device, which includes: A document acquisition module is used to acquire a document to be identified, wherein the document to be identified contains at least one page to be identified; An optical character recognition module is used to recognize the document to be recognized using optical character recognition technology to obtain a first recognition result and a target page, wherein the target page is a page to be recognized containing non-text elements; The model recognition module is used to recognize the target page through the target large language model and obtain a second recognition result; The form input module is used to input the first recognition result and the second recognition result into the target form based on the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, so as to obtain the input target form.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document data entry method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the document data entry method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the document data entry method described above.

[0017] The one or more technical solutions proposed in this application have at least the following technical effects: First, they acquire a document containing at least one page to be recognized, i.e., they acquire complete original information to be entered, avoiding incomplete entry due to missing information; second, they use optical character recognition (OCR) technology to recognize the document to be recognized, obtaining a first recognition result, while accurately locating the target page containing non-text elements, achieving efficient and batch extraction of pure text information from the document, and providing a clear path for the separate processing of non-text elements; third, they use a target large language model to recognize the target page, obtaining a second recognition result, utilizing the powerful multimodal understanding and reasoning capabilities of the large language model to recognize the target page containing non-text elements, effectively compensating for the limitations of OCR technology in rational analysis. This invention addresses the shortcomings of resolving non-standard layouts, handwritten text, or structured content requiring contextual reasoning (such as extracting data relationships from charts), thereby improving the accuracy of document data recognition. Furthermore, based on the similarity between the form fields of the preset target form and the first and second recognition results, the first and second recognition results are entered into the target form, resulting in the entered target form. By calculating the similarity between the text results extracted by OCR (the first recognition result) and the complex content results parsed by LLM (the second recognition result) and the form fields of the target form, information fragments identified from the document to be recognized can be automatically matched and filled into the corresponding fields of the form, automating the form field mapping and improving the accuracy and efficiency of document data entry. This application achieves simultaneous and efficient recognition of text and non-text information through the collaborative work of OCR and a large language model. Combined with an intelligent matching entry mechanism, it improves the efficiency, accuracy, and automation level of document recognition and document data entry. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the document data entry method for this application. Figure 2 This is a flowchart illustrating the second embodiment of the document data entry method for this application. Figure 3 The overall flowchart for document data recognition and data entry provided in Embodiment 2 of this application; Figure 4 This is a schematic diagram of the module structure of the document data entry device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the document data entry method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] It should be noted that in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] Currently, OCR (Optical Character Recognition) technology, commonly used in clinical research data acquisition systems, generally involves three basic steps: image preprocessing, character recognition, and manual data entry. However, it is highly dependent on document format, and its recognition accuracy is easily affected by factors such as document structure, font style, table layout, and background interference. When dealing with non-standardized or unstructured documents such as laboratory reports and examination reports, it often struggles to accurately segment and recognize the entries and related unit information within the tables, leading to an increased error rate in document data entry. Furthermore, manually mapping the recognition results to corresponding fields in the eCRF (Electronic Case Report Form) form—that is, manually entering the recognition results into the eCRF form—is inefficient.

[0026] This application provides a solution. First, it acquires a document containing at least one page to be recognized, i.e., it obtains complete original information to be entered, avoiding incomplete entry due to missing information. Then, it uses optical character recognition (OCR) technology to recognize the document, obtaining a first recognition result, while accurately locating the target page containing non-text elements. This enables efficient and batch extraction of plain text information from the document, while providing a clear path for the separate processing of non-text elements. Next, it uses a target large language model to recognize the target page, obtaining a second recognition result. Leveraging the powerful multimodal understanding and reasoning capabilities of the large language model, it recognizes the target page containing non-text elements, effectively compensating for the limitations of OCR technology in understanding non-standard layouts. This application addresses the shortcomings of handwritten text or structured content requiring contextual reasoning (such as extracting data relationships from charts), thereby improving the accuracy of document data recognition. Furthermore, based on the similarity between the form fields of the preset target form and the first and second recognition results, the first and second recognition results are entered into the target form, resulting in the entered target form. By calculating the similarity between the text results extracted by OCR (first recognition result) and the complex content results parsed by LLM (second recognition result) and the form fields of the target form, information fragments identified from the document to be recognized can be automatically matched and filled into the corresponding fields of the form, automating the form field mapping and improving the accuracy and efficiency of document data entry. This application achieves simultaneous and efficient recognition of text and non-text information through the collaborative work of OCR and a large language model. Combined with an intelligent matching entry mechanism, it improves the efficiency, accuracy, and automation level of document recognition and document data entry.

[0027] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc.

[0028] Based on this, this application provides a document data entry method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the document data entry method of this application.

[0029] In this embodiment, the document data entry method includes steps S10 to S40: Step S10: Obtain the document to be identified, wherein the document to be identified contains at least one page to be identified; A document to be identified refers to a data file stored in a specific digital format that contains one or more pages to be identified. These formats include PDF (Portable Document Format), image formats, DOCX formats, etc. A document to be identified can be represented as a data object with a unique identifier (such as a file path, database ID, etc.). A page to be identified refers to a single page unit that constitutes the document to be identified. Each page is an independent processing object and may contain visual elements such as text, charts, and tables.

[0030] For example, the document to be identified can be specified by the user by selecting a local file, or it can be obtained by scanning a paper document using a scanner.

[0031] Step S20: Using optical character recognition technology, the document to be recognized is identified to obtain a first recognition result and a target page, wherein the target page is the page to be recognized containing non-text elements; Optical character recognition (OCR) technology is a computer technology that uses optical scanning devices (such as scanners or cameras) to convert text images in documents into machine-readable text data. Its workflow includes steps such as image preprocessing, text line detection, and character feature extraction. The first recognition result refers to the structured text data output by the OCR algorithm, which is usually stored in JSON or string format, including the recognized text information and its corresponding text position in the document to be recognized.

[0032] The target page refers to a specific page in the document to be identified that contains non-text elements; while non-text elements refer to visual entities that are classified as non-pure text blocks in layout analysis, including but not limited to: tables (grid areas separated by row and column lines or whitespace), graphics / charts (bar charts, line charts, etc.), natural images (X-ray thumbnails), stamp / signature areas, handwritten areas (whose font model is different from that of printed fonts), etc.

[0033] Optionally, after obtaining the document to be recognized, it can be determined whether it is in image format. If the document to be recognized is not in image format, it needs to be exported as an image to obtain the image to be recognized. If it is, the document to be recognized can be directly identified as the image to be recognized. Then, preprocessing operations such as denoising, binarization, and rotation correction can be performed on the image to be recognized to improve the efficiency and accuracy of subsequent OCR processing.

[0034] For example, a trained OCR recognition model (such as Tesseract OCR model, PaddleOCR model, etc.) can be used to process the document to be recognized or the preprocessed image to be recognized, converting the character images into text data to obtain the first recognition result; at the same time, the page structure of the document to be recognized is analyzed, pages containing non-text elements are marked, the target page is determined, and a target page index is generated according to its page number.

[0035] Understandably, OCR technology quickly extracts text content from documents to be recognized, providing basic data for subsequent data entry. At the same time, by recognizing non-text elements and filtering out target pages, it enables the prediction and early warning of areas with high OCR recognition errors, allowing for targeted enhancement measures to be taken in these areas, thus providing a preliminary guarantee for achieving high-precision data entry.

[0036] Step S30: Identify the target page using the target large language model to obtain a second identification result; A target large language model refers to a deep learning model pre-trained on large-scale text and / or multimodal data, capable of parsing complex content such as unstructured text, tables, and charts, and extracting textual information. Common target large language models include Doubao and Qwen, and this implementation does not impose specific limitations on them. The second recognition result refers to the structured data representation output by the target large language model after processing the target page, including the extraction and summarization of information contained in non-text elements.

[0037] For example, the target large language model is invoked through a preset model call interface, and the target page, the list of extracted fields (the form fields of the target form) and the preset prompt words (such as "Please extract all test items and their corresponding results and units from the following files and output them in JSON format") are input into the target large language model; then the model performs reasoning calculations based on its internal parameters and knowledge, generates structured text output that conforms to the prompt word instructions, and obtains the second recognition result.

[0038] In one possible implementation, prior to step S30, the method further includes: Step S301: Determine the keyword list of the target page based on the first identification result corresponding to the target page; A keyword list is a set of strings that represent the content features of a target page, extracted from the first recognition result corresponding to the target page using natural language processing technology. Common extraction methods include rule-based methods (such as high-frequency words, specific dictionary matching), word frequency-inverse document frequency statistical methods, or lightweight machine learning models (such as TextRank).

[0039] For example, for any target page, all the text content corresponding to it in the first recognition result is read to obtain the target text; then, the target text is analyzed by a preset keyword extraction algorithm to identify element morphology words (such as "bar chart" and "table") and high information content words (such as professional terms), and an ordered keyword list is generated according to their frequency of occurrence or algorithm score.

[0040] Step S302: Determine the target large language model based on the keyword list and the preset mapping relationship between keywords and large language models.

[0041] The mapping relationship between keywords and large language models refers to the association between pre-set specific keywords or keyword combinations and different large language model instances available in the system.

[0042] For example, check whether the keyword list matches a keyword setting in the mapping relationship. For instance, if the keyword setting in the mapping relationship contains "bar chart" or "chart analysis", then determine the visual large language model (such as Doubao) as the target large language model.

[0043] For example, the keyword list can also be treated as a short text, and a lightweight classifier (such as SVM or FastText) can be used to determine its most likely document category (such as "inspection", "imaging", "pathology", "disease course"), while the mapping relationship is pre-defined by a dedicated large language model corresponding to each document category.

[0044] Optionally, the Dify platform can be introduced, which can call the target large language model to process the target page and generate a second recognition result.

[0045] For example, after identifying the target page, it is saved as a temporary file and uploaded to the Dify platform through a preset file upload interface to obtain a unique upload identifier (upload_file_id). Then, based on the first recognition result (as context information), the keyword list, the extracted field list, and the image resource identifier (i.e., the upload identifier of the target page), a request payload is constructed and sent to the Dify platform. Subsequently, the Dify platform determines the target large language model according to the keyword list and the preset mapping relationship between keywords and large language models. On the Dify platform, the target large language model, combined with the first recognition result, the extracted field list, and the preset prompt words, is used to process the data pointed to by the image resource identifier to generate a second recognition result.

[0046] In this embodiment, keyword extraction and model matching are used to accurately select the most suitable large language model for processing the target page, thereby improving the accuracy and efficiency of non-text element recognition.

[0047] In one feasible implementation, the non-text element includes a histogram, the large language model includes a visual large language model, and step S30 includes: Step S31: Using the visual big language model, determine the target bar chart from the target page based on the form fields of the target form; A visual big language model is a model that integrates visual perception capabilities on the basis of a big language model. It can not only process text information, but also understand and analyze visual elements such as images and graphics. It encodes image pixels into visual features that are fused with text tokens, thereby achieving deep visual-language semantic alignment and reasoning.

[0048] A bar chart is a common data visualization graphic that uses a series of vertical bars of varying heights to represent the size of data. You can determine whether a target page contains a bar chart by detecting the density and length of horizontal and vertical lines. If there are many horizontal and vertical lines of varying lengths, then the non-text elements on the target page are likely bar charts.

[0049] A target bar chart refers to a specific bar chart image area selected from all non-text elements (which may include multiple charts, tables, and images) on the target page based on their semantic relevance to the form fields. For example, if the form fields contain "atrial data", then the bar charts related to the atrial data will be determined as the target bar chart.

[0050] For example, the target page and form fields are input into a visual big language model. The visual big language model performs an overall understanding of the target page, identifies all possible bar charts, and interprets their visual semantics. Then, through a cross-attention mechanism, the semantic relevance between the visual semantics of each bar chart and the form fields of the target form is calculated. Bar charts with a relevance greater than a preset threshold are identified as target bar charts, and the position information of the target bar charts on the target page is returned for subsequent information extraction.

[0051] Step S32: Using the visual big language model, identify the column structure of the target histogram and extract the label information of each column structure; A bar chart refers to an independent visual unit representing a single data sequence, such as a rectangular bar; it can be identified through image segmentation techniques and has a specific position, height, width, and color.

[0052] Label information refers to the textual descriptions used to identify the meaning of the bar chart structure, and is usually located near the horizontal or vertical axis of the bar chart.

[0053] For example, the visual big language model identifies the precise outline and position of all bar structures in the target bar chart through an instance segmentation algorithm, and extracts the label text of each bar structure by combining the visual context (such as legend and axis labels) and the first recognition result corresponding to the target bar chart. For example, the visual big language model may match the correspondence between colors and bars from the legend, or directly read the annotation text near the bars.

[0054] Step S33: Using the visual big language model, extract the numerical information of each columnar structure from the preset surrounding area of ​​each columnar structure; The preset surrounding area refers to the predefined search range for each column structure, used to locate and extract numerical information. It is determined by dividing the boundaries according to the conventional layout of the column chart (such as the top of the column, inside the column, or the side of the column), such as the annotation area above or around the column.

[0055] Numerical information refers to the numerical values ​​corresponding to each columnar structure, which can be identified from a preset surrounding area using image recognition technology.

[0056] For example, for each identified columnar structure, the visual large language model scans its preset surrounding area (such as within 50 pixels above the top of the column), detects and reads the text displaying the numerical value, and determines it as the numerical information of the columnar structure.

[0057] Step S34: Associate the label information and numerical information of each columnar structure to obtain the second identification result.

[0058] For example, the extracted label information and numerical information of each state structure are aligned and matched according to the extraction order or spatial position relationship (such as the height of the bars from high to low) to generate a structured data set (such as a JSON array) and obtain the second recognition result.

[0059] In this embodiment, by calling the visual large language model, the label and numerical information of the bar chart are extracted, which realizes the accurate extraction of bar chart information and improves the accuracy of document recognition. At the same time, by associating the label and numerical information of each bar result and outputting them, it is convenient for subsequent field entry.

[0060] In one possible implementation, the non-text element includes a table, and step S30 includes: Step S35: Using the target large language model, determine the target table from the target page based on the form fields of the target form, and identify the arrangement structure of the target table; A table is a two-dimensional data structure composed of rows and columns. Similarly, the presence of a table on a target page can be determined by detecting the density and length of horizontal and vertical lines. For example, if there are many horizontal and vertical lines of similar length, the target page is likely to contain a table. A target table refers to a specific table area selected from all non-text elements on the target page based on its semantic relevance to form fields.

[0061] The table's structure refers to the organization of rows and columns within the table, as well as the relationships between cells, including the number of rows, columns, the order of rows and columns, and the division between the table header and body.

[0062] For example, after determining the target table, the first recognition result corresponding to the target table (i.e., the text content of the target table area) is input into the target large language model. The model uses its understanding of table conventions (such as the table header usually being in the first row / column) to analyze the text content of the target table area, identify whitespace characters, indentation, text alignment patterns, and the semantic category of cell content (such as text or numbers), and infer the logical row and column structure of the target table. The method of determining the target table in this embodiment is the same as the method of determining the target bar chart in step S31 above, so it will not be described again.

[0063] Step S36: According to the arrangement structure, concatenate the data of each row / column in the first recognition result corresponding to the target table into a string to obtain the table string data corresponding to each row / column; Table string data refers to data in string form formed by concatenating the data from each row or column of a target table.

[0064] For example, if the target table is arranged such that the first row is the table header, the text content in the target table area (i.e. the first recognition result corresponding to the target table) is concatenated row by row, and the data in the same row is concatenated into a string to obtain the table string data corresponding to each row.

[0065] Optionally, the data in the target table area can be cleaned before string concatenation, such as removing special characters and keeping Chinese and English characters and commas.

[0066] Step S37: Using the target large language model, parse the table string data to determine the data item name and result value of each row / column, and obtain the second recognition result.

[0067] For example, the table string data obtained by concatenating the data of each row / column is input into the target large language model. The model parses the logical relationship of the data and identifies the item name, the data value corresponding to the item name, and the unit of measurement corresponding to the data value in each string. The item name is determined as the data item name of each row / column, and the data value and unit of measurement are determined as the result value of each row / column, thus obtaining the second recognition result.

[0068] In this embodiment, by using structure-based information splicing, the problem of table structure loss caused by the OCR process is effectively corrected, and disordered text blocks are restored into meaningful row / column records, so as to achieve accurate recognition of table content.

[0069] Step S40: Based on the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, the first recognition result and the second recognition result are entered into the target form to obtain the entered target form.

[0070] A predefined target form refers to a predefined, templated data framework that specifies the fields to be filled in, field types, etc., such as an ECRF (Electronic Case Report Form), to achieve a unified display of information from different documents. A form field, on the other hand, refers to a specific data item in the target form used to record specific information. It typically exists in key-value pairs, where the key is the field name and the value is the data to be entered.

[0071] Similarity refers to the degree of matching between two strings or data items, and can be calculated using specific algorithms (such as cosine similarity, edit distance, etc.). In the data entry process, similarity is used to measure the degree of matching between form fields and data fields in the recognition results.

[0072] In one feasible implementation, step S40 includes: Step S41: For any form field, calculate the similarity between the form field and each data field in the first recognition result and the second recognition result; A data field is a basic data unit that can be matched in the recognition result. For example, in the first recognition result, a data field can be a sentence or paragraph obtained by OCR recognition; in the second recognition result, a data field can be a structured field with a clear data name and data value.

[0073] For example, each form field of the target form, each data field of the first recognition result (such as a text paragraph), and each data field of the second recognition result are encoded to obtain the vectorized representation of each field; then, for any form field, the cosine similarity between its vectorized representation and the vectorized representation of each data field is calculated to obtain the similarity between the form field and each data field in the first and second recognition results.

[0074] Step S42: In the case where there are data fields with a similarity greater than a preset similarity threshold, the data field with the highest similarity is determined as the target field, and the value of the target field is mapped to the value of the form field in the target form to obtain the target form after input.

[0075] The target field refers to the data unit that best matches the current form field, determined through similarity calculation, threshold filtering, and sorting. The value carried by the target field can be extracted to populate the target form.

[0076] For example, for any form field, it is determined whether its similarity with each data field exceeds a threshold. Data fields with similarity exceeding the threshold are identified as candidate fields. The candidate fields are sorted according to their similarity, and the candidate field with the highest similarity is identified as the target field. Then, the corresponding value is extracted from the target field and mapped to the value of the form field, and the value is entered into the target form.

[0077] Optionally, if multiple candidate fields are obtained, the target field that matches the current form field can be determined based on historical mapping records and user mapping correction records.

[0078] Optionally, if the similarity of all data fields is less than the similarity threshold, the target field matching the current form field can be determined based on historical mapping records and user mapping correction records. For example, based on historical mapping records, the field that is mapped most frequently in the current form field can be determined, and based on mapping correction records, it can be determined whether the user has corrected the mapping result. If the user has corrected the mapping result, the corrected field is determined as the target field; if the user has not corrected the mapping result, the field that is mapped most frequently in the historical mapping records is determined as the target field.

[0079] This embodiment provides a document data entry method. By performing OCR recognition on the acquired document to be recognized, it accurately identifies the target page containing non-text elements such as tables and charts while extracting basic text information. Furthermore, based on the content of the target page, a suitable target language model is selected to extract information from the non-text elements in the target page. This effectively compensates for the shortcomings of OCR technology in understanding non-standard layouts, handwritten text, or structured content that requires contextual reasoning (such as extracting data relationships from charts), thus improving the accuracy of document data recognition. Moreover, through adaptive mapping based on semantic similarity, it realizes automated entry of form fields, improving the efficiency and accuracy of form data entry.

[0080] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the document data entry method of this application. After step S30, it further includes: Step A31: Determine the semantic entropy of the second recognition result using the target large language model; Semantic entropy refers to the comprehensive quantitative score of the semantic ambiguity, inherent contradiction, or uncertainty exhibited by a structured data unit (second recognition result) under a given context and domain knowledge, as evaluated by a large language model. The higher the entropy value, the more ambiguous, contradictory, or unreliable the result is, meaning that the current recognition content may be ambiguous. Secondary confirmation can be performed using the original document to be recognized to improve the accuracy of document data recognition.

[0081] For example, the word embedding vectors corresponding to each data field in the second recognition result can be determined by the target large language model, and the mean and covariance matrix of all word embedding vectors can be calculated. The semantic entropy of the second recognition result can be obtained by using the entropy formula of the multivariate Gaussian distribution to calculate the mean and covariance matrix.

[0082] Step A32: Using the target large language model, determine whether the second recognition result conforms to the preset data logic; Data logic refers to the inherent relationships and rules of rationality between data, which can include numerical ranges, correlations between data, logical order, etc. This data logic can be common knowledge or the judgment logic of users in response to common incorrect inputs. This implementation method does not impose specific limitations on it.

[0083] For example, the target large language model loads data logic from a preset rule base, such as the medication end date cannot be earlier than the start date, and performs logical verification on the second recognition result based on this, comparing the data value corresponding to the start date and the data value corresponding to the end date in the second recognition result.

[0084] Step A33: If the semantic entropy is greater than a preset entropy threshold or the second recognition result does not conform to the data logic, adjust the second recognition result according to the document to be recognized.

[0085] Entropy threshold refers to a pre-set critical value used to determine whether semantic entropy exceeds the allowable range. When the semantic entropy calculation result of the second recognition result is greater than the threshold, it can be determined that the model of the second recognition result is too high and needs further adjustment.

[0086] For example, after obtaining the second recognition result, the semantic entropy of the second recognition result is calculated using the target large language model, and the calculated semantic entropy is compared with a preset entropy threshold. At the same time, the second recognition result is checked according to predefined data logic rules to determine whether it meets the requirements. If at least one of the two conditions, "semantic entropy is greater than the preset entropy threshold" or "the second recognition result does not conform to the data logic", the document to be recognized is further obtained, and it is compared and verified with the second recognition result. The second recognition result is adjusted based on the inconsistencies between the two.

[0087] In this embodiment, the semantic entropy of the second recognition result is calculated and its data logic is judged. When the semantic entropy is too high or the second recognition result does not conform to the data logic, it indicates that there may be errors or uncertainties in the recognition result. Adjustments are made according to the document to be recognized to correct errors and supplement missing information, thereby improving the accuracy of document data recognition and data entry.

[0088] For example, to help understand the implementation flow of the document data entry method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 3 , Figure 3A general flowchart for document data recognition and data entry is provided. First, the document to be recognized is acquired (A101), and optical character recognition (OCR) is performed on it (A102) to obtain a first recognition result and a target page containing non-text elements. Then, based on the first recognition result corresponding to the target page, a keyword list for the target page is determined (A103) to identify a suitable target large language model for recognizing the target page (A104). Next, the non-text elements (such as bar charts, tables, etc.) in the target page are recognized using this target large language model (A105) to obtain a second recognition result. Finally, the second recognition result is further processed... The system performs data verification (including semantic entropy judgment and data logic judgment) and determines whether the data verification passes (A106). If the data verification passes, the system directly performs form entry (A108), that is, it calculates the similarity between the form fields of the target form and the first recognition result and the second recognition result, and performs field mapping and data entry based on this. The specific mapping and entry methods can refer to step S40 in the first embodiment, which will not be elaborated here. If the data verification fails, the system re-acquires the document to be recognized and corrects the second recognition result based on this (A107). After correction, the system performs form entry (A108).

[0089] In one possible implementation, after step S40, the method further includes: Step A41: Intercept database write requests for the entered target form through a preset real-time verification interface; A real-time validation interface is a lightweight software gateway or interceptor deployed between the application service layer and the database layer. It listens for all "insert" or "update" operation requests that point to a specific database table (such as an eCRF data table) and suspends the request before it is actually executed, performing data validation on the data payload (i.e., the target form after data entry) contained in the request.

[0090] Step A42: Verify the target form associated with the database write request using the preset data verification rules within the electronic data capture system; Electronic Data Capture (EDC) systems are electronic systems used for data acquisition, management, and reporting in clinical research and trials.

[0091] Data validation rules refer to a set of business rules pre-set in the EDC system to check the logical correctness, format compliance, and consistency with other data of data, including the EditCheck rule base.

[0092] For example, after detecting that the target form has been entered, the electronic device displays it on a preset operation interface so that the user can check it and decide whether to submit it. Then, in response to the triggered submission operation, the electronic device generates a database write request for the target form. The preset real-time verification interface intercepts the request and submits the target form associated with the request to the rule verification engine of the EDC system. The engine loads all the rules that need to be executed at this moment from the EditCheck rule base according to the form type and context, and applies the rules one by one to determine whether they are feasible, thereby realizing the data verification of the target form. If all rules are feasible, the verification is determined to be successful; otherwise, the verification fails.

[0093] Step A43: If the verification is successful, the entered target form is written into the preset database; Step A44: If verification fails, output an alarm message.

[0094] For example, if the data validation of the target form associated with the database write request passes, the real-time validation interface will allow the request and write the target form into the database normally; if the validation fails, the real-time validation interface will return a "validation failed" response to the caller (such as the front-end application) to alert the user.

[0095] Understandably, by using a real-time verification interface, the numerical range and logical conflicts (such as diastolic blood pressure being higher than systolic blood pressure) can be verified in milliseconds before data is written, without having to wait for all the data to be written before verification. This effectively advances the error feedback time and reduces the writing of invalid data.

[0096] In this embodiment, the document recognition data and form input data are further validated by using a large language model and a real-time verification interface, which improves the quality and reliability of the form data and further enhances the accuracy of document data input.

[0097] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the document data entry method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0098] This application also provides a document data input device, please refer to... Figure 4 The document data entry device includes: The document acquisition module 10 is used to acquire a document to be identified, wherein the document to be identified contains at least one page to be identified; The optical character recognition module 20 is used to recognize the document to be recognized by optical character recognition technology to obtain a first recognition result and a target page, wherein the target page is a page to be recognized containing non-text elements; Model recognition module 30 is used to recognize the target page through the target large language model and obtain a second recognition result; The form input module 40 is used to input the first recognition result and the second recognition result into the target form according to the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, so as to obtain the input target form.

[0099] The document data entry device provided in this application, employing the document data entry method described in the above embodiments, can solve the technical problem of how to improve the accuracy of document recognition and data entry. Compared with the prior art, the beneficial effects of the document data entry device provided in this application are the same as those of the document data entry method provided in the above embodiments, and other technical features in the document data entry device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0100] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the document data entry method in the first embodiment described above.

[0101] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0102] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0103] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0104] The electronic device provided in this application, employing the document data entry method described in the above embodiments, can solve the technical problem of how to improve the accuracy of document recognition and data entry. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the document data entry method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0105] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0107] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the document data entry method in the above embodiments.

[0108] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0109] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0110] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the electronic device to: acquire a document to be recognized, wherein the document to be recognized contains at least one page to be recognized; recognize the document to be recognized using optical character recognition technology to obtain a first recognition result and a target page, wherein the target page is a page to be recognized containing non-text elements; recognize the target page using a target large language model to obtain a second recognition result; and input the first recognition result and the second recognition result into a target form based on the similarity between the form fields of a preset target form and the first recognition result and the second recognition result, thereby obtaining the input target form.

[0111] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0114] The readable storage medium provided in this application is a computer-readable storage medium. This computer-readable storage medium stores computer-readable program instructions (i.e., a computer program) for executing the above-described document data entry method, and can solve the technical problem of how to improve the accuracy of document recognition and data entry. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the document data entry method provided in the above embodiments, and will not be repeated here.

[0115] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the document data entry method described above.

[0116] The computer program product provided in this application can solve the technical problem of how to improve the accuracy of document recognition and data entry. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the document data entry method provided in the above embodiments, and will not be repeated here.

[0117] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A document data entry method, characterized in that, The document data entry method includes: Obtain the document to be identified, wherein the document to be identified contains at least one page to be identified; The document to be identified is identified using optical character recognition technology to obtain a first identification result and a target page, wherein the target page is a page to be identified that contains non-text elements; The target page is identified using the target large language model to obtain a second identification result; Based on the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, the first recognition result and the second recognition result are entered into the target form to obtain the entered target form.

2. The document data entry method as described in claim 1, characterized in that, Before the step of identifying the target page through the target large language model and obtaining the second identification result, the method further includes: Based on the first identification result corresponding to the target page, determine the keyword list of the target page; The target large language model is determined based on the keyword list and the preset mapping relationship between keywords and large language models.

3. The document data entry method as described in claim 1, characterized in that, The non-text elements include bar charts, the large language model includes a visual large language model, and the step of identifying the target page through the target large language model to obtain a second identification result includes: Using the visual big language model, the target bar chart is determined from the target page based on the form fields of the target form; The visual big language model is used to identify the column structure of the target histogram and extract the label information of each column structure. The visual big language model is used to extract numerical information of each columnar structure from the preset surrounding area of ​​each columnar structure. The second identification result is obtained by associating the label information and numerical information of each columnar structure.

4. The document data entry method as described in claim 1, characterized in that, The non-text elements include tables, and the step of identifying the target page using the target large language model to obtain the second identification result includes: Using the target large language model, the target table is determined from the target page based on the form fields of the target form, and the arrangement structure of the target table is identified; According to the arrangement structure, the data of each row / column in the first recognition result corresponding to the target table are concatenated into a string to obtain the table string data corresponding to each row / column; By using the target large language model, the table string data is parsed to determine the data item name and result value of each row / column, thus obtaining the second recognition result.

5. The document data entry method as described in claim 1, characterized in that, The step of inputting the first recognition result and the second recognition result into the target form based on the similarity between the form fields of the preset target form and the first recognition result and the second recognition result, to obtain the input target form, includes: For any form field, calculate the similarity between the form field and each data field in the first recognition result and the second recognition result; In the case where there are data fields with a similarity greater than a preset similarity threshold, the data field with the highest similarity is determined as the target field, and the value of the target field is mapped to the value of the form field in the target form to obtain the target form after data entry.

6. The document data entry method as described in claim 1, characterized in that, Following the step of obtaining the second identification result, the method further includes: The semantic entropy of the second recognition result is determined using the target large language model; Using the target large language model, it is determined whether the second recognition result conforms to the preset data logic; If the semantic entropy is greater than a preset entropy threshold or the second recognition result does not conform to the data logic, the second recognition result is adjusted according to the document to be recognized.

7. The document data entry method as described in claim 1, characterized in that, Following the step of obtaining the target form after data entry, the following is also included: The database write request for the target form after it has been entered is intercepted through a preset real-time verification interface. The target form associated with the database write request is verified using preset data verification rules within the electronic data capture system. If the verification is successful, the entered target form will be written into a preset database; If verification fails, an alarm message will be output.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document data entry method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the document data entry method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the document data entry method as described in any one of claims 1 to 7.

Images