Document auditing method and document auditing model training method

Abstract

The embodiment of the invention provides a document auditing method and a document auditing model training method.The document auditing method comprises the steps that in response to a document auditing request for a target document, auditing auxiliary information of the target document is obtained, the target document comprises at least one piece of content to be audited, and the content to be audited is obtained; the auditing auxiliary information is used for providing an auditing basis of the to-be-audited The target document and the auditing auxiliary information are input into a document auditing model, a document auditing result is obtained, the document auditing result comprises content position information, and the content position information is used for describing the position of the to-be-audited content in the target document. The auditing auxiliary information can provide an auditing basis for the document auditing model, the position of the to-be-audited content in the target document is determined based on the auditing auxiliary information, the processing efficiency and the position accuracy of the document auditing model are improved, the position of the to-be-audited content is obtained from the document auditing result, and the processing efficiency is improved. Therefore, the document auditing result of the target document is more visual and readable, and the reading experience of the user is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to document review methods and document review model training methods. Background Technology

[0002] In the document review process, traditional manual review methods rely heavily on the experience of professionals, which is not only time-consuming, labor-intensive, and costly, but also prone to errors due to fatigue or negligence. With the rapid development of artificial intelligence (AI) technology, automating document review through AI technology greatly improves review efficiency and accuracy, freeing up human resources to focus on higher-value strategic analysis and decision-making.

[0003] Among related technologies, AI document review has obvious shortcomings. The review result output by AI is only a piece of text, which needs to be manually compared between the text feedback and the original document, ultimately resulting in a poor user experience. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a document review method. One or more embodiments of this specification also relate to a document review model training method, a document review device, a document review model training device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a document review method is provided, comprising: In response to a document review request for a target document, review assistance information for the target document is obtained, wherein the target document includes at least one piece of content to be reviewed, and the review assistance information is used to provide the basis for reviewing the content to be reviewed; Input the target document and review support information into the document review model to obtain the document review results. The document review results include content location information, which describes the location of the content to be reviewed in the target document.

[0006] According to a second aspect of the embodiments of this specification, a method for training a document review model is provided, comprising: Obtain sample documents, sample review assistance information for sample documents, and sample document review results. The sample document includes at least one sample content to be reviewed. The sample review assistance information is used to provide the basis for reviewing the sample content to be reviewed. The sample document review results include sample content location information, which describes the location of the sample content to be reviewed in the sample document. Input the sample document and sample review auxiliary information into the initial review model to obtain the predicted document review results; Based on the predicted document review results and the sample document review results, the parameters of the initial review model are adjusted to obtain the document review model.

[0007] According to a third aspect of the embodiments of this specification, a document review apparatus is provided, comprising: The first acquisition module is configured to acquire review assistance information for the target document in response to a document review request for the target document. The target document includes at least one piece of content to be reviewed, and the review assistance information is used to provide the basis for reviewing the content to be reviewed. The first input module is configured to input the target document and review auxiliary information into the document review model to obtain the document review result. The document review result includes content location information, which describes the location of the content to be reviewed in the target document.

[0008] According to a fourth aspect of the embodiments of this specification, a document review model training apparatus is provided, comprising: The second acquisition module is configured to acquire a sample document, sample review assistance information of the sample document, and sample document review results. The sample document includes at least one sample content to be reviewed. The sample review assistance information is used to provide the basis for reviewing the sample content to be reviewed. The sample document review results include sample content location information, which describes the location of the sample content to be reviewed in the sample document. The second input module is configured to input sample documents and sample review auxiliary information into the initial review model to obtain the predicted document review results. The first adjustment module is configured to adjust the parameters of the initial review model based on the predicted document review results and the sample document review results to obtain the document review model.

[0009] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first or second aspect above.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0012] This specification provides a document review method according to one embodiment. In response to a document review request for a target document, it obtains review assistance information for the target document. The target document includes at least one piece of content to be reviewed, and the review assistance information provides the basis for reviewing the content to be reviewed. The target document and the review assistance information are input into a document review model to obtain a document review result. The document review result includes content location information, which describes the location of the content to be reviewed within the target document. The review assistance information provides a basis for the document review model. Determining the location of the content to be reviewed within the target document based on the review assistance information improves the processing efficiency and location accuracy of the document review model. Obtaining the location of the content to be reviewed from the document review result makes the document review result of the target document more intuitive and readable, improving the user reading experience. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a document review method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of a student's answer sheet provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a step-by-step rendering provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating a document review model training method provided in one embodiment of this specification; Figure 5 This is a flowchart of a job grading system provided in one embodiment of this specification; Figure 6 This is an architecture diagram of a document review system provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a document review device provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a document review model training device provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” as used in one or more embodiments of this specification means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Artificial intelligence refers to the technology that uses computer systems to simulate human intelligent processes.

[0022] K12: refers to the basic education stage from kindergarten to grade 12, covering preschool education, primary school, junior high school and senior high school. It is a common abbreviation for the basic education market in the domestic and international online education and educational technology industries.

[0023] Multi-modal Large Language Model (MLLM): A large-scale neural network model capable of processing multiple modalities of data such as images and text simultaneously. It typically consists of an image encoder, a text encoder, and a fusion decoder.

[0024] Large Language Model (LLM): A deep learning model trained on large-scale text data, capable of general language understanding and generation, and can be used for tasks such as text correction, question answering, and summarization.

[0025] prompt: A prompt word that refers to the initial text or instruction input to a generative model (such as a large language model or a multimodal model) to guide the model to output results in the expected format, style or content.

[0026] Natural Language Processing (NLP) is a technical field that studies how computers can understand, generate, and interact with human language, encompassing core technologies such as word segmentation, syntactic analysis, semantic understanding, and text generation.

[0027] Convolutional Neural Network (CNN): A type of feedforward neural network used for image feature extraction, often used in tasks such as object detection and image classification.

[0028] Optical Character Recognition (OCR): refers to the technology of converting text in an image into editable text.

[0029] Object detection: a computer vision task that aims to identify all objects of interest in an image and determine their location and category.

[0030] Supervised fine-tuning (SFT) is a method of further training a model based on a pre-trained model. In this method, the model is trained on a dataset containing pairs of inputs and desired outputs so that it learns how to generate responses closer to human-level performance. Supervised fine-tuning is often used to adapt a model to task-specific or domain-specific data, thereby improving its performance on those tasks.

[0031] Region of Interest (ROI): This refers to a localized area in an image or document that has specific significance and requires focused attention or further processing. This region typically contains the key information needed for the task and is distinct from the background or other irrelevant content. An ROI is a two-dimensional region that can be represented using a bounding box, polygon, mask, etc., and its coordinate information (such as the coordinates of the top-left and bottom-right corners) is used for precise localization. For example, in a document recognition task, an ROI could be the title area of ​​a question, an answer key, a table cell, or a region of handwritten text.

[0032] Transformer: A neural network architecture based on self-attention mechanism, widely used in natural language processing and computer vision.

[0033] Multimodal fusion: Integrating information from different modalities (such as images and text) to improve the model's ability to understand and process complex tasks.

[0034] Rendering: refers to the process of overlaying graphics or symbols (such as boxes, colors, icons) onto an image to generate a visual result.

[0035] In K-12 online education, dual-teacher classrooms, photo-based question search, and AI-assisted homework tutoring scenarios, teachers need to use AI tools daily to grade a large number of student assignments and tests uploaded as images. Compared to multiple-choice and fill-in-the-blank questions, word problems are more subjective, with students providing more content and greater freedom in their responses. Furthermore, for word problems with multiple steps, grading only the question itself is insufficient; it's necessary to determine the correctness of each step to maximize the efficiency of both teacher grading and student learning. These characteristics make AI grading of word problems more difficult and less accurate than other question types.

[0036] Word problems typically include: printed or handwritten text (problem stem, student's solution steps); visual elements such as geometric figures, function curves, tables, and coordinate systems; and superimposed information such as handwritten formulas, arrows, markings, and auxiliary lines. Therefore, the system must simultaneously "understand" both text and graphics and provide detailed feedback step by step to meet the "precise to the step" grading requirements of teaching scenarios.

[0037] The AI-based problem-solving approach for word problems mainly follows two technical routes: 1. Traditional detection and recognition + large language model correction solution: (1) Overall approach The entire page of student answers is used to obtain "answer boxes" using object detection technology. The content in each box is transcribed into plain text using an OCR model, and then handed over to a Large Language Model (LLM) for grading in the traditional NLP way.

[0038] (2) Implementation path details a. Image preprocessing: denoising, enhancement, and tilt correction; using a CNN-based object detection network to detect "answer boxes," which typically only recognizes large rectangular areas with dense text; b. Text Recognition: Perform OCR on each answer box and output the string; c. Text concatenation: Concatenate the question stem text, OCR text, and standard answer text into a long prompt; d.LLM grading: Input the prompt into the large language model and ask it to output the text result of "grading the whole question" or "grading by steps"; e. Result display: Usually returned to the front end in plain text or rich text list format, and the front end displays it line by line.

[0039] 2. A multimodal large-scale model grading solution with plain text output: (1) Overall approach The entire student's answer sheet and the question text are fed into a multimodal model with visual-language capabilities, allowing the model to directly output the text-based grading results.

[0040] (2) Implementation path details a. Input structure: Image, containing images of student responses; Text, (optional) question stem + (optional) standard answer text + system prompts; b. Model Inference: The multimodal large model extracts image features through a visual encoder; after concatenation with text features, it is fed into a Transformer Decoder to generate corrected text through autoregression; c. Post-processing: Use regular expressions to extract the correction results from the text and render them to the front end by paragraph.

[0041] 1. Limitations of traditional detection and recognition + large language model grading solutions • Blind spots in graphics: Detectors often discard geometric graphs and function graphs as background noise; if they are forcibly retained, OCR will not be able to interpret the semantics of the graphics, resulting in overall misclassification of graphics questions; • Robustness issues: When handwriting is illegible, formulas are nested inside graphics, or text and graphics intersect, the OCR error rate increases sharply, and the accuracy of LLM correction decreases accordingly. • Scalability bottleneck: Replacing the LLM requires re-tuning the prompt, and because it needs to incorporate student answer recognition results, its length is limited by the model context, and long questions or answers are easily truncated. • Interactive experience: Students receive a text and need to compare it with the original image line by line to find errors, resulting in a poor learning experience.

[0042] 2. Limitations of the multimodal large-scale model grading scheme with plain text output • No coordinate output: Mainstream open-source / closed-source models do not natively support returning image coordinates, making it impossible to "point and shoot" to mark right and wrong on the image; • Output uncertainty: The free text format leads to uncontrollable formatting and makes it difficult to extract structured information; • User experience: Students still need to switch back and forth between "viewing pictures on a vertical screen phone + reading text below", which results in a high cognitive load; • Scalability: If teachers want to introduce custom scoring dimensions or switch to another multimodal model, they need to redesign the prompt and parsing logic, which results in high migration costs.

[0043] The existing solution of "answer box detection + recognition + pure large language model grading" relies on OCR to extract text, which cannot effectively handle questions with graphs such as geometric figures, function graphs, and handwritten formulas, resulting in grading errors or failure to grade.

[0044] While the existing "multimodal large model grading with plain text output" solution can recognize images, the output grading result is only a piece of text, without locating the specific steps on the original answer image. Students find it difficult to intuitively understand whether each step is right or wrong, resulting in a poor user experience.

[0045] Traditional detection and recognition + large language model grading solutions downgrade "images" to "text," making image-based questions inherently ungradable. While multimodal large model grading solutions using pure text output allow the model to "describe images," they struggle to provide "image location + structured step-by-step feedback." Furthermore, both approaches suffer from a core flaw: they cannot provide "step-by-step" visualization results on the original answer image, and they are ill-suited to flexibly adapt to the personalized needs of different teachers or different large models.

[0046] This specification presents an embodiment of a step-by-step grading system for application problems based on a multimodal large model, capable of simultaneously processing multimodal information such as text and graphics, and visually overlaying the grading results onto the original image in a step-by-step positioning manner, thus solving the two major problems of "unable to grade problems with images" and "unintuitive grading results".

[0047] This specification provides an AI grading system that integrates graphics and text processing, step-by-step positioning, result visualization overlay, and flexible and scalable formatting. It achieves the following: coverage of all question types (including geometry and function graphs); step-by-step grading results are rendered onto the original image for intuitive and easy reading; and it supports plug-and-play use of different large models.

[0048] This specification provides a document review method, and also relates to a document review model training method, a document review device, a document review model training device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] It should be noted that the above-mentioned step-by-step grading system or AI grading system is a specific application of the document review method provided in the embodiments of this specification in the field of homework grading. The document review method can also be applied to other fields related to document review besides homework grading, and the embodiments of this specification do not limit this application.

[0050] See Figure 1 , Figure 1 This specification illustrates a flowchart of a document review method according to an embodiment, which specifically includes the following steps: Step 102: In response to a document review request for a target document, obtain review assistance information for the target document. The target document includes at least one content to be reviewed, and the review assistance information is used to provide a basis for reviewing the content to be reviewed.

[0051] It should be noted that the target document refers to a document to be processed that needs to be reviewed. The target document can be a document in different scenarios, such as test papers, exercise books, homework, etc. in the education scenario, or a legal problem-solving document in the legal scenario. The target document can be a document with a hierarchical relationship, such as a test paper including nested large and small questions, or a document without a hierarchical relationship, such as a test paper without nested large and small questions. The target document can also be a single-column layout document or a multi-column layout document. In a multi-column layout document, it can include questions spanning columns or not. The target document can be a document in different formats, such as images, scanned documents, etc. The target document contains multiple questions and usually has a certain layout structure (such as headings, paragraphs, numbers, multi-column layout, etc.). Taking the education scenario as an example, the target document can be an image of a scanned junior high school mathematics final exam paper or a two-column layout English exercise file exported as text.

[0052] A document review request refers to a request to review the content or related attributes of a document. A complete document review request usually includes document identification (document title / name, document number, version number), request information (applicant, application date, deadline), review type (content review, technical review, compliance review, format review), review description (change summary, review focus), and operation guidelines (how to submit feedback, review process). For example, a document review request can be a request to review the content to be reviewed in a document, or a request to review the attributes such as the save type and size of a document. A document review request can also be a request to locate the position of the content to be reviewed in the target document.

[0053] Review assistance information refers to information for assisting in reviewing the target document. During the document review process, it is necessary to refer to the review assistance information to review the target document. The target document includes at least one content to be reviewed. The review assistance information can be the standard answer corresponding to the content to be reviewed, or the initial review result of the content to be reviewed, or the basis for reviewing the content to be reviewed. For example, if the content to be reviewed is "Because a < b < 0, so a < 0, b < 0", the review assistance information is "Step 1: Because a < b < 0, so a < 0, b < 0; Status: Correct; Basis: Formula matching (Inequality Property 1: Transitivity of inequalities), logical equivalence verification passed, and the sign attributes of a and b were accurately extracted".

[0054] Content awaiting review refers to a single unit of content within the target document that requires review. For example, if the target document is a student's answer sheet, the content awaiting review is the student's answer steps for word problems.

[0055] The basis for review refers to the criteria, standards, regulations, or normative documents used in various review activities such as content review, quality inspection, and qualification assessment to determine whether the reviewed object is compliant, qualified, or meets specific standards. In the embodiments of this specification, the basis for review refers to the various standards referenced when making review results on the content to be reviewed.

[0056] In practical applications, document review requests can be sent by users through the client or actively retrieved by the server from the request list.

[0057] There are multiple ways to obtain review assistance information for a target document, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation, the target document carries review assistance information, which can be directly obtained when acquiring the target document. The target document can be acquired by receiving it from a user via a client, or by reading it from another database or data acquisition device.

[0058] In another possible implementation of this specification, obtaining the review assistance information for the target document may include the following steps: By using an audit assistance model, the target document is processed to obtain audit assistance information.

[0059] It should be noted that a review assistance model refers to a trained machine learning or deep learning model used to process target documents and obtain review assistance information. Review assistance models are obtained through supervised fine-tuning based on sample documents and their corresponding sample review assistance information, exhibiting good generalization ability. For example, a review assistance model can be a large-scale multimodal model, with the input being the target document and recognition prompts (used to guide the review assistance model in processing the target document and outputting review assistance information), and the output being the review assistance information. A review assistance model can also be a model specifically designed for the recognition of review assistance information.

[0060] In practical applications, there are various ways to process a target document using an audit assistance model to obtain audit assistance information. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation of this specification, the target document is input into the audit assistance model to obtain audit assistance information.

[0061] In another possible implementation of this specification, the target document and auxiliary prompts are input into the review assistance model to obtain review assistance information, wherein the auxiliary prompts are used to guide the review assistance model to output review assistance information.

[0062] For example, the review assistance model is a multimodal grading model, the target document is the student's answer sheet, and the review assistance information is the grading report. The student's answer sheet and the assistance prompts are input into the multimodal grading model to obtain the grading report of the student's answer sheet.

[0063] The solution implemented in the embodiments of this specification automatically processes the target document using an audit assistance model to obtain audit assistance information, thereby achieving efficient and accurate parsing of the target document, reducing manual intervention, effectively improving the intelligence level and processing efficiency of document processing, and providing reliable data support for subsequent document audits.

[0064] In another possible implementation of this specification, the target document includes images of the answers to the questions; processing the target document using an audit assistance model to obtain audit assistance information for the target document may include the following steps: Obtain the question auxiliary information corresponding to the question answer image; The question's auxiliary information and the image of the question's answer are input into the review assistance model for processing to obtain the review assistance information of the target document.

[0065] It should be noted that the question-answering image refers to an image containing the corresponding answer content. A question is an independent content unit in the target document used to test knowledge or ability, usually with a question number (such as "1.", "(2)"), and may contain components such as the question stem, options, and answer area. The question-answering image contains the answer content and may contain the question.

[0066] Question-related information refers to information related to the question in the question-answer image. This information is used to assist the review and approval model in processing the question-answer image. Question-related information can be the question stem or the standard answer.

[0067] In practical applications, there are various ways to obtain question assistance information corresponding to the question answer image, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on this method. In one possible implementation, question assistance information can be received from the user via a client. In another possible implementation, question assistance information can be read from other databases or data acquisition devices.

[0068] In another possible implementation of this specification, a standard image of the question corresponding to the question answer image is obtained, and the standard image of the question is input into the question recognition model to obtain question auxiliary information. The standard image of the question contains question auxiliary information, and the question recognition model is trained on the initial recognition model based on the sample standard image of the question and the sample question auxiliary information.

[0069] In practical applications, there are multiple ways to input question assistance information and question answer images into the review assistance model for processing to obtain the review assistance information of the target document. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, question assistance information and question answer images are input into the review assistance model for processing to obtain the review assistance information of the target document. In another possible implementation, question assistance information, question answer images, and auxiliary prompt information are input into the review assistance model for processing to obtain the review assistance information of the target document, wherein the auxiliary prompt information is used to guide the review assistance model in outputting the review assistance information.

[0070] For example, the review assistance model is a multimodal grading model, the question answer image is the student's answer image, the question assistance information is the question stem + answer, and the review assistance information is the grading report. The student's answer image, question stem, answer and assistance information are input into the multimodal grading model, and the grading report is output.

[0071] The solution implemented in the embodiments of this specification provides semantic basis for the review assistance model based on the question assistance information. The review assistance model can process the question answer image based on the question assistance information, thereby improving the processing efficiency of the review assistance model and the accuracy of the review assistance information.

[0072] In one optional embodiment of this specification, before processing the target document using the review assistance model to obtain the review assistance information of the target document, the following steps may be included: Obtain the first sample data, which includes the sample document and the sample review auxiliary information of the sample document; Using the initial auxiliary model, the sample documents are processed to obtain predictive review auxiliary information for the sample documents; Based on the predicted review assistance information and the sample review assistance information, the parameters of the initial assistance model are adjusted to obtain the review assistance model.

[0073] It should be noted that the first sample data was used to train the initial auxiliary model to obtain the audit auxiliary model.

[0074] Sample documents refer to a set of real or artificially constructed documents with labeled sample review assistance information used to train the review assistance model. Sample documents can be diverse documents from different scenarios and of different types. They can serve as "teaching materials" for the initial assistance model, helping it understand the structural patterns of different document types. For example, sample documents could be multiple images of exam papers with manually labeled sample review assistance information, or multiple electronic exercise books with manually labeled sample review assistance information. Each sample document can include at least one sample review content, which is an independent content unit within the sample document that needs to be reviewed. Sample review content includes multiple related sample review sub-contents.

[0075] Sample review auxiliary information refers to the information obtained after manually, semi-automatically, or automatically annotating sample documents using authoritative systems. This information serves as the "ground truth" for initial auxiliary model training. Sample review auxiliary information provides the basis for reviewing sample content. It includes multiple sample auxiliary sub-information items, each corresponding one-to-one with a sample review sub-content.

[0076] An initial auxiliary model refers to an untrained or pre-trained model before training begins. It is the initial version of the auxiliary model and does not yet have the ability to process the target document. For example, an initial auxiliary model can be a multimodal pre-trained model for document understanding tasks, or a randomly initialized convolutional neural network and sequence decoder structure model.

[0077] Predictive review support information refers to the output of the initial support model after the sample document is input. This predictive review support information may contain errors. For example, the initial review model may incorrectly review the "correct" opinion for question 1 as an "incorrect" opinion. Predictive review support information includes multiple predictive support sub-information, and each predictive support sub-information corresponds one-to-one with the sample review sub-content.

[0078] Parameter tuning refers to updating the weight parameters of the initial auxiliary model based on the difference between the sample review auxiliary information (real labels) and the predicted review auxiliary information (model output), using methods such as backpropagation, SFT, online learning, or reinforcement learning, to improve its accuracy in document processing. Taking SFT as an example, when tuning the parameters of the initial auxiliary model based on the sample review auxiliary information and the predicted review auxiliary information, an auxiliary loss (such as based on cross-entropy loss) can be calculated. This auxiliary loss is then used to adjust the parameters of the initial auxiliary model to obtain the review auxiliary model.

[0079] In practical applications, there are multiple ways to obtain the first sample data, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on this method in its embodiments. In one possible implementation, the first sample data can be received from a user via a client. In another possible implementation, the first sample data can be read from other databases or data acquisition devices.

[0080] By applying the solution of the embodiments in this specification, sample documents are input into the initial auxiliary model to obtain predicted review assistance information, which is then compared with the actual sample review assistance information. The parameters of the initial auxiliary model are adjusted using the difference information, thus realizing a closed-loop optimization from "preliminary processing" to "precise processing". This significantly improves the processing accuracy of the initial auxiliary model on sample documents, and the final review assistance model has stronger generalization ability and practicality.

[0081] Step 104: Input the target document and review support information into the document review model to obtain the document review results. The document review results include content location information, which describes the location of the content to be reviewed in the target document.

[0082] It's important to note that a document review model refers to a trained machine learning or deep learning model used to automatically identify the visual and textual information of a target document. Document review models are obtained through supervised fine-tuning based on sample documents, sample review assistance information for those sample documents, and sample document review results, exhibiting good generalization ability. For example, a document review model can be a large-scale multimodal model, taking the target document, review assistance information, and review prompts (used to guide the model in reviewing the target document and outputting the review results) as input, and outputting the review results for the target document. A document review model can also be a model specifically designed for performing document review and outputting review results.

[0083] Document review results refer to the output of a document review model after reviewing a target document. Document review results can include content location information, as well as the target review results for the content to be reviewed. Document review results may also include the content to be reviewed and its content number.

[0084] The content location information refers to the information in the target document that describes the location of the content to be reviewed (i.e., the ROI). It reflects the specific geometric location of the content to be reviewed on the document page and is an important basis for document structure recognition and layout analysis. There are various forms of representation for content location information. In one possible implementation, the content location information can be represented based on page area division, such as the content to be reviewed is in the first question type block. In another possible implementation, the content location information can be the content area coordinates of the content to be reviewed in the document page (which can be unnormalized area coordinates or normalized area coordinates), usually represented by a bounding box. For example, [80,120 500,180] indicates that the content to be reviewed is located near the upper left corner of the page.

[0085] In an optional embodiment of this specification, the content to be reviewed includes multiple sub - contents to be reviewed with an associated relationship, the audit auxiliary information includes multiple auxiliary sub - information, the auxiliary sub - information corresponds to the sub - contents to be reviewed one by one, and the content location information includes multiple sub - content location information, and the sub - content location information corresponds to the sub - contents to be reviewed one by one.

[0086] It should be noted that the associated relationship refers to a certain specific connection, link or dependence existing between two or more entities (such as people, companies, things, data points). The associated relationship can be a parallel relationship, a selection relationship, a transitional relationship, a logical progression relationship, a causal relationship or a conditional relationship.

[0087] The sub - content to be reviewed refers to the content unit obtained by dividing the content to be reviewed. For example, if the content to be reviewed is the answer content of an application problem, the sub - content to be reviewed is a single step in the answer content. There is an associated relationship between multiple steps in the answer content. For example, step 1 is "because a < b < 0, so a < 0, b < 0", step 2 is "so ab > 0", and there is a causal relationship between step 1 and step 2, where step 1 is the cause and step 2 is the result.

[0088] The auxiliary sub - information refers to the content unit obtained by dividing the audit auxiliary information. For example, the audit auxiliary information includes the audit basis for multiple steps in an application problem, and the auxiliary sub - information is the audit basis for a single step.

[0089] The auxiliary sub - information corresponds to the sub - contents to be reviewed one by one. For example, the audit basis for step 1 corresponds to step 1 of the student's answer, and the audit basis for step 2 corresponds to step 2 of the student's answer.

[0090] The sub - content location information refers to the content unit obtained by dividing the content location information. For example, if the content location information is the answer location coordinates of an application problem, the sub - content location information is the answer location coordinates of a single step in the application problem.

[0091] The location information of each sub-content corresponds one-to-one with the sub-content to be reviewed. For example, the location coordinates of step 1 correspond to step 1 in the student's answer, and the location coordinates of step 2 correspond to step 2 in the student's answer.

[0092] Figure 2 This diagram illustrates a student answer sheet provided in one embodiment of this specification. The student answer sheet represents the target document, and the answers displayed in the student answer sheet represent content to be reviewed. Figure 2 As shown, the student's answer diagram includes the question stem and the answer content. The question stem is "Grading 7. Given..." Please judge. and The answer should state the size of the object and explain why. , , , , ,answer: Each step in the answer is a sub-content to be reviewed, and there are relationships between different steps.

[0093] For example, Figure 2 The corresponding graded reports for the student answer sheets are shown below: "###Step-by-step correction report" **Step 1:** ,

[0094] Status: Correct Basis: Formula matching (Inequality Property 1: Inequality Transitivity), logical equivalence verification passed, accurately extracted... , Symbolic attributes **Step 2:**

[0095] Status: Correct Basis: Formula matching (negative multiplication rule: negative times negative equals positive), logical equivalence verification passed, correctly judged. symbols **Step 3:** ,

[0096] Status: Error Basis: Formula matching failed (incorrect application of property 3 of inequalities), logical equivalence does not hold. This occurs when both sides of the inequality are multiplied by negative numbers. When the inequality sign is reversed, the correct conclusion should be:

[0097] ###Overall Evaluation of the Questions Final verdict: Incorrect Key test point: New curriculum standard code 7-9-3-2 (basic properties of inequalities) Ability Analysis: Computational ability is at level C (incomplete understanding of symbolic arithmetic rules), reasoning ability is at level C (errors in applying key properties), and the error heatmap shows that step 3 is the main error point.

[0098] The correction report serves as supplementary information for review. Each step in the correction report can be considered as supplementary sub-information, and the step correction report corresponds to a step in the answer content. For example, the step correction report for step 1 corresponds to step 1… , ".

[0099] For example, the step-by-step correction results output by the large multimodal localization model are shown below: “ <ocr-handwriting id="1" roi="0.145,0.167 0.321,0.435">${\because}a{\lt}b{\lt}0$, ${\therefore}a{\lt}0,b{\lt}0$, PASS <ocr-handwriting id="2" roi="0.145,0.435 0.247,0.537"> ${\therefore}ab{\gt}0$< / ocr-handwriting> PASS <ocr-handwriting id="3" roi="0.145,0.556 0.297,0.785">${\because}a{\lt}b$, ${\therefore}b^{2}{\gt}ab$ FAILED <ocr-handwriting id="4" roi="0.234,0.853 0.41,1"> Answer: $b^{2}{\gt}ab$.< / ocr-handwriting> FAILED” <ocr-handwriting> and< / ocr-handwriting> The content enclosed in the label is the recognition result of the student's answer for that step. If it is a drawing step, it is a brief description of the student's drawing. "id" is the sequence number of the student's answer step, "roi" is the relative coordinate position of the bounding box of the student's answer step in the answer graph, and PASS and FAILED at the end of each line are the grading labels for that step.

[0100] Among them, the step-by-step revise result is the document review result, the ROI coordinate is the content location information, the ROI coordinate of each step is the sub-content location information, and the revise label is the target review result.

[0101] The solution implemented in this specification divides the content to be reviewed into multiple related sub-contents, thereby achieving a fine division of the content to be reviewed and enhancing the ability to identify complex content; it also enables precise review of the sub-contents to be reviewed by mapping them to auxiliary sub-information; and it allows for accurate positioning of the sub-contents to be reviewed by mapping them to sub-content location information; ultimately, it improves review efficiency and accuracy.

[0102] In practical applications, there are multiple ways to input the target document and review assistance information into the document review model to obtain the document review result. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation of this specification, the target document, review assistance information, and review prompt information are input into the document review model to obtain the document review result. The review prompt information is used to guide the document review model to output the document review result.

[0103] In another possible implementation of this specification, the audit support information includes the initial audit result and audit basis of the content to be audited, and the document audit result also includes the target audit result of the content to be audited; inputting the target document and audit support information into the document audit model to obtain the document audit result may include the following steps: Input the target document, initial review results, and review criteria into the document review model to obtain content location information and target review results.

[0104] It should be noted that the initial review result refers to a preliminary judgment made on the content to be reviewed. It is not a final decision, but rather an initial classification or suggestion pending further processing. For example, if the content to be reviewed is step 1 of answering an application problem, the initial review result could be "Step 1 is correct".

[0105] The target audit result refers to the final judgment made on the content to be audited. The target audit result may or may not be the same as the initial audit result.

[0106] In practical applications, there are multiple ways to input the target document, initial review result, and review basis into the document review model to obtain content location information and the target review result. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation of this specification, the target document, initial review result, and review basis are input into the document review model to obtain content location information and the target review result.

[0107] In another possible implementation of this specification, the target document, initial review result, review basis, and review prompt information are input into the document review model to obtain content location information and target review result. The review prompt information is used to guide the document review model to output content location information and target review result.

[0108] For example, the target document is the student's answer diagram mentioned above, the initial review result is "correct" or "incorrect" in the above correction report, the review basis is the remaining information in the above correction report, the document review model is a multimodal localization model, and the target review result is the correction result of the above steps.

[0109] By applying the solutions in the embodiments of this specification, the document review model can determine the content location information and target review results of the content to be reviewed based on the initial review results and review criteria, thereby achieving automated review, improving the accuracy of review and the diversity of output content.

[0110] The solution implemented in this specification, in response to a document review request for a target document, obtains review assistance information for the target document. The target document includes at least one piece of content to be reviewed, and the review assistance information provides the basis for reviewing the content. The target document and the review assistance information are input into a document review model to obtain a document review result. The document review result includes content location information, which describes the location of the content to be reviewed within the target document. The review assistance information provides a basis for the document review model, and determining the location of the content to be reviewed within the target document based on the review assistance information improves the processing efficiency and location accuracy of the document review model. Obtaining the location of the content to be reviewed from the document review result makes the document review result of the target document more intuitive and readable, enhancing the user's reading experience.

[0111] In one optional embodiment of this specification, before inputting the target document and review assistance information into the document review model to obtain the document review result, the following steps may also be included: Obtain second sample data, which includes sample documents, sample review assistance information for the sample documents, and sample document review results; Input the sample document and sample review auxiliary information into the initial review model to obtain the predicted document review results; Based on the predicted document review results and the sample document review results, the parameters of the initial review model are adjusted to obtain the document review model.

[0112] It should be noted that the second sample data was used to train the initial review model to obtain the document review model.

[0113] The sample document review results refer to the information obtained after manually, semi-automatically, or automatically annotating sample documents using authoritative systems. This information serves as the "ground truth" for training the initial review model. The sample document review results include sample content location information, which describes the position of the reviewed content within the sample document. The sample document review results may also include the target review results for the reviewed content.

[0114] An initial review model refers to an untrained or pre-trained model before the start of training. It is the initial version of the document review model and does not yet have the ability to review target documents. For example, an initial review model can be a multimodal pre-trained model for document understanding tasks, or it can be a randomly initialized convolutional neural network and sequence decoder structure model.

[0115] In practical applications, there are various ways to obtain the second sample data, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on this method. In one possible implementation, the second sample data can be received from a user via a client. In another possible implementation, the second sample data can be read from other databases or data acquisition devices.

[0116] By applying the solution in the embodiments of this specification, the sample document and sample review auxiliary information are input into the initial review model to obtain the predicted document review result, which is then compared with the actual sample document review result. The parameters of the initial review model are adjusted using the difference information, thus realizing a closed-loop optimization from "preliminary review" to "precise review". This significantly improves the accuracy of the initial review model in reviewing sample documents, and the final document review model has stronger generalization ability and practicality.

[0117] In one optional embodiment of this specification, after inputting the target document and review assistance information into the document review model and obtaining the document review result, the following steps may also be included: Based on the document review results, add review markers to the target document. The review markers include at least one of the following: a content area box for the content to be reviewed and content review comments.

[0118] It should be noted that review tags refer to information marked on the target document, used to visually / intuitively display relevant data of the content to be reviewed during the review process.

[0119] The content region box refers to the bounding box of the area corresponding to the content to be reviewed, used to locate and identify the scope of the content to be reviewed. The content region box corresponds to the content location information. For example, the content location information includes the coordinates of the upper left corner and the lower right corner of the content region box.

[0120] Content review comments refer to the written conclusions and comments formed after reviewing the answers based on established standards, specifications, or requirements. For example, content review comments include PASS (correct), FAILED (incorrect), and UNDONE (not answered).

[0121] In practical applications, there are various ways to add review tags to target documents based on document review results. The specific method chosen depends on the actual situation, and this specification does not limit the specific method used in this embodiment. In one possible implementation, the document review results and the target document are input into the tag-adding model, and the output is the target document with the added review tags. The tag-adding model is trained on an initial tagging model based on sample review results, sample documents, and sample documents with added sample review tags.

[0122] In another possible implementation of this specification, a third-party tagging tool is called to tag the target document based on the document review results, resulting in a target document with added review tags.

[0123] Figure 3 This diagram illustrates a step-by-step rendering of an embodiment of this specification. Figure 3 As shown, in Figure 2 Based on this, add the content area box and content review comments for each step to the student's answer sheet. The content area box is... Figure 3 The content review comments for the rectangle in the image are: Figure 3 The "√" or "×" in the text.

[0124] By applying the solutions in the embodiments of this specification and adding review marks to the target document, the review results of the content to be reviewed can be displayed intuitively, thereby improving the display effect of the review results.

[0125] See Figure 4 , Figure 4 This specification illustrates a flowchart of a document review model training method according to an embodiment, which specifically includes the following steps: Step 402: Obtain the sample document, the sample review auxiliary information of the sample document, and the sample document review result. The sample document includes at least one sample content to be reviewed. The sample review auxiliary information is used to provide the basis for reviewing the sample content to be reviewed. The sample document review result includes the sample content location information, which describes the location of the sample content to be reviewed in the sample document.

[0126] Step 404: Input the sample document and sample review auxiliary information into the initial review model to obtain the predicted document review results.

[0127] Step 406: Based on the predicted document review results and the sample document review results, adjust the parameters of the initial review model to obtain the document review model.

[0128] It should be noted that the implementation methods of steps 402 to 406 are different from those of... Figure 1 The training method for the "document review model" in the document review methods shown is the same, so the embodiments in this specification will not be described again.

[0129] By applying the solutions in the embodiments of this specification, the parameters of the initial review model are adjusted based on the review results of the sample documents and the predicted review results, which significantly improves the ability of the initial review model to review sample documents, obtains accurate predicted review results, and the final document review model has stronger generalization ability and practicality.

[0130] See Figure 5 , Figure 5 A flowchart of a job grading system according to one embodiment of this specification is shown. Figure 5 As shown, the entire grading system pipeline has a serial structure. The core components of the system include: a multimodal grading model, which generates a text-based grading report based on mixed image and text information. This part can be flexibly replaced by other open-source or closed-source models; a multimodal localization model, which takes the grading report and student answer image output from the previous stage as input, breaks down and locates the steps of the student's handwritten answer, and provides right and wrong labels; and a fusion rendering module, which overlays the localization results onto the original image to form a step-by-step grading rendering image.

[0131] The system inputs, outputs, and intermediate results include: student answer image, which contains images of student answers and may also include question stems and illustrations; question stem + answer (optional), which supplements necessary information when the question in the student answer image is incomplete, and can be input in text or image format; grading report, a text-based grading report generated by the multimodal grading model, with no strict format requirements, containing key semantic information for grading student answers; step-by-step grading results, the output of the multimodal localization model, which includes the bounding box coordinates of each student answer step, student identification content, and step grading labels (including three categories: PASS, FAILED, and UNDONE), one line per step; and a step grading rendering image, which overlays the step grading results on the student answer image, drawing different colored bounding boxes and icons at the original position of each step according to the step grading labels.

[0132] The correction process is as follows: 1. Grading Report Generation: The multimodal grading model simultaneously reads student answer diagrams, question stems, and answers. Through the fusion of visual and textual features, it outputs a grading report covering overall comments and step-by-step error descriptions. This report provides semantic support for subsequent localization models and limits their focus areas and error types.

[0133] 2. Step-by-Step Correction and Localization: The multimodal localization model takes the student's answer sheet and the correction report generated in the previous step as input. Utilizing semantic clues in the report, it sequentially outlines mentioned or implied incorrect steps, correct steps, and unanswered areas on the student's answer sheet and assigns corresponding labels. The existence of the correction report allows the localization model to concentrate resources on key areas, improving localization accuracy and speed.

[0134] 3. Integration and rendering of correction results: The fusion module uses Python to deduplicate, merge, and verify the consistency of the coordinates and labels output by the localization model. It then plots these on the original image: correct steps are indicated by green bounding boxes with green checkmarks; incorrect steps are indicated by red bounding boxes with red crosses; unanswered areas are indicated by gray bounding boxes with gray question marks. All bounding boxes maintain the original image's text and graphics' clarity, ensuring students can easily compare and contrast.

[0135] The scalability mechanism is as follows: the system adopts a unified interface, which supports the replacement of multimodal grading large models of different sizes or from different sources without changing the overall process; the input end of the multimodal positioning large model has no mandatory requirements on the grading report format, the model obtains key information on its own and outputs step grading results in a unified format, reducing migration and maintenance costs.

[0136] Performance and robustness: On real primary and secondary school application problem test sets, the average recall and accuracy of step localization both reach over 95%; the system has fault tolerance capabilities for common issues such as photo distortion, illegible handwriting, red pen marks, and low resolution.

[0137] In the embodiments of this specification, a serial dual-model collaborative mechanism is proposed: a serial process of dual multimodal large models consisting of a correction model and a positioning model is proposed. First, the multimodal correction model outputs a correction report containing step-by-step semantics. Then, the multimodal positioning model uses the report as a guide to accurately frame the position of each step and assign right or wrong labels, thus solving the problem of "being able to correct but not being able to locate" or "being able to locate but having no basis for correction".

[0138] Natural language correction report-driven step localization: Using unstructured natural language correction reports as necessary input to the localization model, it directly focuses on the mentioned error areas, significantly reducing the false bounding box rate and achieving a one-to-one direct mapping between semantics and coordinates.

[0139] Step-by-step visualization with lossless overlay of original images: While maintaining the integrity of the students' original answer images, the grading results are rendered step by step using bounding boxes and symbols, forming an intuitive feedback on whether all steps are correct or not in the original image, overcoming the inefficient experience of repeatedly comparing with plain text output.

[0140] Flexible input format and replaceable model architecture: The grading report does not require preset fields or templates, and any natural language can be extracted by the model. Therefore, the grading large model part supports the replacement of models of different sizes or different manufacturers without retraining, which significantly reduces the subsequent upgrade and maintenance costs.

[0141] Comprehensive coverage of mixed text and image question types: The same process can simultaneously handle text, formulas, geometric figures, function curves, tables, and any combination of these question types, breaking through the inherent blind spots of traditional technical approaches for graphic questions and achieving true step-by-step grading for all question types.

[0142] 1. Question Type Coverage: Related technologies such as "answer box detection + OCR + pure large language model grading" can only handle pure text or simple formulas, and cannot recognize visual elements such as geometric figures and function curves, resulting in the inability to grade questions with images. The embodiments in this manual directly understand images and text through a multimodal large model, and can simultaneously handle complex question types such as text, graphics, formulas, and tables, achieving comprehensive coverage of questions with images and those requiring diagramming.

[0143] 2. Presentation of Grading Results: Related technologies such as "Answer Box Detection + OCR + Pure Large Language Model Grading" and "Plain Text Output Multimodal Large Model Grading" can only return a single text or list. Students need to compare the text back and forth between the text and the original image, making it difficult to locate the correct answer. This embodiment in the manual precisely overlays the grading result (correct / incorrect / unanswered) for each step onto the corresponding position in the original student's answer image using colored bounding boxes and icons. Students can easily locate the incorrect step at a glance, making it intuitive and easy to read.

[0144] 3. Scalability: Related technologies such as "answer box detection + OCR + pure large language model grading" and "multimodal large model grading with plain text output" have strict requirements on the format of the input grading report (e.g., fixed fields are required). Changing the large model necessitates rewriting the parsing logic, resulting in high migration costs. This specification's embodiments adopt a relaxed text input and standardized interface design. The grading report can be in any natural language. The model automatically extracts key information, and the multimodal grading large model portion supports replacement of large models of different sizes or from different vendors without requiring retraining of the multimodal localization large model, resulting in low scalability.

[0145] 4. User Experience: The existing technology of "answer box detection + OCR + pure large language model grading" often results in poor user experience due to blind spots in graphics and fragmented steps, leading to students frequently receiving "cannot be graded" or "zero marks for the entire question." While "multimodal large model grading with pure text output" can grade image-text questions, the output format is uncontrollable, requiring students to manually compare the text and image, resulting in a heavy cognitive load. The embodiments in this manual directly visualize the correctness or incorrectness of each step on the original image, eliminating the need for students to compare and search, significantly improving the learning experience.

[0146] The multimodal grading model in this grading system can be replaced by multimodal models of different sizes and structures.

[0147] The multimodal localization model in this correction system can be replaced by other multimodal models; only retraining is required, and the training cost is low.

[0148] Considering the large number of parameters in the document review model and the limited computing resources on the client side, the document review method proposed in the embodiments of this specification can be applied to, for example... Figure 6The document review system shown is not limited to this. See also Figure 6 , Figure 6 This specification illustrates an architecture diagram of a document review system according to an embodiment of the present specification. The document review system may include a client 602 and a server 604. Client 602 is used to send a document review request for the target document to server 604; The server-side 604 response is used to respond to a document review request for a target document, obtain review assistance information for the target document, wherein the target document includes at least one piece of content to be reviewed, and the review assistance information is used to provide the basis for reviewing the content to be reviewed; input the target document and the review assistance information into the document review model to obtain the document review result, wherein the document review result includes content location information, which is used to describe the location of the content to be reviewed in the target document; and send the document review result to the client 602. Client 602 is also used to receive document review results sent by server 604.

[0149] like Figure 6 As shown, the document review model is deployed in server 604. Server 604 can connect to one or more clients 602 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Data transmitted by client 602 may require encoding, transcoding, compression, or other processing before being published to server 604. Client 602 can also interact with users through a graphical user interface to invoke the document review model, thereby implementing the document review method provided in this embodiment. Multiple clients 602 can establish communication connections through server 604. In the document review scenario, server 604 provides document review services between multiple clients 602. Multiple clients 602 can act as senders or receivers, communicating through server 604. Users can interact with server 604 through client 602 to receive data sent by other clients 602, or send data to other clients 602, etc. In a document review scenario, a user can publish a target document to a server 604 via client 602. The server 604 then generates a document review result based on the target document and pushes the document review result to other clients that have established communication.

[0150] Client 602 can be a browser, application (APP), or web application such as HyperText Markup Language 5 (H5) application, or a lightweight application (also known as a mini-program), or cloud application, etc. Client 602 can be developed based on the software development kit (SDK) of the corresponding service provided by server 604, such as based on the Real-Time Communication (RTC) SDK. Client 602 can be deployed in electronic devices and depends on the device to run or on certain APPs on the device. Electronic devices may have a display screen and support information browsing, such as personal mobile terminals such as mobile phones, tablets, and personal computers (PCs). Various other types of applications can also be configured in electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0151] Server-side 604 can include servers providing various services, such as servers providing communication services for multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server-side 604 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0152] It is worth noting that the document review method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the document review model, the client can also have similar functions to the server, thereby executing the document review method provided in the embodiments of this specification. In other embodiments, the document review method provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0153] Corresponding to the above-described document review method embodiments, this specification also provides embodiments of a document review device. Figure 7 A schematic diagram of a document review device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The first acquisition module 702 is configured to acquire review assistance information of the target document in response to a document review request for the target document, wherein the target document includes at least one piece of content to be reviewed, and the review assistance information is used to provide the basis for reviewing the content to be reviewed. The first input module 704 is configured to input the target document and review auxiliary information into the document review model to obtain the document review result. The document review result includes content location information, which describes the location of the content to be reviewed in the target document.

[0154] Optionally, the content to be reviewed includes multiple related sub-contents to be reviewed, the review auxiliary information includes multiple auxiliary sub-information, the auxiliary sub-information corresponds one-to-one with the sub-content to be reviewed, and the content location information includes multiple sub-content location information, the sub-content location information corresponds one-to-one with the sub-content to be reviewed.

[0155] Optionally, the audit support information includes the initial audit result and audit basis of the content to be audited, and the document audit result also includes the target audit result of the content to be audited; the first input module is also configured to input the target document, the initial audit result and the audit basis into the document audit model to obtain the content location information and the target audit result.

[0156] Optionally, the first acquisition module is also configured to use an audit assistance model to process the target document and obtain audit assistance information for the target document.

[0157] Optionally, the target document includes a question-and-answer image; the first acquisition module is further configured to acquire question-assistance information corresponding to the question-and-answer image; and input the question-assistance information and the question-and-answer image into the review assistance model for processing to obtain the review assistance information of the target document.

[0158] Optionally, the device further includes: a second adjustment module configured to acquire first sample data, wherein the first sample data includes a sample document and sample review assistance information of the sample document; process the sample document using an initial assistance model to obtain predicted review assistance information of the sample document; and adjust the parameters of the initial assistance model based on the predicted review assistance information and the sample review assistance information to obtain a review assistance model.

[0159] Optionally, the device further includes: a third adjustment module configured to acquire second sample data, wherein the second sample data includes a sample document, sample review assistance information of the sample document, and sample document review results; input the sample document and sample review assistance information into an initial review model to obtain a predicted document review result; and adjust the parameters of the initial review model based on the predicted document review result and the sample document review result to obtain a document review model.

[0160] Optionally, the first input module is also configured to add review marks to the target document based on the document review results, wherein the review marks include at least one of the content area box of the content to be reviewed and the content review comments.

[0161] By applying the solutions in the embodiments of this specification, the audit assistance information can provide audit basis for the document audit model. Based on the audit assistance information, the location of the content to be audited in the target document can be determined, thereby improving the processing efficiency and location accuracy of the document audit model. The location of the content to be audited can be obtained from the document audit results, making the document audit results of the target document more intuitive and readable, and improving the user reading experience.

[0162] The above is an illustrative scheme of a document review device according to this embodiment. It should be noted that the technical solution of this document review device and the technical solution of the document review method described above belong to the same concept. For details not described in detail in the technical solution of the document review device, please refer to the description of the technical solution of the document review method described above.

[0163] Corresponding to the above-described document review model training method embodiments, this specification also provides embodiments of document review model training devices. Figure 8 A schematic diagram of a document review model training device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes: The second acquisition module 802 is configured to acquire a sample document, sample review assistance information of the sample document, and sample document review result. The sample document includes at least one sample content to be reviewed. The sample review assistance information is used to provide the basis for reviewing the sample content to be reviewed. The sample document review result includes sample content location information, which is used to describe the location of the sample content to be reviewed in the sample document. The second input module 804 is configured to input sample documents and sample review auxiliary information into the initial review model to obtain the predicted document review results. The first adjustment module 806 is configured to adjust the parameters of the initial review model based on the predicted document review results and the sample document review results to obtain a document review model.

[0164] By applying the solutions in the embodiments of this specification, the parameters of the initial review model are adjusted based on the review results of the sample documents and the predicted review results, which significantly improves the ability of the initial review model to review sample documents, obtains accurate predicted review results, and the final document review model has stronger generalization ability and practicality.

[0165] The above is an illustrative scheme of a document review model training device according to this embodiment. It should be noted that the technical solution of this document review model training device and the technical solution of the document review model training method described above belong to the same concept. For details not described in detail in the technical solution of the document review model training device, please refer to the description of the technical solution of the document review model training method described above.

[0166] Figure 9 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0167] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0168] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0169] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers. The computing device 900 can also be a mobile or stationary server.

[0170] The processor 920 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the document review method or document review model training method described above.

[0171] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the document review method and the document review model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the document review method or the document review model training method described above.

[0172] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the document review method or document review model training method described above.

[0173] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the document review method and document review model training method described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the document review method or document review model training method described above.

[0174] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described document review method or document review model training method.

[0175] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the document review method and the document review model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the document review method or the document review model training method described above.

[0176] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0177] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0178] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of document review, the method comprising: The method comprises the following steps: obtaining auditing auxiliary information of a target document in response to a document auditing request for the target document, wherein the target document comprises at least one content to be audited, and the auditing auxiliary information is used to provide an auditing basis for the content to be audited; inputting the target document and the auditing auxiliary information into a document auditing model to obtain a document auditing result, wherein the document auditing result comprises content position information used to describe a position of the content to be audited in the target document.

2. The method of claim 1, wherein, The content to be audited comprises a plurality of sub-contents to be audited that have a correlation relationship, the auditing auxiliary information comprises a plurality of auxiliary sub-information that corresponds to the sub-contents to be audited one by one, and the content position information comprises a plurality of sub-content position information that corresponds to the sub-contents to be audited one by one.

3. The method of claim 1, wherein, The auditing auxiliary information comprises an initial auditing result of the content to be audited and the auditing basis, and the document auditing result further comprises a target auditing result of the content to be audited. The step of inputting the target document and the auditing auxiliary information into the document auditing model to obtain the document auditing result comprises: inputting the target document, the initial auditing result and the auditing basis into the document auditing model to obtain the content position information and the target auditing result.

4. The method of claim 1, wherein, The step of obtaining the auditing auxiliary information of the target document comprises: processing the target document by using an auditing auxiliary model to obtain the auditing auxiliary information of the target document.

5. The method of claim 4, wherein, The target document comprises a question answering image. The step of processing the target document by using the auditing auxiliary model to obtain the auditing auxiliary information of the target document comprises: obtaining question auxiliary information corresponding to the question answering image; inputting the question auxiliary information and the question answering image into the auditing auxiliary model for processing to obtain the auditing auxiliary information of the target document.

6. The method of claim 4, wherein, Before the step of processing the target document by using the auditing auxiliary model to obtain the auditing auxiliary information of the target document, the method further comprises: obtaining first sample data, wherein the first sample data comprises a sample document and sample auditing auxiliary information of the sample document; processing the sample document by using an initial auxiliary model to obtain predicted auditing auxiliary information of the sample document; adjusting parameters of the initial auxiliary model according to the predicted auditing auxiliary information and the sample auditing auxiliary information to obtain the auditing auxiliary model.

7. The method according to any one of claims 1 to 6, characterized in that, Before the step of inputting the target document and the auditing auxiliary information into the document auditing model to obtain the document auditing result, the method further comprises: obtaining second sample data, wherein the second sample data comprises a sample document, sample auditing auxiliary information of the sample document and a sample document auditing result; inputting the sample document and the sample auditing auxiliary information into an initial auditing model to obtain a predicted document auditing result; adjusting parameters of the initial auditing model according to the predicted document auditing result and the sample document auditing result to obtain the document auditing model.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: According to the document review result, adding a review mark on the target document, wherein the review mark includes at least one of a content area frame of the content to be reviewed and a content review opinion. 9.A method for training a document review model, the method comprising: The method further includes: Obtaining a sample document, sample review auxiliary information of the sample document, and a sample document review result, wherein the sample document includes at least one sample content to be reviewed, the sample review auxiliary information is used to provide a review basis for the sample content to be reviewed, and the sample document review result includes sample content position information used to describe a position of the sample content to be reviewed in the sample document; Inputting the sample document and the sample review auxiliary information into an initial review model to obtain a predicted document review result; According to the predicted document review result and the sample document review result, performing parameter adjustment on the initial review model to obtain the document review model.

10. A computing device, comprising: The method further includes: A memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, realize steps of the method in any one of claims 1 to 9.

11. A computer readable storage medium, characterized in that, The memory has stored computer programs / instructions, and the computer programs / instructions, when executed by the processor, realize steps of the method in any one of claims 1 to 9.

12. A computer program product, characterised in that, The computer programs / instructions, when executed by the processor, realize steps of the method in any one of claims 1 to 9.

Images