A method, system and terminal for document modification analysis based on fusion of multi-modal visual understanding and a storage medium

Abstract

This invention relates to the field of computer vision technology and discloses a method, system, terminal, and storage medium for document modification analysis that integrates multimodal visual understanding. The method includes: acquiring user-input text, image links, and front-end directly transmitted images; performing image normalization processing on the text, image links, and front-end directly transmitted images according to dual-channel normalization rules to obtain standardized multimodal input data; inputting the standardized multimodal input data into a large visual model, where the large visual model recognizes the standardized multimodal input data to obtain structured element information; acquiring RAG retrieval rules, and performing modification analysis on the structured element information according to the RAG retrieval rules to obtain document modification analysis results. This invention, based on dual-channel normalization fusion of multimodal input, uses visual model recognition and RAG retrieval-driven analysis to obtain document modification analysis results, achieving a precise closed-loop analysis from image input to recognition results.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, terminal, and computer-readable storage medium for document modification analysis that integrates multimodal visual understanding. Background Technology

[0002] Current intelligent question answering systems mostly rely on plain text input: users must manually enter or copy the content of contracts and documents into text before submitting them to the system for retrieval and analysis. Some systems support file uploads, but only perform text extraction (OCR, Optical Character Recognition), which is independent of and separate from text knowledge base retrieval. The materials available to workers are often contract screenshots, scanned copies, and photographed document images. Plain text input requires manual transcription first, which is inefficient and prone to omissions and errors. Image text extraction and legal clause analysis are disconnected, lacking an integrated process that directly combines the recognition of image content with legal retrieval and analysis. The existing interaction is inconvenient: it is difficult to quickly input screenshots from the clipboard or multiple pages of contract images into the analysis. Multimodal input is not integrated with legal vertical domain retrieval, and the recognition results cannot be automatically applied to compliance clause analysis.

[0003] Existing intelligent document analysis methods (such as legal question-and-answer systems based on plain text input or independent OCR text extraction) suffer from several drawbacks. First, image recognition and legal retrieval are separate processes (independent legal retrieval is only performed after text extraction). Second, they cannot directly drive compliance analysis in vertical fields during the multimodal input stage. Third, they may introduce additional errors and efficiency losses due to manual transcription or step-by-step processing. Fourth, they are difficult to achieve accurate closed-loop analysis from image input to recognition results in multimodal document scenarios such as contract screenshots and scanned documents. This has become a pressing problem to be solved.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a document modification analysis method, system, terminal, and computer-readable storage medium that integrates multimodal visual understanding, aiming to solve the problem in the prior art that it is impossible to achieve accurate closed-loop analysis from image input to recognition result in multimodal document scenarios such as contract screenshots and scanned documents.

[0006] To achieve the above objectives, the present invention provides a document modification analysis method that integrates multimodal visual understanding. The document modification analysis method that integrates multimodal visual understanding includes the following steps: The system obtains user-inputted text, image links, and front-end transmitted images. It then performs image normalization processing on the text, image links, and front-end transmitted images according to a dual-channel normalization rule to obtain standardized multimodal input data. The standardized multimodal input data is input into a large visual model, which then identifies the standardized multimodal input data to obtain structured element information. Obtain the RAG retrieval rules, and modify and analyze the structured element information according to the RAG retrieval rules to obtain the document modification analysis results.

[0007] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein acquiring user-input text, image links, and front-end directly transmitted images, and performing image normalization processing on the text, image links, and front-end directly transmitted images according to dual-channel normalization rules to obtain standardized multimodal input data, specifically includes: The system obtains text, image links, and images input by the user; extracts the image links according to a dual-channel normalization rule to obtain an initial file; and encodes the initial file according to a preset encoding format to obtain a target file. The front-end direct transmission image is prefixed according to the dual-channel normalization rule to obtain the processed front-end direct transmission image. The text, the target file, and the processed front-end direct-transmitted image are fused through an encoding layer to obtain standardized multimodal input data.

[0008] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein the standardized multimodal input data is input into a large visual model, and the large visual model identifies the standardized multimodal input data to obtain structured element information, specifically includes: Obtain the image values ​​of the standardized multimodal input data, and input the image values ​​and the text into the large visual model; The large visual model fuses the image values ​​and the text through an encoding layer to obtain fused feature information; The visual big model maps the fused feature information to data distribution through a transformation layer to obtain rule terms and text content; The rules and text content are structured to obtain structured element information.

[0009] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein the step of inputting the standardized multimodal input data into a large visual model, and the large visual model recognizing the standardized multimodal input data to obtain structured element information, further includes: The text is captured based on recurrent neural networks and long short-term memory networks to generate context sequence data. The context sequence data is then processed according to preset rules to obtain temporal recognition features. The temporal recognition features are input into the visual large model for training to obtain the trained visual large model. The trained large visual model is used to identify the standardized multimodal input data.

[0010] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein the standardized multimodal input data is input into a large visual model, the large visual model identifies the standardized multimodal input data to obtain structured element information, and then further includes: The temporal recognition features are divided according to a preset division ratio to obtain a training set and a test set. The visual large model is trained according to the training set to obtain an initial visual large model. The loss value is obtained by calculating the difference between the structured element information and the parameters of the initial visual large model; The initial large visual model is back-optimized based on the test set and the loss value to obtain the optimal large visual model.

[0011] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein obtaining RAG retrieval rules and analyzing the structured element information according to the RAG retrieval rules to obtain document modification analysis results specifically includes: Obtain RAG retrieval rules, and perform a retrieval in a pre-built knowledge base based on the RAG retrieval rules and the key points of the structured element information to obtain a set of retrieval results; The search result set is compared with the structured element information to identify compliance omissions, thereby obtaining compliance deviation analysis results. Based on the compliance deviation analysis results, the structured element information is modified and analyzed to obtain the document modification analysis results.

[0012] Optionally, the document modification analysis method integrating multimodal visual understanding, wherein the step of modifying the structured element information based on the compliance deviation analysis results to obtain the document modification analysis results specifically includes: Based on the anomalies identified in the compliance deviation analysis results, the structured element information is located to obtain the missing content. Obtain the original input image, and map the abnormal missing content and the necessary missing content of the compliance deviation analysis result to the original input image to obtain the modified positioning information; Based on the preset standard contract template and preset rule content, the abnormal missing content is replaced according to the modification location information and the preset rule content to obtain the document modification analysis results.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a document modification analysis system that integrates multimodal visual understanding, wherein the document modification analysis system that integrates multimodal visual understanding: The multimodal data acquisition module is used to acquire user-input text, image links, and front-end directly transmitted images, and to perform image normalization processing on the text, image links, and front-end directly transmitted images according to the dual-channel normalization rule to obtain standardized multimodal input data. The data recognition module is used to input the standardized multimodal input data into the visual large model, and the visual large model recognizes the standardized multimodal input data to obtain structured element information; The data modification module is used to obtain RAG retrieval rules, modify and analyze the structured element information according to the RAG retrieval rules, and obtain document modification analysis results.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a document modification analysis program that integrates multimodal visual understanding, and when the document modification analysis program that integrates multimodal visual understanding is executed by a processor, it implements the steps of the document modification analysis method that integrates multimodal visual understanding as described above.

[0015] This invention acquires user-input text, image links, and directly transmitted images from the front end. Based on a dual-channel normalization rule, the text, image links, and directly transmitted images are subjected to image normalization processing to obtain standardized multimodal input data. This standardized multimodal input data is then input into a large-scale visual model, which identifies the standardized multimodal input data to obtain structured element information. RAG retrieval rules are acquired, and the structured element information is modified and analyzed according to the RAG retrieval rules to obtain document modification analysis results. This invention, based on dual-channel normalization fusion of multimodal input, and driven by visual model recognition and RAG retrieval analysis, achieves a precise closed-loop analysis from image input to recognition results. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the document modification analysis method integrating multimodal visual understanding of the present invention; Figure 2 This is a schematic diagram of the call sequence of a preferred embodiment of the document modification analysis method integrating multimodal visual understanding of the present invention; Figure 3 This is a schematic diagram of multimodal document analysis, which is a preferred embodiment of the document modification analysis method integrating multimodal visual understanding of the present invention. Figure 4This is a schematic diagram of dual-channel image normalization, representing a preferred embodiment of the document modification analysis method integrating multimodal visual understanding of the present invention. Figure 5 This is a structural diagram of a preferred embodiment of the document modification analysis system integrating multimodal visual understanding of the present invention; Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] Existing intelligent document analysis methods (such as legal question-and-answer systems based on plain text input or independent OCR text extraction) suffer from several drawbacks. Firstly, image recognition and legal retrieval are separate processes (independent legal retrieval is only performed after text extraction). Secondly, they cannot directly drive compliance analysis in vertical fields during the multimodal input stage. Thirdly, they may introduce additional errors and efficiency losses due to manual transcription or step-by-step processing. These limitations make it difficult to achieve accurate closed-loop analysis from image input to recognition results in multimodal document scenarios such as contract screenshots and scanned documents. Therefore, a document modification analysis method integrating multimodal visual understanding is needed. This method is based on dual-channel normalized fusion of multimodal input, driven by visual large-model recognition and RAG retrieval, to obtain document modification analysis results and achieve accurate closed-loop analysis from image input to recognition results.

[0019] The preferred embodiment of the document modification analysis method integrating multimodal visual understanding described in this invention, such as... Figure 1 and Figure 2 As shown, the document modification analysis method integrating multimodal visual understanding includes the following steps: Step S10: Obtain the text, image link, and front-end transmitted image input by the user; perform image normalization processing on the text, image link, and front-end transmitted image according to the dual-channel normalization rule to obtain standardized multimodal input data.

[0020] Step S10 includes: Step S11: Obtain the text, image links, and images input by the user; extract the image links according to the dual-channel normalization rule to obtain the initial file; encode the initial file according to the preset encoding format to obtain the target file. Step S12: Perform prefix processing on the front-end direct transmission image according to the dual-channel normalization rule to obtain the processed front-end direct transmission image; Step S13: The text, the target file, and the processed front-end direct-transmitted image are fused through the encoding layer to obtain standardized multimodal input data.

[0021] Specifically, the system acquires user-input text, image links, and images (multimodal input receives user input, which can be text, image URLs, or base64 images directly transmitted from the front end (supporting multiple images at once)). The image links are extracted according to a dual-channel normalization rule to obtain an initial file. The initial file is then encoded according to a preset encoding format to obtain a target file (the corresponding image file on the local disk is located based on the file identifier in the initial file; the image file is read and encoded into a base64 string, ensuring that the image from the disk URL and the image transmitted directly from the front end have the same encoding format). The image transmitted directly from the front end is prefixed according to a dual-channel normalization rule to obtain a processed image transmitted directly from the front end (for images transmitted directly from the front end, if it is a data URI, the prefix "data:image / xxx;base64," is removed). The text, the target file, and the processed image transmitted directly from the front end are then fused through an encoding layer to obtain standardized multimodal input data (the text, the target file, and the processed image transmitted directly from the front end are aggregated and merged into a unified base64 image list, ensuring that multiple input sources have a consistent data format and supporting batch processing, resulting in standardized multimodal input data).

[0022] In this embodiment, as Figure 2 As shown, users submit text, multiple image URLs, or base64 files, supporting multiple documents. The system first normalizes the input using modules 701 / 702, converting all images or attachments into a unified base64 format. Then, the 703 visual big data model recognizes the normalized images, extracting text, tables, and elements. Next, modules 704 / 705 perform structured analysis, further organizing the identified text, tables, and elements. Simultaneously, the 705 semantic retrieval / answer retrieval module performs answer retrieval based on the structured information. Afterward, the system summarizes all recognition results. Based on this, it again calls the 703 visual big data model to complete element structuring (corresponding to module 704), retrieves relevant documents and cases by element, generates analysis conclusions, and finally synthesizes the analysis conclusions and suggestions (corresponding to module 705), returning the final results to the user. The entire process proceeds sequentially, ensuring that multiple documents can be processed in batches and compliance analysis conclusions can be generated.

[0023] For example, such as Figure 3As shown, the document processing workflow begins with user input, supporting the submission of text, multiple image URLs, or base64 encoded files. The system first receives multimodal input through module 701, then performs image normalization processing on all images in module 702, converting images of different formats into the standard base64 format. Next, module 703 calls a large visual model to recognize the normalized images, extracting text, tables, and key elements. After recognition, module 704 structures and organizes these results to form standardized element data. Then, module 705 performs semantic retrieval in conjunction with the knowledge base, matching relevant documents and cases based on the extracted elements. Finally, module 706 performs compliance analysis and generates conclusions, combining the retrieval results and structured elements to output the final analysis conclusions and recommendations. In the entire process, if multiple images are input, they are processed sequentially in batches between steps 702 and 705, automatically moving to the next image after each one is processed, until all are completed.

[0024] For example, such as Figure 4 As shown, the system supports two input sources: URL images from the disk and base64 images directly uploaded from the front end. For disk URL images, the system first extracts the field (field identifier), then reads the corresponding local file based on the field, and encodes the file content into base64 format. For base64 images uploaded from the front end, the system first removes the data URI prefix (e.g., data:image / ...;base64,) to obtain a clean base64 string. After both processing steps are completed, all the obtained base64 data are merged into a unified image list. Finally, this list is sent to the visual large model (module 703) for recognition. The entire process ensures that image data from different sources is standardized into a unified base64 format for subsequent batch processing.

[0025] Step S20: Input the standardized multimodal input data into the visual large model. The visual large model identifies the standardized multimodal input data to obtain structured element information.

[0026] Step S20 includes: Step S21: Obtain the image values ​​of the standardized multimodal input data, and input the image values ​​and the text into the large visual model; Step S22: The visual large model performs fusion processing on the image values ​​and the text through the encoding layer to obtain fused feature information; Step S23: The visual big model maps the data distribution of the fused feature information through a transformation layer to obtain the rule terms and text content; Step S24: Structure the rule clauses and the text content to obtain structured element information.

[0027] Specifically, prior to step S20, the method further includes: capturing the text based on recurrent neural networks and long short-term memory networks to generate context sequence data (e.g., document terminology, clause keywords, and document structure markers); processing the context sequence data according to preset rules to obtain temporal recognition features (e.g., text semantic association features, clause position encoding features, and legal entity annotation features); inputting the temporal recognition features into the visual big model for training to obtain the trained visual big model (e.g., possessing cross-modal association recognition capabilities for legal text and images); and using the trained visual big model to recognize the standardized multimodal input data (e.g., recognizing clause text, signing elements, and layout structure information in contract screenshots) to obtain the standardized... The image values ​​of the multimodal input data are input into a large visual model along with the text (e.g., the image is input in the form of a unified base64 encoded list, and the text is input in the form of parallel embedded vectors). The large visual model fuses the image values ​​and the text through an encoding layer to obtain fused feature information (unified features, such as image-text cross-modal alignment features and joint representation features of legal elements). The large visual model maps the fused feature information to a data distribution through a transformation layer to obtain rule clauses and text content (e.g., contract clause text, text description, and risk point annotations). The rule clauses and text content are then structured to obtain structured element information (e.g., a list of key points, clause compliance check items, and structured fields of document elements).

[0028] In this embodiment, after step S20, the method further includes dividing the temporal recognition features according to a preset division ratio to obtain a training set and a test set. The visual large model is then trained using the training set to obtain an initial visual large model (e.g., a preliminary model with basic image text and legal clause recognition capabilities, capable of outputting clause content and element locations; in each training cycle, the model extracts image features and recognizes text through forward propagation, calculates the loss based on the difference between the output and the actual annotation, and then performs backpropagation on the training set to optimize the parameters). The difference between the structured element information and the parameters of the initial visual large model is calculated to obtain a loss value (e.g., the initial...). The cross-entropy and cross-union ratio differences between the clause recognition text and element location boxes output by the visual big model and the actual legal points and annotation positions in the structured element information are calculated to generate a loss value for measuring the clause recognition accuracy and element location accuracy. The initial visual big model is back-optimized based on the test set and the loss value to obtain the optimal visual big model (for example, the clause recognition accuracy and element location accuracy are periodically measured on the test set, and the model parameters are optimized by backpropagation based on the loss value, and hyperparameters such as learning rate and batch size are adjusted to prevent overfitting, and finally the optimal visual big model that can accurately recognize legal clauses and text content and stably output structured element information is obtained).

[0029] Step S30: Obtain RAG retrieval rules, and perform modification analysis on the structured element information according to the RAG retrieval rules to obtain document modification analysis results.

[0030] Step S30 includes: Step S31: Obtain RAG retrieval rules, and perform a retrieval in a pre-built knowledge base based on the RAG retrieval rules and the key points of the structured element information to obtain a set of retrieval results; Step S32: Compare the set of search results with the structured element information to identify compliance omissions and obtain compliance deviation analysis results; Step S33: Based on the compliance deviation analysis results, modify and analyze the structured element information to obtain the document modification analysis results.

[0031] Specifically, the process involves acquiring RAG (Retrieval-Augmented Generation) retrieval rules, performing a search in a pre-built knowledge base based on the RAG retrieval rules and the key points of the structured element information, and obtaining a set of search results (e.g., a set of search results containing relevant provisions, textual explanations, and similar cases related to the key points of the document, serving as the basis for compliance comparison). The search result set is then compared with the structured element information for compliance omissions to obtain compliance deviation analysis results (e.g., compliance deviation analysis results containing clause anomalies, compliance omissions, and risk levels, used to identify the compliance status of the structured element information). Based on the compliance deviation analysis results, the structured element information is modified and analyzed to obtain document modification analysis results (e.g., document modification analysis results containing clause amendment suggestions, supplementary clause templates, and risk statements, used to guide document compliance adjustments).

[0032] Step S33 includes: Step S331: Locate the structured element information based on the anomalies in the compliance deviation analysis results to obtain the missing content; Step S332: Obtain the original input image, map the abnormal missing content and the necessary missing content of the compliance deviation analysis result to the original input image to obtain the modified positioning information; Step S333: Based on the preset standard contract template and preset rule content, replace the abnormal missing content according to the modification location information and the preset rule content to obtain the document modification analysis result.

[0033] Specifically, based on the anomalies in the compliance deviation analysis results, the structured element information is located to obtain abnormal missing content (e.g., abnormal clauses in the structured element information that differ from regulatory requirements and missing necessary clauses). The original input image is obtained, and the abnormal missing content and the necessary missing content from the compliance deviation analysis results are mapped to the original input image to obtain modification positioning information (e.g., modification positioning information including the page coordinates, clause area identifiers, and corresponding page numbers of the abnormal missing content in the original input image). Based on a preset standard contract template and preset rule content, the abnormal missing content is replaced according to the modification positioning information and the preset rule content to obtain document modification analysis results (e.g., document modification analysis results including modification comparisons and compliance explanations generated after replacing the abnormal missing content with standard clauses according to the preset rule content).

[0034] Beneficial effects: 1. Direct image input: Document screenshots / images can be analyzed without manual transcription, reducing missing words and typos and improving efficiency; 2. Dual-channel normalization: Compatible with both disk URL images and front-end direct base64 uploads, and automatically removes the dataURI prefix, making it easy to integrate; 3. Supports clipboard pasting and batch processing of multiple pages at once: This is well-suited for the practical scenarios where document workers need to process multiple pages of documents at once; 4. Integrated closed loop: The recognition results are automatically applied to RAG retrieval and compliance analysis in the document domain, forming a complete link of images, clauses, regulations, and conclusions.

[0035] Furthermore, such as Figure 5 As shown, based on the above-mentioned document modification analysis method integrating multimodal visual understanding, the present invention also provides a document modification analysis system integrating multimodal visual understanding, wherein the document modification analysis system integrating multimodal visual understanding includes: The multimodal data acquisition module 51 is used to acquire text, image links and front-end direct-transmitted images input by the user, and perform image normalization processing on the text, image links and front-end direct-transmitted images according to the dual-channel normalization rule to obtain standardized multimodal input data; Data recognition module 52 is used to input the standardized multimodal input data into the visual large model, and the visual large model recognizes the standardized multimodal input data to obtain structured element information; The data modification module 53 is used to obtain RAG retrieval rules, modify and analyze the structured element information according to the RAG retrieval rules, and obtain document modification analysis results.

[0036] Furthermore, such as Figure 6 As shown, based on the above-mentioned document modification analysis method and system that integrates multimodal visual understanding, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0037] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a document modification analysis program 40 that integrates multimodal visual understanding. This document modification analysis program 40 that integrates multimodal visual understanding can be executed by the processor 10, thereby implementing the document modification analysis method integrating multimodal visual understanding in this application.

[0038] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the document modification analysis method that integrates multimodal visual understanding.

[0039] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0040] In one embodiment, when the processor 10 executes the document modification analysis program 40 that integrates multimodal visual understanding in the memory 20, the following steps are performed: The system obtains user-inputted text, image links, and front-end transmitted images. It then performs image normalization processing on the text, image links, and front-end transmitted images according to a dual-channel normalization rule to obtain standardized multimodal input data. The standardized multimodal input data is input into a large visual model, which then identifies the standardized multimodal input data to obtain structured element information. Obtain RAG retrieval rules, and modify and analyze the structured element information according to the RAG retrieval rules to obtain document modification analysis results; Specifically, the process of acquiring user-input text, image links, and front-end directly transmitted images, and performing image normalization processing on the text, image links, and front-end directly transmitted images according to dual-channel normalization rules to obtain standardized multimodal input data, includes: The system obtains text, image links, and images input by the user; extracts the image links according to a dual-channel normalization rule to obtain an initial file; and encodes the initial file according to a preset encoding format to obtain a target file. The front-end direct transmission image is prefixed according to the dual-channel normalization rule to obtain the processed front-end direct transmission image. The text, the target file, and the processed front-end direct-transmitted image are fused through an encoding layer to obtain standardized multimodal input data.

[0041] Specifically, the step of inputting the standardized multimodal input data into a large visual model, and the large visual model recognizing the standardized multimodal input data to obtain structured element information, includes: Obtain the image values ​​of the standardized multimodal input data, and input the image values ​​and the text into the large visual model; The large visual model fuses the image values ​​and the text through an encoding layer to obtain fused feature information; The visual big model maps the fused feature information to data distribution through a transformation layer to obtain rule terms and text content; The rules and text content are structured to obtain structured element information.

[0042] The process of inputting the standardized multimodal input data into a large visual model, and the large visual model recognizing the standardized multimodal input data to obtain structured feature information, further includes the following steps: The text is captured based on recurrent neural networks and long short-term memory networks to generate context sequence data. The context sequence data is then processed according to preset rules to obtain temporal recognition features. The temporal recognition features are input into the visual large model for training to obtain the trained visual large model. The trained large visual model is used to identify the standardized multimodal input data.

[0043] The process includes inputting the standardized multimodal input data into a large visual model, where the large visual model identifies the standardized multimodal input data to obtain structured feature information, and then further includes: The temporal recognition features are divided according to a preset division ratio to obtain a training set and a test set. The visual large model is trained according to the training set to obtain an initial visual large model. The loss value is obtained by calculating the difference between the structured element information and the parameters of the initial visual large model; The initial large visual model is back-optimized based on the test set and the loss value to obtain the optimal large visual model.

[0044] The step of obtaining RAG retrieval rules and analyzing the structured element information according to the RAG retrieval rules to obtain document modification analysis results specifically includes: Obtain RAG retrieval rules, and perform a retrieval in a pre-built knowledge base based on the RAG retrieval rules and the key points of the structured element information to obtain a set of retrieval results; The search result set is compared with the structured element information to identify compliance omissions, thereby obtaining compliance deviation analysis results. Based on the compliance deviation analysis results, the structured element information is modified and analyzed to obtain the document modification analysis results.

[0045] Specifically, the step of modifying the structured element information based on the compliance deviation analysis results to obtain document modification analysis results includes: Based on the anomalies identified in the compliance deviation analysis results, the structured element information is located to obtain the missing content. Obtain the original input image, and map the abnormal missing content and the necessary missing content of the compliance deviation analysis result to the original input image to obtain the modified positioning information; Based on the preset standard contract template and preset rule content, the abnormal missing content is replaced according to the modification location information and the preset rule content to obtain the document modification analysis results.

[0046] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a document modification analysis program that integrates multimodal visual understanding, and the document modification analysis program that integrates multimodal visual understanding implements the steps of the document modification analysis method that integrates multimodal visual understanding as described above when executed by a processor.

[0047] In summary, this invention provides a method, system, terminal, and storage medium for document modification analysis that integrates multimodal visual understanding. The method includes: acquiring user-input text, image links, and front-end directly transmitted images; performing image normalization processing on the text, image links, and front-end directly transmitted images according to dual-channel normalization rules to obtain standardized multimodal input data; inputting the standardized multimodal input data into a large visual model, where the large visual model identifies the standardized multimodal input data to obtain structured element information; acquiring RAG retrieval rules, and performing modification analysis on the structured element information according to the RAG retrieval rules to obtain document modification analysis results. This invention, based on dual-channel normalization and multimodal input fusion, uses large visual model recognition and RAG retrieval-driven analysis to obtain document modification analysis results, achieving a precise closed-loop analysis from image input to recognition results.

[0048] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal system that includes that element.

[0049] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0050] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A document modification analysis method integrating multimodal visual understanding, characterized in that, The document modification analysis method that integrates multimodal visual understanding includes: The system obtains user-inputted text, image links, and front-end transmitted images. It then performs image normalization processing on the text, image links, and front-end transmitted images according to a dual-channel normalization rule to obtain standardized multimodal input data. The standardized multimodal input data is input into a large visual model, which then identifies the standardized multimodal input data to obtain structured element information. Obtain the RAG retrieval rules, and modify and analyze the structured element information according to the RAG retrieval rules to obtain the document modification analysis results.

2. The document modification analysis method integrating multimodal visual understanding according to claim 1, characterized in that, The process of acquiring user-input text, image links, and front-end directly transmitted images, and then performing image normalization processing on the text, image links, and front-end directly transmitted images according to a dual-channel normalization rule to obtain standardized multimodal input data, specifically includes: The system obtains text, image links, and images input by the user; extracts the image links according to a dual-channel normalization rule to obtain an initial file; and encodes the initial file according to a preset encoding format to obtain a target file. The front-end direct transmission image is prefixed according to the dual-channel normalization rule to obtain the processed front-end direct transmission image. The text, the target file, and the processed front-end direct-transmitted image are fused through an encoding layer to obtain standardized multimodal input data.

3. The document modification analysis method integrating multimodal visual understanding according to claim 1, characterized in that, The process involves inputting the standardized multimodal input data into a large visual model, whereby the large visual model identifies the standardized multimodal input data to obtain structured element information. Specifically, this includes: Obtain the image values ​​of the standardized multimodal input data, and input the image values ​​and the text into the large visual model; The large visual model fuses the image values ​​and the text through an encoding layer to obtain fused feature information; The visual big model maps the fused feature information to data distribution through a transformation layer to obtain rule terms and text content; The rules and text content are structured to obtain structured element information.

4. The document modification analysis method integrating multimodal visual understanding according to claim 3, characterized in that, The process of inputting the standardized multimodal input data into a large visual model, and the large visual model recognizing the standardized multimodal input data to obtain structured feature information, further includes: The text is captured based on recurrent neural networks and long short-term memory networks to generate context sequence data. The context sequence data is then processed according to preset rules to obtain temporal recognition features. The temporal recognition features are input into the visual large model for training to obtain the trained visual large model. The trained large visual model is used to identify the standardized multimodal input data.

5. The document modification analysis method integrating multimodal visual understanding according to claim 4, characterized in that, The process involves inputting the standardized multimodal input data into a large visual model, where the large visual model identifies the standardized multimodal input data to obtain structured feature information. The process then further includes: The temporal recognition features are divided according to a preset division ratio to obtain a training set and a test set. The visual large model is trained according to the training set to obtain an initial visual large model. The loss value is obtained by calculating the difference between the structured element information and the parameters of the initial visual large model; The initial large visual model is back-optimized based on the test set and the loss value to obtain the optimal large visual model.

6. The document modification analysis method integrating multimodal visual understanding according to claim 2, characterized in that, The process of obtaining RAG retrieval rules and analyzing the structured element information based on those rules to obtain document modification analysis results specifically includes: Obtain RAG retrieval rules, and perform a retrieval in a pre-built knowledge base based on the RAG retrieval rules and the key points of the structured element information to obtain a set of retrieval results; The search result set is compared with the structured element information to identify compliance omissions, thereby obtaining compliance deviation analysis results. Based on the compliance deviation analysis results, the structured element information is modified and analyzed to obtain the document modification analysis results.

7. The document modification analysis method integrating multimodal visual understanding according to claim 6, characterized in that, The step of modifying and analyzing the structured element information based on the compliance deviation analysis results to obtain document modification analysis results specifically includes: Based on the anomalies identified in the compliance deviation analysis results, the structured element information is located to obtain the missing content. Obtain the original input image, and map the abnormal missing content and the necessary missing content of the compliance deviation analysis result to the original input image to obtain the modified positioning information; Based on the preset standard contract template and preset rule content, the abnormal missing content is replaced according to the modification location information and the preset rule content to obtain the document modification analysis results.

8. A document modification analysis system integrating multimodal visual understanding, characterized in that, The document modification analysis system that integrates multimodal visual understanding includes: The multimodal data acquisition module is used to acquire user-input text, image links, and front-end directly transmitted images, and to perform image normalization processing on the text, image links, and front-end directly transmitted images according to the dual-channel normalization rule to obtain standardized multimodal input data. The data recognition module is used to input the standardized multimodal input data into the visual large model, and the visual large model recognizes the standardized multimodal input data to obtain structured element information; The data modification module is used to obtain RAG retrieval rules, modify and analyze the structured element information according to the RAG retrieval rules, and obtain document modification analysis results.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a document modification analysis program that integrates multimodal visual understanding, stored in the memory and executable on the processor. When the document modification analysis program that integrates multimodal visual understanding is executed by the processor, it implements the steps of the document modification analysis method that integrates multimodal visual understanding as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a document modification analysis program that integrates multimodal visual understanding. When the document modification analysis program that integrates multimodal visual understanding is executed by a processor, it implements the steps of the document modification analysis method that integrates multimodal visual understanding as described in any one of claims 1-7.

Images