Multi-modal document analysis system based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an artificial intelligence-based multi-modal document analysis system, which consists of an image preprocessing module, a multi-modal feature extraction module, a natural language processing module, a modal fusion and analysis module and a document structure identification module. According to the method, the image quality is improved through image sharpening and noise and blurring removal, text, table and image areas are extracted through masks respectively, different document contents are separated accurately, visual features and extracted area information are combined together, a comprehensive feature vector is generated, the document contents are understood more comprehensively, and the document quality is improved. According to the method, the text extracted from the image is analyzed, semantic analysis and keyword extraction are carried out, so that text content is understood, keywords are extracted from the analyzed text, main topics and key points of the document are recognized, and the beneficial effect that the analysis result is more accurate due to multi-modal analysis of the document is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal document parsing system based on artificial intelligence. Background Art

[0002] In real life, for some documents, reviewers need to patiently read their contents to extract the required elements in the documents and realize manual analysis of such documents; for example, in the case trial process, in order to quickly, effectively and accurately try the case, the judge usually pays attention to the key factors affecting the case characterization and the procedural matters affecting the case trial. However, since the details of specific cases are different, there are differences between each case to be tried, which means that the judge needs to read all the case materials (documents) carefully and review each element in the case materials when trying each case. Obviously, such an operation will consume a lot of judges' time and work efficiency is low.

[0003] In order to solve the defects of the prior art, a document parsing method, system, device and storage medium based on image recognition are found through searching, and the publication number is: CN 112989786A, wherein the method preprocesses the document to be parsed to convert the image information in the document to be parsed into text information; performs preliminary parsing on the preprocessed document to be parsed to divide the preprocessed document to be parsed into different types of materials, and determines the title of each type of material; determines the document type of the preprocessed document to be parsed based on the title of each type of material, and queries the preset element label factor system table through the document type to obtain the key element label corresponding to the document type; extracts elements from the preprocessed document to be parsed based on the key element label, and forms a key element table according to the key parsing elements. The technical solution provided by the present invention can solve the problem of low efficiency in manual reading of documents and parsing elements.

[0004] The aforementioned technologies mainly solve the problem of low efficiency in manual reading and parsing of documents, and use image recognition and document parsing technologies to automate these processes. However, if the image quality of the document to be parsed is not high, such as blur or noise, the effect of the preprocessing stage will be affected, thereby affecting the accuracy of the entire parsing process. Summary of the invention

[0005] Technical issues solved:

[0006] In view of the shortcomings of the prior art, the present invention provides a multimodal document parsing system based on artificial intelligence, which has the advantages of multimodal parsing documents to make the parsing results more accurate and avoid the impact of document ambiguity or noise on the parsing results, thereby solving the problems of the above-mentioned technologies.

[0007] Technical solution:

[0008] To achieve the above object, the present invention provides the following technical solutions: a multimodal document parsing system based on artificial intelligence, the parsing system is composed of an image preprocessing module, a multimodal feature extraction module, a natural language processing module, a modality fusion and parsing module and a document structure recognition module;

[0009] The image preprocessing module is used to perform denoising, deblurring and image enhancement preprocessing operations on the input document image to improve the image quality;

[0010] The multimodal feature extraction module is used to extract visual features from the document image while combining the structural information of the document: text area, table and image;

[0011] The natural language processing module is used to parse the text extracted from the image, perform semantic analysis, keyword extraction and title recognition;

[0012] The modality fusion and analysis module is used to fuse image features and text features, conduct comprehensive analysis, identify document types, extract key elements, and construct an element label system;

[0013] The document structure recognition module is used to recognize structural information in the document, including titles, paragraphs, tables and images;

[0014] The image enhancement expression in the image preprocessing module is:

[0015] I'(x,y)=αI(x,y)+β

[0016] Where: I(x,y) is the original image, I′(x,y) is the enhanced image, α is the contrast adjustment factor, and β is the brightness adjustment factor.

[0017] Preferably, the image preprocessing module further includes denoising, and the expression of denoising is:

[0018]

[0019] Where: k is the size of the filter.

[0020] Preferably, the deblurring expression in the image preprocessing module is:

[0021]

[0022] in: represents the Fourier transform, represents inverse Fourier transform, I(x,y) is the blurred image, and K(x,y) is the blur kernel.

[0023] Preferably, the visual feature extraction expression in the multimodal feature extraction is:

[0024] F visual =Φ(I(x,y))

[0025] Among them: F visual Represents the visual features extracted from the image I(x,y);

[0026] The text area extraction expression is:

[0027] R text =I(x, y)·M text

[0028] Where: R text represents the extracted text area image; M text is the mask of the text area used to extract the text area in the image;

[0029] The table feature extraction expression is: R table =I(x, y)·M table , R table represents the extracted table area image, M table is the mask of the table area, which is used to extract the table area in the image;

[0030] The image feature extraction expression is R image =I(x, y)·M image , where R image represents the extracted image region image, M image is the mask of the image region, which is used to extract the image region in the image:

[0031] The combined structural information expression is: F multi =[F visual , R text , R table , R image ] Among them F multi is the combined multimodal feature vector.

[0032] Preferably, the text analysis expression in the natural language processing module is:

[0033] T=Parse(R text )

[0034] T represents the text data obtained by parsing. Parse is to parse the text area extracted from the image to generate usable text data. text is the text region extracted from the image.

[0035] Preferably, the keyword extraction expression in the natural language processing module is:

[0036] K = ExtractKeywords(T)

[0037] K represents a set of keywords extracted from the text data T.

[0038] Preferably, the expression for fusion of image features and text features in the state fusion and analysis module is:

[0039] F=Fuse(I,T)

[0040] Among them: Fuse(I,T) means fusing the image feature I and the text feature T into a comprehensive feature F;

[0041] The comprehensive analysis expression is:

[0042] C=Analyze(F)

[0043] Among them: Analyze(F) means to perform a comprehensive analysis on the fusion feature F to obtain the analysis result C.

[0044] Preferably, the state fusion and parsing module identifies the document type expression as:

[0045] Type = IdentifyType (C)

[0046] Wherein: IdentifyType(C) indicates identifying the document type from the analysis result C to obtain the identified document type Type.

[0047] Preferably, the expression for extracting key elements is:

[0048] KeyElements=ExtractKeyElements(C)

[0049] Among them: ExtractKeyElements(C) means extracting key elements from the analysis result C to obtain key elements KeyElements.

[0050] Preferably, the construction element label system expression is:

[0051] LabelSystem=BuildLabelSystem(KeyElements)

[0052] Among them: BuildLabelSystem(KeyElements) means building an element label system according to the extracted key elements KeyElements to obtain a label system LabelSystem.

[0053] Compared with the prior art, the present invention provides a multimodal document parsing system based on artificial intelligence, which has the following beneficial effects:

[0054] 1. The present invention improves image quality by clarifying images, removing noise and blur, and uses masks to extract text, tables and image areas respectively, so as to accurately separate different document contents, and combines visual features with extracted area information to generate a comprehensive feature vector, so as to more comprehensively understand the content of the document, parse the text extracted from the image, perform semantic analysis and keyword extraction, so as to understand the text content, extract keywords from the parsed text, help identify the main theme and key points of the document, and achieve the beneficial effect of multimodal parsing documents to make the parsing results more accurate.

[0055] 2. The present invention uses a denoising algorithm to remove noise from the image, reduce random noise in the image, improve the clarity of the image, and make subsequent feature extraction more accurate. It improves image quality by means of contrast adjustment, brightness enhancement, etc., making text and images easier to distinguish. The enhanced image provides a clearer basis for subsequent analysis and reduces the possibility of misidentification. When performing feature extraction on the image, appropriate regional masks are used to specifically extract text, charts and other important information areas. This precise regional extraction reduces interference caused by blur and noise, ensuring that the extracted content is as true and accurate as possible. When parsing text, semantic analysis and grammar checking are performed on the text extracted from the image to ensure that even in the case of poor image quality, the text information can be identified and understood as accurately as possible, thereby achieving the beneficial effect of avoiding the impact of document blur or noise on the parsing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the structure of the present invention; DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 , a multimodal document parsing system based on artificial intelligence, the parsing system consists of an image preprocessing module, a multimodal feature extraction module, a natural language processing module, a modality fusion and parsing module and a document structure recognition module;

[0059] The image preprocessing module is used to perform denoising, deblurring and image enhancement preprocessing operations on the input document image to improve the image quality;

[0060] The multimodal feature extraction module is used to extract visual features from the document image while combining the structural information of the document: text area, table and image;

[0061] The natural language processing module is used to parse the text extracted from the image, perform semantic analysis, keyword extraction and title recognition;

[0062] The modality fusion and analysis module is used to fuse image features and text features, conduct comprehensive analysis, identify document types, extract key elements, and construct an element label system;

[0063] The document structure recognition module is used to recognize structural information in the document, including titles, paragraphs, tables and images;

[0064] The image enhancement expression in the image preprocessing module is:

[0065] I′(x,y)=αI(x,y)+β

[0066] Where: I(x,y) is the original image, I′(x,y) is the enhanced image, α is the contrast adjustment factor, and β is the brightness adjustment factor.

[0067] By sharpening the image and removing noise and blur, the image quality is improved. This helps subsequent feature extraction and analysis become more accurate. For example, the enhanced image shows text and image content more clearly, thereby improving the accuracy of feature extraction.

[0068] Use filters to smooth the image and remove noise from the image to make the image cleaner.

[0069] Special mathematical methods are used to remove blur from images, making image details clearer.

[0070] Extract visual features such as color, shape, and texture from images to capture the details of the image.

[0071] Use masks to extract text, table, and image areas separately, allowing you to accurately separate different document contents.

[0072] Combine structural information: Combine visual features with extracted regional information to generate a comprehensive feature vector, which provides a more comprehensive understanding of the content of the document.

[0073] Parse the text extracted from the image, perform semantic analysis and keyword extraction to understand the text content.

[0074] Extract keywords from the parsed text to help identify the main topics and key points of the document.

[0075] Fusion of image and text features: Combining image features and text features to form a comprehensive feature. This fusion allows the system to better consider visual information and text information.

[0076] Analyze the fused features, identify the document type, extract key elements, and establish a labeling system to more accurately parse the document content.

[0077] Determine the layout of headings, paragraphs, tables, and images in a document, helping the system better understand the structure of the document and thus more efficiently process and analyze the document content.

[0078] Preferably, the image preprocessing module further includes denoising, and the expression of denoising is:

[0079]

[0080] Where: k is the size of the filter.

[0081] Preferably, the deblurring expression in the image preprocessing module is:

[0082]

[0083] in: represents the Fourier transform, represents inverse Fourier transform, I(x,y) is the blurred image, and K(x,y) is the blur kernel.

[0084] Preferably, the visual feature extraction expression in the multimodal feature extraction is:

[0085] F visual =Φ(I(x,y))

[0086] Among them: F visual Represents the visual features extracted from the image I(x,y);

[0087] The text area extraction expression is:

[0088] R text =I(x,y)·M text

[0089] Where: R text represents the extracted text area image; M text is the mask of the text area used to extract the text area in the image;

[0090] The table feature extraction expression is: R table =I(x,y)·Mtable , R table represents the extracted table area image, M table is the mask of the table area, which is used to extract the table area in the image;

[0091] The image feature extraction expression is R image =I(x,y)·M image , where R image represents the extracted image region image, M image is the mask of the image region, which is used to extract the image region in the image:

[0092] The combined structural information expression is: F multi =[F visual ,R text ,R table ,R image ], where F multi is the combined multimodal feature vector.

[0093] Preferably, the text analysis expression in the natural language processing module is:

[0094] T=Parse(R text )

[0095] T represents the text data obtained by parsing. Parse is to parse the text area extracted from the image to generate usable text data. text is the text region extracted from the image.

[0096] Preferably, the keyword extraction expression in the natural language processing module is:

[0097] K = ExtractKeywords(T)

[0098] K represents a set of keywords extracted from the text data T.

[0099] Preferably, the expression for fusion of image features and text features in the state fusion and analysis module is:

[0100] F=Fuse(I,T)

[0101] Among them: Fuse(I,T) means fusing the image feature I and the text feature T into a comprehensive feature F;

[0102] The comprehensive analysis expression is:

[0103] C=Analyze(F)

[0104] Among them: Analyze(F) means to perform a comprehensive analysis on the fusion feature F to obtain the analysis result C.

[0105] Preferably, the state fusion and parsing module identifies the document type expression as:

[0106] Type = IdentifyType (C)

[0107] Wherein: IdentifyType(C) indicates identifying the document type from the analysis result C to obtain the identified document type Type.

[0108] Preferably, the expression for extracting key elements is:

[0109] KeyElements=ExtractKeyElements(C)

[0110] Among them: ExtractKeyElements(C) means extracting key elements from the analysis result C to obtain key elements KeyElements.

[0111] Preferably, the construction element label system expression is:

[0112] LabelSystem=BuildLabelSystem(KeyElements)

[0113] Among them: BuildLabelSystem(KeyElements) means building an element label system according to the extracted key elements KeyElements to obtain a label system LabelSystem.

[0114] Improve image quality by adjusting contrast and enhancing brightness, making text and images easier to distinguish. The enhanced image provides a clearer basis for subsequent analysis and reduces the possibility of misidentification.

[0115] When extracting features from images, appropriate region masks are used to specifically extract text, graphics, and other important information areas. This precise region extraction reduces interference caused by blur and noise, ensuring that the extracted content is as real and accurate as possible.

[0116] Combining visual features and structural information, the inherent characteristics in the image are effectively utilized to improve the parsing accuracy of low-quality image content.

[0117] When parsing text, semantic analysis and grammar checking are performed on the text extracted from the image to ensure that the text information can be recognized and understood as accurately as possible even in the case of poor image quality. This approach enables the system to maintain a certain degree of parsing ability in the face of noise and blur.

[0118] By comprehensively analyzing image features and text features, the system can better distinguish potential errors caused by blur or noise. Multimodal information fusion helps to complement each other and improve the robustness of the overall analysis.

[0119] During the processing, if the system detects that the image quality is low, it will adopt a reconstruction solution to truly restore the distorted part to improve the accuracy of the analysis.

[0120] By recognizing the structure of the document, the system can determine which parts are important information areas and prioritize these areas while ignoring other content affected by noise or ambiguity. This priority processing improves the effectiveness and accuracy of information extraction

[0121] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal document parsing system based on artificial intelligence, characterized in that: The analysis system consists of an image preprocessing module, a multimodal feature extraction module, a natural language processing module, a modality fusion and analysis module, and a document structure recognition module; The image preprocessing module is used to perform denoising, deblurring and image enhancement preprocessing operations on the input document image to improve the image quality; The multimodal feature extraction module is used to extract visual features from the document image while combining the structural information of the document: text area, table and image; The natural language processing module is used to parse the text extracted from the image, perform semantic analysis, keyword extraction and title recognition; The modality fusion and analysis module is used to fuse image features and text features, conduct comprehensive analysis, identify document types, extract key elements, and construct an element label system; The document structure recognition module is used to recognize structural information in the document, including titles, paragraphs, tables and images; The image enhancement expression in the image preprocessing module is: I′(x,y)=αI(x,y)+β Where: I(x,y) is the original image, I′(x,y) is the enhanced image, α is the contrast adjustment factor, and β is the brightness adjustment factor.

2. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The image preprocessing module also includes denoising, and the expression of denoising is: Where: k is the size of the filter.

3. The multimodal document parsing system based on artificial intelligence according to claim 2, characterized in that: The expression for deblurring in the image preprocessing module is: in: represents the Fourier transform, represents inverse Fourier transform, I(x,y) is the blurred image, and K(x,y) is the blur kernel.

4. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The visual feature extraction expression in the multimodal feature extraction is: F visual =Φ(I(x,y)) Among them: F visual Represents the visual features extracted from the image I(x,y); The text area extraction expression is: R text =I(x,y)·M text Where: R text represents the extracted text area image; M text is the mask of the text area used to extract the text area in the image; The table feature extraction expression is: R table =I(x,y)·M table , R table represents the extracted table area image, M table is the mask of the table area, which is used to extract the table area in the image; The image feature extraction expression is R image =I(x,y)·M image , where R image represents the extracted image region image, M image is the mask of the image region, which is used to extract the image region in the image: The combined structural information expression is: F multi =[F visual ,R text ,R table ,R image ], where F multi is the combined multimodal feature vector.

5. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The text analysis expression in the natural language processing module is: T=Parse(R text ) T represents the text data obtained by parsing. Parse is to parse the text area extracted from the image to generate usable text data. text is the text region extracted from the image.

6. The multimodal document parsing system based on artificial intelligence according to claim 5, characterized in that: The keyword extraction expression in the natural language processing module is: K = ExtractKeywords(T) K represents a set of keywords extracted from the text data T.

7. The multimodal document parsing system based on artificial intelligence according to claim 6, characterized in that: The expression of fusion image features and text features in the state fusion and analysis module is: F=Fuse(I,T) Among them: Fuse(I,T) means fusing the image feature I and the text feature T into a comprehensive feature F; The comprehensive analysis expression is: C=Analyze(F) Among them: Analyze(F) means to perform a comprehensive analysis on the fusion feature F to obtain the analysis result C.

8. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The expression for identifying the document type by the state fusion and parsing module is: Type = IdentifyType (C) Wherein: IdentifyType(C) indicates identifying the document type from the analysis result C to obtain the identified document type Type.

9. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The key element expression is extracted as follows: KeyElements=ExtractKeyElements(C) Among them: ExtractKeyElements(C) means extracting key elements from the analysis result C to obtain key elements KeyElements.

10. The multimodal document parsing system based on artificial intelligence according to claim 1, characterized in that: The expression for constructing the element label system is: LabelSystem=BuildLabelSystem(KeyElements) Among them: BuildLabelSystem(KeyElements) means building an element label system according to the extracted key elements KeyElements to obtain a label system LabelSystem.

Images