Multi-modal document analysis method and device, electronic equipment and storage medium

By classifying and module screening, content analysis and topic extraction of multimodal documents, and combining information location information for information merge, the problem of low accuracy of multimodal documents in traditional technology is solved, and more efficient document analysis effect is achieved .

CN118917300APending Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411060589.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional document analysis technology cannot effectively parse potential associations between different types of content in multimodal documents, resulting in low parsing accuracy.

Method used

By obtaining multimodal documents, classifying them according to the original modal categories, filtering out corresponding analysis modules for content analysis, extracting topic information, and combining information based on topic information and content location information to mine the relationship between content.

Benefits of technology

It improves the analytical accuracy of multimodal documents and can more accurately understand and integrate the associations of multiple types of content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917300A_ABST
    Figure CN118917300A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a multi-modal document analysis method and device, electronic equipment and a storage medium, and belongs to the technical field of document processing. The method comprises the following steps: acquiring a target multi-modal document; wherein the multi-modal document comprises documents of at least two original modal categories; classifying the target multi-modal document according to the original modal category to obtain target classification information; wherein the target classification information comprises target content data, a target modal category and target content position information; screening out a target analysis module from a preset content analysis module according to the target modal category; performing content analysis on the target content data based on a target analysis module to obtain target content analysis information; performing theme extraction according to the target content analysis information to obtain target theme information; and performing information combination on the target content analysis information according to the target theme information and the target content position information to obtain a target analysis document. According to the embodiment of the invention, the multi-modal document can be accurately analyzed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of document processing technology, and in particular to a multimodal document parsing method and device, electronic equipment and storage medium. Background Art

[0002] A multimodal document is a document that includes at least two types of documents, such as images, text, and videos. In traditional document parsing technology, images, text, and videos can only be parsed separately and the parsing results are simply combined together, without considering the potential correlation between multiple types of content, resulting in low parsing accuracy of the generated parsed documents. Therefore, how to accurately parse multimodal documents has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a multimodal document parsing method and device, electronic device and storage medium, aiming to accurately parse multimodal documents.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a multimodal document parsing method, the method comprising:

[0005] Acquire a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories;

[0006] Classifying the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, a target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document;

[0007] Filtering a target parsing module from preset content parsing modules according to the target modality category;

[0008] Performing content analysis on the target content data based on the target analysis module to obtain target content analysis information;

[0009] Extracting the subject according to the target content parsing information to obtain target subject information;

[0010] The target content parsing information is merged according to the target subject information and the target content location information to obtain a target parsing document.

[0011] In some embodiments, the step of merging the target content parsing information according to the target subject information and the target content location information to obtain a target parsing document includes:

[0012] Performing importance evaluation on the target content parsed information according to the target subject information to obtain information importance evaluation data;

[0013] The target content parsed information is merged according to the target subject information, the information importance evaluation data and the target content location information to obtain the target parsed document.

[0014] In some embodiments, the step of merging the target content parsed information according to the target subject information, the information importance evaluation data and the target content location information to obtain the target parsed document includes:

[0015] Compressing the target content analysis information according to the information importance evaluation data to obtain updated content analysis information;

[0016] The updated content parsing information is merged according to the target subject information and the content location information to obtain the target parsed document.

[0017] In some embodiments, performing content parsing on the target content data based on the target parsing module to obtain target content parsing information includes:

[0018] Performing data parsing on the target content data based on the target parsing module to obtain candidate content parsing information;

[0019] Obtaining satisfaction evaluation data of the target parsing module;

[0020] The candidate content analysis information is screened according to the satisfaction evaluation data to obtain the target content analysis information.

[0021] In some embodiments, the performing data parsing on the target content data based on the target parsing module to obtain candidate content parsing information includes:

[0022] Obtain the number of target parsing modules to obtain the number of modules;

[0023] Creating threads according to the number of modules to obtain target threads; wherein the number of target threads is equal to the number of modules;

[0024] Performing thread configuration on the target parsing module according to the target thread to obtain a target parsing thread;

[0025] The target content data is parsed based on the target parsing thread within a preset parsing time to obtain the candidate content parsing information.

[0026] In some embodiments, extracting a topic according to the target content parsing information to obtain target topic information includes:

[0027] Performing clustering processing according to the target content analysis information to obtain clustering areas;

[0028] Extracting topics according to the clustering regions to obtain candidate topic information;

[0029] Calculate the word frequency according to the target content analysis information to obtain the keyword frequency;

[0030] The candidate topic information is screened according to the keyword frequency to obtain the target topic information.

[0031] In some embodiments, performing clustering processing according to the target content parsing information to obtain clustering areas includes:

[0032] Perform word splitting according to the target content analysis information to obtain a text vector;

[0033] Performing summary extraction processing on the text vector by using a preset summary extraction model to obtain target summary information;

[0034] Clustering is performed according to the target summary information to obtain the clustering area.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a multimodal document parsing device, the device comprising:

[0036] An acquisition module, used to acquire a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories;

[0037] A classification module, configured to classify the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, a target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document;

[0038] A screening module, used for screening out a target parsing module from preset content parsing modules according to the target modality category;

[0039] A parsing module, configured to perform content parsing on the target content data based on the target parsing module to obtain target content parsing information;

[0040] An extraction module, used to extract the subject according to the target content parsing information to obtain target subject information;

[0041] The merging module is used to merge the target content parsing information according to the target subject information and the target content location information to obtain a target parsing document.

[0042] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0044] The present application proposes a multimodal document parsing method and device, electronic device and storage medium, which obtains a target modal document including at least two original modal categories, and classifies the target multimodal document according to the original modal category to obtain target content data, target modal category and target content location information, and then selects a target parsing module from a preset content parsing module according to the target modal category, and then performs content parsing on the target content data based on the target parsing module to obtain target content parsing information, thereby parsing the target content data according to the modal category of the target content data and improving the accuracy of the parsing. Furthermore, the subject is extracted according to the target parsing information to obtain the target subject information, and then the target content parsing information is merged according to the target subject information and the target content location information, thereby mining the potential association between the target content parsing information and the target subject information and the target content location information, and the information is merged based on the potential association, thereby improving the accuracy of parsing the multimodal document. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a multimodal document parsing method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Flow chart of step S104 in FIG.

[0047] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0048] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0049] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0050] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0051] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.

[0052] Figure 8 It is a structural diagram of a multimodal document parsing device provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Multimodal Document: refers to a document that contains multiple different types of information expressions, including text, images, audio, and video. Multimodal documents combine various forms of information to provide more comprehensive and rich information content and context. Its core lies in integrating content from different modes to achieve more efficient information transfer and deeper understanding. For example, a multimodal document may contain an article (text), accompanying pictures, related video clips, and background music.

[0059] Clustering: is a data mining technique used to divide a group of objects into multiple clusters, so that objects in the same cluster are more similar in some sense, while objects in different clusters are as different as possible. The purpose of cluster analysis is to discover potential patterns and structures in data. It is widely used in unsupervised learning, which clusters similar data points together without pre-labeled data.

[0060] Thread: It is the basic unit for operating system to schedule operations. It is a basic unit that is smaller than a process and can run independently. A process can be composed of multiple threads. These threads share the resources of the process, such as memory space, file handles, etc., but each thread has its own stack space and registers. The main function of threads is to improve the efficiency of program execution, especially in multi-processor systems, where threads can realize parallel computing and thus make better use of system resources. Threads can communicate with each other through shared memory, which is faster than inter-process communication.

[0061] A multimodal document is a document that includes at least two types of documents, such as images, text, and videos. In traditional document parsing technology, images, text, and videos can only be parsed separately and the parsing results are simply combined together, without considering the potential correlation between multiple types of content, resulting in low parsing accuracy of the generated parsed documents. Therefore, how to accurately parse multimodal documents has become an urgent problem to be solved.

[0062] Based on this, the embodiments of the present application provide a multimodal document parsing method and device, an electronic device and a storage medium, aiming to accurately parse multimodal documents.

[0063] A multimodal document parsing method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the multimodal document parsing method in the embodiments of the present application is described.

[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0066] The multimodal document parsing method provided in the embodiment of the present application relates to the field of document processing technology. The multimodal document parsing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the multimodal document parsing method, etc., but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0069] Figure 1 is an optional flowchart of the multimodal document parsing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0070] Step S101, obtaining a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories;

[0071] Step S102, classifying the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document;

[0072] Step S103, selecting a target parsing module from preset content parsing modules according to the target modality category;

[0073] Step S104, performing content analysis on the target content data based on the target analysis module to obtain target content analysis information;

[0074] Step S105, extracting the subject according to the target content parsing information to obtain target subject information;

[0075] Step S106: merging the target content analysis information according to the target subject information and the target content location information to obtain a target analysis document.

[0076] Steps S101 to S106 shown in the embodiment of the present application are obtained by obtaining a target modal document including at least two original modal categories, and classifying the target multimodal document according to the original modal category to obtain target content data, target modal category and target content location information, and then filtering out a target parsing module from a preset content parsing module according to the target modal category, and then performing content parsing on the target content data based on the target parsing module to obtain target content parsing information, thereby parsing the target content data according to the modal category of the target content data and improving the accuracy of the parsing. Further, topic extraction is performed based on the target parsing information to obtain target topic information, and then information merging is performed on the target content parsing information based on the target topic information and the target content location information, thereby mining the potential association between the target content parsing information and the target topic information and the target content location information, and information merging is performed based on the potential association, thereby improving the accuracy of parsing the multimodal document.

[0077] In step S101 of some embodiments, the target multimodal document is a document containing multiple different modal categories, such as text, image, audio, and video. Multimodal documents combine multiple modal categories to provide more comprehensive and rich information content and context. For example, a presentation document (Power Point, PPT) includes video, audio, text, and images. The original modal category is the category of the content in the target multimodal document. For example, if PPT includes video, audio, and text, the original modal category is video, audio, and text.

[0078] In step S102 of some embodiments, the target multimodal document is classified according to the original modal category to obtain the content corresponding to each original modal category. For example, the first to fifth lines of the first page of the PPT are text, below the fifth line is a picture, and the second page is a video. The target multimodal document is classified according to the text to obtain the text, i.e., the target content data, the text is located in the first to fifth lines of the first page, i.e., the target content location information, and the target modal category is confirmed to be text. The target multimodal document is classified according to the picture to obtain the picture, i.e., the target content data, the picture is behind the fifth line of the first page, i.e., the target content location information, and the target modal category is confirmed to be a picture. The target multimodal document is classified according to the video to obtain the video, i.e., the target content data, the video is located on the second page, i.e., the target content location information, and the target modal category is confirmed to be a video.

[0079] In step S103 of some embodiments, the preset content analysis module is a content analysis module corresponding to each modality category. For example, for the text category, the preset content analysis module may include a first text content analysis module and a second text content analysis module. Among them, the first content analysis module is a convolutional neural network, and the second content analysis module is a fully connected layer deep learning network. According to the target modality category, the corresponding multiple target analysis modules are selected from the preset content analysis module, which improves the analysis accuracy for subsequent content analysis.

[0080] See also Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S203:

[0081] Step S201, performing data analysis on target content data based on a target analysis module to obtain candidate content analysis information;

[0082] Step S202, obtaining satisfaction evaluation data of the target analysis module;

[0083] Step S203: Screen candidate content analysis information according to the satisfaction evaluation data to obtain target content analysis information.

[0084] Steps S201 to S203 shown in the embodiment of the present application are to obtain candidate content analysis information by performing data analysis on the target content data based on the target analysis module, and then obtain satisfaction evaluation data of the target analysis module. Finally, the candidate content analysis information is screened according to the satisfaction evaluation data to obtain target content analysis information, thereby completing the screening of multiple candidate content analysis information according to the satisfaction evaluation data, obtaining the target content analysis information with the highest satisfaction, and improving the analysis accuracy of multimodal documents.

[0085] See also Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S304:

[0086] Step S301, obtaining the number of target parsing modules to obtain the number of modules;

[0087] Step S302, creating threads according to the number of modules to obtain target threads; wherein the number of target threads is equal to the number of modules;

[0088] Step S303, thread configuration is performed on the target parsing module according to the target thread to obtain the target parsing thread;

[0089] Step S304: parse the target content data based on the target parsing thread within a preset parsing time to obtain candidate content parsing information.

[0090] Steps S301 to S304 shown in the embodiment of the present application are to obtain the number of modules by acquiring the number of target parsing modules, and to create threads according to the number of modules to obtain the target thread, and then to configure the threads of the target parsing modules according to the target thread to obtain the target parsing thread, and finally to perform data parsing on the target content data based on the target parsing thread within a preset parsing time to obtain candidate content parsing information, thereby realizing the process of performing content parsing based on multiple target parsing modules in an asynchronous manner, thereby reducing the waiting time for content parsing, and at the same time, setting a preset parsing time to avoid continuous waiting for the target parsing thread.

[0091] In step S301 of some embodiments, the number of target parsing modules includes at least one, for example, for a text category, there is a first text content parsing module and a second text content parsing module. The first content parsing module is a convolutional neural network, and the second content parsing module is a fully connected layer deep learning network, that is, the number of modules is two.

[0092] In step S302 of some embodiments, threads are created according to the number of modules, that is, a corresponding thread is created according to each target parsing module to obtain a target thread. It should be noted that threads can be created according to a thread pool, or a new thread can be created directly, and this application does not make specific restrictions.

[0093] In step S303 of some embodiments, the target parsing module is thread-configured according to the target thread, that is, the task of the target thread is set as the target parsing module to parse the target content, and the target parsing module is thread-configured according to the target thread to obtain the target parsing thread.

[0094] In step S304 of some embodiments, a target parsing thread is started, and the execution time of the target parsing thread is monitored. When the execution time is longer than a preset parsing time, such as 10 seconds, the target parsing thread is terminated. When the parsing thread completes the parsing of the target content data within the execution time, the parsing result is placed in a preset content list. That is, multiple target parsing threads are started, and the parsing results of the threads that complete the content parsing within the preset parsing time are collected.

[0095] In one embodiment, the target resolution module is located in multiple servers in different geographical locations. When the target resolution thread performs content resolution, it needs to transmit over the network to complete the sending and receiving of the request. Therefore, a preset resolution time is set to avoid continuously waiting for the target resolution thread to perform content resolution when the network is poor.

[0096] In one embodiment, the target parsing modules are located in different physical servers, some of which have fewer available resources, resulting in a longer parsing time for the target parsing thread. By setting a preset parsing time, the situation in which the parsing time is too long when some physical servers have fewer available resources can be avoided.

[0097] In step S202 of some embodiments, the satisfaction evaluation data is the evaluation feedback data of the target parsing module by the historical users. For example, the scoring information is obtained by the historical users scoring the information parsed by the target parsing module. For example, the evaluation information is obtained by the historical users evaluating the information parsed by the target parsing module, and then judging the satisfaction of the historical users with the target parsing module based on the preset sentiment judgment model.

[0098] In step S203 of some embodiments, the candidate content analysis information is screened according to the satisfaction evaluation data, specifically to obtain the highest satisfaction evaluation data, and obtain the corresponding content analysis module according to the highest satisfaction evaluation data, and obtain the corresponding analysis information, and obtain the target content analysis information.

[0099] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S404:

[0100] Step S401, performing clustering processing according to target content analysis information to obtain clustering areas;

[0101] Step S402, extracting topics according to the clustered areas to obtain candidate topic information;

[0102] Step S403, calculating the word frequency according to the target content analysis information to obtain the keyword word frequency;

[0103] Step S404, screening candidate topic information according to keyword frequency to obtain target topic information.

[0104] Steps S401 to S404 shown in the embodiment of the present application perform clustering processing based on the target content analysis information to obtain clustering areas, and perform topic extraction based on the clustering areas to obtain candidate topic information, then perform word frequency calculation based on the target content analysis information to obtain keyword word frequency, and finally screen the candidate topic information based on the keyword word frequency to obtain target topic information, thereby accurately extracting topics from the target content analysis information, providing an information basis for subsequent information merging based on the target topic information.

[0105] See also Figure 5 In some embodiments, step S401 includes but is not limited to steps S501 to S503:

[0106] Step S501, performing word splitting according to target content parsing information to obtain text vectors;

[0107] Step S502, extracting summary from the text vector using a preset summary extraction model to obtain target summary information;

[0108] Step S503: performing clustering processing according to the target summary information to obtain clustering areas.

[0109] In steps S501 to S503 shown in the embodiment of the present application, word splitting is performed according to the target content analysis information to obtain a text vector, and then a summary extraction process is performed on the text vector through a preset summary extraction model to obtain target summary information, and finally clustering is performed according to the target summary information to obtain clustering areas, thereby accurately clustering the target content analysis information, and providing an information basis for subsequent clustering-based topic extraction.

[0110] In step S501 of some embodiments, word splitting is to perform keyword splitting on the target parsed information, including removing stop words, punctuation marks, keyword extraction, part-of-speech restoration and stem extraction, so as to obtain a text vector from the target content parsed information.

[0111] In step S502 of some embodiments, the summary extraction model is a model for extracting a summary from a text vector. In one embodiment, the summary extraction model is a sequence-to-sequence model (Seq2Seq, S2S), and S2S includes an encoder and a decoder. The encoder is used to extract key information from the text vector and obtain the encoded information. The decoder decodes based on the encoded information to obtain target summary information. In one embodiment, the summary extraction model is a text ranking model (TextRank), which takes each keyword in the graph as a node, and then determines the edge weight between two keywords based on the sentence, thereby selecting the word with the highest edge weight in the keyword as the target word, and generating text summary information based on the target word.

[0112] In step S503 of some embodiments, clustering is a process of grouping text summary information, where the text summary information is divided into multiple groups, where the sentences in each group are similar in meaning and there are significant differences in the sentences between the groups. This application does not specifically limit the specific method of clustering.

[0113] In step S402 of some embodiments, topic extraction is to identify and extract main topics from cluster regions, which is used to determine the core content of cluster regions, and candidate topic information is the main core content of cluster regions. In one embodiment, topic extraction is based on latent Dirichlet allocation (LDA) to extract topic information from cluster regions to obtain candidate topic information. In one embodiment, topic extraction is based on the term frequency-inverse document frequency (TF-IDF) method to identify important words or phrases to extract topics.

[0114] In step S403 of some embodiments, the word frequency calculation is to calculate the number of times each keyword appears in the target content analysis information, that is, to obtain the keyword word frequency. The calculation method is to obtain the ratio of the number of keyword appearances to the total number of document words to obtain the keyword word frequency.

[0115] In step S404 of some embodiments, candidate topics are screened according to keyword frequency. The screening method is to screen out target keywords according to a preset frequency threshold, and calculate the similarity of candidate topic information based on the target keywords, and select the candidate topic information with the highest similarity to obtain the target topic information.

[0116] See also Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S602:

[0117] Step S601, evaluating the importance of target content parsed information according to target subject information to obtain information importance evaluation data;

[0118] Step S602: merging the target content parsing information according to the target subject information, the information importance evaluation data and the target content location information to obtain a target parsing document.

[0119] In steps S601 to S602 shown in the embodiment of the present application, importance evaluation is performed on the target content analysis information according to the target subject information to obtain information importance evaluation data, and then information merging is performed on the target content analysis information according to the target subject information, the information importance evaluation data and the target content location information, thereby improving the merging accuracy when merging the target content analysis information according to the information importance evaluation data.

[0120] In step S601 of some embodiments, importance assessment is used to confirm the importance of target content parsed information to obtain information importance assessment data. Information importance assessment data can be in the form of priority or percentage, and this application does not impose specific restrictions. In one embodiment, importance assessment is based on word frequency statistics to determine information importance assessment data. In one embodiment, importance assessment is based on a support vector machine (SVM) to predict the importance of information to obtain information importance assessment data.

[0121] See also Figure 7 In some embodiments, step S602 may include but is not limited to steps S701 to S702:

[0122] Step S701, compressing the target content analysis information according to the information importance evaluation data to obtain updated content analysis information;

[0123] Step S702: merge the updated content resolution information according to the target subject information and the content location information to obtain a target resolution document.

[0124] Steps S701 to S702 in the embodiment of the present application are to obtain updated content analysis information by compressing the target content analysis information according to the information importance evaluation data.

[0125] In step S701 of some embodiments, content compression is to reduce the target content parsing information to more concise information while retaining key information, that is, to obtain updated content parsing information.

[0126] In step S702 of some embodiments, the updated content resolution information is merged according to the target subject information and the content location information to obtain the target resolution document. In one embodiment, the information is merged based on the Long Short-Term Memory network (LSTM), and the LSTM can effectively merge the updated content resolution information based on the target subject information and the content location information, and complete the information merging by capturing the location connection between the updated content resolution information and the potential association with the subject, thereby obtaining the target resolution document.

[0127] See also Figure 8 The embodiment of the present application also provides a multimodal document parsing device, which can implement the above multimodal document parsing method, and the device includes:

[0128] The acquisition module 801 is used to acquire a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories;

[0129] The classification module 802 is used to classify the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document;

[0130] A screening module 803 is used to screen out a target parsing module from preset content parsing modules according to a target modality category;

[0131] The parsing module 804 is used to perform content parsing on the target content data based on the target parsing module to obtain target content parsing information;

[0132] Extraction module 805, used to extract the subject according to the target content parsing information to obtain the target subject information;

[0133] The merging module 806 is used to merge the target content parsing information according to the target subject information and the target content location information to obtain a target parsing document.

[0134] The specific implementation of the multimodal document parsing device is basically the same as the specific implementation of the multimodal document parsing method described above, and will not be repeated here.

[0135] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multimodal document parsing method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0136] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0137] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0138] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the multimodal document parsing method of the embodiment of this application;

[0139] Input / output interface 903, used to implement information input and output;

[0140] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0141] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0142] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0143] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the multimodal document parsing method described above is implemented.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The multimodal document parsing method, multimodal document parsing device, electronic device and storage medium provided in the embodiment of the present application obtain a target modal document including at least two original modal categories, and classify the target multimodal document according to the original modal category to obtain target content data, target modal category and target content location information, then filter out a target parsing module from a preset content parsing module according to the target modal category, and then perform content parsing on the target content data based on the target parsing module to obtain target content parsing information, thereby parsing the target content data according to the modal category of the target content data and improving the accuracy of the parsing. Furthermore, the subject extraction is performed according to the target parsing information to obtain the target subject information, and then the target content parsing information is merged according to the target subject information and the target content location information, thereby mining the potential association between the target content parsing information and the target subject information and the target content location information, and the information is merged based on the potential association, thereby improving the accuracy of parsing the multimodal document.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multimodal document parsing method, characterized in that: The method comprises: Acquire a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories; Classifying the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, a target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document; Filtering a target parsing module from preset content parsing modules according to the target modality category; Performing content analysis on the target content data based on the target analysis module to obtain target content analysis information; Extracting the subject according to the target content parsing information to obtain target subject information; The target content parsing information is merged according to the target subject information and the target content location information to obtain a target parsing document.

2. The method according to claim 1, characterized in that The step of merging the target content parsing information according to the target subject information and the target content location information to obtain a target parsing document includes: Performing importance evaluation on the target content parsed information according to the target subject information to obtain information importance evaluation data; The target content parsed information is merged according to the target subject information, the information importance evaluation data and the target content location information to obtain the target parsed document.

3. The method according to claim 2, characterized in that The step of merging the target content parsed information according to the target subject information, the information importance evaluation data and the target content location information to obtain the target parsed document includes: Compressing the target content analysis information according to the information importance evaluation data to obtain updated content analysis information; The updated content parsing information is merged according to the target subject information and the content location information to obtain the target parsed document.

4. The method according to claim 1, characterized in that: The performing content analysis on the target content data based on the target analysis module to obtain target content analysis information includes: Performing data parsing on the target content data based on the target parsing module to obtain candidate content parsing information; Obtaining satisfaction evaluation data of the target parsing module; The candidate content analysis information is screened according to the satisfaction evaluation data to obtain the target content analysis information.

5. The method according to claim 4, characterized in that The step of performing data parsing on the target content data based on the target parsing module to obtain candidate content parsing information includes: Obtain the number of target parsing modules to obtain the number of modules; Creating threads according to the number of modules to obtain target threads; wherein the number of target threads is equal to the number of modules; Performing thread configuration on the target parsing module according to the target thread to obtain a target parsing thread; The target content data is parsed based on the target parsing thread within a preset parsing time to obtain the candidate content parsing information.

6. The method according to any one of claims 1 to 5, characterized in that: The subject extraction is performed according to the target content parsing information to obtain target subject information, including: Performing clustering processing according to the target content analysis information to obtain clustering areas; Extracting topics according to the clustering regions to obtain candidate topic information; Calculate the word frequency according to the target content analysis information to obtain the keyword frequency; The candidate topic information is screened according to the keyword frequency to obtain the target topic information.

7. The method according to claim 6, characterized in that The clustering process is performed according to the target content parsing information to obtain a clustering area, including: Perform word splitting according to the target content analysis information to obtain a text vector; Performing summary extraction processing on the text vector by using a preset summary extraction model to obtain target summary information; Clustering is performed according to the target summary information to obtain the clustering area.

8. A multimodal document parsing device, characterized in that: The device comprises: An acquisition module, used to acquire a target multimodal document; wherein the multimodal document includes documents of at least two original modality categories; A classification module, configured to classify the target multimodal document according to the original modality category to obtain target classification information; wherein the target classification information includes target content data, a target modality category and target content location information, the target modality category is one of the original modality categories, and the target content location information indicates the location of the target content data in the target multimodal document; A screening module, used for screening out a target parsing module from preset content parsing modules according to the target modality category; A parsing module, configured to perform content parsing on the target content data based on the target parsing module to obtain target content parsing information; An extraction module, used to extract the subject according to the target content parsing information to obtain target subject information; The merging module is used to merge the target content parsing information according to the target subject information and the target content location information to obtain a target parsing document.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multimodal document parsing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal document parsing method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • RAG multi-mode document analysis method and device and medium

    CN120087357A