Data processing method, computer and storage medium

Abstract

The invention discloses a data processing method, a computer and a storage medium, and the method comprises the steps: detecting that a directory structure does not exist in a multi-modal document based on the text structure information of the multi-modal document, and obtaining a first theme of each page in the multi-modal document through semantic recognition; based on the first topic of each page in the multi-modal document, when it is detected that the similarity of the first topics of the adjacent pages in the multi-modal document is larger than a similarity threshold value, classifying the adjacent pages into the same page group; obtaining a second theme of each page group according to the first theme of the page in each page group in the multi-modal document; and obtaining a directory of each page group according to the second theme of each page group, and combining the directories of the plurality of page groups to obtain a directory structure of the multi-modal document. According to the method, grouping is carried out based on page similarity, the catalog is generated according to the grouped theme, the document core information can be fully reflected, the matching degree of the catalog and the multi-modal document content is ensured, and the catalog generation accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a data processing method, a computer, and a storage medium. Background Art

[0002] Cross-modal document data parsing has long been a key challenge and a challenge for algorithms and data engineering. The vast majority of documents compiled in real life present information in a cross-modal format. When multimodal documents lack a table of contents, users face significant difficulties in finding and locating specific information. Existing solutions for generating tables of contents for multimodal documents fail to fully extract and parse relevant content from all modalities. Consequently, the generated tables of contents based on the extracted document content fail to fully reflect the document's overall structure and core information, resulting in low accuracy. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, a computer, and a storage medium. After obtaining the first topic of all pages through semantic recognition, the pages are grouped based on the similarity of adjacent pages, and a directory is generated based on the second topic of the grouping. This method can fully reflect the overall structure and core information of the document, ensure the matching degree between the generated directory and the actual content of the multimodal document, and improve the accuracy of the generated directory.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0005] By performing structural analysis on a multimodal document, text structure information of the multimodal document is obtained; based on the text structure information of the multimodal document, it is detected that no directory structure exists in the multimodal document, and the first topic of each page in the multimodal document is obtained through semantic recognition; based on the first topic of each page in the multimodal document, it is detected that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, and the adjacent pages are classified into the same page group; based on the first topic of the page in each page group in the multimodal document, the second topic of each page group is obtained; based on the second topic of each page group, a directory of each page group is obtained, and the directories of the multiple page groups are combined to obtain the directory structure of the multimodal document.

[0006] In conjunction with the first aspect, in a possible implementation, obtaining text structure information of a multimodal document by performing structural analysis on the multimodal document includes:

[0007] Determine the file type of the multimodal document based on the file suffix of the multimodal document; determine the first structure parsing file for the multimodal document based on the configuration table of the structure parsing file and the file type of the multimodal document; parse the multimodal document based on the first structure parsing file to obtain text structure information of the multimodal document.

[0008] In conjunction with the first aspect, in one possible implementation, a mapping relationship between a file type of a multimodal document and a parsed file is configured in a configuration table of a structure parsing file, and a first structure parsing file for the multimodal document is determined based on the configuration table of the structure parsing file and the file type of the multimodal document, including:

[0009] According to the file type of the multimodal document, a query is performed from the configuration table of the structure parsing file to obtain a first structure parsing file corresponding to the file type of the multimodal document.

[0010] In conjunction with the first aspect, in one possible implementation, the text structure information includes format features and hierarchical relationships of the text, and the parsing of the multimodal document according to the first structure parsing file to obtain the text structure information of the multimodal document includes:

[0011] The multimodal document is traversed through the first structural parsing file to obtain format features and hierarchical relationships of text in the multimodal document.

[0012] In conjunction with the first aspect, in one possible implementation, the method further includes:

[0013] According to the content type and content layout of the page group, the content of the page group under each directory in the directory structure is split to obtain multiple module contents of the page group; if there is a first module content that does not match the page group among the multiple module contents of the page group, the page group that matches the first module content is determined; and the first module content is adjusted to the page group that matches the first module content.

[0014] In conjunction with the first aspect, in one possible implementation, if there is a first module content that does not match the page group among the multiple module contents of the page group, determining a page group that matches the first module content includes:

[0015] The third topic of each module content is obtained through semantic recognition; the similarity between the third topic of each module content and the second topic of the page grouping in the multimodal document is calculated; if the maximum similarity among the similarities is not the similarity between the third topic of the first module content and the second topic of the page grouping of the first module content, it is determined that the first module content and the page grouping of the first module content do not match; in the case that the first module content and the page grouping of the first module content do not match, the page grouping corresponding to the maximum similarity is determined as the page grouping that matches the first module content.

[0016] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0017] An acquisition module, configured to obtain text structure information of a multimodal document by performing structural analysis on the multimodal document;

[0018] an identification module configured to detect, based on text structure information of the multimodal document, that no directory structure exists in the multimodal document, and to obtain a first topic of each page in the multimodal document through semantic recognition; and to, based on the first topic of each page in the multimodal document, detect that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, and to group the adjacent pages into the same page group;

[0019] The acquisition module is also used to obtain the second topic of each page group in the multimodal document based on the first topics of all pages in each page group; and to obtain the directory of each page group based on the second topic of each page group, and combine the directories of the multiple page groups to obtain the directory structure of the multimodal document.

[0020] In a third aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0021] A memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program is configured to implement part or all of the steps described in any method in the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, some or all of the steps described in any method in the first aspect are implemented.

[0023] In a fifth aspect, embodiments of the present application provide a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform some or all of the steps described in any method of the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0024] By implementing the embodiment of the present application, the text structure information of the multimodal document is first obtained by structurally parsing the multimodal document; then, based on the text structure information of the multimodal document, it is detected that there is no directory structure in the multimodal document, and the first topic of each page in the multimodal document is obtained by semantic recognition; then, based on the first topic of each page in the multimodal document, it is detected that the similarity of the first topics of adjacent pages in the multimodal document is greater than the similarity threshold, and the adjacent pages are classified into the same page group; then, according to the first topic of each page in each page group in the multimodal document, the second topic of each page group is obtained; finally, according to the second topic of each page group, the directory of each page group is obtained, and the directories of the multiple page groups are combined to obtain the directory structure of the multimodal document. After obtaining the first topics of all pages through semantic recognition, they are grouped based on the similarity of adjacent pages, and a directory is generated according to the second topic of the group, which can fully reflect the overall structure and core information of the document, ensure the matching degree between the generated directory and the actual content of the multimodal document, and improve the accuracy of the generated directory. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0026] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0027] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present application;

[0028] Figure 3 This is a schematic diagram of the architecture of a large visual language model provided in an embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of module content in a multimodal document provided in an embodiment of the present application;

[0030] Figure 5 This is a schematic diagram of a data organization form of a four-layer architecture provided in an embodiment of the present application;

[0031] Figure 6This is a schematic diagram of adjusting the page grouping and module content matching provided by an embodiment of the present application;

[0032] Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0033] Figure 8 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work should fall within the scope of protection of the present invention.

[0035] The terms "first," "second," and "third," etc. in the specification, claims, and drawings of this application are used to distinguish between different objects, not to describe a particular order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0036] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] Cross-modal document data parsing has long been a key challenge and a challenge for algorithms and data engineering. In real life, the vast majority of documents compiled by people present information in a cross-modal format. When multimodal documents lack a table of contents, users face significant difficulties in finding and locating specific information.

[0038] Common methods for generating multimodal document directories are as follows:

[0039] Rule-matching-based method: Parse the multimodal document according to pre-set rules to generate a table of contents. The pre-set rules are usually formulated based on the common features and structures of various modes in the document. For example, for text modality, specific font formats (such as bold, font size, etc.), punctuation marks (such as specific title number formats) or specific keywords may be set to identify titles of different levels; for image modality, its association with the text content and its position in the table of contents may be determined based on the image's annotation information, the image's position in the document (such as whether it is at the beginning of a chapter), and the image's type (such as schematic diagrams, photos, etc.).

[0040] Template matching: Multimodal documents are compared and matched against pre-designed templates. Templates are created based on the common layouts of different types of multimodal documents and define the location, format, and logical relationships of the various modal content. During the matching process, the actual document content is compared against the corresponding locations and elements in the template, and the document's directory structure is determined based on the degree of match.

[0041] It can be seen that for multimodal documents, although existing solutions can generate directories, they cannot fully cover the complex situations of all multimodal documents. They cannot accurately analyze the content that does not conform to the preset rules or templates, resulting in low accuracy of the generated directory.

[0042] In response to the above problems, the embodiment of the present application provides a data processing method, a computer and a storage medium, which first obtains the text structure information of the multimodal document by structurally parsing the multimodal document; then, based on the text structure information of the multimodal document, it is detected that there is no directory structure in the multimodal document, and the first topic of each page in the multimodal document is obtained by semantic recognition; then, based on the first topic of each page in the multimodal document, it is detected that the similarity of the first topics of adjacent pages in the multimodal document is greater than the similarity threshold, and the adjacent pages are classified into the same page group; then, based on the first topic of each page in each page group in the multimodal document, the second topic of each page group is obtained; finally, based on the second topic of each page group, the directory of each page group is obtained, and the directories of the multiple page groups are combined to obtain the directory structure of the multimodal document. After obtaining the first topics of all pages through semantic recognition, they are grouped based on the similarity of adjacent pages, and a directory is generated based on the second topic of the group, which can fully reflect the overall structure and core information of the document, ensure the matching degree between the generated directory and the actual content of the multimodal document, and improve the accuracy of the generated directory.

[0043] The data processing method, computer and storage medium provided in the embodiments of the present application can be applied to Figure 1 In the data processing system shown, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application. Data processing system 100 includes client 101 and server 102. Client 101 can communicate with server 102 via a network. Client 101 refers to a user's device, such as a smartphone or computer. In this solution, client 101 is primarily responsible for sending multimodal documents to be processed to server 102. Client 101 is also used to receive multimodal documents containing a directory structure after server 102 completes directory generation.

[0044] Server 102 is a remote computer used to process large amounts of computing tasks and store data. In this solution, server 102 is responsible for structurally parsing the multimodal document. When it detects that a directory structure does not exist in the multimodal document, server 102 uses semantic recognition to determine the first topic of each page. It then groups adjacent pages whose first topic similarity exceeds a threshold into the same group, resulting in multiple page groups. Based on the second topic of each page group, server 102 determines a directory for each page group. Finally, server 102 combines the directories of the multiple page groups to obtain the directory structure of the multimodal document.

[0045] Based on this, the present application provides a data processing method, a computer and a storage medium, and the present application is described in detail below with reference to the accompanying drawings.

[0046] See also Figure 2 , Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0047] S201 , obtaining text structure information of a multimodal document by performing structural analysis on the multimodal document.

[0048] Among them, the execution subject of this method can be Figure 1 Data processing system 100 is shown with server 102 .

[0049] A multimodal document is one that contains information in two or more different modalities, where modality refers to the form of information presentation or perception channel. Common modalities in multimodal documents can be text, images, audio, video, charts, animations, etc. For example, presentations (PowerPoint, PPT) often contain both text and images, and Portable Document Format (PDF) files often contain a mixture of text, images, audio, etc. These are all multimodal documents.

[0050] Among them, the directory structure in the document is usually located at the beginning of the document. The directory structure is an index to the overall structure and content of the document. The directory can be composed of multiple titles and page numbers corresponding to the titles. The directory can be a hierarchical structure, and the hierarchical relationship between titles is reflected through indentation, font size, numbering, etc. For example, when there are multiple levels of titles, the first-level title font is set larger and aligned to the top of the page, and the second-level title font is set smaller and indented to indicate that the second-level title is the subordinate content of the first-level title.

[0051] In a possible implementation, obtaining text structure information of a multimodal document by performing structural analysis on the multimodal document includes:

[0052] Determine the file type of the multimodal document based on the file suffix of the multimodal document; determine the first structure parsing file for the multimodal document based on the configuration table of the structure parsing file and the file type of the multimodal document; parse the multimodal document based on the first structure parsing file to obtain text structure information of the multimodal document.

[0053] The file extension of a document is a mechanism used by the operating system to identify the file type. It's typically located at the end of the file name and consists of a period and several letters. For example, .doc is the file format used by Microsoft Word versions 97-2003, .docx is the default file format used by Word 2007 and later, .xls is the file format used by Excel versions 97-2003, .xlsx is the default format for Excel 2007 and later, .pdf is the Portable Document Format, and .txt is a plain text file.

[0054] The configuration table of the structure analysis file stores the structure analysis file corresponding to each file type. The configuration table of the structure analysis file can be configured by the user in Figure 1 Client 101 of data processing system 100 is shown as being configured or modified.

[0055] Among them, after structural analysis of the multimodal document, the text structure information obtained includes the format features and hierarchical relationships of the text. Furthermore, the multimodal document is parsed according to the first structural analysis file, and the method for obtaining the text structure information of the multimodal document can be: through the first structural analysis file, the multimodal document is traversed to obtain the format features and hierarchical relationships of the text in the multimodal document.

[0056] Specifically, when traversing the multimodal document through the first structure parsing file, the format features of the text obtained can be the display attributes of the text such as font, font size, color, bold, italic, underline, or the typesetting attributes of the text such as alignment, line spacing, and paragraph spacing; the hierarchical relationship of the text obtained can be the affiliation between different chapters or paragraphs in the document, such as the nested relationship between chapters, sections, and subsections.

[0057] As can be seen, in this example, the file type of a multimodal document is determined by its file suffix. The first structure parsing file is then found based on the configuration table and file type, and the multimodal document is then parsed to obtain text structure information. This operation based on file characteristics and the configuration table allows for effective processing of different types of multimodal documents, ensuring the accuracy of the text structure information required for subsequent use.

[0058] In one possible implementation, a mapping relationship between a file type of a multimodal document and a parsed file is configured in a configuration table of the structure parsing file, and determining a first structure parsing file for the multimodal document according to the configuration table of the structure parsing file and the file type of the multimodal document includes:

[0059] According to the file type of the multimodal document, a query is performed from the configuration table of the structure parsing file to obtain a first structure parsing file corresponding to the file type of the multimodal document.

[0060] Among them, the configuration table of the structure parsing file is configured with a mapping relationship between the file type of the multimodal document and the parsing file. The mapping relationship can be one-to-one, one-to-many or many-to-one, and is not limited here.

[0061] The configuration table of the structure parsing file may be as shown in Table 1 below. The mapping relationship between the file type of the multimodal document and the parsing file in Table 1 is one-to-one. The structure parsing file in Table 1 is a third-party package called by the parsing file:

[0062] Table 1 Configuration table of structure analysis file

[0063] File Type Structural analysis file (three-party package used for analysis) Word files Python-docx PDF files PyMuPDF TXT file Python native file processing package

[0064] In Table 1, the file suffix corresponding to the Word file type may be .docx, the file suffix corresponding to the PDF file type may be .pdf, and the file suffix corresponding to the TXT file type may be .txt.

[0065] The file structure parsing the Word file type in Table 1 is Python-docx, a third-party package required for parsing Word files. Python-docx is a Python third-party package for processing Microsoft Word files. Python-docx can read, write, and modify .docx documents. Users can use Python-docx to extract text content, paragraph formatting, table information, and more.

[0066] The structure parsing file corresponding to the PDF file type in Table 1 is PyMuPDF, which is a third-party package that needs to be called when parsing the PDF file type. PyMuPDF is a third-party package for Python. It is based on the MuPDF library and provides a series of functions for processing document formats such as PDF, XPS, OpenXPS, EPUB, FB2 and CBZ. In this solution, PyMuPDF is mainly used to process PDF files. Specifically, the get_outlines() method in PyMuPDF can be called to determine whether the PDF file has a directory structure.

[0067] The structure parsing file corresponding to the TXT file type in Table 1 is a Python native file processing package. Among them, Python native file processing mainly relies on the built-in open() function. By calling this function, operations such as reading and writing files can be performed. Specifically, each line of the TXT file can be read by calling the open() function, and it can be judged whether each line starts with "No." and contains "Chapter", or contains a number and ".", so as to judge whether the TXT file contains a directory structure. For example, when the TXT file read by calling the open() function includes "Chapter 1", "1.1", etc., it is considered that the TXT file contains a directory structure.

[0068] It can be seen that in this example, different types of documents have different structures and characteristics. By determining the corresponding structure parsing file according to the file type and mapping relationship, accurate structure parsing can be performed on documents of different file types.

[0069] In one possible implementation, the method further includes:

[0070] If the multimodal document does not have a file suffix, it is determined that there is no directory structure in the multimodal document; if there is no structure parsing file corresponding to the file type of the multimodal document in the configuration table of the structure parsing file, it is determined that there is no directory structure in the multimodal document.

[0071] You can check whether a document suffix exists directly through the document name, or use a specific command line to check the document's file suffix.

[0072] Among them, in this scheme, if the document does not have a file suffix, it can be considered that the document is not completely saved or damaged, and it is directly considered that there is no directory structure in the document; if there is no structure parsing file corresponding to the file type of the document in the configuration table of the structure parsing file, it can be directly considered that there is no directory structure in the document, or the document can be parsed using the default configuration method. For example, if there is no structure parsing file corresponding to the file type of the document in the configuration table of the structure parsing file, directly try to use the Python native file processing package to perform structural parsing on the document, and then further determine whether the document has a directory structure based on the parsing results.

[0073] It can be seen that in this example, when the document does not have a file suffix or the structure parsing file corresponding to the file type of the document does not exist in the configuration table of the structure parsing file, it is directly determined that the document does not have a directory structure, and the subsequent directory generation operation for the document is directly performed, which can simplify the processing flow and improve data processing efficiency.

[0074] S202: Based on the text structure information of the multimodal document, it is detected that no directory structure exists in the multimodal document, and a first topic of each page in the multimodal document is obtained through semantic recognition.

[0075] Among them, semantic recognition refers to the technology that computer systems use to analyze, understand and process text data, extract semantic information from it, convert human language into a form that computers can understand, and then achieve tasks such as classification, understanding, reasoning and generation of text content.

[0076] In this solution, Large Vision-Language Models (LVLMs) can be used to perform semantic recognition on each page to obtain the first topic of each page. LVLMs are artificial intelligence models that integrate vision and language processing capabilities.

[0077] Specifically, see Figure 3 , Figure 3 This is a schematic diagram of the architecture of a large visual language model provided by an embodiment of the present application. Figure 3 As shown in the figure, the architecture of a large visual language model usually includes a visual processing module, a language processing module, a cross-modal interaction module and an output module. The visual processing module and the language processing module process the image and text data respectively, and input the obtained visual features and language features into the cross-modal interaction module for fusion processing. The fused features output by the cross-modal interaction module are then input into the output module, which performs task-related prediction and post-processing to obtain the final output result.

[0078] Among them, the visual processing module may include an image input layer, a feature extraction layer and a visual feature encoder. The image input layer is used to receive input image data and pass information such as the pixel value of the image to the feature extraction layer. The feature extraction layer performs a convolution operation on the image based on the convolutional layer of the Convolutional Neural Network (CNN), extracts low-level features such as the edge of the image, and then compresses and downsamples the features through the pooling layer to retain key features. The feature extraction layer can also divide the image into multiple patches based on the Vision Transformer (ViT), map each patch to a vector, and form a vector sequence. The visual feature encoder can further process the retained key features in the CNN-based model to generate more advanced semantic features. The visual feature encoder can also further deeply encode the vector sequence in the ViT-based model to output a representative visual feature representation.

[0079] The language processing module includes a text input layer, a word embedding layer, and a language encoder. The text input layer receives input text data, performs preliminary processing on information such as words or characters in the text, and passes it to the word embedding layer. The word embedding layer converts the words in the text into low-dimensional vector representations and introduces position embedding to specifically process the order information of words and capture the sequential characteristics of the text. The language encoder can be an encoder based on the Transformer architecture, consisting of multiple stacked Transformer layers. Each layer contains a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism allows each word to pay attention to other words in the text, capturing the long-term dependencies and semantic relationships of the text. The feedforward neural network further processes the output of the attention mechanism, enhances the feature representation, and outputs a language feature vector.

[0080] The cross-modal interaction module's interaction methods include attention mechanisms and fusion strategies. The attention mechanisms include self-attention and cross-attention. Self-attention captures dependencies within each modality within visual and linguistic features, respectively. Cross-attention enables mutual attention between visual and linguistic features, allowing visual features to focus on related linguistic information, and linguistic features to focus on corresponding visual content. For example, a word in the text is associated with a related area in the image, and objects in the image are also associated with the textual vocabulary that describes them. Fusion strategies include early fusion, late fusion, and hybrid fusion. Early fusion combines visual and linguistic features for processing at the early stages of feature extraction. Late fusion fuses the features obtained after the visual and linguistic modules have fully processed their respective data. Hybrid fusion fuses cross-modal features at multiple different processing stages.

[0081] The output module consists of a prediction head and a post-processing layer. The prediction head is designed according to the task requirements. The post-processing layer processes the output of the prediction head. For example, in a classification task, the output can be converted into a probability distribution using the Softmax function to represent the likelihood of belonging to each category.

[0082] In this solution, optical character recognition (OCR) can also be used to recognize and extract characters on each page.

[0083] S203: Based on the first topic of each page in the multimodal document, if it is detected that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, the adjacent pages are grouped into the same page group.

[0084] The similarity can be obtained directly by calculating the number or proportion of identical words or phrases in different first topics, or by first understanding the first topic through natural language processing technology, converting the first topic into a vocabulary vector, and calculating the vector similarity.

[0085] In one possible implementation, the similarity of different first topics can be measured by calculating the cosine similarity of different first topics. Specifically, irrelevant information (such as special characters, punctuation marks, etc.) in different first topics is first removed, and then the first topics are converted into vector representations using a bag-of-words model, word frequency-inverse document frequency, or a word embedding model. Finally, the cosine similarity between different vectors is directly calculated to obtain the similarity between different first topics.

[0086] Taking the bag-of-words model as an example, it first constructs a vocabulary containing all the different words that appear in the text. It then represents each text as a vector, where each dimension of the vector corresponds to a word in the vocabulary, and the value of the vector represents the number of times the word appears in the text. For example, if Text 1 reads "I love spring" and Text 2 reads "I love summer," the vocabulary constructed is [I, love, spring, day, summer]. In this case, the vector representation of Text 1 is [1, 1, 1, 1, 0], and the vector representation of Text 2 is [1, 1, 0, 1, 1].

[0087] The calculation formula of cosine similarity is as follows:

[0088]

[0089] in, is the vector obtained by transforming the first topic 1, is the vector obtained by transforming the first topic 2, Respectively and The modulus of the cosine similarity is between -1 and 1. The closer the value is to 1, the more similar the two vectors (i.e., the two first topics) are, and the closer the value is to -1, the less similar the two vectors (i.e., the two first topics) are.

[0090] S204: Obtain a second topic of each page group according to the first topic of the pages in each page group in the multimodal document.

[0091] Among them, a page group includes one or more pages, each page has its own first topic. At this time, the method for obtaining the second topic of the corresponding page group based on the first topics of all pages in each page group can be cluster analysis, keyword extraction, or topic model.

[0092] Specifically, cluster analysis converts the page's primary topic into a vector representation. Using a clustering algorithm (such as K-Means clustering), pages with similar primary topics are grouped into clusters, each corresponding to a secondary topic. For example, using TF-IDF to convert page text into vectors, the vectors are then clustered using the K-Means algorithm. Each cluster in the clustering results represents the secondary topic of a page group.

[0093] Specifically, the steps of generating the second topic of the page grouping through keyword extraction include: data preprocessing, removing special characters, stop words and other useless information in the first topic of the page through text cleaning, and then splitting the first topic into words, English text is separated by spaces, and Chinese text is processed using a word segmentation tool. Keyword extraction, using word frequency statistics or TD-IDF methods, extracts one or more high-frequency words from all first topics. Generate the second topic, and obtain the second topic of the page grouping by combining one or more extracted high-frequency words. For example, high-frequency words include "artificial intelligence", "transportation", and "application". The second topic generated at this time can be "The application of artificial intelligence in the field of transportation".

[0094] Among them, a topic model is a statistical model used to discover potential topic structures in data such as text, such as the Latent Dirichlet Allocation (LDA) model. Specifically, the LDA model is used to analyze the first topics of multiple pages. The LDA model can mine potential and meaningful second topics from these first topics. After obtaining the second topics, by calculating the probability distribution of each page's first topic on each second topic, the page is classified into the page group corresponding to the second topic with the highest probability, ultimately achieving a unique second topic corresponding to each page group.

[0095] S205 , obtaining a directory of each page group according to the second theme of each page group, and combining the directories of the multiple page groups to obtain a directory structure of the multimodal document.

[0096] After obtaining the second theme of each page group, the directory of the page group can be directly obtained from the second theme, or keywords can be extracted from the second theme again to obtain the directory of the page group.

[0097] Specifically, after obtaining the directory of each page group, the directories of different page groups can be arranged and combined according to the original arrangement order of each page group in the document (such as page numbers from small to large) to obtain the directory structure of the multimodal document. For example, the directories of multiple page groups are "Mobile Phone Size" (corresponding to document pages 1-6), "Mobile Phone Color" (corresponding to document pages 7-8), "Mobile Phone Introduction" (corresponding to document pages 9-10), and "Mobile Phone Material" (corresponding to document pages 11-12). At this time, the directory structure of the multimodal document is "Mobile Phone Size", "Mobile Phone Color", "Mobile Phone Introduction", and "Mobile Phone Material".

[0098] Alternatively, after obtaining the directory for each page group, further semantic analysis or keyword analysis can be performed on the directories of all page groups, and directories with similar semantics or high keyword similarity can be arranged and combined in sequence to form a directory structure for the multimodal document. For example, the directories of multiple page groups are "Mobile Phone Size", "Tablet Color", "Mobile Phone Material", and "Tablet Function". In this case, according to keyword analysis, "Mobile Phone Size" and "Mobile Phone Material" both include the keyword "Mobile Phone", and "Tablet Color" and "Tablet Function" both include the keyword "Tablet". Therefore, the directory structure of the multimodal document can be formed as "Mobile Phone Size", "Mobile Phone Material", "Tablet Color", and "Tablet Function".

[0099] Furthermore, after obtaining the directory for each page group, the directories of multiple page groups can be analyzed to determine whether there is a relationship of inclusion and being included. In this case, the directory that contains one or more other directories can be used as the upper directory, and the other one or more directories contained in this directory can be used as the lower directory, finally forming the directory structure of the multimodal document. For example, the directories of multiple page groups are "Mobile Phone Size", "Mobile Phone Color", "Mobile Phone Introduction", and "Mobile Phone Material". In this case, after semantic analysis, it is determined that the "Mobile Phone Introduction" directory can be used as the upper directory, including the lower directories "Mobile Phone Size", "Mobile Phone Color", and "Mobile Phone Material". The directory structure of the multimodal document formed at this time is the main directory of "Mobile Phone Introduction", including the subdirectories "Mobile Phone Size", "Mobile Phone Color", and "Mobile Phone Material".

[0100] In one possible implementation, the method further includes:

[0101] According to the content type and content layout of the page group, the content of the page group under each directory in the directory structure is split to obtain multiple module contents of the page group; if there is a first module content that does not match the page group among the multiple module contents of the page group, the page group that matches the first module content is determined; and the first module content is adjusted to the page group that matches the first module content.

[0102] Among them, the multimodal document has a four-layer architecture of document-directory-page-module. The multimodal document is structurally decomposed according to the directory structure to obtain multiple modules of the multimodal document page. The module content can be text content, image content, or graphic content, etc.

[0103] Among them, see Figure 4 , Figure 4 This is a schematic diagram of module content in a multimodal document provided by an embodiment of the present application, such as Figure 4 As shown, the left side is the directory in the four-layer structure of the document, including directory one to directory five, and the right side is the page in the four-layer structure of the document. The page includes 3 modules, from top to bottom, module one, module two, and module three. Among them, module one is image content, module two is text content, and module three is text content. Module one and module two are divided by different modes, and module two and module three are divided by different paragraphs.

[0104] To distinguish different modules on the same page, you can first divide the document by paragraphs, perform semantic recognition on the content of each paragraph, and group paragraphs with similar semantics into modules. Alternatively, you can divide the content on the page into modules based on different modalities, dividing image content into separate modules and text content into separate modules.

[0105] Among them, see Figure 5 , Figure 5 This is a schematic diagram of a data organization form of a four-layer architecture provided in an embodiment of the present application, such as Figure 5 As shown, a document includes multiple directories, each directory includes multiple pages, and each page can have one or more modules. For module 1 of page 1 under directory 1, module 1 has multiple attributes, including type, image address and content. If the module type is text, the image address is empty, and the content can be directly parsed through code; if the current module type is image, the address where the image is saved locally will be recorded, and the description of the current image by the large visual model will be recorded in the content.

[0106] Among them, after understanding the relationship between modules and pages, since the module content is parsed from the page, and each page is in a page group, it is possible to further determine whether the module content matches the page group to which the module content belongs. The judgment method can be to perform semantic recognition on the module content, extract key information and keywords, and then determine whether the module content matches the page group based on the second theme of the page group where the module content is located and the key information and keywords of the module content. For example, if the second theme of the page group is "Mobile Phone Introduction", and the keywords of the module content under the page group are "Car" and "Driving", it is considered that the module content does not match the page group.

[0107] Among them, after determining that the module content does not match the page group where the module content is located, the page group that best matches the module content can be further found from all page groups based on the similarity between the second topic of other page groups and the key information and keywords of the module content, and the module content can be adjusted to the corresponding page group.

[0108] Among them, when adjusting the module content to another page group, the module content can be directly placed at the end of the multiple module contents of the page group, or the similarity between the multiple module contents under the page group and the second theme of the page group can be analyzed in turn, and the multiple module contents can be arranged in order from large to small according to the similarity.

[0109] It can be seen that in this example, by structurally decomposing the multimodal document and adjusting the unmatched module content in the page grouping to another matching page grouping, the content of each part of the document can be accurately corresponded to the page group to which it belongs, thereby improving the accuracy of the subsequent directory generated based on the second topic of the page grouping.

[0110] In a possible implementation, if there is a first module content that does not match the page group among the plurality of module contents of the page group, determining a page group that matches the first module content includes:

[0111] The third topic of each module content is obtained through semantic recognition; the similarity between the third topic of each module content and the second topic of the page grouping in the multimodal document is calculated; if the maximum similarity among the similarities is not the similarity between the third topic of the first module content and the second topic of the page grouping of the first module content, it is determined that the first module content and the page grouping of the first module content do not match; in the case that the first module content and the page grouping of the first module content do not match, the page grouping corresponding to the maximum similarity is determined as the page grouping that matches the first module content.

[0112] Among them, after obtaining the third topic of the module content through semantic recognition, the matching degree of the module content and the page grouping can be measured by calculating the similarity between the third topic of the module content and the second topic of the page grouping in the multimodal document. Specifically, irrelevant information (such as special characters, punctuation marks, etc.) in the third topics of different module contents is first removed, and then the third topic of the module content is converted into a vector representation using a bag-of-words model, word frequency-inverse document frequency or word embedding model. Finally, the cosine similarity between different vectors and the vectors converted from the second topic of the page grouping in the multimodal document is directly calculated to obtain the similarity between the module content and the page grouping in the multimodal document.

[0113] Taking the bag-of-words model as an example, it first constructs a vocabulary containing all the different words that appear in the text. It then represents each text as a vector, where each dimension of the vector corresponds to a word in the vocabulary, and the value of the vector represents the number of times the word appears in the text. For example, if Text 1 reads "I love spring" and Text 2 reads "I love summer," the vocabulary constructed is [I, love, spring, day, summer]. In this case, the vector representation of Text 1 is [1, 1, 1, 1, 0], and the vector representation of Text 2 is [1, 1, 0, 1, 1].

[0114] The calculation formula of cosine similarity is as follows:

[0115]

[0116] in, is the vector obtained by transforming the third theme of module content 1, is the vector obtained by converting any page group in a multimodal document, Respectively and The modulus of the cosine similarity is between -1 and 1. The closer the value is to 1, the more similar the two vectors (i.e., module content and page grouping) are. The closer the value is to -1, the less similar the two vectors (i.e., module content and page grouping) are.

[0117] A similarity threshold can be set to determine whether the module content matches the page group based on the similarity between the third topic of the module content and the second topic of the page group and the similarity threshold. For example, if the similarity between the module content and the page group is 0.6 and the similarity threshold is 0.8, the module content is considered to not match the page group.

[0118] It can be seen that in this example, by using semantic recognition to calculate the similarity between the third topic of the module content and the second topic of the page group to which the module content belongs to judge whether the module content matches the page group. This can accurately locate the mismatched module content, facilitate subsequent adjustments to the mismatched modules, and avoid incorrect matching of module content and page grouping.

[0119] Among them, if the maximum similarity between the first module content and the page grouping in the multimodal document is not the similarity between the third topic of the first module content and the second topic of the page grouping of the first module content, it is determined that the first module content and the page grouping of the first module content do not match. At this time, the first module content is adjusted to the page grouping with the largest calculated similarity.

[0120] Among them, see Figure 6 , Figure 6 This is a schematic diagram of page grouping-module content matching adjustment provided by an embodiment of the present application, such as Figure 6 As shown, the document includes page group A and page group B. Initially, page group A includes module content 1, module content 2, and module content 3, and page group B includes module content 4 and module content 5. At this time, the similarity between each module content and page group A and page group B is calculated respectively. The calculation results are shown in Table 2 below:

[0121] Table 2 Page grouping-module similarity

[0122] Similarity with page group A Similarity with page group B Module Content 1 0.75 0.6 Module Content 2 0.85 0.3 Module Content 3 0.3 0.9 Module Content 4 0.2 0.9 Module Content 5 0.3 0.85

[0123] Among them, it can be seen from Table 2 that although module content 3 is now in page group A, the similarity between module content 3 and page group A is 0.3, and the similarity between module content 3 and page group B is 0.9. Since 0.3<0.9, module content 3 will be moved to page group B. At this time, page group A includes module content 1 and module content 2, and page group B includes module content 3, module content 4, and module content 5.

[0124] Among them, when it is determined that module content 3 needs to be adjusted to page group B, the similarity between module content 3 and other module contents under page group B can be calculated first, and module content 3 can be adjusted to the front or back of the module content with the highest similarity. For example, the similarity between module content 3 and module content 4 is 0.4, and the similarity with module content 5 is 0.8, then module content 3 is adjusted to the back of module content 5.

[0125] Among them, the matching degree between module contents can be measured by calculating the cosine similarity between modules. Specifically, irrelevant information (such as special characters, punctuation marks, etc.) in the third topics of different module contents is first removed, and then the third topics of the module contents are converted into vector representations using the bag-of-words model, word frequency-inverse document frequency, or word embedding model. Finally, the cosine similarity between different vectors is directly calculated to obtain the similarity of different module contents.

[0126] Taking the bag-of-words model as an example, it first constructs a vocabulary containing all the different words that appear in the text. It then represents each text as a vector, where each dimension of the vector corresponds to a word in the vocabulary, and the value of the vector represents the number of times the word appears in the text. For example, if Text 1 reads "I love spring" and Text 2 reads "I love summer," the vocabulary constructed is [I, love, spring, day, summer]. In this case, the vector representation of Text 1 is [1, 1, 1, 1, 0], and the vector representation of Text 2 is [1, 1, 0, 1, 1].

[0127] The calculation formula of cosine similarity is as follows:

[0128]

[0129] in, is the vector obtained by transforming the third theme of module content 1, is the vector obtained by converting the third theme of module content 2, Respectively and The modulus of the cosine similarity is between -1 and 1. The closer the value is to 1, the more similar the two vectors (i.e., module content 1 and module content 2) are. The closer the value is to -1, the less similar the two vectors (i.e., module content 1 and module content 2) are.

[0130] Alternatively, you can directly set the module content adjustment rules, that is, when the module content needs to be adjusted to another directory, the module content will be adjusted to the front, back or fixed position of the corresponding directory by default.

[0131] It can be seen that in this example, by calculating the similarity between the content of the first module and all page groups in the multimodal document and determining the page grouping that matches it, the most suitable page grouping can be accurately found for the module content, ensuring the matching degree between the page grouping and the module, and further improving the accuracy of the subsequent directory generated by the second topic based on the page grouping.

[0132] See also Figure 7 , Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application, such as Figure 7 As shown, the data processing device 700 includes:

[0133] An acquisition module 701 is configured to obtain text structure information of a multimodal document by performing structural analysis on the multimodal document;

[0134] Identification module 702 is configured to detect, based on text structure information of the multimodal document, that no directory structure exists in the multimodal document, and to obtain a first topic of each page in the multimodal document through semantic recognition; and to, based on the first topic of each page in the multimodal document, detect that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, and to group the adjacent pages into the same page group.

[0135] The acquisition module 701 is also used to obtain the second topic of each page group in the multimodal document based on the first topics of all pages in each page group; and to obtain the directory of each page group based on the second topic of each page group, and combine the directories of the multiple page groups to obtain the directory structure of the multimodal document.

[0136] In one possible implementation, in terms of obtaining the text structure information of the multimodal document by performing structural parsing on the multimodal document, the acquisition module 701 is specifically used to: determine the file type of the multimodal document based on the file suffix of the multimodal document; determine the first structural parsing file for the multimodal document based on the configuration table of the structural parsing file and the file type of the multimodal document; and parse the multimodal document according to the first structural parsing file to obtain the text structure information of the multimodal document.

[0137] In one possible implementation, a mapping relationship between the file type of a multimodal document and a parsed file is configured in a configuration table of the structure parsing file. In determining the first structure parsing file for the multimodal document based on the configuration table of the structure parsing file and the file type of the multimodal document, the acquisition module 701 is specifically used to: query the configuration table of the structure parsing file based on the file type of the multimodal document to obtain the first structure parsing file corresponding to the file type of the multimodal document.

[0138] In one possible implementation, the text structure information includes the format features and hierarchical relationships of the text. In parsing the multimodal document according to the first structure parsing file to obtain the text structure information of the multimodal document, the acquisition module 701 is specifically used to: traverse the multimodal document through the first structure parsing file to obtain the format features and hierarchical relationships of the text in the multimodal document.

[0139] In one possible implementation, the acquisition module 701 is further used to: split the content of the page group under each directory in the directory structure according to the content type and content layout of the page group to obtain multiple module contents of the page group; if there is a first module content that does not match the page group in the multiple module contents of the page group, determine the page group that matches the first module content; and adjust the first module content to the page group that matches the first module content.

[0140] In one possible implementation, if there is a first module content in the multiple module contents of the page grouping that does not match the page grouping, then determining the page grouping that matches the first module content, the acquisition module 701 is specifically used to: obtain the third topic of each module content through semantic recognition; calculate the similarity between the third topic of each module content and the second topic of the page grouping in the multimodal document; if the maximum similarity among the similarities is not the similarity between the third topic of the first module content and the second topic of the page grouping of the first module content, then determining that the first module content does not match the page grouping of the first module content; in the case that the first module content does not match the page grouping of the first module content, determining the page group corresponding to the maximum similarity as the page grouping that matches the first module content.

[0141] It is worth noting that the specific functional implementation of the data processing device 700 is shown in the above Figure 2 The description of the data processing method shown in FIG. 7 is as follows, for example, the acquisition module 701 is used to implement the relevant content of executing S201, S204, and S205, and the identification module 702 is used to implement the relevant content of executing S202-S203. Each unit or module in the data processing device 700 can be separately or completely merged into one or more other units or modules to form a structure, or one (or more) of the units or modules can be further split into multiple smaller units or modules to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided according to logical functions. In actual applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0142] According to the description of the above method embodiment and related device embodiment, please refer to Figure 8 , Figure 8 is a structural diagram of a data processing device provided in an embodiment of the present application, Figure 8The data processing device 800 shown includes a processor 801 , a memory 802 , a communication interface 803 , and a bus 804 . The processor 801 , the memory 802 , and the communication interface 803 are communicatively connected to each other via the bus 804 .

[0143] Optionally, the memory 802 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0144] The memory 802 can store executable program codes. When the executable program codes stored in the memory 802 are executed by the processor 801, the processor 801 and the communication interface 803 are used to execute the program codes. Figure 2 The various steps of the data processing method of the embodiment are shown.

[0145] The processor 801 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits to execute relevant programs to perform the data processing method of the embodiment of the method of the present application.

[0146] Processor 801 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the data processing method of the present application can be completed by hardware integrated logic circuits in processor 801 or by software instructions. Optionally, processor 801 is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The processor can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor is a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The optional software module is located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media well-known in the art. The storage medium is located in memory 802. Processor 801 reads information in memory 802 and, in conjunction with its hardware, completes the functions required to be performed by the modules included in a data processing device 700 of the embodiments of the present application, or executes the data processing method of the method embodiments of the present application.

[0147] The communication interface 803 uses, for example but not limited to, a transceiver or other transceiver-related device.

[0148] The bus 804 may include a path for transmitting information between various components of the data processing device 800 (eg, the memory 802 , the processor 801 , and the communication interface 803 ).

[0149] It should be noted that although Figure 8The data processing device 800 shown only shows a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the data processing device 800 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the data processing device 800 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the data processing device 800 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 8 All devices shown in .

[0150] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program for electronic data exchange. The computer program includes execution instructions, and the execution instructions are used to execute part or all of the steps of any one of the data processing methods described in the above-mentioned data processing method embodiments. The above-mentioned computer includes an electronic client device.

[0151] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to perform part or all of the steps of any data processing method recorded in the above method embodiments. The computer program product can be a software installation package.

[0152] It should be noted that for the sake of simplicity, the embodiments of any of the aforementioned data processing methods are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0153] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of a data processing method, a computer, and a storage medium of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, based on the idea of ​​a data processing method, a computer, and a storage medium of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The memory may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps. The fact that certain measures are recited in different dependent claims does not mean that these measures cannot be combined to produce good results.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned data processing methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0158] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in an embodiment of a data processing method of the present application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in the present application.

[0159] Obviously, those skilled in the art may make various modifications and variations to the data processing method, computer, and storage medium provided in this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining text structure information of the multimodal document by performing structural analysis on the multimodal document; Based on the text structure information of the multimodal document, it is detected that no directory structure exists in the multimodal document, and then a first topic of each page in the multimodal document is obtained through semantic recognition; Based on the first topic of each page in the multimodal document, detecting that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, then grouping the adjacent pages into the same page group; Obtaining a second topic of each page group according to the first topic of the pages in each page group in the multimodal document; A directory of each page group is obtained according to the second theme of each page group, and the directories of multiple page groups are combined to obtain a directory structure of the multimodal document.

2. The method according to claim 1, wherein The step of performing structural analysis on the multimodal document to obtain text structure information of the multimodal document includes: Determining the file type of the multimodal document according to the file suffix of the multimodal document; Determining a first structure parsing file for the multimodal document according to a configuration table of the structure parsing file and a file type of the multimodal document; The multimodal document is parsed according to the first structure parsing file to obtain text structure information of the multimodal document.

3. The method according to claim 2, wherein The configuration table of the structure parsing file is configured with a mapping relationship between the file type of the multimodal document and the parsing file, and determining the first structure parsing file for the multimodal document according to the configuration table of the structure parsing file and the file type of the multimodal document includes: According to the file type of the multimodal document, a query is performed from the configuration table of the structure parsing file to obtain a first structure parsing file corresponding to the file type of the multimodal document.

4. The method according to claim 2 or 3, wherein: The text structure information includes format features and hierarchical relationships of the text. The parsing of the multimodal document according to the first structure parsing file to obtain the text structure information of the multimodal document includes: The multimodal document is traversed through the first structural parsing file to obtain format features and hierarchical relationships of text in the multimodal document.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Splitting the content of the page group under each directory in the directory structure according to the content type and content layout of the page group to obtain multiple module contents of the page group; If there is a first module content that does not match the page group among the multiple module contents of the page group, determining a page group that matches the first module content; The first module content is adjusted to a page group that matches the first module content.

6. The method according to claim 5, wherein If there is a first module content that does not match the page group among the plurality of module contents of the page group, determining a page group that matches the first module content includes: The third theme of each module content is obtained through semantic identification; Calculating the similarity between the third topic of each module content and the second topic of the page group in the multimodal document; If the maximum similarity among the similarities is not the similarity between the third theme of the first module content and the second theme of the page group of the first module content, it is determined that the first module content and the page group of the first module content do not match; In the case that the first module content does not match the page group of the first module content, the page group corresponding to the maximum similarity is determined as the page group matching the first module content.

7. A data processing device, characterized in that: The device comprises: An acquisition module, configured to obtain text structure information of a multimodal document by performing structural analysis on the multimodal document; an identification module configured to detect, based on text structure information of the multimodal document, that no directory structure exists in the multimodal document, and to obtain a first topic of each page in the multimodal document through semantic recognition; and to, based on the first topic of each page in the multimodal document, detect that the similarity of the first topics of adjacent pages in the multimodal document is greater than a similarity threshold, and to group the adjacent pages into the same page group; The acquisition module is also used to obtain the second topic of each page group in the multimodal document based on the first topics of all pages in each page group; and to obtain the directory of each page group based on the second topic of each page group, and combine the directories of multiple page groups to obtain the directory structure of the multimodal document.

8. A data processing device, characterized in that: The device comprises: A memory, a processor, and an executable program code stored in the memory and capable of running on the processor, wherein the processor executes the steps of the data processing method according to any one of claims 1 to 6 when executing the executable program code.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable program code, which includes execution instructions for executing the steps of the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and the computer program is used to enable a computer to execute the steps of the data processing method according to any one of claims 1 to 6.

Images