Information filtering method and device for multi-type documents
Through the analysis of OOXML and ofd format documents and the establishment of shared thesaurus, combined with improved OCR processing and prefix tree matching detection, the problems of slow response speed and recognition accuracy of multi-type document information filtering systems are solved, and automated Document filtering and identification improves work efficiency and information security.
Patent Information
- Application Number
- CN202411577808.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art responds slowly when processing multi-type documents, and the information filtering system is difficult to meet user needs. At the same time, there are problems of missing and pseudo-identification of identifying key information, especially in the processing of image information and text information.
Use xml parsing and binary parsing to parse OOXML and ofd format documents, establish a shared thesaurus, and use improved OCR processing and failed prefix tree construction compass to perform character matching detection, organize the position information and context information of the string, and output it to the front end .
It improves the response speed and identification accuracy of the document information filtering system, can automatically identify and filter documents, reduce manual intervention, reduce the risk of human error, meet the diversified management needs of enterprises, and protect the core confidentiality and privacy information of enterprises.
Smart Images

Figure CN119066188A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses an information filtering method and device for multi-type documents, and relates to the technical field of intelligent production. Background Art
[0002] Trie tree is also called prefix tree, dictionary tree, word search tree, a variant of hash tree, usually used for string matching, to solve the problem of quickly finding a string in a set of strings. Its advantages are: using the common prefix of the string to reduce the query time, minimize unnecessary string comparisons, and higher query efficiency than hash tree.
[0003] Aho-Corasick (AC) automaton is an improved algorithm based on Trie tree. It is a multi-pattern matching algorithm invented by Alfred V. Aho and Margaret J. Corasick, researchers at Bell Labs. AC automaton algorithm uses Trie tree to store the prefix of pattern string and uses failed match pointer (mismatch pointer) to handle the jump of failed match.
[0004] OCR (Optical Character Recognition) refers to the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text.
[0005] Redis (Remote Dictionary Server), also known as remote dictionary service, is an open source log-type, Key-Value database written in ANSI C language, supporting the network, memory-based and persistent, and providing APIs in multiple languages.
[0006] Office Open XML, also known as OpenXML or OOXML, is an XML-based format for office documents, including word processing documents, spreadsheets, presentations, as well as charts, graphs, shapes, and other graphical material. The specification was developed by Microsoft and adopted by ECMA International as ECMA-376 in 2006. The second version was released in December 2008, and the third version of the standard was released in June 2011. The specification has been adopted by ISO and IEC as ISO / IEC 29500.
[0007] The existing technology has a slow response speed. The information filtering system needs to process a huge amount of data. It is very difficult to maintain the system's response speed while ensuring that user needs are met. This system optimizes the retrieval algorithm, performs multi-process processing on image information and text information, and uses real-time display to solve the problem of user lag.
[0008] Some image information or text may be confused or replaced with synonyms, which may cause missed recognition and false recognition of key information. This system optimizes the phrase and phrase segmentation algorithm, and adds vertical and tilted image recognition to make the content recognition more accurate. Summary of the invention
[0009] In view of the problems of the prior art, the present invention provides an information filtering method and device for multi-type documents, and the technical solution adopted is: In a first aspect, a method for filtering information of multiple types of documents is provided, the method comprising: S1, according to the document structure of the streaming document and the layout document, parse the OOXML format document and the ofd format document respectively through XML parsing and binary parsing to obtain the document to be identified; S2, establishing a shared word segmentation library according to the document to be identified, by saving the text information in the document to be identified; S3, according to the image information in the document to be identified, save it into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; S4, building a corresponding prefix tree according to the table of filtered phrases to be detected as needed; S5, constructing a failure compass to perform character matching detection according to the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
[0010] In some implementations, the S2 specifically includes: S21, saving text position information and context information according to the text information of the document to be recognized; S22, performing word segmentation processing by a maximum matching algorithm according to the stored information to obtain a word sequence; S23, storing the word sequence in a shared word segmentation library and establishing a first-word index.
[0011] In some implementations, the S5 specifically includes: S51, constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; S52, according to the node access position of the failed compass, determine the longest suffix node by following the failed compass of its parent node, and check whether there is a character matching the current node in the child nodes of the longest suffix node; S53: If there is no match, search for the next longest suffix by continuing to follow the subsequent pointer until the current node is matched and the root node is reached.
[0012] In some implementations, the S5 further includes: S54, performing corresponding state transition through each character in the input text according to the root node; S55, when the current node has a child node corresponding to the character, converting the failure compass downward; S56, when the current node has no child node corresponding to the character and has a dictionary suffix, converting the failure compass along the dictionary suffix.
[0013] In a second aspect, an embodiment of the present invention provides an information filtering device for multi-type documents, the device comprising: The type parsing module is used to parse the OOXML format document and the OFD format document respectively through XML parsing and binary parsing according to the document structure of the streaming document and the layout document to obtain the document to be identified; A document creation module is used to create a shared word segmentation library according to the document to be identified by saving the text information in the document to be identified; A picture saving module is used to save the picture information in the document to be identified into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; A prefix tree module, used to detect the table of filtered phrases as needed and establish a corresponding prefix tree; A matching detection module, used to construct a failure compass to perform character matching detection based on the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
[0014] In some implementations, the document creation module specifically includes: An information processing unit, used to save text position information and context information according to the text information of the document to be recognized; A word segmentation processing unit, used to perform word segmentation processing according to the stored information through a maximum matching algorithm to obtain a word sequence; The index processing unit is used to store the word sequence in a shared word segmentation library and establish a first word index.
[0015] In some implementations, the matching detection module specifically includes: A compass construction unit, used for constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; A node matching unit, for determining a longest suffix node by following the failed compass of its parent node according to the node access position of the failed compass shown, and checking whether there is a character matching the current node in the child nodes of the longest suffix node; The suffix matching unit is used to search for the next longest suffix by continuing to follow the subsequent pointer if there is no match until the current node is matched and the root node is reached.
[0016] In some implementations, the matching detection module further includes: A character conversion unit, used for performing corresponding state conversion through each character in the input text according to the root node; A down conversion unit, used for down converting the failure compass when the current node has a child node corresponding to the character; A backward conversion unit is used to convert the failure compass along the dictionary suffix when the current node has no child node corresponding to the character and has a dictionary suffix.
[0017] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the method described in the first aspect above is implemented.
[0018] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the first aspect.
[0019] One or more embodiments of the present invention can bring at least the following beneficial effects: This article provides an information filtering method for multiple types of documents. It can identify and search for search terms of interest in batch-entered documents, and can also filter out files containing set vocabulary. By identifying and filtering specific types of documents, such as confidential files, sensitive data, etc., it can effectively prevent these files from being illegally accessed or leaked, and protect the company's core confidentiality and privacy information.
[0020] The method of the present invention can automatically identify and filter documents, reduce manual intervention, and improve work efficiency. At the same time, the automated processing method can also reduce the risk of human error. By setting different filtering rules and strategies, documents of different types and levels can be managed in a refined manner to meet the diverse management needs of the enterprise. By filtering out the information required by the user, the file interference and burden of the user can be reduced, and the location of the required information can be quickly located. At the same time, it can be connected to systems such as network disks and cloud disks to quickly filter the information of network disks and cloud disks and obtain the required files in the network disks. The multi-type document information recognition and filtering system has significant beneficial effects in terms of enterprise information security, work efficiency, resource utilization, compliance, and user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1 It is an implementation flow chart of an information filtering method for multi-type documents provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention. Embodiment 1:
[0024] Figure 1 A flowchart of an implementation method for filtering information of multiple types of documents is shown, as Figure 1 The implementation flow chart of the method shown in the figure, the information filtering method for multi-type documents provided in this embodiment includes: S1, according to the document structure of the streaming document and the layout document, parse the OOXML format document and the ofd format document respectively through XML parsing and binary parsing to obtain the document to be identified; S2, establishing a shared word segmentation library according to the document to be identified, by saving the text information in the document to be identified; S3, according to the image information in the document to be identified, save it into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; S4, building a corresponding prefix tree according to the table of filtered phrases to be detected as needed; S5, constructing a failure compass to perform character matching detection according to the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
[0025] First, according to S1, in view of the different document structures of streaming documents and layout documents, different file formats are decoded using common decoding methods, such as XML parsing for OOXML format documents, binary parsing for OFD documents, and document standardization parsing for OFD documents.
[0026] The next step, according to S2, includes: S21, saving text position information and context information according to the text information of the document to be recognized; S22, performing word segmentation processing by a maximum matching algorithm according to the stored information to obtain a word sequence; S23, storing the word sequence in a shared word segmentation library and establishing a first-word index.
[0027] Create a shared word segmentation database based on the document to be recognized, and store the text information in the document. Record the location and context information of the text. Use the maximum matching algorithm to perform word segmentation processing, obtain the word sequence, store it in the word segmentation shared data table, and create an index based on the first word; Next, according to S3, the improved OCR processing is performed on the images in the document to identify the text information in the images, record the image address information of the text information, store it in the shared word segmentation library, and mark the word segmentation source address as coming from the image; Next, according to S4, a table of phrases to be detected and filtered is established, and a prefix tree is established based on the table of phrases to be detected. The data structure of the prefix tree is constructed to store the information of the predecessor and successor nodes, the frequency of occurrence of the characters, etc.
[0028] The next step, according to S5, specifically includes: S51, constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; S52, according to the node access position of the failed compass, determine the longest suffix node by following the failed compass of its parent node, and check whether there is a character matching the current node in the child nodes of the longest suffix node; S53, if there is no match, searching for the next longest suffix by continuing to follow the subsequent pointer until the current node is matched until the root node is reached; S54, performing corresponding state transition through each character in the input text according to the root node; S55, when the current node has a child node corresponding to the character, converting the failure compass downward; S56, when the current node has no child node corresponding to the character and has a dictionary suffix, converting the failure compass along the dictionary suffix.
[0029] Add a failure pointer. Build a failure pointer for the node pointing to the longest possible strict suffix. When you visit a node, you can find its longest suffix node by following the failure pointer of its parent node and check whether there is a character matching the current node in the child node of the suffix node. If there is no match, continue to follow the suffix pointer to find the next longest suffix until a match is found or the root node is reached; Add dictionary suffixes. Each node can be indirectly traversed through the failure pointer, and a dictionary suffix is established to facilitate quick search; Text matching is performed through AC automata. During the parsing and storage process, a search subthread is synchronously established to improve the processing speed. Starting from the root node, the state transition is performed for each character in the input text. If the current node has a child node corresponding to the character, the transition is performed downward. If there is no corresponding child node, the transition is performed along the failure pointer. If there is a dictionary suffix, the transition is performed along the dictionary suffix. Finally, according to S6, the position information and context information of the successfully matched string are sorted out, and a list is created and output to the front end. Embodiment 2:
[0030] In a second aspect, an embodiment of the present invention provides an information filtering device for multi-type documents, the device comprising: The type parsing module is used to parse the OOXML format document and the OFD format document respectively through XML parsing and binary parsing according to the document structure of the streaming document and the layout document to obtain the document to be identified; A document creation module is used to create a shared word segmentation library according to the document to be identified by saving the text information in the document to be identified; A picture saving module is used to save the picture information in the document to be identified into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; A prefix tree module, used to detect the table of filtered phrases as needed and establish a corresponding prefix tree; A matching detection module, used to construct a failure compass to perform character matching detection based on the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
[0031] Furthermore, the document creation module specifically includes: An information processing unit, used to save text position information and context information according to the text information of the document to be recognized; A word segmentation processing unit, used to perform word segmentation processing according to the stored information through a maximum matching algorithm to obtain a word sequence; The index processing unit is used to store the word sequence in a shared word segmentation library and establish a first word index.
[0032] Furthermore, the matching detection module specifically includes: A compass construction unit, used for constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; A node matching unit, for determining a longest suffix node by following the failed compass of its parent node according to the node access position of the failed compass shown, and checking whether there is a character matching the current node in the child nodes of the longest suffix node; The suffix matching unit is used to search for the next longest suffix by continuing to follow the subsequent pointer if there is no match until the current node is matched and the root node is reached.
[0033] Furthermore, the matching detection module further includes: A character conversion unit, used for performing corresponding state conversion through each character in the input text according to the root node; A down conversion unit, used for down converting the failure compass when the current node has a child node corresponding to the character; A backward conversion unit is used to convert the failure compass along the dictionary suffix when the current node has no child node corresponding to the character and has a dictionary suffix. Embodiment three:
[0034] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions implement the method of embodiment 1 when executed by the processor; In practical applications, the processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller unit (MCU), a microprocessor or other electronic components to execute the methods in the above embodiments.
[0035] The method implemented in this embodiment is as shown in the content of the first embodiment. Embodiment 4:
[0036] This embodiment further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method of the first embodiment is implemented; Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0037] The method implemented in this embodiment is as shown in the content of the first embodiment.
[0038] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed system and method can also be implemented in other ways. The system and method embodiments described above are merely illustrative.
[0039] It should be noted that, in this article, the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0040] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.
Claims
1. A method for filtering information of multiple types of documents, characterized in that: The method comprises: S1, according to the document structure of the streaming document and the layout document, parse the OOXML format document and the ofd format document respectively through XML parsing and binary parsing to obtain the document to be identified; S2, establishing a shared word segmentation library according to the document to be identified, by saving the text information in the document to be identified; S3, according to the image information in the document to be identified, save it into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; S4, building a corresponding prefix tree according to the table of filtered phrases to be detected as needed; S5, constructing a failure compass to perform character matching detection according to the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
2. The method according to claim 1, characterized in that The S2 specifically includes: S21, saving text position information and context information according to the text information of the document to be recognized; S22, performing word segmentation processing by a maximum matching algorithm according to the stored information to obtain a word sequence; S23, storing the word sequence in a shared word segmentation library and establishing a first-word index.
3. The method according to claim 2, characterized in that The S5 specifically includes: S51, constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; S52, according to the node access position of the failed compass, determine the longest suffix node by following the failed compass of its parent node, and check whether there is a character matching the current node in the child nodes of the longest suffix node; S53: If there is no match, search for the next longest suffix by continuing to follow the subsequent pointer until the current node is matched and the root node is reached.
4. The method according to claim 3, characterized in that The S5 further includes: S54, performing corresponding state transition through each character in the input text according to the root node; S55, when the current node has a child node corresponding to the character, converting the failure compass downward; S56, when the current node has no child node corresponding to the character and has a dictionary suffix, converting the failure compass along the dictionary suffix.
5. An information filtering device for multi-type documents, characterized in that: The device comprises: The type parsing module is used to parse the OOXML format document and the OFD format document respectively through XML parsing and binary parsing according to the document structure of the streaming document and the layout document to obtain the document to be identified; A document creation module is used to create a shared word segmentation library according to the document to be identified by saving the text information in the document to be identified; A picture saving module is used to save the picture information in the document to be identified into the shared word segmentation library through improved OCR processing, and mark the source address as a picture; A prefix tree module, used to detect the table of filtered phrases as needed and establish a corresponding prefix tree; A matching detection module, used to construct a failure compass to perform character matching detection based on the prefix tree; S6, according to the node traversed by the failed compass, by establishing a dictionary suffix for the node; S7, according to the character string for which the character matching is successfully detected, by sorting out the position information and context information of the character string, establishing a corresponding list and outputting it to the front end.
6. The device according to claim 5, characterized in that The document creation module specifically includes: An information processing unit, used to save text position information and context information according to the text information of the document to be recognized; A word segmentation processing unit, used to perform word segmentation processing according to the stored information through a maximum matching algorithm to obtain a word sequence; The index processing unit is used to store the word sequence in a shared word segmentation library and establish a first word index.
7. The device according to claim 6, characterized in that The matching detection module specifically includes: A compass construction unit, used for constructing a failure compass pointing to the longest suffix node according to the nodes of the prefix tree; A node matching unit, for determining a longest suffix node by following the failed compass of its parent node according to the node access position of the failed compass shown, and checking whether there is a character matching the current node in the child nodes of the longest suffix node; The suffix matching unit is used to search for the next longest suffix by continuing to follow the subsequent pointer if there is no match until the current node is matched and the root node is reached.
8. The device according to claim 7, characterized in that The matching detection module further includes: A character conversion unit, used for performing corresponding state conversion through each character in the input text according to the root node; A down conversion unit, used for down converting the failure compass when the current node has a child node corresponding to the character; A backward conversion unit is used to convert the failure compass along the dictionary suffix when the current node has no child node corresponding to the character and has a dictionary suffix.
9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method as described in any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method as described in any one of claims 1 to 4 above.
Citation Information
Patent Citations
Method and device for rapidly extracting organization name in text
CN114676214A
Text sensitivity detection method and device, electronic equipment and readable storage medium
CN117313695A
Method and system for identifying variant words in text and extracting sensitive words
CN117725161A
Encrypted query method, system and equipment based on data desensitization sensitive word and medium
CN118332571A
