Text data extraction method, system and equipment based on multi-modal fusion and self-evolution learning and medium

Abstract

The invention relates to the technical field of text data processing, and discloses a text data extraction method, system, equipment and medium based on multi-modal fusion and self-evolution learning, which comprises the following steps of: performing feature extraction and spatial alignment on a printed text, a handwritten annotation and a dynamic table of a mixed format document to obtain a semantic feature of an image-text table, and inputting the semantic feature into a dynamic analysis layer; analyzing metaphor expressions and synonymous heterogeneous fields through field extraction and a context semantic reasoning mechanism, and outputting structured data; performing grammar compliance verification by adopting a regularization engine, and performing comparison verification through a federal learning mechanism; and inputting the verified data into the reinforcement learning model, updating the analysis rule and the model parameters through strategy iteration, and feeding back the updated analysis rule and model parameters to the dynamic analysis layer to complete closed-loop optimization. According to the method, the processing precision and efficiency of the complex document are greatly improved, the manual intervention requirement is remarkably reduced, and meanwhile, the privacy protection and compliance requirements are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text data processing, and in particular to a text data extraction method, system, device and medium based on multi-modal fusion and self-evolution learning. BACKGROUND

[0002] How to accurately and efficiently extract structured data from complex and diverse documents is directly related to the optimization of business processes and decision-making quality of enterprises. However, the document data in real business scenarios often presents the characteristics of multi-modal mixing, various expression methods, and complex and variable formats, which makes the traditional text processing technology face serious challenges in actual application.

[0003] The existing technical system shows obvious limitations in dealing with these complex requirements. The traditional text recognition method based on a single modality cannot effectively process composite documents containing printed, handwritten and table data, and the fragmentation between different modal information leads to poor overall recognition effect. In terms of semantic understanding, the conventional natural language processing method is limited by fixed dictionaries and rule bases, and has insufficient ability to analyze professional terms, metaphorical expressions and synonymous heterogeneous expressions, making it difficult to accurately capture the deep semantics of the text. More notably, the existing technical solutions generally lack adaptive ability, and when faced with new document templates or special business scenarios, a large amount of manual work is required for rule adjustment and model optimization, not only increasing the implementation cost, but also seriously affecting the response speed of the system. In terms of data security and collaborative processing, the traditional centralized architecture cannot meet the increasingly stringent data privacy protection requirements, and it is also difficult to achieve efficient data comparison and consistency maintenance across systems.

[0004] The existence of the above-mentioned existing technical bottlenecks makes enterprises have to rely on a large amount of manual intervention when processing massive and diverse text data, which seriously restricts the automation level and processing efficiency of business processes. Therefore, developing an intelligent text data extraction method that can integrate multi-modal information, have semantic depth understanding ability, support adaptive optimization and meet the requirements of privacy protection has become an urgent need for the development of current text processing technology. SUMMARY

[0005] In view of the above-mentioned existing problems, the present application is proposed.

[0006] Therefore, the present application provides a text data extraction method, system, device and medium based on multi-modal fusion and self-evolution learning, which solves the problem that the existing text recognition technology cannot effectively process multi-modal mixed documents and understand complex semantic expressions, and lacks adaptive optimization ability, resulting in low recognition accuracy and processing efficiency in complex scenarios.

[0007] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a text data extraction method based on multi-modal fusion and self-evolution learning, comprising: receiving a mixed format document, performing feature extraction and spatial alignment on printed text, handwritten annotations and dynamic tables in the mixed format document to obtain graphic table semantic features; inputting the graphic table semantic features into a dynamic analysis layer, analyzing metaphorical expressions and synonymous heterogeneous fields through field extraction and context semantic reasoning mechanism, and outputting structured data; performing syntax compliance verification on the structured data by using a regular engine, performing comparison verification on the structured data by using a federal learning mechanism, and inputting the verified data into a reinforcement learning model to update the analysis rules and model parameters through policy iteration; feedback the updated analysis rules and model parameters to the dynamic analysis layer to complete the closed-loop optimization.

[0008] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, wherein the feature extraction step comprises: performing character contour feature matching and context semantic completion on the printed text in the mixed format document by using an optical character recognition engine to obtain text semantic features; extracting spatial features of the handwritten region image in the mixed format document by using a convolutional neural network, inputting the spatial features into a long short-term memory network after flattening them into a sequence form for learning and recognition to obtain image semantic features; the loss function of the long short-term memory network is a sequence labeling loss function, and the sequence labeling loss function allows learning character sequences without accurate alignment labels; performing target detection algorithm positioning on the table frame and cells in the dynamic table in the mixed format document, and outputting a bounding box, and preliminarily establishing a table row and column structure by performing spatial clustering and sorting on all cells in the bounding box; inputting each cell region into the optical character recognition engine to extract text features, inputting the text features and position information as cell features into a self-attention mechanism learning model to calculate the dependency relationship between cells, and obtaining structure-aware table semantic features.

[0009] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, wherein the spatial alignment step comprises: normalizing the original coordinates of the printed text, handwritten annotations and dynamic tables in the mixed format document to a unified image space; calculating the overlap degree of the normalized image space and the relative distance of the position information; According to the relative distance of the image space overlap degree and position information, a corresponding relationship is established between the text box, image area and handwriting track point to realize spatial alignment.

[0010] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, the output structured data includes: Based on the dynamically generated neural network structure, the context-aware field boundary identification is performed on the input graphic-text table semantic features to extract the key fields in the contract. The pre-constructed domain knowledge graph is introduced, and the extracted key fields are converted into standardized field names through a synonym and isomeric field mapping library, and the mapping relationship can be represented as: Among them, represents the unified standard field name, represents a graph-driven semantic standardization function, and S represents an original field or phrase. The unstructured text is input into a pre-trained language model for context encoding to generate feature representations containing semantic and position information, and a classifier is used to predict the entity type of each encoded position. The standardized field name and the entity prediction result are integrated into structured data in the form of key-value pairs, and output to a federal verification layer.

[0011] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, the syntax compliance verification of the structured data by the regular engine includes: The syntax compliance verification of the key fields of the structured data is performed by the regular engine, and if the field format does not meet the requirements, it is not compliant. The compliant verification data is compared with the local business data to determine whether there are identical data records through hash indexing, and a data set with correct format and no duplication is output.

[0012] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, the comparison and verification of the structured data by the federal learning mechanism includes: For data that cannot be verified locally, the federal learning mechanism is introduced, and the feature parameters are encrypted by an encryption algorithm and sent to a central server for distributed data comparison with a human resource management system, an enterprise resource planning system and a blockchain storage system, and the comparison result is output. The central server identifies whether it is the same business record according to the feature similarity of the comparison result The formula is represented as: wherein, and respectively represent the feature vectors extracted before the encryption of two system data. If the feature similarity of the comparison result is greater than a first set threshold, it is indicated that the two data items represent the same business object.

[0013] As a preferred scheme of the text data extraction method based on multi-modal fusion and self-evolution learning, wherein: the step of updating the parsing rule and the model parameter in the reinforcement learning model through policy iteration after the verified data is inputted comprises: constructing a reinforcement learning model for each field extraction task, discretizing the model parameter space into a state set, and each state corresponding to a different rule combination; taking the field extraction strategy as the objective function, taking the extraction accuracy as the reward signal, updating the Q value table by using the Bellman equation, and the formula is represented as: wherein, represents the objective function of taking action s in state a , represents a learning rate, and controls the update step; r represents the reward value obtained by the agent after executing the action a , and γ represents a discount factor, and is used to balance the immediate reward and the long-term effect; represents a new state after executing the action, represents the action taken in the new state, represents the objective function of taking action in state ; when the verification set accuracy is improved, the current optimal parameter combination is stored in the experience pool, the optimal extraction strategy is gradually converged through dynamic programming, the updated field parsing rule set and the adjusted model parameter are outputted, and the self-evolution iteration is completed.

[0014] In a second aspect, the application provides a text data extraction system based on multi-modal fusion and self-evolution learning, comprising: a multi-modal perception module, configured to receive a mixed format document, perform feature extraction and spatial alignment on printed text, handwritten notes and dynamic tables in the mixed format document, and obtain graphic table semantic features; a dynamic parsing module, configured to input the graphic table semantic features into a dynamic parsing layer, parse metaphorical expressions and synonymous heterogeneous fields through field extraction and context semantic reasoning mechanism, and output structured data; a federal verification module configured to perform syntax compliance verification on the structured data using a regular engine and comparison verification on the structured data through a federated learning mechanism; a self-evolution module configured to input the verified data into a reinforcement learning model to update the parsing rules and model parameters through policy iteration, and feed the updated parsing rules and model parameters back to the dynamic parsing layer to complete closed-loop optimization.

[0015] In a third aspect, the present application provides an electronic device comprising a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to implement the steps of the text data extraction method based on multi-modal fusion and self-evolution learning when executing the computer executable instructions.

[0016] In a fourth aspect, the present application provides a computer readable storage medium storing computer executable instructions, which implement the steps of the text data extraction method based on multi-modal fusion and self-evolution learning when executed by a processor.

[0017] Compared with the prior art, the beneficial effects of the present application are as follows: the text data extraction method, system, device and medium based on multi-modal fusion and self-evolution learning of the present application can significantly improve the intelligent level of complex document processing by constructing an intelligent text recognition system based on multi-modal fusion and self-evolution learning. The four-dimensional closed-loop architecture of "quantum-level perception-neural analysis-federal verification-evolution optimization" is adopted to realize the whole-process intelligent processing from raw document collection to dynamic optimization. The multi-modal perception layer breaks through the feature extraction capability of printed matter, handwriting and table data, and ensures the complete capture of various types of information through spatial alignment and cross-modal collaborative mechanism; the dynamic analysis layer effectively solves the recognition problems of metaphorical expression and synonymous heterogeneous fields based on deep semantic understanding and knowledge graph technology; the federal verification layer realizes real-time data collaboration across systems while protecting data privacy; the self-evolution layer continuously optimizes the model parameters through reinforcement learning, enabling the system to adapt to new scenarios autonomously. The technical solution provided by the present application not only greatly improves the processing accuracy and efficiency of complex documents, but also significantly reduces the need for manual intervention, while meeting the requirements of privacy protection and compliance, providing strong technical support for the digital transformation of enterprises. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1A whole flow logical schematic diagram of a text data extraction method based on multi-modal fusion and self-evolution learning is provided for an embodiment of the present application.

[0020] Figure 2 A flow architecture schematic diagram of the text data extraction method based on multi-modal fusion and self-evolution learning is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0022] Embodiment 1, refer to Figures 1-2 For an embodiment of the present application, a text data extraction method based on multi-modal fusion and self-evolution learning is provided, as shown in Figure 1 specifically comprising the following steps: S100: receiving a mixed format document, extracting and spatially aligning features of printed text, handwritten annotations and dynamic tables in the mixed format document to obtain graphic table semantic features; S200: inputting the graphic table semantic features into a dynamic analysis layer, analyzing metaphorical expressions and synonymous heterogeneous fields through field extraction and context semantic reasoning mechanism, and outputting structured data; S300: using a regular engine to perform syntax compliance verification on the structured data, and performing comparison and verification on the structured data through a federated learning mechanism; S400: inputting the verified data into a reinforcement learning model to update the analysis rules and model parameters through policy iteration, and feeding back the updated analysis rules and model parameters to the dynamic analysis layer to complete closed-loop optimization.

[0023] It should be noted that, in order to solve the problems that the existing text recognition technology cannot effectively process multi-modal mixed documents, understand complex semantic expressions, and lacks self-adaptive optimization capability, resulting in low recognition accuracy and processing efficiency in complex scenarios, the above steps S100-S400 construct an intelligent text recognition system based on multi-modal fusion and self-evolution learning, which can significantly improve the intelligent level of complex document processing. As shown in Figure 2As shown, the application adopts a four-dimensional closed-loop architecture of "quantum-level perception-neural analysis-federal verification-evolution optimization", realizes the whole-process intelligent processing from original document collection to dynamic optimization. The multi-modal perception layer breaks through the feature extraction capability of printed matter, handwriting and table data, and ensures the complete capture of various types of information through spatial alignment and cross-modal coordination mechanism; the dynamic analysis layer effectively solves the identification problems of metaphorical expression and synonymous heterogeneous fields based on deep semantic understanding and knowledge graph technology; the federal verification layer realizes real-time data collaboration across systems on the premise of protecting data privacy; the self-evolution layer continuously optimizes model parameters through reinforcement learning, enabling the system to adapt to new scenarios independently. The technical scheme provided by the application not only greatly improves the processing accuracy and efficiency of complex documents, but also significantly reduces the need for manual intervention, while meeting the requirements of privacy protection and compliance, providing strong technical support for the digital transformation of enterprises.

[0024] It should be noted that, as Figure 2 As shown, the multi-modal perception layer receives mixed format original documents, accurately extracts structured data in mixed format documents, fuses multi-dimensional information of scanned text, handwritten annotations and dynamic tables, realizes efficient collection of full-format data, and outputs coordinate-aligned graphic table semantic units to the dynamic analysis layer; the dynamic analysis layer generates a standardized field structure based on semantic units, analyzes metaphorical expressions and synonymous heterogeneous fields, realizes automatic extraction of fields with context awareness, and inputs the analysis results to the federal verification layer; the federal verification layer performs compliance verification and cross-system deduplication on the fields, and outputs the verified business data to the self-evolution layer; the self-evolution layer constructs a reinforcement learning model according to the verification results, updates the analysis rules through policy iteration, and feeds back the optimized model parameters to the dynamic analysis layer in real time.

[0025] In the embodiment of the application, the above step S100 receives mixed format documents, and performs feature extraction and spatial alignment on printed text, handwritten annotations and dynamic tables in the mixed format documents to obtain graphic table semantic features, including the following sub-steps A1 and A2: In A1: feature extraction is performed on the printed text, handwritten annotations and dynamic tables in the mixed format documents; the specific extraction steps include the following sub-steps A11-A13: In A11: an optical character recognition engine is used to perform character contour feature matching and context semantic completion on the printed text in the mixed format documents to obtain text semantic features; In A12: the spatial features of the handwritten region image in the mixed format document are extracted by a convolutional neural network, and the formula is: Among them, I represents the input image, FFor the extracted spatial feature map, the CNN represents the extraction operation of the convolutional neural network; Further, the spatial features are flattened into a sequence form and input into a long short-term memory network for learning and recognition to obtain image semantic features, wherein the long short-term memory network is represented as: wherein, represents the input feature at time t , is the current hidden state, is the hidden state at the previous time, LSTM represents the extraction operation of the long short-term memory network; Further, the loss function of the long short-term memory network is a sequence labeling loss function, which allows learning of character sequences without accurate alignment labels.

[0026] In A13: A target detection algorithm is used to locate the table frame and cell in the mixed format document, and a set of bounding boxes output can be represented as: wherein, represents the center coordinates of the i-th bounding box, and represent the width and height of the center coordinates of the i-th bounding box, respectively, represents the category of the center coordinates of the i-th bounding box, and N is the number of detected table elements; Further, the table row and column structure is preliminarily established by spatial clustering and sorting of all cells in the bounding box; the text features of each cell region are input into an optical character recognition engine to extract text features, and the text features and position information are input into a self-attention mechanism learning model as cell features to calculate the dependency between cells to obtain structure-aware table semantic features, which are represented by the formula: wherein, Q, K, and V represent query, key, and value matrices, respectively, represents the dimension of the key vector, and softmax represents the normalization exponential function.

[0027] It should be noted that after encoding by the self-attention mechanism learning model, each cell obtains a structure-aware context representation.

[0028] In A2: The printed text, handwritten notes and dynamic tables in the mixed format document are spatially aligned; the specific steps include: The original coordinates of the printed text, handwritten notes and dynamic tables in the mixed format document are normalized to a unified image space, which is represented by the formula: in,( x ′, y ′) is the result after normalization, ( x , y ) represents the coordinate position of an element in the original document, and W and H represent the width and height of the original image, respectively; after normalization, document elements of different resolutions and sources are uniformly transformed into the standard spatial coordinate system; Calculate the spatial overlap and relative distance of the location information in the normalized image; Based on the spatial overlap of the images and the relative distance of the positional information, a correspondence is established between the text boxes, image areas and handwritten trajectory points to achieve spatial alignment.

[0029] It should be noted that step S100 above adopts a cross-modal collaborative mechanism, which uses multimodal feature extraction and spatial alignment technology to achieve unified processing of printed text, handwritten annotations and dynamic tables in mixed format documents, effectively solving the information fragmentation problem that exists in traditional methods when processing multimodal mixed documents.

[0030] In this embodiment of the invention, step S200 inputs the semantic features of the image and text table into the dynamic parsing layer, and parses the metaphorical expressions and synonymous heterogeneous fields through field extraction and contextual semantic reasoning mechanisms, outputting structured data including the following sub-steps B1~B4: In B1: Based on a dynamically generated neural network structure, context-aware field boundary recognition is performed on the semantic features of the input graphic and textual table to extract key fields from the contract; Specifically, it is built based on Java bytecode dynamic generation technology, adjusts the network structure in real time, automatically generates extraction subnetworks for undefined fields, and infers field boundaries through contextual semantics.

[0031] In B2: A pre-built domain knowledge graph is introduced, which defines various contract entities and their synonyms and hierarchical relationships; the extracted key fields are converted into standardized field names through a synonym-heterogeneous field mapping library, and the mapping relationship can be represented as: in, This represents the standardized field name after unification. This represents a graph-driven semantic normalization function, where S represents the original field or phrase; In B3: Unstructured text is input into a pre-trained language model and context-encoded to generate feature representations containing semantic and positional information. The entity type at each encoded position is then predicted by a classifier. Specifically, the unstructured text is input into a pre-trained language model for context encoding to enhance the understanding of Chinese long words and professional terms, and the context encoding process of the pre-trained language model is as follows: wherein, T is the input contract text sequence, represents the context encoding representation at position i, covering its semantic meaning and context clues, and BERT represents a bidirectional encoder transformation operation.

[0032] Specifically, the entity type of each encoding position is predicted by a classifier: wherein, W , b are the weight and bias parameters of the classifier, respectively, represents the predicted label of position i , and the output entity category; In B4: the standardized field name and the entity prediction result are integrated into structured data in the form of key-value pairs, and output to the federal verification layer.

[0033] It should be noted that the above step S200 can accurately identify and analyze the metaphorical expressions, professional terms and synonymous heterogeneous fields in the contract by combining the domain knowledge graph and the context reasoning mechanism, and convert the unstructured text data into standardized structured output, thereby significantly improving the adaptability of the system to diversified contract templates and avoiding the problem of frequent manual adjustment required by traditional rule-based methods.

[0034] In the embodiment of the present application, the above step S300 adopts a regular engine to perform syntax compliance verification on the structured data, and the structured data is compared and verified by a federal learning mechanism, including the following sub-steps C1 and C2: In C1: the regular engine is used to perform syntax compliance verification on the structured data; Specifically, the regular engine is used to perform syntax compliance verification on the key fields of the structured data, and if the field format does not meet the requirements, it is not compliant; the compliant verification data is compared with the local business data to determine whether there are identical data records by hash index, and a data set with correct format and no duplication is output.

[0035] It should be noted that step C1 first defines regular matching rules for different business scenarios for key fields (such as ID number, date, amount, etc.), verifies whether the field format meets the standard requirements through pattern matching, and automatically filters out data items with incorrect format. For data that passes syntax verification, the system further uses hash index technology to quickly compare it with the local business database. By calculating the feature hash value of the data and establishing an index, efficient deduplication detection is achieved. This ensures the format specification of the data and avoids the entry of duplicate data. Finally, a set of valid data with correct format and uniqueness is output.

[0036] In C2: comparison and verification of structured data through federated learning mechanism; the specific steps include: For data that cannot be verified locally, introduce a federated learning mechanism, encrypt the feature parameters through encryption algorithms, and send them to the central server. Distributed data comparison with human resource management systems, enterprise resource planning systems, and blockchain storage systems is performed to output the comparison results. The central server compares the feature similarity of the comparison results Identifies whether it is the same business record, and the formula is: Where, and represent the feature vectors extracted before encryption of the two system data. If the feature similarity of the comparison results is greater than the first set threshold, it means that the two data items represent the same business object.

[0037] Specifically, the first set threshold needs to consider multiple factors such as business scenario requirements, data feature distribution, and system fault tolerance. It is set by the following methods: based on historical business data to construct positive and negative sample pairs, calculate the similarity distribution of their feature vectors; use the receiver operating characteristic curve analysis or precision-recall balance method to select the similarity value that achieves the optimal balance point of false matching rate and missed matching rate as the initial threshold; adjust it according to the fault tolerance requirements of the specific business scenario, generally in the range of 0.7-0.9.

[0038] It should be noted that in actual application, the first set threshold also needs to be dynamically adjusted according to the evolution of business data features. The system will periodically evaluate the accuracy of the comparison results, and automatically trigger the threshold optimization process when performance degradation is found, ensuring optimal recognition effect at different times.

[0039] It should be noted that the double-checking mechanism adopted in the above step S300 ensures the accuracy and reliability of the data extraction result. The regular engine strictly checks the syntax compliance of the key fields, effectively filtering the data with format errors. The federated learning mechanism realizes cross-system data consistency verification under the premise of protecting data privacy. It not only meets the high requirements of enterprises for data quality, but also complies with the current strict data privacy protection regulations.

[0040] In the embodiment of the present application, the above step S400 inputs the verified data into the reinforcement learning model to update the parsing rules and model parameters through policy iteration, and feeds back the updated parsing rules and model parameters to the dynamic parsing layer to complete the closed-loop optimization, including the following sub-steps D1-D3: In D1: for each field extraction task, a reinforcement learning model is constructed, and the model parameter space is discretized into a state set, each state corresponding to a different rule combination; In D2: the field extraction strategy is taken as the objective function, and the extraction accuracy is taken as the reward signal. The Bellman equation is used to update the Q value table, which is expressed as: wherein, represents the objective function of taking action s in state a , represents the learning rate, which controls the update step size; r represents the reward value obtained by the agent after performing action a , and γ represents the discount factor, which is used to balance the immediate reward and long-term effect; represents the new state after performing action , represents the action taken in the new state, represents the objective function of taking action in state ; In D3: when the verification set accuracy is improved, the current optimal parameter combination is stored in the experience pool, and the optimal extraction strategy is gradually converged through dynamic programming. The updated field parsing rule set and adjusted model parameters are output, and the self-evolution iteration is completed.

[0041] It should be noted that the above step S400 converts the verification result in the actual business scenario into a training sample, and continuously adjusts the model parameters and parsing rules through policy iteration to realize the self-improvement of system performance. It effectively overcomes the defects of traditional system performance solidification and difficulty in adapting to new scenarios, and ensures that the system can maintain a high level of recognition accuracy when processing new documents or special business needs.

[0042] In embodiment 2, a text data extraction system based on multi-modal fusion and self-evolution learning is provided, which comprises: a multi-modal perception module, configured to receive a mixed format document, perform feature extraction and spatial alignment on printed text, handwritten annotations and dynamic tables in the mixed format document, and obtain graphic-table-text semantic features; a dynamic analysis module, configured to input the graphic-table-text semantic features into a dynamic analysis layer, analyze metaphorical expressions and synonymously heterogeneous fields through field extraction and context semantic reasoning mechanisms, and output structured data; a federal verification module, configured to perform syntax compliance verification on the structured data by using a regular engine, and perform comparison verification on the structured data by using a federal learning mechanism; a self-evolution module, configured to input the verified data into a reinforcement learning model to update analysis rules and model parameters through policy iteration, and feed the updated analysis rules and model parameters back to the dynamic analysis layer to complete closed-loop optimization.

[0043] It should be noted that the technical scheme of the text data extraction system based on multi-modal fusion and self-evolution learning is the same as the technical scheme of the text data extraction method based on multi-modal fusion and self-evolution learning described above. The details of the technical scheme of the text data extraction system based on multi-modal fusion and self-evolution learning in this embodiment are not described in detail, and can be referred to the description of the technical scheme of the text data extraction method based on multi-modal fusion and self-evolution learning.

[0044] The above-mentioned unit modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0045] The embodiment also provides an electronic device, which includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the electronic device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement the text data extraction method based on multi-modal fusion and self-evolution learning. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device. In addition, the input device can be an external keyboard, touchpad or mouse, etc.

[0046] The embodiment further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method proposed in the above embodiment.

[0047] The storage medium proposed in the embodiment belongs to the same inventive concept as the method proposed in the above embodiment, and the technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0048] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory, a random access memory, a flash memory, a hard disk or an optical disk, etc., and includes a number of instructions to make an electronic device (which can be a personal computer, a server, or a network device, etc.) execute the method of the embodiments of the present application.

[0049] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limit the present application, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A text data extraction method based on multimodal fusion and self-evolutionary learning, characterized in that, include: Receive a mixed-format document, and perform feature extraction and spatial alignment on the printed text, handwritten annotations and dynamic tables in the mixed-format document to obtain the semantic features of the image, text and table. The semantic features of the image and text table are input into the dynamic parsing layer, and the metaphorical expressions and synonymous heterogeneous fields are parsed through field extraction and contextual semantic reasoning mechanisms to output structured data. The structured data is grammatically validated using a regular expression engine, compared and validated using a federated learning mechanism, and the validated data is input into a reinforcement learning model to update the parsing rules and model parameters through policy iteration. The updated parsing rules and model parameters are fed back to the dynamic parsing layer to complete closed-loop optimization.

2. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 1, characterized in that, The feature extraction steps include: An optical character recognition engine is used to perform character contour feature matching and contextual semantic completion on the printed text in the mixed-format document to obtain text semantic features; Spatial features of handwritten regions in the hybrid format document are extracted by a convolutional neural network. The spatial features are flattened into a sequence and then input into a long short-term memory network for learning and recognition to obtain image semantic features. The loss function of the Long Short-Term Memory network is the sequence labeling loss function, which allows for the learning of character sequences without precisely aligned labels; For the dynamic tables in the mixed-format document, an object detection algorithm is used to locate the table borders and cells, and the bounding box is output. By performing spatial clustering and sorting on all cells in the bounding box, the table row and column structure is initially established. Each cell region is input into the optical character recognition engine to extract text features. The text features and location information are then used as cell features and input into a self-attention mechanism learning model to calculate the dependencies between cells in order to obtain structure-aware table semantic features.

3. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 2, characterized in that, The spatial alignment steps include: The original coordinates of the printed text, handwritten annotations, and dynamic tables in the mixed-format document are normalized to a unified image space; Calculate the spatial overlap and relative distance of the location information in the normalized image; Based on the spatial overlap of the images and the relative distance of the positional information, a correspondence is established between the text box, the image area, and the handwritten trajectory points to achieve spatial alignment.

4. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 3, characterized in that, The output structured data includes: Based on a dynamically generated neural network structure, context-aware field boundary recognition is performed on the semantic features of the input graphic and textual tables to extract key fields from the contract. A pre-built domain knowledge graph is introduced, and the extracted key fields are converted into standardized field names through a synonym-heterogeneous field mapping library. The mapping relationship can be represented as follows: in, This represents the standardized field name after unification. This represents a graph-driven semantic normalization function, where S represents the original field or phrase; Unstructured text is input into a pre-trained language model and context-encoded to generate feature representations containing semantic and positional information. A classifier is then used to predict the entity type at each encoded position. The standardized field names and entity prediction results are integrated into structured data in key-value pair form and output to the federated verification layer.

5. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 4, characterized in that, The step of using a regular expression engine to perform syntax compliance checks on the structured data includes: A regular expression engine is used to perform syntax compliance checks on the key fields of the structured data; fields that do not meet the requirements are considered non-compliant. The compliance verification data is compared with the local business data to remove duplicates. A hash index is used to determine whether there are duplicate data records, and a set of data with correct format and no repetition is output.

6. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 5, characterized in that, The comparison and verification of the structured data through the federated learning mechanism includes: For data that cannot be verified locally, a federated learning mechanism is introduced. The feature parameters are encrypted using an encryption algorithm and then sent to the central server for distributed data comparison with the human resource management system, enterprise resource planning system, and blockchain evidence storage system, and the comparison results are output. The central server bases the comparison results on the feature similarity. The formula for identifying whether they are the same business record is as follows: in, and These represent the feature vectors extracted from the two systems before data encryption; if the feature similarity of the comparison results... If the values ​​are greater than the first set threshold, it means that the two data items represent the same business object.

7. The text data extraction method based on multimodal fusion and self-evolutionary learning as described in claim 6, characterized in that, The step of inputting the verified data into the reinforcement learning model and updating the parsing rules and model parameters through policy iteration includes: For each field extraction task, a reinforcement learning model is built, and the model parameter space is discretized into a set of states, with each state corresponding to a different combination of rules. Using the field extraction strategy as the objective function and the extraction accuracy as the reward signal, the Bellman equation is used to update the Q-value table, as expressed by the formula: in, Indicates the state s Take action below a The objective function, This represents the learning rate and controls the update step size. r This indicates that the agent is performing an action. a The reward value obtained later, where γ represents the discount factor, is used to balance immediate rewards and long-term effects; This indicates the new state after the action is performed. This indicates the action taken in the new state. Indicates the state Take action below The objective function; When the accuracy of the validation set improves, the current optimal parameter combination is stored in the experience pool. Through dynamic programming, it gradually converges to the optimal extraction strategy, outputs the updated field parsing rule set and the adjusted model parameters, and completes the self-evolutionary iteration.

8. A text data extraction system based on multimodal fusion and self-evolutionary learning, employing the text data extraction method based on multimodal fusion and self-evolutionary learning as described in any one of claims 1 to 7, characterized in that, include: A multimodal perception module is used to receive mixed-format documents, extract features and spatially align printed text, handwritten annotations and dynamic tables in the mixed-format documents to obtain semantic features of the text, images and tables. The dynamic parsing module is used to input the semantic features of the image and text table into the dynamic parsing layer, and parse metaphorical expressions and synonymous heterogeneous fields through field extraction and contextual semantic reasoning mechanisms to output structured data; The federated verification module is used to perform syntax compliance verification on the structured data using a regular expression engine and to compare and verify the structured data through a federated learning mechanism. The self-evolution module is used to input the verified data into the reinforcement learning model and update the parsing rules and model parameters through policy iteration; the updated parsing rules and model parameters are fed back to the dynamic parsing layer to complete closed-loop optimization.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and when the processor executes the computer-executable instructions, it implements the steps of the text data extraction method based on multimodal fusion and self-evolutionary learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the computer-executable instructions are executed by the processor, they implement the steps of the text data extraction method based on multimodal fusion and self-evolutionary learning as described in any one of claims 1 to 7.

Images