Information extraction method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence and the technical field of finance, and relates to an information extraction method, which comprises the steps of obtaining to-be-processed data, inputting the to-be-processed data into a pre-trained key information extraction model which comprises an image coding layer and an image decoding layer, image feature analysis is carried out on image data through the image coding layer to obtain an image feature vector, dictionary mapping is carried out on text data through a cuckoo hash algorithm to obtain an information dictionary, feature conversion is carried out on the image feature vector through the image decoding layer to obtain a text feature vector, and the text feature vector is decoded through the image decoding layer. And performing dictionary decoding on the text feature vector based on the information dictionary to obtain text key information. The invention further provides an information extraction device, computer equipment and a storage medium. According to the method, the end-to-end key information extraction process from picture input to demand value output is realized, and the manual participation degree in the business process is reduced.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to an information extraction method, apparatus, computer equipment and storage medium. Background Art

[0002] In today's era when insurance business is becoming more and more popular, the automated and high-precision extraction of key information from insurance invoices has become an important step towards the digital transformation of finance.

[0003] At present, invoice recognition generally extracts the text on the invoice through high-precision text recognition, and then filters and selects the results through information extraction methods. This method can show particularly good extraction accuracy in scenarios with high text recognition accuracy, and also makes it possible to automatically extract key information from invoices.

[0004] However, the current invoice recognition technology uses a two-stage approach that prolongs the overall algorithm computing time and cannot achieve an end-to-end key information extraction process from image input to required value output. In addition, post-processing is prone to show its shortcomings when the scene changes. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose an information extraction method, apparatus, computer equipment and storage medium, the main purpose of which is to realize an end-to-end key information extraction process from image input to demand value output, thereby reducing manual participation in the business process.

[0006] In order to solve the above technical problems, the present application embodiment provides an information extraction method, which adopts the following technical solution:

[0007] Acquire data to be processed, wherein the data to be processed includes image data and text data;

[0008] Inputting the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer;

[0009] Performing image feature analysis on the image data through the image coding layer to obtain an image feature vector;

[0010] Perform dictionary mapping on the text data by using a cuckoo hash algorithm to obtain an information dictionary;

[0011] Performing feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector;

[0012] The text feature vector is subjected to dictionary decoding based on the information dictionary to obtain text key information.

[0013] Furthermore, before the step of inputting the data to be processed into the pre-trained key information extraction model, the method further includes:

[0014] Obtaining a CLIP large model source code, and constructing an image coding layer based on the CLIP large model source code;

[0015] Construct an image decoding layer based on the self-attention mechanism and the fully connected network;

[0016] The image encoding layer and the image decoding layer are integrated and deployed in a cloud server to obtain a key information extraction model to be trained;

[0017] A key information training set is obtained, and model training is performed on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model.

[0018] Furthermore, the step of constructing the image coding layer based on the CLIP large model source code specifically includes:

[0019] Decoupling the image encoding module in the CLIP large model source code to obtain a first target encoding module;

[0020] Separating the parameters of the first target coding module according to the class name specification to obtain a second target coding module;

[0021] The second target module is connected to a preset neural network layer to obtain the image coding layer.

[0022] Furthermore, the image decoding layer includes several decoding levels, including an embedding layer, an attention mechanism layer, a forward propagation layer, and a short connection layer. The step of constructing the image decoding layer based on the self-attention mechanism and the fully connected network specifically includes:

[0023] Connecting the embedding layer to the attention mechanism layer to obtain a first network layer, wherein the output of the embedding layer is the input of the attention mechanism layer;

[0024] Perform layer normalization on the first network layer through a forward propagation layer to obtain a second network layer;

[0025] Performing a residual connection between the second network layer and the short connection layer to obtain a third network layer;

[0026] The third network layer of each decoding level is subjected to serial expansion processing to obtain the image decoding layer.

[0027] Furthermore, the step of performing model training on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model specifically includes:

[0028] Classify the key information training set by labels to obtain an information training set, an information test set and an information verification set;

[0029] Importing preset training parameters into the image coding layer and using the information training set to train the general model to obtain a key information extraction model to be trained;

[0030] and importing the information test set into the key information extraction model to be trained, and outputting the prediction result;

[0031] Based on the prediction result and the information verification set, a connectionist temporal classification loss function and a cross entropy loss function are used for fitting to obtain a prediction error, wherein the connectionist temporal classification loss function is a main loss function and the cross entropy loss function is an auxiliary loss function;

[0032] The prediction error is compared with a preset threshold. If the prediction error is greater than the preset threshold, the key information extraction model to be trained is iteratively updated until the prediction error is less than or equal to the preset threshold, thereby obtaining a pre-trained key information extraction model.

[0033] Furthermore, the step of performing feature conversion on the image feature vector through the image decoding layer to obtain the text feature vector specifically includes:

[0034] Vectorizing the image feature vector through an embedding layer to obtain a weak text feature vector;

[0035] Performing a first matrix operation on the image feature vector through the attention mechanism layer to obtain a first target matrix, where the first target matrix includes a first branch matrix and a second branch matrix;

[0036] Perform a second matrix operation on the weak text feature vector and the attention mechanism layer to obtain a second target matrix;

[0037] Normalizing the first branch matrix and the second target matrix row by row through the forward propagation layer to obtain a target weight matrix;

[0038] The target weight matrix and the second branch matrix are fused and accumulated through the short connection layer by using the Strassen algorithm to obtain the text feature vector.

[0039] Furthermore, the step of performing dictionary decoding on the text feature vector based on the information dictionary to obtain text key information specifically includes:

[0040] Preprocessing the text feature vector to obtain a target text vector;

[0041] Taking the information dictionary as an anchor point, the target text vector is located by a sparse reconstruction algorithm to obtain the key information of the text.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides an information extraction device, which adopts the following technical solution:

[0043] An acquisition module, used for acquiring data to be processed, wherein the data to be processed includes image data and text data;

[0044] An input module, used for inputting the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer;

[0045] A parsing module, used for performing image feature parsing on the image data through the image coding layer to obtain an image feature vector;

[0046] A mapping module, used for performing dictionary mapping on the text data through a cuckoo hash algorithm to obtain an information dictionary;

[0047] A conversion module, used for performing feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector;

[0048] A decoding module is used to perform dictionary decoding on the text feature vector based on the information dictionary to obtain text key information.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the information extraction method when executing the computer-readable instructions.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0052] A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the information extraction method are implemented.

[0053] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0054] The embodiment of the present application obtains data to be processed, inputs the data to be processed into a pre-trained key information extraction model, and the pre-trained key information extraction model includes an image encoding layer and an image decoding layer. The image encoding layer performs image feature analysis on the image data to obtain an image feature vector, performs dictionary mapping on the text data through a cuckoo hash algorithm to obtain an information dictionary, performs feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector, and performs dictionary decoding on the text feature vector based on the information dictionary to obtain text key information, thereby realizing an end-to-end key information extraction process from image input to demand value output, and directly returns the text content of the demand field by using the difference information between features, thereby reducing the manual participation in the business process and avoiding the problems of manual line-by-line detection and low recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 is a flow chart of an embodiment of the information extraction method according to the present application;

[0058] Figure 3 is a schematic structural diagram of an embodiment of an information extraction device according to the present application;

[0059] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0066] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0067] It should be noted that the information extraction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the information extraction device is generally arranged in the server / terminal device.

[0068] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0069] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for extracting information according to the present application. The information extraction method comprises the following steps:

[0070] Step S201, obtaining data to be processed.

[0071] In this embodiment, the information extraction method can be deployed in an information extraction platform. The information extraction platform can be constructed by a server or a server cluster. The server or server cluster can be any electronic device with image recognition, image processing, text processing, data transmission, data storage and other functions. The information extraction method runs on the electronic device (for example, Figure 1 The server / terminal device shown in the figure can receive the data to be processed through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0072] In this embodiment, the data to be processed includes image data and text data. Specifically, the image data may be images such as insurance invoices, insurance invoice receipts, insurance list images, and insurance bills, and the text data may be historical invoice texts and historical bill texts in the insurance company database, and the historical invoice texts and historical bill texts may be text information such as amount, date, taxpayer identification number, buyer and seller information waiting to be extracted.

[0073] Step S202: input the data to be processed into a pre-trained key information extraction model.

[0074] In this embodiment, the pre-trained key information extraction model may be an artificial intelligence model for converting information in a picture format into information in a text format, and the pre-trained key information extraction model includes an image encoding layer and an image decoding layer.

[0075] Specifically, the image encoding layer is used to convert the input image data from its original pixel form into an image feature vector or a feature map. The image feature vector or feature map is mainly used to capture the text character information in the image data. The image encoding layer can be implemented by a convolutional neural network (CNN) such as VGG, ResNet or Inception; the image decoding layer can be a feature vector that converts the image feature vector or feature map into text form, and then generates a descriptive text related to the input image data. The image decoding layer can be implemented by a neural network such as a fully connected network, a recursive neural network (RNN) and a long short-term memory network (LSTM).

[0076] Step S203, performing image feature analysis on the image data through the image coding layer to obtain an image feature vector.

[0077] In this embodiment, image feature analysis can be implemented through feature matching algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF). The above-mentioned image feature vector can be a multidimensional vector, in which each element or a group of elements represents the characteristics of the above-mentioned image data such as color, texture, edge, shape, etc. The scale of the above-mentioned image feature vector is nx 512, where n represents the number of image blocks in the image data.

[0078] In a possible embodiment, the VGG19 model can be selected as the image encoding layer, and the invoice image to be processed is first compressed by mathematical processing techniques such as bilinear interpolation, nearest neighbor interpolation or Lanczos resampling to obtain a first invoice image, where the pixel size of the first invoice image is 224x224 pixels. The pixel values ​​of the first invoice image are then normalized, and the pixel values ​​are converted from the range of [0,255] to [0,1] to obtain a second invoice image. Then, according to the pre-training settings of the VGG19 model, the mean and standard deviation of the second invoice image are standardized to obtain a target invoice image (i.e., the above-mentioned image data), and the pre-trained VGG19 model is loaded. The pre-trained VGG19 model removes the classification layer and only retains the last convolution layer. The target invoice image is forward-transferred through the pre-trained VGG19 model to obtain the above-mentioned image feature vector.

[0079] The above-mentioned mean value may refer to the average value of all pixel values ​​in the second invoice image, and the above-mentioned mean value may be obtained by calculating the mean value of each color channel (red, green, and blue) in the second invoice image. The above-mentioned standard deviation may be a statistic for measuring the distribution range of pixel values. The larger the standard deviation, the greater the change in color or brightness of the image. The above-mentioned standardization processing may be to set the mean value and standard deviation of the second invoice image as follows:

[0080] Mean:

[0081] R(red):0.485

[0082] G(green):0.456

[0083] B(blue):0.406

[0084] Standard Deviation:

[0085] R(red):0.229

[0086] G(green):0.224

[0087] B(blue):0.225

[0088] Step S204, dictionary mapping is performed on the text data using the cuckoo hash algorithm to obtain an information dictionary.

[0089] In this embodiment, the above-mentioned Cuckoo Hashing is a hash algorithm used to resolve conflicts in a hash table and is suitable for scenarios that require fast search, insertion and deletion. The above-mentioned information dictionary can be a mapping table of an abstract data type, which is specifically used to store a set of key-value pairs, where each key is associated with at most one value, and each of the above-mentioned keys and values ​​can be stored in character type.

[0090] In a possible embodiment, the text data may be segmented to obtain a text data set, which may include multiple sentences, paragraphs or word lists, and the text data set may be cleaned (which may include removing punctuation marks, converting text to lowercase or uppercase, removing stop words, etc.) to obtain a target text data set, and then at least two hash functions are selected, which may be encrypted hash functions such as MD5, SHA-1 or SHA-256, or Python's built-in hash() function, and the hash function may be used to map each text item in the text data set to a hash value, and then insert the mapping relationship into a preset information mapping table to obtain an information dictionary. Specifically, the preset information mapping table contains keys and values, the keys are the cleaned text items, and the values ​​are the corresponding hash values. The insertion operation involves putting the key-value pair into the hash table. If the position is already occupied, the cuckoo hashing strategy is used to remove the old key value and try to move it to the position specified by another hash function.

[0091] Using the cuckoo hash algorithm to perform dictionary mapping on text data of text type and create an information dictionary can greatly reduce conflicts and improve the space utilization of the hash table, thereby speeding up data access. Secondly, this mapping method supports efficient insertion, deletion, and search operations, and is particularly suitable for dynamically changing data sets, such as real-time message processing and large-scale text analysis (such as invoice analysis in insurance business), and can quickly respond to data changes.

[0092] Step S205, performing feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector.

[0093] In this embodiment, the text feature vector may be a set of numerical features that can be used in a machine learning model by converting text data (such as words, phrases, or entire documents).

[0094] Specifically, regarding the specific implementation process of performing feature conversion on the image feature vector through the image decoding layer to obtain the text feature vector, this application will provide further detailed descriptions in subsequent specific embodiments, and will not be elaborated on here.

[0095] Step S206, performing dictionary decoding on the text feature vector based on the information dictionary to obtain key information of the text.

[0096] The above-mentioned text key information may be text information in the invoice image (the above-mentioned image data) about the insurance amount, date, taxpayer identification number, buyer and seller information waiting to be extracted.

[0097] Specifically, regarding the specific implementation process of performing dictionary decoding on the text feature vector based on the information dictionary to obtain the text key information, this application will provide further detailed descriptions in subsequent specific embodiments, and will not be elaborated on here.

[0098] The present application obtains data to be processed, and inputs the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer, performs image feature analysis on the image data through the image encoding layer to obtain an image feature vector, performs dictionary mapping on the text data through the cuckoo hash algorithm to obtain an information dictionary, performs feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector, and performs dictionary decoding on the text feature vector based on the information dictionary to obtain text key information, thereby realizing an end-to-end key information extraction process from image input to demand value output, and directly returns the text content of the demand field by using the difference information between features, thereby reducing the manual participation in the business process and avoiding the problems of manual line-by-line detection and low recognition efficiency.

[0099] In some optional implementations, before step S202, the following steps are included:

[0100] Get the CLIP large model source code, and build the image coding layer based on the CLIP large model source code;

[0101] Construct an image decoding layer based on the self-attention mechanism and the fully connected network;

[0102] The image encoding layer and the image decoding layer are integrated and deployed in the cloud server to obtain the key information extraction model to be trained;

[0103] A key information training set is obtained, and model training is performed on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model.

[0104] In this embodiment, the above-mentioned CLIP (Contrastive Language–Image Pre-training) is a large multimodal model developed by OpenAI, which can be used to simultaneously understand visual and language information by learning extensive associations between images and related text labels.

[0105] In this embodiment, the cloud server may be a virtual server such as Amazon Web Services (AWS), Microsoft Azure or Google Cloud Platform (GCP).

[0106] Specifically, regarding the specific implementation process of constructing the image decoding layer based on the self-attention mechanism and the fully connected network, this application will provide further detailed descriptions in subsequent specific embodiments, and will not be elaborated in detail here.

[0107] Specifically, regarding the specific implementation process of constructing the image decoding layer based on the self-attention mechanism and the fully connected network, this application will provide further detailed descriptions in subsequent specific embodiments, and will not be elaborated in detail here.

[0108] Specifically, regarding the above-mentioned acquisition of the key information training set, the specific implementation process of performing model training on the key information extraction model to be trained based on the key information training set to obtain the pre-trained key information extraction model, this application will provide further detailed descriptions of this in subsequent specific embodiments, and will not be elaborated in detail here.

[0109] This application constructs a pre-trained key information extraction model, which can automatically and accurately extract key information from images in various applications, such as automatic annotation, content review, and extraction of important data in insurance documents, thereby greatly improving the efficiency and accuracy of information processing. The above-mentioned key information extraction model can accurately identify and extract key information from images by utilizing the cross-modal learning ability of CLIP, and uses the self-attention mechanism to enhance the model's attention to different areas of the image, so that it can better understand the relationship between image content and key information. In addition, the key information extraction model deployed in the cloud ensures high availability and easy scalability, and is suitable for processing large-scale data.

[0110] In some optional implementations, the step of “obtaining the CLIP large model source code and constructing the image coding layer based on the CLIP large model source code” includes the following steps:

[0111] Decouple the image encoding module in the CLIP large model source code to obtain the first target encoding module;

[0112] Separate the parameters of the first target encoding module according to the class name specification to obtain a second target encoding module;

[0113] The second target module is connected to the preset neural network layer to obtain an image coding layer.

[0114] In this embodiment, the above-mentioned CLIP large model may include a visual encoder (i.e., the above-mentioned image encoding module) and a text encoder. The above-mentioned visual encoder may be a structure based on ResNet or ViT (Vision Transformer) for extracting feature representations from images, and the above-mentioned text encoder may be a Transformer-based architecture for extracting feature representations from text (such as descriptions or labels).

[0115] In this embodiment, the class name specifications may include Upper CamelCase, Snake Case, Kebab Case, and Lower CamelCase.

[0116] In this embodiment, the above-mentioned preset neural network layer may include a fully connected layer, an activation layer and a short connection layer.

[0117] In a possible embodiment, the source code of the CLIP large model can be used to analyze and identify the specific coupling points and dependencies between the image coding module and other parts (such as the text encoder) in the CLIP large model. Based on the above dependencies, the code is reorganized to separate the tightly coupled functions into independent modules (i.e., the first target coding module). The first target coding module stores all the functions of the image coding module in the above CLIP large model. The class name of the first target coding module is standardized in a snake-like naming manner, and then the model parameters of the first target coding module are separated, and only the parameters related to the image coding module in the first target coding module (and the above parameter separation) are retained to obtain the second target coding module. The second target coding module is connected through a fully connected layer, an activation layer, and a short connection layer to construct an image coding layer.

[0118] This application decouples and reintegrates the image encoding module of the CLIP large model source code, allowing developers to control the various components of the CLIP model, optimize its performance, and integrate with other neural network layers to enhance functionality and adaptability, which can significantly improve the flexibility and maintainability of the image encoding layer.

[0119] In some optional implementations, constructing an image decoding layer based on a self-attention mechanism and a fully connected network includes the following steps:

[0120] Connect the embedding layer to the attention mechanism layer to obtain the first network layer;

[0121] The first network layer is normalized through the forward propagation layer to obtain the second network layer;

[0122] Perform residual connection between the second network layer and the short connection layer to obtain the third network layer;

[0123] The third network layer of each decoding level is extended in series to obtain an image decoding layer.

[0124] In an embodiment of the present application, the above-mentioned image decoding layer includes several decoding levels, and the above-mentioned decoding levels include an embedding layer, an attention mechanism layer, a forward propagation layer and a short connection layer.

[0125] Specifically, the above-mentioned embedding layer can be a pre-trained word embedding such as Word2Vec, GloVe or a directly randomly initialized embedding layer, the above-mentioned attention mechanism layer can be a self-attention (Self-Attention), such as the model used in Transformer, the above-mentioned forward propagation layer can be a fully connected layer including activation functions such as ReLU and LeakyReLU, and the above-mentioned short connection layer is implemented by ResNet (Residual Network), which can be used to directly add the input of a certain layer (or multiple layers) to the output of the following layers, wherein the output of the above-mentioned embedding layer is the input of the attention mechanism layer, and the above-mentioned series expansion processing can be to connect the third network layer of each decoding level in sequence to form a longer or more complex network structure (the above-mentioned image decoding layer).

[0126] In a possible embodiment, in each decoding level, the Word2Vec model is connected to the Transformer (attention mechanism layer) as an embedding layer, the output of the above embedding layer is used as the input of the attention mechanism layer to obtain a first network layer, the first network layer is connected to the forward propagation layer, the mean and standard deviation of the output data (which can be an image feature vector) in the first network layer are calculated as statistical data, and then the first network layer is normalized using the above statistical data to obtain a second network layer; then the second network layer is superimposed with the input of the first network layer and the second network layer and the output of ResNet through ResNet (short connection layer) to obtain a third network layer, and finally the third network layer of each decoding level is cascaded and expanded to obtain an image decoding layer.

[0127] In this application, by using a combination of word embedding and self-attention layers, rich semantic information can be effectively captured and processed, which is crucial for understanding image content and its related text descriptions. At the same time, by introducing layer normalization and residual connections in each decoding level, the model not only improves the stability and efficiency of the training process, but also can better solve the gradient vanishing problem, so that the network can be effectively trained even if it is very deep. In addition, through serial expansion processing, the output of each decoding level can be further integrated to form a complex and deep network structure, which enhances the model's ability to capture details and overall prediction performance.

[0128] In some optional implementations, the step of performing model training on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model includes the following steps:

[0129] Classify the key information training set by labeling to obtain the information training set, information test set and information verification set;

[0130] Importing the preset training parameters into the image coding layer, and using the information training set to train the general model to obtain the key information extraction model to be trained;

[0131] And import the information test set into the key information extraction model to be trained, and output the prediction results;

[0132] Based on the prediction results and the information validation set, the connectionist temporal classification loss function and the cross entropy loss function are used for fitting to obtain the prediction error;

[0133] The prediction error is compared with a preset threshold. If the prediction error is greater than the preset threshold, the key information extraction model to be trained is iteratively updated until the prediction error is less than or equal to the preset threshold, thereby obtaining a pre-trained key information extraction model.

[0134] In this embodiment, the above-mentioned label classification can be that business personnel collect invoices within the bank from the in-bank data system, label the invoice data, and divide the invoice data into an information training set, an information test set and an information verification set in a ratio of 5:1:1; the above-mentioned general model can be a model that can perform well in a variety of different tasks, data sets or environments. The above-mentioned general model has a high degree of versatility and adaptability, enabling it to cross the boundaries of a single application or scenario and handle and solve a wide range of problems.

[0135] In this embodiment, the above-mentioned connectionist temporal classification loss function (Connectionist Temporal Classification Loss, referred to as CTC Loss) can be a loss function used to solve the problem that the input and output sequences are of different lengths and cannot be aligned, and the above-mentioned cross-entropy loss function (Cross-Entropy Loss) can be a loss function used to measure the difference between the actual output and the predicted output. Specifically, by fitting the connectionist temporal classification loss function as the main loss function and the cross-entropy loss function as the auxiliary loss function, the advantages of both can be comprehensively utilized: CTC Loss handles the alignment problem of the time series, while Cross-Entropy Loss focuses on the prediction accuracy of each time step or category, thereby improving the overall performance and adaptability of the key information extraction model in complex time series tasks.

[0136] In a possible embodiment, the preset training parameters are imported into the image coding layer and fixed, and the preset training parameters are not updated. The general model is then trained using the above-mentioned information training set to obtain the key information extraction model to be trained. The information test set is imported into the key information extraction model to be trained, and the prediction result is output. The connectionist temporal classification loss function is used as the main loss function, and the cross entropy loss function is used as the auxiliary loss function (the weight of the above-mentioned connectionist temporal classification loss function is 0.9, and the weight of the above-mentioned cross entropy loss function is 0.1). The prediction result is compared with the information verification set to obtain a prediction error between the two, and the prediction error is compared with a preset threshold. If the prediction error is greater than the preset threshold, the training parameters are updated through the adamW optimizer, using an Lr of 0.0001 and a gradient descent algorithm until the prediction error is less than or equal to the preset threshold, thereby obtaining a pre-trained key information extraction model.

[0137] This application separates the training set, test set and validation set from the actual business data and adopts label classification to ensure that the data coverage of the model training is wide and has practical application value. By fixing the preset training parameters and using a general model for training, the existing pre-training advantages can be effectively utilized to speed up the training speed and improve the initial performance of the model. By combining the connectionist time series classification loss function and the cross entropy loss function, the accuracy of the model in classification tasks is enhanced.

[0138] In some optional implementations, step S205 includes the following steps:

[0139] The image feature vector is vectorized through the embedding layer to obtain a weak text feature vector;

[0140] The image feature vector is passed through the attention mechanism layer to perform the first matrix operation to obtain the first target matrix.

[0141] Perform a second matrix operation on the weak text feature vector and the attention mechanism layer to obtain a second target matrix;

[0142] The first branch matrix and the second target matrix are normalized row by row through the forward propagation layer to obtain a target weight matrix;

[0143] The target weight matrix and the second branch matrix are fused and accumulated through the short connection layer using the Strassen algorithm to obtain the text feature vector.

[0144] In an embodiment of the present invention, the above-mentioned vectorization processing can be achieved by passing the above-mentioned image feature vector together with two learnable embeddings through an embedding layer to realize the vectorization of text. The above-mentioned embeddings can be artificial intelligence models such as word embeddings model, positional embeddings model, contextual embeddings model and image feature embeddings model.

[0145] In an embodiment of the present invention, the Strassen algorithm may be an improved matrix multiplication algorithm. The Strassen algorithm utilizes a divide-and-conquer strategy and a recursive technique to reduce the number of multiplication operations required in traditional matrix multiplication.

[0146] In an embodiment of the present invention, the above-mentioned attention mechanism layer includes three kernel functions q, k, and v, where q is the query kernel function, k is the key kernel function, and v is the value kernel function. The above-mentioned first target matrix includes a first branch matrix and a second branch matrix.

[0147] Specifically, the first matrix operation can be to pass the image feature vector through the attention mechanism layer, and perform matrix multiplication operation with the above-mentioned key kernel function and the above-mentioned value kernel function to obtain q_out (that is, the above-mentioned first branch matrix) and v_out (that is, the above-mentioned second branch matrix); the above-mentioned second matrix operation can be to pass the weak text feature vector through the attention mechanism layer, and perform matrix multiplication operation with the above-mentioned query kernel function to obtain k_out (that is, the above-mentioned second target matrix).

[0148] In this embodiment, the normalization process may be to normalize the first branch matrix and the second target matrix row by row through a softmax layer to obtain a matrix (ie, the weight matrix).

[0149] In a possible embodiment, the text is vectorized by passing the image feature vector together with two learnable contextual embedding (Contextual Embeddings) models and the image feature embedding (Image Feature Embeddings) through an embedding layer to obtain a weak text feature vector, and the image feature vector is passed through an attention mechanism layer to perform matrix multiplication with the key kernel function and the value kernel function to obtain q_out (i.e., the first branch matrix) and v_out (i.e., the second branch matrix), and the weak text feature vector is passed through an attention mechanism layer to perform matrix multiplication with the query kernel function to obtain k_out (i.e., the second target matrix), and the first branch matrix and the second target matrix are matrix multiplied through a forward propagation layer, and normalized by row through softmax to obtain matrix (i.e., the weight matrix), and the matrix and the v_out matrix are fused and accumulated through the Strassen algorithm to obtain a text feature vector, and each field of the text feature vector occupies 64 bits.

[0150] In this application, by combining advanced embedding technology, attention mechanism and efficient matrix multiplication algorithm (Strassen algorithm), efficient and accurate fusion and processing of image and text features are achieved. Through vectorization processing and attention layer in the deep learning framework, key information in the data can be captured and emphasized more accurately, thereby improving the accuracy of feature extraction and the response speed of the model. The Strassen algorithm is used to further improve the efficiency of matrix operations, reduce computing costs and time, and make processing large-scale data more feasible.

[0151] In some optional implementations, step S206 includes the following steps:

[0152] Preprocess the text feature vector to obtain the target text vector;

[0153] Taking the information dictionary as the anchor point, the target text vector is located through the sparse reconstruction algorithm to obtain the key information of the text.

[0154] In an embodiment of the present invention, the above-mentioned preprocessing can be standardization processing such as unifying the case of text words, converting or deleting numbers and currency symbols, replacing and removing special characters and symbols, and can also be text preprocessing means such as word form restoration, stem extraction, grammatical dependency analysis, and dimensionality reduction processing.

[0155] In the embodiment of the present invention, the above-mentioned sparse reconstruction algorithm can be implemented by algorithms such as Lasso (Least Absolute Shrinkage and Selection Operator), Sparse PCA (Principal Component Analysis), Matching Pursuit and Orthogonal Matching Pursuit.

[0156] In a possible embodiment, the above-mentioned text feature vector is first subjected to dimensionality reduction processing through linear discriminant analysis (LDA) to obtain a target text vector. Based on the above-mentioned information dictionary, a sparse reconstruction algorithm, such as a LASSO regression algorithm or a matching pursuit algorithm, is applied. The hash value in the information dictionary is used as an anchor point. By minimizing the reconstruction error between the target text vector and the hash value vector stored in the dictionary, the key information in the text is determined to obtain the text key information.

[0157] In the present application, by preprocessing the text feature vector and using the information dictionary as an anchor point for sparse reconstruction algorithm, the extraction accuracy and processing efficiency of key information in the text can be effectively improved in fields such as automated invoice processing and insurance invoice analysis. The preprocessing step optimizes the data expression form through dimensionality reduction and standardization to ensure that subsequent algorithms can more effectively identify and reconstruct key features. The sparse reconstruction algorithm uses the hash values ​​pre-stored in the information dictionary to accurately locate and extract key information in the text, so that the system can ensure the high efficiency of information extraction when processing large-scale text data, and can also significantly reduce the consumption of computing resources.

[0158] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned text key information, the above-mentioned text key information can also be stored in a blockchain node.

[0159] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0160] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0161] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0163] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0164] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an information extraction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0165] like Figure 3As shown, the information extraction device 300 described in this embodiment includes: an acquisition module 301, an input module 302, a parsing module 303, a mapping module 304, a conversion module 305 and a decoding module 306. Among them:

[0166] An acquisition module 301 is used to acquire data to be processed, wherein the data to be processed includes image data and text data;

[0167] An input module 302 is used to input the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer;

[0168] The analysis module 303 is used to perform image feature analysis on the image data through the image coding layer to obtain an image feature vector;

[0169] A mapping module 304 is used to perform dictionary mapping on the text data by using a cuckoo hash algorithm to obtain an information dictionary;

[0170] The conversion module 305 is used to perform feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector;

[0171] The decoding module 306 is used to perform dictionary decoding on the text feature vector based on the information dictionary to obtain text key information.

[0172] Before the input module 302, it also includes:

[0173] A source code acquisition module, used to acquire the CLIP large model source code, and construct an image coding layer based on the CLIP large model source code;

[0174] Building module for building image decoding layer based on self-attention mechanism and fully connected network;

[0175] A fusion deployment module, used for fusing the image encoding layer and the image decoding layer and deploying them in a cloud server to obtain a key information extraction model to be trained;

[0176] The training module is used to obtain a key information training set, and perform model training on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model.

[0177] The source code acquisition module includes:

[0178] A decoupling submodule, used for decoupling the image encoding module in the CLIP large model source code to obtain a first target encoding module;

[0179] A parameter separation submodule, used for performing parameter separation on the first target coding module according to a class name specification to obtain a second target coding module;

[0180] The first connection submodule is used to connect the second target module with a preset neural network layer to obtain the image coding layer.

[0181] The building blocks include:

[0182] A second connection submodule is used to connect the embedding layer to the attention mechanism layer to obtain a first network layer, wherein the output of the embedding layer is the input of the attention mechanism layer;

[0183] A normalization submodule, used for performing layer normalization on the first network layer through a forward propagation layer to obtain a second network layer;

[0184] A residual connection submodule, used to perform a residual connection on the second network layer and the short connection layer to obtain a third network layer;

[0185] The serial expansion submodule is used to perform serial expansion processing on the third network layer of each decoding level to obtain the image decoding layer.

[0186] The training module comprises:

[0187] A label classification submodule is used to classify the key information training set into labels to obtain an information training set, an information test set and an information verification set;

[0188] A first import submodule is used to import preset training parameters into the image coding layer and use the information training set to train the general model to obtain a key information extraction model to be trained;

[0189] The second import submodule is used to import the information test set into the key information extraction model to be trained and output the prediction result;

[0190] A fitting submodule, for fitting based on the prediction result and the information verification set using a connectionist temporal classification loss function and a cross entropy loss function to obtain a prediction error, wherein the connectionist temporal classification loss function is a main loss function and the cross entropy loss function is an auxiliary loss function;

[0191] The comparison submodule is used to compare the prediction error with a preset threshold. If the prediction error is greater than the preset threshold, the key information extraction model to be trained is iteratively updated until the prediction error is less than or equal to the preset threshold, thereby obtaining a pre-trained key information extraction model.

[0192] The conversion module 305 includes:

[0193] A vectorization submodule, used for vectorizing the image feature vector through an embedding layer to obtain a weak text feature vector;

[0194] A first operator module, configured to perform a first matrix operation on the image feature vector through the attention mechanism layer to obtain a first target matrix, wherein the first target matrix includes a first branch matrix and a second branch matrix;

[0195] A second operator module is used to perform a second matrix operation on the weak text feature vector and the attention mechanism layer to obtain a second target matrix;

[0196] A third operator module is used to normalize the first branch matrix and the second target matrix by row through the forward propagation layer to obtain a target weight matrix;

[0197] The accumulation submodule is used to fuse and accumulate the target weight matrix and the second branch matrix through the short connection layer by using the Strassen algorithm to obtain the text feature vector.

[0198] The decoding module 306 includes:

[0199] A preprocessing submodule, used for preprocessing the text feature vector to obtain a target text vector;

[0200] The positioning submodule is used to use the information dictionary as an anchor point to locate the target text vector through a sparse reconstruction algorithm to obtain the text key information.

[0201] In this embodiment, by obtaining data to be processed, the data to be processed is input into a pre-trained key information extraction model, and the pre-trained key information extraction model includes an image encoding layer and an image decoding layer. The image encoding layer performs image feature analysis on the image data to obtain an image feature vector, and the cuckoo hash algorithm is used to perform dictionary mapping on the text data to obtain an information dictionary. The image decoding layer performs feature conversion on the image feature vector to obtain a text feature vector, and the text feature vector is dictionary decoded based on the information dictionary to obtain text key information. An end-to-end key information extraction process from image input to demand value output is implemented, and the difference information between features is used to directly return the text content of the demand field, thereby reducing the manual participation in the business process and avoiding the problems of manual line-by-line detection and low recognition efficiency.

[0202] In this embodiment, the operations performed by the above units or modules correspond one-to-one to the steps of the information extraction method in the above implementation mode, and will not be repeated here.

[0203] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0204] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0205] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0206] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the information extraction method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0207] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the information extraction method.

[0208] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0209] The present embodiment provides a computer device, which obtains data to be processed and inputs the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer, performs image feature analysis on the image data through the image encoding layer to obtain an image feature vector, performs dictionary mapping on the text data through a cuckoo hash algorithm to obtain an information dictionary, performs feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector, and performs dictionary decoding on the text feature vector based on the information dictionary to obtain text key information, thereby realizing an end-to-end key information extraction process from image input to demand value output, and directly returns the text content of the demand field by using the difference information between features, thereby reducing the manual participation in the business process and avoiding the problems of manual line-by-line detection and low recognition efficiency.

[0210] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the information extraction method as described above.

[0211] The present embodiment provides a computer-readable storage medium, which obtains data to be processed and inputs the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer, performs image feature analysis on the image data through the image encoding layer to obtain an image feature vector, performs dictionary mapping on the text data through a cuckoo hash algorithm to obtain an information dictionary, performs feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector, and performs dictionary decoding on the text feature vector based on the information dictionary to obtain text key information, thereby realizing an end-to-end key information extraction process from image input to demand value output, and directly returns the text content of the demand field by using the difference information between features, thereby reducing the manual participation in the business process and avoiding the problems of manual line-by-line detection and low recognition efficiency.

[0212] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0213] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An information extraction method, characterized in that: The steps include: Acquire data to be processed, wherein the data to be processed includes image data and text data; Inputting the data to be processed into a pre-trained key information extraction model, wherein the pre-trained key information extraction model includes an image encoding layer and an image decoding layer; Performing image feature analysis on the image data through the image coding layer to obtain an image feature vector; Perform dictionary mapping on the text data by using a cuckoo hash algorithm to obtain an information dictionary; Performing feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector; The text feature vector is subjected to dictionary decoding based on the information dictionary to obtain text key information.

2. The information extraction method according to claim 1, characterized in that: Before the step of inputting the to-be-processed data into the pre-trained key information extraction model, the method further includes: Obtaining a CLIP large model source code, and constructing an image coding layer based on the CLIP large model source code; Construct an image decoding layer based on the self-attention mechanism and the fully connected network; The image encoding layer and the image decoding layer are integrated and deployed in a cloud server to obtain a key information extraction model to be trained; A key information training set is obtained, and model training is performed on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model.

3. The information extraction method according to claim 2, characterized in that: The step of constructing the image coding layer based on the CLIP large model source code specifically includes: Decoupling the image encoding module in the CLIP large model source code to obtain a first target encoding module; Separating the parameters of the first target coding module according to the class name specification to obtain a second target coding module; The second target module is connected to a preset neural network layer to obtain the image coding layer.

4. The information extraction method according to claim 2, characterized in that: The image decoding layer includes several decoding levels, including an embedding layer, an attention mechanism layer, a forward propagation layer, and a short connection layer. The steps of constructing the image decoding layer based on the self-attention mechanism and the fully connected network specifically include: Connecting the embedding layer to the attention mechanism layer to obtain a first network layer, wherein the output of the embedding layer is the input of the attention mechanism layer; Perform layer normalization on the first network layer through a forward propagation layer to obtain a second network layer; Performing a residual connection between the second network layer and the short connection layer to obtain a third network layer; The third network layer of each decoding level is subjected to serial expansion processing to obtain the image decoding layer.

5. The information extraction method according to claim 2, characterized in that: The step of performing model training on the key information extraction model to be trained based on the key information training set to obtain a pre-trained key information extraction model specifically includes: Classify the key information training set by labels to obtain an information training set, an information test set and an information verification set; Importing preset training parameters into the image coding layer, and using the information training set to train the general model to obtain a key information extraction model to be trained; Importing the information test set into the key information extraction model to be trained, and outputting the prediction result; Based on the prediction result and the information verification set, a connectionist temporal classification loss function and a cross entropy loss function are used for fitting to obtain a prediction error, wherein the connectionist temporal classification loss function is a main loss function and the cross entropy loss function is an auxiliary loss function; The prediction error is compared with a preset threshold. If the prediction error is greater than the preset threshold, the key information extraction model to be trained is iteratively updated until the prediction error is less than or equal to the preset threshold, thereby obtaining a pre-trained key information extraction model.

6. The information extraction method according to claim 4, characterized in that: The step of performing feature conversion on the image feature vector through the image decoding layer to obtain the text feature vector specifically includes: Vectorizing the image feature vector through an embedding layer to obtain a weak text feature vector; Performing a first matrix operation on the image feature vector through the attention mechanism layer to obtain a first target matrix, where the first target matrix includes a first branch matrix and a second branch matrix; Perform a second matrix operation on the weak text feature vector and the attention mechanism layer to obtain a second target matrix; Normalizing the first branch matrix and the second target matrix row by row through the forward propagation layer to obtain a target weight matrix; The target weight matrix and the second branch matrix are fused and accumulated through the short connection layer by using the Strassen algorithm to obtain the text feature vector.

7. The information extraction method according to claim 1, characterized in that: The step of performing dictionary decoding on the text feature vector based on the information dictionary to obtain text key information specifically includes: Preprocessing the text feature vector to obtain a target text vector; Taking the information dictionary as an anchor point, the target text vector is located by a sparse reconstruction algorithm to obtain the key information of the text.

8. An information extraction device, characterized in that: include: An acquisition module, used for acquiring data to be processed, wherein the data to be processed includes image data and text data; An input module, used for inputting the to-be-processed data into a pre-trained key information extraction model, wherein the pre-trained key information extraction model comprises an image encoding layer and an image decoding layer; A parsing module, used for performing image feature parsing on the image data through the image coding layer to obtain an image feature vector; A mapping module, used for performing dictionary mapping on the text data through a cuckoo hash algorithm to obtain an information dictionary; A conversion module, used for performing feature conversion on the image feature vector through the image decoding layer to obtain a text feature vector; A decoding module is used to perform dictionary decoding on the text feature vector based on the information dictionary to obtain text key information.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the information extraction method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the information extraction method according to any one of claims 1 to 7 are implemented.

Images