Medical bill image processing method and device, electronic device, and storage medium

Abstract

The present invention relates to the field of artificial intelligence and smart medical care, and discloses a medical bill image processing method and device, electronic equipment, and storage medium, including: obtaining text position information and text content information of text in a target medical bill image; inputting the target medical bill image, the text content information, and the text position information into a self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; wherein the self-attention mechanism model is obtained by training a machine translation model based on a self-attention mechanism and a document understanding pre-training model using a medical bill image sample set containing a variety of layouts. Through the present invention, the technical problems in the related art such as low efficiency and low accuracy of the existing scheme for extracting information from medical bills due to the complexity of the layout types of medical bills are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and smart medicine, and in particular to a medical bill image processing method and device, electronic equipment, and storage medium. Background Art

[0002] With the development and progress of science and technology, and the increasing perfection of artificial intelligence technology, the automated recognition of medical invoices in commercial insurance claims and medical insurance reimbursement has become the mainstream direction in the industry. The automated recognition of medical bills effectively saves manpower costs and improves service efficiency.

[0003] In the current medical invoice recognition system, after text detection and text recognition based on deep learning, it is necessary to carry out customized development of data structuring according to different layouts in different regions. Different layouts need to be equipped with a set of different data post-processing methods. Because the current customized data structuring method has shown great disadvantages in many aspects such as data quantity, R&D manpower investment, annotation manpower investment, and R&D cycle. For example, in the absence of data, it is difficult to develop a system that matches the national layout; in order to cover the various layouts of medical bills launched in real time in various regions, a large amount of R&D manpower has been invested, resulting in high costs; in addition, when the traditional medical invoice recognition solution encounters problems such as unstable text position or image quality in the layout, the accuracy of data structuring is low.

[0004] With respect to the above-mentioned problems existing in the related technologies, no effective solutions have been found so far. Summary of the invention

[0005] The embodiments of the present invention provide a medical bill image processing method and device, electronic device, and storage medium, so as to at least solve the technical problems in the related art that the existing solutions for extracting information from medical bills have low efficiency and low accuracy due to the complex layout types of medical bills.

[0006] According to one embodiment of the present invention, a medical bill image processing method is provided, the method comprising: obtaining text position information and text content information of text in a target medical bill image; inputting the target medical bill image, the text content information and the text position information into a self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and a field type corresponding to each structured text field; wherein the self-attention mechanism model is obtained by training a machine translation model based on a self-attention mechanism and a document understanding pre-training model using a sample set of medical bill images containing multiple layouts.

[0007] Optionally, obtaining text position information and text content information of text in a target medical bill image includes: inputting the target medical bill image into a pre-trained text detection model for text box detection, determining at least one text box containing text in the target medical bill image, and marking the text box in the target medical bill image; inputting the marked target medical bill image into a pre-trained text recognition model for text recognition, and obtaining text position information and text content information corresponding to the text box.

[0008] Optionally, the text recognition model includes a convolutional neural network, a long short-term memory model, and a neural network-based time series classification model; the step of inputting the marked target medical bill image into a pre-trained text recognition model for feature learning to obtain text position information and text content information corresponding to the text box includes: inputting the marked target medical bill image into the convolutional neural network for feature extraction to obtain image convolution features corresponding to the marked target medical bill image; inputting the image convolution features into the long short-term memory model for feature extraction to obtain sequence features corresponding to the image convolution features; inputting the sequence features into the neural network-based time series classification model for text alignment to obtain text content information in the text box and text position information corresponding to each text content.

[0009] Optionally, before inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field, the method also includes: collecting the sample set of medical bill images containing multiple layouts; inputting the medical bill image sample set into a pre-trained text detection model and a pre-trained text recognition model in sequence to extract the text position information and text content information of the text in each medical bill image in the medical bill image sample set; inputting the medical bill image sample set and the text position information and text content information of the text in each medical bill image into a pre-trained model based on the self-attention mechanism for training to obtain the self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

[0010] Optionally, the step of inputting the target medical bill image, the text content information and the text position information into a self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field includes: inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model; using the machine translation model based on the self-attention mechanism to perform feature extraction on the target medical bill image, the text content information and the text position information to generate a feature combination vector corresponding to the target medical bill image; using the document understanding pre-training model to perform a pre-training task on the feature combination vector to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field.

[0011] Optionally, the use of the self-attention mechanism-based machine translation model to perform feature extraction on the target medical bill image, the text content information and the text position information to generate a feature combination vector corresponding to the target medical bill image includes: using the global average pooling principle and the linear projection principle to extract the graph feature vector corresponding to the target medical bill image; using the text splitting principle to extract the text content feature vector corresponding to the text content information; and, by constructing a coordinate system of the text box corresponding to the text position information, extracting the text position feature vector corresponding to the text position information and the relative position feature vector between different texts; and performing feature combination on the graph feature vector, the text content feature vector, the text position feature vector and the relative position feature vector to obtain the feature combination vector.

[0012] Optionally, the use of the document understanding pre-training model to perform a pre-training task on the feature combination vector to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field includes: inputting the feature combination vector into the document understanding pre-training model; obtaining matching information between the text content information and the target medical bill image by judging whether the text content information matches the target medical bill image; and / or obtaining text blackening judgment information by judging whether the text content information is blackened; and / or obtaining text occlusion judgment information by judging whether the text content information is obscured; performing feature fusion on at least one of the matching information, the text blackening judgment information and the text occlusion judgment information with the feature combination vector to obtain a feature fusion vector; obtaining a modal alignment relationship between the at least one structured text field and the field type of the at least one structured text field by learning the context of the target medical bill image and the feature fusion vector; and outputting the at least one structured text field and the field type corresponding to each structured text field.

[0013] According to one embodiment of the present invention, a medical bill image processing device is provided, comprising: an acquisition module, used to acquire text position information and text content information of text in a target medical bill image; a learning module, used to input the target medical bill image, the text content information and the text position information into a self-attention mechanism model for feature learning, to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; wherein the self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a medical bill image sample set containing multiple layouts.

[0014] Optionally, the acquisition module includes: a determination unit, used to input the target medical bill image into a pre-trained text detection model for text box detection, determine at least one text box containing text in the target medical bill image, and mark the text box in the target medical bill image; a first learning unit, used to input the marked target medical bill image into a pre-trained text recognition model for text recognition, and obtain text position information and text content information corresponding to the text box.

[0015] Optionally, the text recognition model includes a convolutional neural network, a long short-term memory model and a neural network-based time series classification model; the first learning unit includes: a first extraction subunit, used to input the marked target medical bill image into the convolutional neural network for feature extraction, and obtain the image convolution features corresponding to the marked target medical bill image; a second extraction subunit, used to input the image convolution features into the long short-term memory model for feature extraction, and obtain the sequence features corresponding to the image convolution features; a processing subunit, used to input the sequence features into the neural network-based time series classification model for text alignment, and obtain the text content information in the text box and the text position information corresponding to each text content.

[0016] Optionally, the device also includes: an acquisition module, which is used to collect the medical bill image sample set containing multiple layouts before the learning module inputs the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; an extraction module, which is used to input the medical bill image sample set into a pre-trained text detection model and a pre-trained text recognition model in sequence to extract the text position information and text content information of the text in each medical bill image in the medical bill image sample set; a training module, which is used to input the medical bill image sample set and the text position information and text content information of the text in each medical bill image into a pre-trained model based on the self-attention mechanism for training to obtain the self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

[0017] Optionally, the learning module includes: an input unit, used to input the target medical bill image, the text content information and the text position information into the self-attention mechanism model; a generation unit, used to use the machine translation model based on the self-attention mechanism to perform feature extraction on the target medical bill image, the text content information and the text position information, and generate a feature combination vector corresponding to the target medical bill image; an execution unit, used to use the document understanding pre-training model to perform a pre-training task on the feature combination vector, and obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field.

[0018] Optionally, the generation unit includes: a third extraction subunit, used to extract the graph feature vector corresponding to the target medical bill image by utilizing the global average pooling principle and the linear projection principle; to extract the text content feature vector corresponding to the text content information by utilizing the text segmentation principle; and to extract the text position feature vector corresponding to the text position information and the relative position feature vector between different texts by constructing a coordinate system of the text box corresponding to the text position information; a feature combination subunit, used to perform feature combination on the graph feature vector, the text content feature vector, the text position feature vector and the relative position feature vector to obtain the feature combination vector.

[0019] Optionally, the execution unit includes: an input subunit, used to input the feature combination vector into the document understanding pre-training model; a judgment subunit, used to obtain the matching information between the text content information and the target medical bill image by judging whether the text content information matches the target medical bill image; and / or, by judging whether the text content information is blackened, to obtain text blackening judgment information; and / or, by judging whether the text content information is blocked, to obtain text occlusion judgment information; a feature fusion subunit, used to perform feature fusion on at least one of the matching information, the text blackening judgment information and the text occlusion judgment information with the feature combination vector to obtain a feature fusion vector; a learning subunit, used to obtain the modal alignment relationship between the at least one structured text field and the field type of the at least one structured text field by learning the context of the target medical bill image and the feature fusion vector; an output subunit, used to output the at least one structured text field and the field type corresponding to each structured text field.

[0020] According to another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0021] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned device embodiments when running.

[0022] Through the present invention, medical bills of various layouts are used to train a machine translation model based on the self-attention mechanism and a document understanding pre-training model to generate a self-attention mechanism model; the medical bill image to be identified and the text content information and text position information in the image are input into the self-attention mechanism model for feature learning to obtain at least one structured text field and the field type corresponding to each structured text field, so that unstructured medical bill images of various layouts can be accurately converted into structured text fields, without considering customized development of data structuring for medical bills of different layouts; in addition, due to the highly parallelizable computing power, the self-attention mechanism can reduce the training time of the model, greatly improve the processing efficiency of data structuring, and solve the technical problems in the related prior art that the existing schemes for extracting information from medical bills are low in efficiency and low in accuracy due to the complex layout types of medical bills. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 It is a hardware structure block diagram of a medical bill image processing method provided by an embodiment of the present invention applied to a computer terminal;

[0025] Figure 2 is a flow chart of a medical bill image processing method provided according to an embodiment of the present invention;

[0026] Figure 3 is a structural block diagram of a medical bill image processing device provided according to an embodiment of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a server, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1FIG. 1 is a hardware structure diagram of a medical bill image processing method provided by an embodiment of the present invention applied to a computer terminal. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the medical bill image processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory, and may also include a volatile memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] Figure 2 is a flowchart of a medical bill image processing method provided according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0034] Step S202, obtaining text position information and text content information of the text in the target medical bill image;

[0035] Among them, medical bills refer to receipts issued by non-profit medical and health institutions when they provide medical services to patients in outpatient, emergency, first aid, hospitalization, physical examination, etc. and obtain medical income. They are also called "medical fee bills". The layout types of medical bills in different regions are different. Medical bills mainly include outpatient fee bills and inpatient fee bills; for example, the basic content of inpatient fee bills includes bill name, bill code, business serial number, hospital type, billing time, name, gender, medical insurance type, medical insurance payment method, social security number, item, amount, total, prepaid amount, supplementary payment amount, refund amount, medical insurance pooling payment, personal account payment, other medical insurance payment, self-payment, collection unit, payee and other information.

[0036] In an optional scheme in the present case, the target medical bill image is input into a pre-trained text detection model for text box detection, at least one text box containing text in the target medical bill image is determined, and the text box is marked in the target medical bill image; the marked target medical bill image is then input into a pre-trained text recognition model for text recognition, and the text position information and text content information corresponding to the text box are obtained.

[0037] Preferably, the above text detection model uses the DBnet algorithm (full name Differentiable Binarization (DB), differentiable binarization algorithm). Specifically, using the DBnet algorithm to perform text detection on the target medical bill image includes: inputting the target medical bill image into the feature pyramid backbone for feature extraction; upsampling the pyramid features to the same size F, predicting the probability map P and the threshold map T based on the feature map F, and calculating the approximate binary map using the feature map F and the threshold map T; in the training stage, supervising the probability map, the threshold map and the approximate binary map, wherein the probability map and the approximate binary map share a common supervision; in the inference process, through the box (box) formulation module, the text bounding box is obtained from the approximate binary map or the probability map, so as to accurately locate the text box containing the text in the medical bill.

[0038] Preferably, the above-mentioned text recognition model includes a convolutional neural network, a long short-term memory model, and a time series classification model based on a neural network. Specifically, the marked target medical bill image is input into the convolutional neural network for feature extraction to obtain the image convolution features corresponding to the marked target medical bill image; the image convolution features are input into the long short-term memory model for feature extraction to obtain the sequence features corresponding to the image convolution features; the sequence features are input into the time series classification model based on a neural network for text alignment to obtain the text content information in the text box and the text position information corresponding to each text content.

[0039] In an optional example of the above embodiment, the above text recognition model uses the CRNN+CTC algorithm, namely CNN (full name Convolutional Neural Networks) + RNN (full name recurrent neural network) + CTC (full name Connectionist temporal classification, i.e. a temporal classification model based on a neural network).

[0040] First, determine the size of the input medical bill image marked with a text box (for example, 32, 100, 3), (height, width, channel) format, that is, (height, width, channel);

[0041] Next, the convlutional feature maps of the input image are extracted using convolutional layers, and the image size (32,100,3) is converted to a convolutional feature matrix of size (1,25,512). Furthermore, the image is scaled to [32*W*3] with a fixed aspect ratio ([W] represents any width). After passing through CNN, it becomes [1*(W / 4)*512]. For LSTM, [T=(W / 4)] is set to input the features into LSTM.

[0042] Furthermore, Recurrent layers are used to extract text sequence features based on convolutional features; among them, Recurrent layers is a deep bidirectional LSTM network.

[0043] Then, the text sequence features are output using Transcription layers.

[0044] In addition, the embodiment of the present invention introduces CTC to replace the common Softmax Loss, and the training samples do not need to be aligned; by introducing blank characters, the problem of no characters in some positions can be solved.

[0045] According to the above example, the text content information and text position information in the text box are output based on the above CNN+RNN+CTC algorithm.

[0046] In another optional embodiment of the present invention, the data of the above text content and text location can be acquired and processed based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0047] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] Step S204, inputting the target medical bill image, text content information and text position information into the self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and the field type corresponding to each structured text field;

[0049] Among them, the self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a sample set of medical bill images containing various layouts.

[0050] In this embodiment, the document understanding pre-training model preferably uses the LayoutLM2 model. LayoutLM2 is a new generation of multimodal document understanding pre-training model. It directly introduces image information in the input stage and uses the multimodal pre-training framework to jointly learn text, pictures and text positions. It can learn the local invariance information of different document template types. When the model needs to be migrated to another template type, the text in the template can be structured by annotating a small number of samples, thereby extracting the structured text fields in the medical bills and the field types of the structured text fields.

[0051] In an optional embodiment of the present case, before obtaining at least one structured text field in the target medical bill image and the field type corresponding to each structured text field, a sample set of medical bill images containing multiple layouts is collected; the sample set of medical bill images is sequentially input into a pre-trained text detection model and a pre-trained text recognition model to extract text position information and text content information of the text in each medical bill image in the sample set of medical bill images; the sample set of medical bill images and the text position information and text content information of the text in each medical bill image are input into a pre-trained model based on a self-attention mechanism for training to obtain a self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

[0052] Through the embodiment of the present invention, a machine translation model based on a self-attention mechanism and a document understanding pre-training model are trained using medical bills of various layouts to generate a self-attention mechanism model; the medical bill image to be identified and the text content information and text position information in the image are input into the self-attention mechanism model for feature learning to obtain at least one structured text field and the field type corresponding to each structured text field, so that unstructured medical bill images of various layouts can be accurately converted into structured text fields, without considering customized development of data structuring for medical bills of different layouts; in addition, due to the highly parallelizable computing power, the self-attention mechanism can reduce the training time of the model, greatly improve the processing efficiency of data structuring, and solve the technical problems of low efficiency and low accuracy in the existing schemes for extracting information from medical bills due to the complex layout types of medical bills in the related prior art.

[0053] In an optional embodiment of the present case, the above step S204 specifically includes: inputting the target medical bill image, text content information and text position information into the self-attention mechanism model; using a machine translation model based on the self-attention mechanism to perform feature extraction on the target medical bill image, text content information and text position information, and generate a feature fusion vector corresponding to the target medical bill image; using a document understanding pre-training model to perform a pre-training task on the feature combination vector to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field.

[0054] In this embodiment, the text content information, text position information and target medical bill image are input into the self-attention mechanism model for feature extraction. The self-attention mechanism model can be trained by a machine translation model based on the self-attention mechanism and a document understanding pre-training model. In this embodiment, the machine translation model transformer model is constructed based on the self-attention mechanism, and the distance between any two positions in the sequence is reduced to a constant; the transformer model is applied to the post-processing process of automatic recognition of medical bill images, and a data structuring model (i.e., the above-mentioned self-attention mechanism model) is obtained after training, so that the results of text detection and text recognition are structured through the data structuring model, and finally structured medical bill data is obtained; the document understanding pre-training model is used to learn the local invariance information in bills of different layouts, and the structured text field marked with the field type is output, so that the unstructured information in the bill image is converted into structured information.

[0055] Furthermore, in an optional scheme of an embodiment of the present invention, the global average pooling principle and the linear projection principle are used to extract the graph feature vector corresponding to the target medical bill image; the text content feature vector corresponding to the text content information is extracted using the text segmentation principle; and the text position feature vector corresponding to the text position information and the relative position feature vector between different texts are extracted by constructing a coordinate system of a text box corresponding to the text position information; the graph feature vector, the text content feature vector, the text position feature vector and the relative position feature vector are feature combined to obtain a feature combination vector.

[0056] In this embodiment, the transformer model is used to extract features of text content information, text position information (layout), and medical bill images, and convert them into corresponding text content feature vectors, text position feature vectors, graph feature vectors, and relative position feature vectors (i.e., the relative position relationship between different text blocks in the image), and each feature vector is submitted to the encoder network in the transformer model to splice the text content feature vector, text position feature vector, graph feature vector, and relative position feature vector, that is, to splice the image and text sequences to obtain a feature combination vector.

[0057] In a possible implementation of the present case, the extraction of the above-mentioned text content feature vector includes: using a text segmentation Wordpiece to segment the text content; using [CLS] and [SEP] to add tags, and using [PAD] to pad the length to obtain a text input sequence; based on the text input sequence, combining the word vector, the one-dimensional position vector, and the segment vector to obtain a text content feature vector.

[0058] In a possible implementation of the present case, the extraction of the above-mentioned graph feature vector includes: extracting the feature graph of the target medical bill image; performing average pooling on the feature graph to a fixed size (W*H); expanding the feature graph after average pooling by rows; obtaining the graph feature sequence corresponding to the image through linear projection; and adding the graph feature sequence, the one-dimensional position vector, and the segmentation vector to obtain the final graph feature vector.

[0059] In a possible implementation of the present case, the extraction of the above-mentioned text position feature vector includes: constructing a coordinate system of the text box parallel to the text position information; using the four boundary coordinate values, width, and height of the text box to represent the position of the text box and the relative position between different text blocks, and finally outputting the layout information of the text, namely the text position feature vector and the relative position feature vector.

[0060] Furthermore, the above-mentioned feature combination vector is input into the encoder network, and the encoder network is used to: (1) encode the image or text content; (2) encode the 2D two-dimensional position information; (3) encode the 1D one-dimensional position information; (4) encode the image classification to confirm the classification of the information segment, such as the image information is C and the text content is A.

[0061] In an optional embodiment of the present case, the above-mentioned document understanding pre-training model is used to perform a pre-training task on the feature combination vector to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field, specifically including: inputting the feature combination vector into the document understanding pre-training model; by judging whether the text content information matches the target medical bill image, obtaining the matching information between the text content information and the target medical bill image; and / or, by judging whether the text content information is blackened, obtaining the text blackening judgment information; and / or, by judging whether the text content information is blocked, obtaining the text occlusion judgment information; performing feature fusion on at least one of the matching information, the text blackening judgment information and the text occlusion judgment information with the feature combination vector to obtain a feature fusion vector; by learning the context of the target medical bill image and the feature fusion vector, obtaining the modal alignment relationship between at least one structured text field and the field type of at least one structured text field; and outputting at least one structured text field and the field type corresponding to each structured text field.

[0062] In this embodiment, after the image feature vector, the text content feature vector, the text position feature vector and the relative position feature vector are spliced ​​to obtain the feature combination vector, the document understanding pre-training model is used to perform a pre-training task on the combination vector, specifically including: whether the image matches the text information, whether the text content is blackened and whether the text content is blocked, to obtain the fused image pixel information, text position, content information, and text-related features; and the matching information of the text information and the image information, the text blackening judgment information and the occluded text prediction information are at least one of the fused feature vectors to obtain the feature fusion vector of each sample; and then based on the self-supervised learning method, the previous word order is used to predict the probability of occurrence of the next word to learn the context-related representation, and for blocked sentences or words, disrupted word orders, etc., the original data can be reconstructed to improve the text recognition accuracy.

[0063] For example, in an example of text-image alignment, a portion of the text is randomly covered in lines on the image, and the text output of the model is used to perform word-level binary classification to predict whether each word is covered, so as to align the position information of the text and the image. For example, if the content T1 of the field Text1 is T1 and the position boxT1 is T1, the output should be True (correct), and if the input is T1, boxT3, the output should be False (wrong).

[0064] In an example of text-image matching, the model’s document-level binary classification method is used to predict whether the image and text match in order to align the content information of the text and image.

[0065] In another example, in order to determine whether the image pixel information and text content information are correct, the features that match the image pixel information and content are slowly trained; the weight of the image pixel information and the matching text information after the transformer will be large, otherwise it will be small.

[0066] Furthermore, the feature fusion vector is input into LayoutLM2. The LayoutLM2 model learns the modal alignment relationship between text position and text semantics (i.e., field type) based on the context of the bill image, the feature fusion vector, and the inferred occluded vocabulary, and then obtains the modal alignment relationship between the text field and the field type, and obtains the structured text field with field type (i.e., structured information), such as "April 19, 2021" is marked as the date of consultation, "0013853632" is marked as the invoice number, etc. Through the pre-training task of the text field, the classification of each text segment (i.e., the final structured data) can be finally obtained.

[0067] This solution can also extract structured text fields such as name, price, quantity, etc. from medical bill images based on pre-defined key information entities (such as name, price, quantity, etc.).

[0068] The existing medical invoice recognition solution requires the redesign of post-processing for each invoice layout, that is, each set of invoices needs to be equipped with a set of post-processing algorithms. The disadvantage of this method is that the layout information of the medical invoice must be clear before subsequent development can be carried out. Then, due to the situation that the medical invoice version of a certain region cannot be obtained, that is, the training samples cannot be obtained, and development is more difficult without training samples. Through this solution, the LayoutLM2 model is used to model the traditional post-processing part, and the layout is no longer distinguished. There is no need to match a set of post-processing for different layouts. One model is adapted to all regional layouts, solving the problem of medical invoice recognition in layouts across the country, saving R&D manpower, and improving development timeliness.

[0069] Based on the above embodiments, this case proposes a method for medical invoice recognition based on the self-attention mechanism, which replaces the traditional post-processing development part with a modeling method, and uses a unified model to adapt to medical invoices of various layouts across the country; the update iteration is fast, and each region will promote a new version of the medical bill, which can quickly adapt to the new layout without the need for additional R&D investment, greatly saving R&D manpower and improving development timeliness.

[0070] This solution applies the transformer algorithm in natural language processing to the field of visual algorithms to solve the problem of text classification in OCR post-processing, reduce errors caused by text position changes and image quality in traditional post-processing methods, and improve the accuracy of data structuring.

[0071] Based on the medical bill image processing method provided in each of the above embodiments, based on the same inventive concept, a medical bill image processing device is also provided in this embodiment, which is used to implement the above embodiments and preferred implementations, and will not be repeated for what has been described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0072] Figure 3 is a structural block diagram of a medical bill image processing device provided according to an embodiment of the present invention. Figure 3As shown, the device includes: an acquisition module 30, used to obtain text position information and text content information of the text in the target medical bill image; a learning module, used to input the target medical bill image, text content information and text position information into the self-attention mechanism model for feature learning, and obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; wherein the self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a medical bill image sample set containing multiple layouts.

[0073] Optionally, the acquisition module 30 includes: a determination unit, used to input the target medical bill image into a pre-trained text detection model for text box detection, determine at least one text box containing text in the target medical bill image, and mark the text box in the target medical bill image; a first learning unit, used to input the marked target medical bill image into a pre-trained text recognition model for text recognition, and obtain text position information and text content information corresponding to the text box.

[0074] Optionally, the text recognition model includes a convolutional neural network, a long short-term memory model, and a time series classification model based on a neural network; the first learning unit includes: a first extraction subunit, used to input the marked target medical bill image into the convolutional neural network for feature extraction, and obtain the image convolution features corresponding to the marked target medical bill image; a second extraction subunit, used to input the image convolution features into the long short-term memory model for feature extraction, and obtain the sequence features corresponding to the image convolution features; a processing subunit, used to input the sequence features into the time series classification model based on a neural network for text alignment, and obtain the text content information in the text box and the text position information corresponding to each text content.

[0075] Optionally, the above-mentioned device also includes: an acquisition module, which is used to collect a sample set of medical bill images containing multiple layouts before the learning module inputs the target medical bill image, text content information and text position information into the self-attention mechanism model for feature learning to obtain at least one text field with a classification identifier; an extraction module, which is used to input the medical bill image sample set into a pre-trained text detection model and a pre-trained text recognition model in sequence, and extract the text position information and text content information of the text in each medical bill image in the medical bill image sample set; a training module, which is used to input the medical bill image sample set and the text position information and text content information of the text in each medical bill image into a pre-trained model based on the self-attention mechanism for training to obtain a self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

[0076] Optionally, the learning module 32 includes: an input unit, used to input the target medical bill image, text content information and text position information into the self-attention mechanism model; a generation unit, used to use a machine translation model based on the self-attention mechanism to perform feature extraction on the target medical bill image, text content information and text position information, and generate a feature combination vector corresponding to the target medical bill image; an execution unit, used to perform a pre-training task on the feature combination vector using a document understanding pre-training model, and obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field.

[0077] Optionally, the generation unit includes: a third extraction subunit, used to extract the graph feature vector corresponding to the target medical bill image using the global average pooling principle and the linear projection principle; to extract the text content feature vector corresponding to the text content information using the text segmentation principle; and to extract the text position feature vector corresponding to the text position information and the relative position feature vector between different texts by constructing a coordinate system of the text box corresponding to the text position information; a feature combination subunit, used to perform feature combination on the graph feature vector, the text content feature vector, the text position feature vector and the relative position feature vector to obtain a feature combination vector.

[0078] Optionally, the execution unit includes: an input subunit, used to input the feature combination vector into the document understanding pre-training model; a judgment subunit, used to obtain matching information between the text content information and the target medical bill image by judging whether the text content information matches the target medical bill image; and / or, by judging whether the text content information is blackened, to obtain text blackening judgment information; and / or, by judging whether the text content information is occluded, to obtain text occlusion judgment information; a feature fusion subunit, used to perform feature fusion on at least one of the matching information, text blackening judgment information and text occlusion judgment information with the feature combination vector to obtain a feature fusion vector; a learning subunit, used to obtain a modal alignment relationship between at least one structured text field and the field type of at least one structured text field by learning the context of the target medical bill image and the feature fusion vector; an output subunit, used to output at least one structured text field and the field type corresponding to each structured text field.

[0079] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0080] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0081] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0082] S1, obtaining text position information and text content information of the text in the target medical bill image;

[0083] S2, inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; wherein the self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a sample set of medical bill images containing multiple layouts.

[0084] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0085] Based on the above Figure 2 The method shown and Figure 3 In order to achieve the above-mentioned purpose, the present application also provides an electronic device, such as Figure 4 As shown, the system includes a memory 42 and a processor 41, wherein the memory 42 and the processor 41 are both arranged on a bus 43. The memory 42 stores a computer program, and the processor 41 executes the computer program to realize Figure 2 The medical bill image processing method shown.

[0086] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a memory (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0087] Optionally, the device may also be connected to a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0088] Those skilled in the art will appreciate that the structure of an electronic device provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0089] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0090] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical bill image processing method, characterized in that: include: Obtaining text position information and text content information of the text in the target medical bill image; Inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; The self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a sample set of medical bill images containing various layouts; Before inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field, the method further includes: Collecting the sample set of medical bill images containing multiple layouts; Inputting the medical bill image sample set into a pre-trained text detection model and a pre-trained text recognition model in sequence, extracting text position information and text content information of the text in each medical bill image in the medical bill image sample set; The medical bill image sample set and the text position information and text content information of the text in each medical bill image are input into a pre-trained model based on a self-attention mechanism for training to obtain the self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

2. The method according to claim 1, characterized in that: The step of obtaining the text position information and text content information of the text in the target medical bill image includes: Inputting the target medical bill image into a pre-trained text detection model to perform text box detection, determining at least one text box containing text in the target medical bill image, and marking the text box in the target medical bill image; The marked target medical bill image is input into a pre-trained text recognition model for text recognition to obtain text position information and text content information corresponding to the text box.

3. The method according to claim 2, characterized in that The text recognition model includes a convolutional neural network, a long short-term memory model, and a time series classification model based on a neural network; the marked target medical bill image is input into a pre-trained text recognition model for feature learning, and the text position information and text content information corresponding to the text box are obtained, including: Inputting the marked target medical bill image into the convolutional neural network for feature extraction to obtain image convolution features corresponding to the marked target medical bill image; Inputting the image convolution feature into the long short-term memory model for feature extraction to obtain sequence features corresponding to the image convolution feature; The sequence features are input into the neural network-based time series classification model for text alignment to obtain text content information in the text box and text position information corresponding to each text content.

4. The method according to claim 1, characterized in that: The step of inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field includes: Inputting the target medical bill image, the text content information and the text position information into the self-attention mechanism model; Using the machine translation model based on the self-attention mechanism to perform feature extraction on the target medical bill image, the text content information, and the text position information, to generate a feature combination vector corresponding to the target medical bill image; The document understanding pre-training model is used to perform a pre-training task on the feature combination vector to obtain at least one structured text field in the target medical bill image and a field type corresponding to each structured text field.

5. The method according to claim 4, characterized in that The using of the machine translation model based on the self-attention mechanism to extract features from the target medical bill image, the text content information, and the text position information to generate a feature combination vector corresponding to the target medical bill image includes: The image feature vector corresponding to the target medical bill image is extracted by using the global average pooling principle and the linear projection principle; the text content feature vector corresponding to the text content information is extracted by using the text segmentation principle; and the text position feature vector corresponding to the text position information and the relative position feature vectors between different texts are extracted by constructing a coordinate system of the text box corresponding to the text position information; The image feature vector, the text content feature vector, the text position feature vector and the relative position feature vector are feature combined to obtain the feature combination vector.

6. The method according to claim 4, characterized in that The pre-training task of using the document understanding pre-training model to pre-train the feature combination vector to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field includes: Inputting the feature combination vector into the document understanding pre-training model; By judging whether the text content information matches the target medical bill image, the matching information of the text content information and the target medical bill image is obtained; and / or, by judging whether the text content information is blackened, the text blackening judgment information is obtained; and / or, by judging whether the text content information is blocked, the text blocking judgment information is obtained; Performing feature fusion on at least one of the matching information, the text blackening judgment information and the text occlusion judgment information with the feature combination vector to obtain a feature fusion vector; By learning the context of the target medical bill image and the feature fusion vector, a modal alignment relationship between the at least one structured text field and the field type of the at least one structured text field is obtained; The at least one structured text field and a field type corresponding to each structured text field are output.

7. A medical bill image processing device, characterized in that: include: An acquisition module, used to acquire text position information and text content information of text in a target medical bill image; A learning module, used for inputting the target medical bill image, the text content information and the text position information into a self-attention mechanism model for feature learning, and obtaining at least one structured text field in the target medical bill image and a field type corresponding to each structured text field; The self-attention mechanism model is obtained by training a machine translation model based on the self-attention mechanism and a document understanding pre-training model using a sample set of medical bill images containing various layouts; The device also includes: an acquisition module, which is used to acquire the sample set of medical bill images containing multiple layouts before the learning module inputs the target medical bill image, the text content information and the text position information into the self-attention mechanism model for feature learning to obtain at least one structured text field in the target medical bill image and the field type corresponding to each structured text field; an extraction module, which is used to input the sample set of medical bill images into a pre-trained text detection model and a pre-trained text recognition model in sequence to extract the text position information and text content information of the text in each medical bill image in the sample set of medical bill images; a training module, which is used to input the sample set of medical bill images and the text position information and text content information of the text in each medical bill image into a pre-trained model based on the self-attention mechanism for training to obtain the self-attention mechanism model; wherein the pre-trained model based on the self-attention mechanism is obtained by sequentially connecting a preset machine translation model and a preset document understanding pre-trained model.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images