Method, apparatus and medium for analyzing medical document image

Abstract

The application relates to the fields of artificial intelligence and medical health, and discloses a medical document image analysis method, device, equipment and medium, which comprises the following steps: acquiring a medical document image and demand information corresponding to the medical document image; if the demand information comprises image segmentation demand information, processing the medical document image and the image segmentation demand information through an image segmentation model to obtain candidate document images corresponding to the image segmentation demand information; if the demand information does not comprise image segmentation demand information, performing document image segmentation on the medical document image through a pre-trained image segmentation model to obtain candidate document images from which the background of the medical document image is removed; and processing the candidate document images and text demand information in the demand information through an information extraction model to obtain target information corresponding to the text demand information. The candidate document images are free of noise, and the accuracy of the target information extracted from the candidate document images is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and healthcare, and in particular to a method, apparatus, device, and medium for analyzing medical document images. Background Technology

[0002] Documents are the most important data carriers in today's society. Each document carries important business or personal data. With the development of digitalization, more and more paper documents are being converted into digital images, that is, document images as information carriers. Intelligent analysis of document images is crucial for many fields that use document images. However, the clarity of some photographed document images is limited, affecting the analysis results.

[0003] For example, in the healthcare field, storing, retrieving, and parsing images of medical documents such as examination reports and medical records is much more convenient than using paper documents, and it also avoids the loss of medical document images. Since some medical document images are only available in paper form, such as doctor-written medical records, these images can be converted from paper documents and saved as images. However, the clarity of these images and the background often pose significant challenges to the parsing process, affecting the accuracy of the analysis. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for analyzing medical document images to address the current problem of low accuracy in resolving medical document images.

[0005] In a first aspect, embodiments of the present invention provide a method for analyzing medical document images, the method comprising:

[0006] Acquire medical document images and the corresponding requirement information for the medical document images;

[0007] If the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed by a pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information;

[0008] If the requirement information does not include image segmentation requirement information, then the medical document image is segmented using a pre-trained image segmentation model to obtain candidate document images from which the background has been removed.

[0009] The candidate document image and the textual demand information in the demand information are processed by a pre-trained information extraction model to obtain target information corresponding to the textual demand information, wherein the target information includes target text and / or target document image.

[0010] Secondly, embodiments of the present invention also provide a medical document image analysis device, the device comprising:

[0011] An information acquisition module is used to acquire medical document images and corresponding demand information, wherein each medical document image includes at least one medical text page;

[0012] The first image segmentation module is used to process the medical document image and the image segmentation requirement information through a pre-trained image segmentation model if the requirement information includes image segmentation requirement information, so as to obtain a candidate document image corresponding to the image segmentation requirement information.

[0013] The second image segmentation module is used to extract the document image from the medical document image using a pre-trained image segmentation model if the requirement information does not include image segmentation requirement information, thereby obtaining a candidate document image from the medical document image after removing the background.

[0014] The target information acquisition module is used to process the candidate document image and the textual demand information in the demand information through a pre-trained information extraction model to obtain target information corresponding to the textual demand information, wherein the target information includes target text and / or target document image.

[0015] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above method.

[0017] In the above-mentioned medical document image analysis method, device, equipment and medium, the medical document image and the corresponding requirement information can be obtained through the client. If the requirement information includes image segmentation requirement information, the medical document image and the image segmentation requirement information are processed by the pre-trained image segmentation model to obtain the candidate document image corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirement information, a pre-trained image segmentation model is used to segment the medical document image, obtaining candidate document images from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document images and the text requirement information in the requirement information to obtain target information corresponding to the text requirement information. This target information is then fed back to the client. In this invention, the following steps are achieved: acquiring medical document images and requirement information; obtaining candidate document images corresponding to the image segmentation requirement information when the requirement information includes it; and obtaining candidate document images with the background removed when the requirement information does not include it. The pre-trained information extraction model processes the candidate document images and the text requirement information in the requirement information to obtain target information. By segmenting the medical document image or removing the background, noise can be removed from the obtained candidate document images, thereby improving the accuracy of target information acquisition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the application environment of a medical document image analysis method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a method for analyzing medical document images according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a medical document image analysis device according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0023] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The present invention provides a method for analyzing medical document images, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive medical document images and corresponding requirement information from the client. If the requirement information includes image segmentation requirements, a pre-trained image segmentation model is used to process the medical document image and the image segmentation requirement information to obtain candidate document images corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirements, the pre-trained image segmentation model is used to segment the medical document image to obtain candidate document images from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document images and the text requirements in the requirement information to obtain target information corresponding to the text requirements information, which is then fed back to the client.

[0026] Medical data, such as medical records filled out by doctors, can be converted from paper documents into images and saved as medical document images by taking photos. However, these images often present significant challenges to analysis due to factors such as image clarity and background. This invention achieves the acquisition of medical document images and requirement information. When the requirement information includes image segmentation requirements, candidate document images corresponding to the image segmentation information are obtained. If the requirement information does not include image segmentation requirements, candidate document images with background removed are obtained. A pre-trained information extraction model processes the textual requirement information in the candidate document images and requirement information to obtain the target information. This is then used for medical document image segmentation or background removal, resulting in noise-free candidate document images. Finally, the target information is obtained from the candidate document images, improving the accuracy of target information acquisition.

[0027] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0028] Please see Figure 2 As shown, Figure 2A flowchart illustrating a method for analyzing medical document images provided in an embodiment of the present invention includes the following steps:

[0029] S110. Obtain medical document images and corresponding requirement information for the medical document images.

[0030] The medical document images include medical records, physical examination reports, etc. The requirements information can include image segmentation requirements for the medical document images, or text requirements. Image segmentation requirements refer to information for segmenting the medical document images. Text requirements include obtaining the text in the medical document images and the corresponding solutions. For example, if the medical document images include two documents, left and right, the image segmentation requirements include obtaining the image of the left document. The text requirements information can include extracting disease features from candidate document images and obtaining corresponding solutions based on these features.

[0031] In this embodiment of the invention, medical document images and corresponding demand information are obtained, so that subsequent steps can process the medical document images according to the demand information to obtain target information.

[0032] S120. If the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed by a pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information.

[0033] The image segmentation model is pre-trained and is used to segment medical document images to obtain candidate document images.

[0034] Specifically, if the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed using an image segmentation model to obtain candidate document images corresponding to the image segmentation requirement information. In this embodiment of the invention, medical document images can be segmented based on the image segmentation requirement information. The image segmentation requirement information can be obtained from the client, making the obtained candidate document images more targeted and improving the user experience.

[0035] S130. If the requirement information does not include image segmentation requirement information, then the medical document image is segmented using a pre-trained image segmentation model to obtain candidate document images from which the background has been removed.

[0036] By segmenting medical document images using an image segmentation model, background images are removed, resulting in candidate document images with background removed. This process eliminates noise in the candidate document images, leading to more accurate extraction of target information in subsequent steps.

[0037] S140. Using a pre-trained information extraction model, the candidate document image and the textual demand information in the demand information are processed to obtain target information corresponding to the textual demand information.

[0038] The information extraction model is pre-trained and can extract target information from candidate document images. Target information includes target text and / or target document images. For example, the target text includes symptom features and corresponding answer information. For instance, the target document image includes text information obtained from a medical document image based on the text requirement information, and then the target document image is obtained based on the document format and text information of the medical document image. For example, if the target information is to label the symptoms in the medical document image, the text information is first the symptoms, and then the target document image is obtained based on the document format and the location of the symptoms in the medical document image.

[0039] Specifically, through the information extraction model, the textual demand information in the candidate document images and demand information is processed to obtain the target information corresponding to the textual demand information. Since noise is removed or image segmentation is performed on the medical document images, the obtained candidate document images are more targeted. Then, the target information is extracted from the candidate document images, making the target information more accurate.

[0040] The technical solution of this invention involves acquiring a medical document image and corresponding requirement information. If the requirement information includes image segmentation requirement information, a pre-trained image segmentation model is used to process the medical document image and the image segmentation requirement information to obtain a candidate document image corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirement information, a pre-trained image segmentation model is used to segment the medical document image to obtain a candidate document image from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document image and the text requirement in the requirement information to obtain target information corresponding to the text requirement information. In this invention, medical document images and demand information are acquired. When the demand information includes image segmentation demand information, candidate document images corresponding to the image segmentation information are obtained. If the demand information does not include image segmentation demand information, candidate document images with background removed are obtained. Through a pre-trained information extraction model, the text demand information in the candidate document images and demand information is processed to obtain target information. Then, the medical document images are segmented or the background is removed, so that the obtained candidate document images can be free of noise. In this way, the target information is obtained from the candidate document images, improving the accuracy of target information acquisition.

[0041] In another embodiment of the invention, the image segmentation model includes a first encoder and a first decoder; the step of processing the medical document image and the image segmentation requirement information using the pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information includes: converting the medical document image into a first image vector using the first encoder; converting the image segmentation requirement information into a segmentation text vector; and segmenting the first image vector based on the segmentation text vector using the first decoder to obtain a candidate document image corresponding to the image segmentation requirement information.

[0042] The system comprises a first encoder for converting a medical document image into a first image vector, and a first decoder for segmenting the first image vector based on the segmented text vector to obtain candidate document images. It should be noted that the specific models of the first encoder and first decoder are not limited; for example, the first encoder could be a convolutional neural network, or a model for converting a medical document image into an embedded vector. The first decoder could be a mask decoder.

[0043] In this embodiment of the invention, a medical document image is converted into a first image vector by a first encoder, and image segmentation requirement information is converted into segmentation text vector. Then, a first decoder segments the first image vector based on the segmentation text vector to obtain candidate document images. Since the image segmentation requirement information can be set as needed, the flexibility of obtaining candidate document images is improved.

[0044] For example, the first encoder includes a cue word encoder and an image encoder. The image encoder extracts the medical document image into an embedded vector, i.e., the first image vector. The cue word encoder can employ sparse coding and dense coding. The cue word encoder is used to convert image segmentation requirement information into segmented text vectors. Optionally, the cue word encoder and image encoder in this embodiment of the invention can refer to the cue word encoder and image encoder in the Segment Anything Model (SAM). For example, the form of image segmentation requirement information includes coordinate points, coordinate regions, or text of the medical document image. For example, for a 500*500 pixel image, if the image segmentation requirement information is a coordinate point of 200*300, the cue word encoder uses this coordinate point as a cue word to convert the coordinates into the corresponding segmented text vector.

[0045] In another embodiment of the invention, the information extraction model includes a second encoder and a second decoder. The text information extraction requirement information includes at least one of obtaining at least one target text in a candidate document image, determining the position of the target text, and obtaining answer information corresponding to the target text. The answer information refers to information retrieved from a preset database that corresponds to the target text. The step of processing the candidate document image and the text requirement information in the requirement information through the pre-trained information extraction model to obtain target information corresponding to the text requirement information includes: performing vector transformation on the candidate document image and the text requirement information respectively through the second encoder to obtain a candidate document image vector and a text requirement information vector, concatenating the candidate document image vector and the text requirement information vector to obtain a target vector; and extracting target text and / or target document image from the target vector through the second decoder to obtain target information corresponding to the text requirement information.

[0046] The specific models of the second encoder and the second decoder are not limited. For example, the second encoder may include a convolutional neural network model, which can convert candidate document images and text requirement information into vectors respectively, and then concatenate the vectors to obtain the target vector. The second decoder may also be a convolutional neural network model, which acquires the target text and / or the target document image.

[0047] In this embodiment of the invention, a second encoder encodes the candidate document image and text requirement information respectively, obtaining a candidate document image vector and a text requirement information vector corresponding to the candidate document image. These vectors are then concatenated to obtain the target vector. Optionally, the candidate document image and text requirement information can be sequentially input into the second encoder, which outputs the concatenated target vector. A second decoder extracts the target text and / or target document image from the target vector to obtain the target information corresponding to the text requirement information. By using the second encoder and second decoder to process the candidate text image and text requirement information to obtain the target information, the accuracy of target information acquisition is improved, manual intervention is eliminated, and work efficiency is increased.

[0048] In another embodiment of the invention, the second decoder includes a text decoder and a layout decoder; the step of extracting target text and / or target document image from the target vector using the second decoder to obtain target information corresponding to the text requirement information includes: extracting text from the target vector using the text decoder to obtain at least one target text; if the text requirement information includes determining the position information of the target text, obtaining the position information of the target text using the layout decoder; marking the target text in the candidate document image based on the position information to obtain a target document image including the marked target text; if the text requirement information includes obtaining the answer information of the target text, retrieving the answer information corresponding to the target text from the database according to the target text, wherein the target information includes the answer information.

[0049] It should be understood that answer information is also a type of text information; that is, the target text in the target information can also refer to answer information.

[0050] Specifically, a text decoder extracts text from the target vector to obtain at least one target text. If the text requirement information includes determining the location information of the target text, a layout decoder can be used to obtain the location information of the target text. Then, based on the location information, the target text in the candidate document image is marked to obtain a target document image including the marked target text. If the text requirement information includes obtaining the answer information of the target text, the answer information corresponding to the target text is retrieved from the database based on the target text. This realizes the acquisition of the target text of the target vector, the determination of the location information of the target text, and the acquisition of the final answer information. This allows users to obtain the answer information from the candidate document image when they need it, thus improving the user experience.

[0051] In another embodiment of the invention, when the candidate document images include multiple images, the step of performing vector transformation on the candidate document images and the text requirement information respectively through the second encoder to obtain candidate document image vectors and text requirement information vectors, and then concatenating the candidate document image vectors and the text requirement information vectors to obtain a target vector, includes: obtaining text semantic information and page information of each candidate document image, wherein the page information includes page number information of the candidate document image; determining whether the candidate document images are related based on the text semantic information and the page information; if there are at least two candidate document images that are related, concatenating the related candidate document images based on the relationship to obtain a target candidate document image; and performing vector transformation on the target candidate document image and the text requirement information respectively through the second encoder to obtain target candidate document image vectors and text requirement information vectors, and then concatenating the target candidate document image vectors and the text requirement information vectors to obtain a target vector.

[0052] Specifically, the text semantic information and page information of each candidate document image are obtained. Based on the text semantic information and page information, it is determined whether there is a relationship between the candidate document images. If there is, the candidate document images with the relationship are stitched together to obtain the target candidate document image. The target candidate document image and the text requirement information are vectorized by the second encoder respectively. The target candidate document image vector and the text requirement information vector are stitched together to obtain the target vector. This realizes the stitching of medical document images with the relationship. In this way, when obtaining target information, it can be obtained from multiple medical document images, making the target information more accurate and complete.

[0053] In another embodiment of the invention, the training step of the image segmentation model includes: acquiring a first training sample set, wherein the first training sample set includes each medical document sample image, segmentation requirement sample information corresponding to the medical document sample image, and a standard image corresponding to the medical document sample image; processing each of the medical document sample images and the segmentation requirement sample information corresponding to the medical document sample image in the first training sample set through a first initial model to obtain a predicted image corresponding to each medical document sample image; obtaining a first loss value based on each predicted image and the standard image corresponding to each predicted image, and updating the network parameters of the first initial model based on the first loss value until the first loss value satisfies a first loss condition, and then using the first initial model as the image segmentation model.

[0054] Among them, the sample requirement information corresponds to the medical document sample image, which refers to the segmentation information of a part of the image in the medical document sample image. It can be described by coordinate points, coordinate regions or text. For example, the text description is: Segment out the document on the left side of the medical document sample image. Of course, the medical document sample image includes two documents arranged left and right.

[0055] In this embodiment of the invention, a first training sample set is first obtained. A first initial model processes each medical document sample image and its corresponding segmentation requirement sample information within the first training sample set to obtain a predicted image for each medical document sample image. Each predicted image is compared with its corresponding labeled image to obtain a first loss value. The network parameters of the first initial model are updated based on the first loss value until the first loss value satisfies a first loss condition. The first initial model at this point is then used as the image segmentation model. This embodiment of the invention implements the training process of the first initial model, ensuring the accuracy of the first initial model training, thereby improving the accuracy of the image segmentation model.

[0056] In another embodiment of the invention, the training step of the information extraction model includes: acquiring a second training sample set, wherein the second training sample set includes each sample document image, sample demand information corresponding to the sample document image, and standard information corresponding to the sample demand information, wherein each sample image includes only one document; processing each sample document image and the sample demand information in the second training sample set through a second initial model to obtain prediction information corresponding to each sample demand information; obtaining a second loss value based on the prediction information and the standard information, and updating the network parameters of the second initial model based on the second loss value, until the second loss value satisfies the second loss condition, and then using the second initial model as the information extraction model.

[0057] The sample requirement information includes obtaining predicted text information or predicted document images from sample document images. For example, the sample requirement information could be to obtain disease A, obtain the corresponding answer information for disease A, or obtain a document image with labeled disease A from a sample document image. Each sample requirement information is set for a specific sample document image.

[0058] Specifically, the second initial model processes the sample document images and sample demand information in the second training sample set to obtain predicted information corresponding to each sample demand information. A second loss value is obtained based on the predicted information and standard information. The network parameters of the second initial model are updated based on the second loss value. When the second loss value satisfies the second loss condition, the second initial model at this point is used as the information extraction model. This embodiment of the invention achieves the training of the second initial model, ensuring training accuracy and thus improving the accuracy of information extraction by the information extraction model.

[0059] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In another embodiment of the invention, a medical document image analysis device is provided, which corresponds one-to-one with the medical document image analysis method in the above embodiments. For example... Figure 3 As shown, the medical document image analysis device includes: an information acquisition module 410, an information processing module 420, a medical document image segmentation module 430, and a target information acquisition module 440. Detailed descriptions of each functional module are as follows:

[0061] The information acquisition module 410 is used to acquire medical document images and corresponding requirement information; the information processing module 420 is used to process the medical document images and the image segmentation requirement information using a pre-trained image segmentation model if the requirement information includes image segmentation requirement information, to obtain candidate document images corresponding to the image segmentation requirement information; the medical document image segmentation module 430 is used to segment the medical document images using a pre-trained image segmentation model if the requirement information does not include image segmentation requirement information, to obtain candidate document images from which the background has been removed; the target information acquisition module 440 is used to process the candidate document images and the text requirement information in the requirement information using a pre-trained information extraction model, to obtain target information corresponding to the text requirement information, wherein the target information includes target text and / or target document images.

[0062] In another embodiment of the invention, the image segmentation model includes a first encoder and a first decoder; the information processing module 420 is further configured to:

[0063] The medical document image is converted into a first image vector by a first encoder; the image segmentation requirement information is converted into a segmentation text vector; and the first image vector is segmented based on the segmentation text vector by a first decoder to obtain a candidate document image corresponding to the image segmentation requirement information.

[0064] In another embodiment of the invention, the information extraction model includes a second encoder and a second decoder, and the text information extraction requirement information includes at least one of obtaining at least one target text in a candidate document image, determining the position of the target text, and obtaining answer information corresponding to the target text, wherein the answer information refers to information corresponding to the target text retrieved from a preset database;

[0065] The target information acquisition module 440 is also used for:

[0066] The second encoder performs vector transformation on the candidate document image and the text requirement information respectively to obtain candidate document image vector and text requirement information vector. The target vector is obtained by concatenating the candidate document image vector and the text requirement information vector. The second decoder extracts target text and / or target document image from the target vector to obtain target information corresponding to the text requirement information.

[0067] In another embodiment of the invention, the second decoder includes a text decoder and a layout decoder;

[0068] The target information acquisition module 440 is also used for:

[0069] The text decoder is used to extract text from the target vector to obtain at least one target text.

[0070] If the text requirement information includes determining the location information of the target text, the location information of the target text is obtained through the layout decoder;

[0071] Based on the location information, the target text in the candidate document image is marked to obtain a target document image including the marked target text;

[0072] If the text requirement information includes obtaining the answer information for the target text, then the answer information corresponding to the target text is retrieved from the database according to the target text, wherein the target information includes the answer information.

[0073] In another embodiment of the invention, when the candidate document images include multiple images, the target information acquisition module 440 is further configured to:

[0074] Obtain the text semantic information and page information of each candidate document image, wherein the page information includes the page number information of the candidate document image;

[0075] Based on the text semantic information and the page information, it is determined whether the candidate document images are related.

[0076] If there are at least two candidate document images that are related, the candidate document images that are related are stitched together based on the relationship to obtain the target candidate document image.

[0077] The second encoder performs vector transformation on the target candidate document image and the text requirement information respectively to obtain the target candidate document image vector and the text requirement information vector. The target vector is obtained by concatenating the target candidate document image vector and the text requirement information vector.

[0078] In another embodiment of the invention, the training module of the image segmentation model is used for:

[0079] Obtain a first training sample set, wherein the first training sample set includes each medical document sample image, segmentation requirement sample information corresponding to the medical document sample image, and a standard image corresponding to the medical document sample image;

[0080] Using the first initial model, each of the medical document sample images in the first training sample set and the segmentation requirement sample information corresponding to the medical document sample images are processed to obtain the predicted image corresponding to each medical document sample image.

[0081] A first loss value is obtained based on each predicted image and the standard image corresponding to each predicted image, and the network parameters of the first initial model are updated based on the first loss value until the first loss value satisfies the first loss condition, at which point the first initial model is used as the image segmentation model.

[0082] In another embodiment of the invention, the training module of the information extraction model is used for:

[0083] Obtain a second training sample set, wherein the second training sample set includes each sample document image, sample requirement information corresponding to the sample document image, and standard information corresponding to the sample requirement information, and each sample image includes only one document;

[0084] The second initial model is used to process the sample document images and sample demand information in the second training sample set to obtain prediction information corresponding to each sample demand information.

[0085] A second loss value is obtained based on the predicted information and the standard information, and the network parameters of the second initial model are updated based on the second loss value until the second loss value satisfies the second loss condition, at which point the second initial model is used as the information extraction model.

[0086] The technical solution of this invention can obtain medical document images and corresponding requirement information through a client. If the requirement information includes image segmentation requirement information, the medical document images and image segmentation requirement information are processed by a pre-trained image segmentation model to obtain candidate document images corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirement information, a pre-trained image segmentation model is used to segment the medical document image, obtaining candidate document images from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document images and the text requirement information in the requirement information to obtain target information corresponding to the text requirement information. This target information is then fed back to the client. In this invention, the following steps are achieved: acquiring medical document images and requirement information; obtaining candidate document images corresponding to the image segmentation requirement information when the requirement information includes it; and obtaining candidate document images with the background removed when the requirement information does not include it. The pre-trained information extraction model processes the candidate document images and the text requirement information in the requirement information to obtain target information. By segmenting the medical document image or removing the background, noise can be removed from the obtained candidate document images, thereby improving the accuracy of target information acquisition.

[0087] Specific limitations regarding the medical document image analysis device can be found in the limitations of the medical document image analysis method described above, and will not be repeated here. Each module in the aforementioned medical document image analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0088] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a medical document image analysis method on the server side.

[0089] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a medical document image analysis method.

[0090] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0091] Acquire medical document images and the corresponding requirement information for the medical document images;

[0092] If the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed by a pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information;

[0093] If the requirement information does not include image segmentation requirement information, then the medical document image is segmented using a pre-trained image segmentation model to obtain candidate document images from which the background has been removed.

[0094] The candidate document image and the textual demand information in the demand information are processed by a pre-trained information extraction model to obtain target information corresponding to the textual demand information, wherein the target information includes target text and / or target document image.

[0095] In the computer device of this invention, when the processor executes a computer program, it can acquire medical document images and corresponding requirement information. If the requirement information includes image segmentation requirement information, the medical document images and image segmentation requirement information are processed by a pre-trained image segmentation model to obtain candidate document images corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirement information, a pre-trained image segmentation model is used to segment the medical document image, obtaining candidate document images from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document images and the text requirement information in the requirement information to obtain target information corresponding to the text requirement information. In this invention, the following steps are achieved: acquiring medical document images and requirement information; obtaining candidate document images corresponding to the image segmentation requirement information when the requirement information includes image segmentation requirement information; and obtaining candidate document images from which the background has been removed when the requirement information does not include image segmentation requirement information. The pre-trained information extraction model processes the text requirement information in the candidate document images and the requirement information to obtain target information. By performing medical document image segmentation or background removal, the obtained candidate document images can be free of noise, thereby obtaining target information from the candidate document images and improving the accuracy of target information acquisition.

[0096] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0097] Acquire medical document images and the corresponding requirement information for the medical document images;

[0098] If the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed by a pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information;

[0099] If the requirement information does not include image segmentation requirement information, then the medical document image is segmented using a pre-trained image segmentation model to obtain candidate document images from which the background has been removed.

[0100] The candidate document image and the textual demand information in the demand information are processed by a pre-trained information extraction model to obtain target information corresponding to the textual demand information, wherein the target information includes target text and / or target document image.

[0101] The computer-readable storage medium of this invention allows a computer program, when executed by a processor, to acquire medical document images and corresponding requirement information. If the requirement information includes image segmentation requirement information, a pre-trained image segmentation model is used to process the medical document images and the image segmentation requirement information to obtain candidate document images corresponding to the image segmentation requirement information. If the requirement information does not include image segmentation requirement information, a pre-trained image segmentation model is used to segment the medical document image, obtaining candidate document images from which the background has been removed. A pre-trained information extraction model is then used to process the candidate document images and the text requirement information in the requirement information to obtain target information corresponding to the text requirement information. In this invention, the following steps are achieved: acquiring medical document images and requirement information; obtaining candidate document images corresponding to the image segmentation requirement information when the requirement information includes image segmentation requirement information; and obtaining candidate document images from which the background has been removed when the requirement information does not include image segmentation requirement information. The pre-trained information extraction model processes the text requirement information in the candidate document images and the requirement information to obtain target information. By performing medical document image segmentation or background removal, the obtained candidate document images can be free of noise, thereby obtaining target information from the candidate document images and improving the accuracy of target information acquisition.

[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0105] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for analyzing images in medical documents, characterized in that, include: Acquire medical document images and the corresponding requirement information for the medical document images; If the requirement information includes image segmentation requirement information, then the medical document image and the image segmentation requirement information are processed by a pre-trained image segmentation model to obtain a candidate document image corresponding to the image segmentation requirement information; the image segmentation requirement information refers to the information for segmenting the medical document image, and the information includes coordinate regions; If the requirement information does not include image segmentation requirement information, then the medical document image is segmented using a pre-trained image segmentation model to obtain candidate document images from which the background has been removed. The candidate document image and the textual demand information in the demand information are processed by a pre-trained information extraction model to obtain target information corresponding to the textual demand information. The target information includes target text and / or target document image. The target text is the symptom features and the answer information corresponding to the symptom features. The location of the symptom features is marked in the target document image.

2. The method for analyzing medical document images according to claim 1, characterized in that, The image segmentation model includes a first encoder and a first decoder; the pre-trained image segmentation model processes the medical document image and the image segmentation requirement information to obtain candidate document images corresponding to the image segmentation requirement information, including: The medical document image is converted into a first image vector using the first encoder; The image segmentation requirement information is converted into a segmentation text vector; The first decoder segments the first image vector based on the segmented text vector to obtain candidate document images corresponding to the image segmentation requirement information.

3. The method for analyzing medical document images according to claim 1, characterized in that, The information extraction model includes a second encoder and a second decoder. The text requirement information includes at least one of the following: obtaining at least one target text in a candidate document image, determining the position of the target text, and obtaining answer information corresponding to the target text. The answer information refers to information corresponding to the target text retrieved from a preset database. The process of using a pre-trained information extraction model to process the candidate document image and the textual demand information in the demand information to obtain target information corresponding to the textual demand information includes: The second encoder performs vector transformation on the candidate document image and the text requirement information respectively to obtain the candidate document image vector and the text requirement information vector. The target vector is obtained by concatenating the candidate document image vector and the text requirement information vector. The second decoder extracts target text and / or target document images from the target vector to obtain target information corresponding to the text requirement information.

4. The method for analyzing medical document images according to claim 3, characterized in that, The second decoder includes a text decoder and a layout decoder; the step of extracting target text and / or target document image from the target vector using the second decoder to obtain target information corresponding to the text requirement information includes: The text decoder is used to extract text from the target vector to obtain at least one target text. If the text requirement information includes determining the location information of the target text, the location information of the target text is obtained through the layout decoder; Based on the location information, the target text in the candidate document image is marked to obtain a target document image including the marked target text; If the text requirement information includes obtaining the answer information for the target text, then according to the target text, the answer information corresponding to the target text is retrieved from the database, wherein the target information also includes the answer information.

5. The method for analyzing medical document images according to claim 4, characterized in that, When there are multiple candidate document images, the step of performing vector transformation on the candidate document images and the text requirement information respectively through the second encoder to obtain candidate document image vectors and text requirement information vectors, and then concatenating the candidate document image vectors and text requirement information vectors to obtain the target vector, includes: Obtain the text semantic information and page information of each candidate document image, wherein the page information includes the page number information of the candidate document image; Based on the text semantic information and the page information, it is determined whether the candidate document images are related. If there are at least two candidate document images that are related, the candidate document images that are related are stitched together based on the relationship to obtain the target candidate document image. The second encoder performs vector transformation on the target candidate document image and the text requirement information respectively to obtain the target candidate document image vector and the text requirement information vector. The target vector is obtained by concatenating the target candidate document image vector and the text requirement information vector.

6. The method for analyzing medical document images according to claim 1, characterized in that, The training steps of the image segmentation model include: Obtain a first training sample set, wherein the first training sample set includes each medical document sample image, image segmentation requirement information corresponding to the medical document sample image, and a standard image corresponding to the medical document sample image; The first initial model is used to process each of the medical document sample images in the first training sample set and the image segmentation requirement information corresponding to the medical document sample images to obtain the predicted image corresponding to each medical document sample image. A first loss value is obtained based on each predicted image and the standard image corresponding to each predicted image, and the network parameters of the first initial model are updated based on the first loss value until the first loss value satisfies the first loss condition, at which point the first initial model is used as the image segmentation model.

7. The method for analyzing medical document images according to claim 1, characterized in that, The training steps of the information extraction model include: Obtain a second training sample set, wherein the second training sample set includes each sample document image, sample requirement information corresponding to the sample document image, and standard information corresponding to the sample requirement information, and each sample document image includes only one document; The second initial model is used to process the sample document images and sample demand information in the second training sample set to obtain prediction information corresponding to each sample demand information. A second loss value is obtained based on the predicted information and the standard information, and the network parameters of the second initial model are updated based on the second loss value until the second loss value satisfies the second loss condition, at which point the second initial model is used as the information extraction model.

8. A medical document image analysis device, characterized in that, include: The information acquisition module is used to acquire medical document images and corresponding demand information. The information processing module is used to process the medical document image and the image segmentation requirement information using a pre-trained image segmentation model if the requirement information includes image segmentation requirement information, to obtain a candidate document image corresponding to the image segmentation requirement information; the image segmentation requirement information refers to information for segmenting the medical document image, and the information includes coordinate regions; The medical document image segmentation module is used to segment the medical document image using a pre-trained image segmentation model if the requirement information does not include image segmentation requirement information, thereby obtaining candidate document images from which the background has been removed. The target information acquisition module is used to process the candidate document image and the textual demand information in the demand information through a pre-trained information extraction model to obtain target information corresponding to the textual demand information. The target information includes target text and / or target document image. The target text is the symptom features and the answer information corresponding to the symptom features. The target document image is marked with the location of the symptom features.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images