Method and apparatus for extracting image information, and non-transitory storage medium

Abstract

The application discloses a method and device for extracting image information and a nonvolatile storage medium. The method comprises the following steps: obtaining a to-be-processed image, and preprocessing the to-be-processed image to obtain an initial image; performing text analysis processing on the initial image to obtain a text analysis processing result; determining an image type of the to-be-processed image according to the to-be-processed image and the text analysis processing result; and processing the to-be-processed image, the text analysis processing result and the image type by using a graphic-text multi-modal model to obtain a processing result output by the graphic-text multi-modal model, wherein the processing result comprises a plurality of target entities and a relationship between the target entities; determining a target communication subgraph corresponding to the to-be-processed image according to the image type of the to-be-processed image, and determining a preliminary information extraction result of the to-be-processed image according to the processing result output by the graphic-text multi-modal model and the target communication subgraph; and determining a final information extraction result of the to-be-processed image according to the preliminary information extraction result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for extracting image information, and a non-volatile storage medium. Background Technology

[0002] With the development of medical informatization, related technologies mainly rely on electronic medical record (EHR) systems to manage medical records. However, EHR systems typically only focus on structured text data, neglecting the large amount of diverse and unstructured image data generated during patient diagnosis and treatment, such as laboratory reports, ultrasounds, electrocardiograms, admission and discharge diagnoses, surgical records, and diagnostic certificates. They cannot automatically extract unstructured data. When faced with medical images from different hospitals and of different types provided by patients, medical staff mainly rely on manual analysis to address the unstructured data contained in these images. Therefore, when extracting information from medical record images, there are problems of low information extraction efficiency and low information extraction accuracy.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method and apparatus for extracting image information, as well as a non-volatile storage medium, to at least solve the technical problem that in related technologies, only structured data can be extracted from medical record images, resulting in incomplete information extraction and low accuracy of information extraction from medical record images.

[0005] According to one aspect of the embodiments of this application, a method for extracting image information is provided, comprising: acquiring an image to be processed and preprocessing the image to be processed to obtain an initial image; performing text analysis processing on the initial image to obtain a text analysis processing result, wherein the text analysis processing result includes: multiple text boxes and related information for each text box, different text boxes being used to select different text content contained in the initial image, and the related information including: the position information of the text box in the initial image, and the text content located within the text box; determining the image type of the image to be processed based on the image to be processed and the text analysis processing result; and using a graph-text multimodal model to process the image to be processed, the text analysis processing result, and the image type to obtain a processing result output by the graph-text multimodal model, wherein the processing result includes: multiple target entities, target... The relationships between entities are defined, with each text box containing text content generating a target entity. The image-text multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images, each type of historical medical record image including different entity categories, and different entity categories corresponding to different loss functions. The target connected subgraph corresponding to the image to be processed is determined according to the image type of the image to be processed, and the preliminary information extraction result of the image to be processed is determined according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting entities as edges. The final information extraction result of the image to be processed is determined according to the preliminary information extraction result.

[0006] Optionally, the training data also includes: multiple ground truth labels, where each ground truth label corresponds to a historical medical record image. Each ground truth label records the image type of the historical medical record image, multiple entities generated from the historical medical record image, multiple relationships, and the location information of each entity. Multiple entities of the same category contain multiple training text boxes. Each relationship includes: the association between multiple entities and the association between multiple training text boxes containing entities of the same category. Each training text box is used to select the text content contained in the historical medical record image; different training text boxes select different text content. The location information indicates the position of the training text box in the historical medical record image. The image-text multimodal model is trained using the following method: combining historical medical record images belonging to the same image type with the ground truth labels of the historical medical record images... The real labels are determined as the training set corresponding to the image type. For each image type, an initial multimodal model is used to perform a preset task on the historical medical record images contained in the training set to obtain the prediction results output by the initial multimodal model. The initial multimodal model outputs different prediction results when performing different preset tasks. The preset tasks include: entity prediction task and relationship prediction task. The prediction results include: entity prediction results, relationship head entities and relationship tail entities. Relationship head entities and relationship tail entities jointly indicate the relationship between entities. The initial loss function is iteratively updated based on the real labels and multiple prediction results until the change in the initial loss function is less than a preset threshold to obtain the image-text multimodal model. The change is the difference between the latest update result of the initial loss function and the previous update result of the latest update result.

[0007] Optionally, the initial multimodal model includes multiple input layers and multiple task processing layers. The input layers include: a text box embedding layer, an image embedding layer, a location embedding layer, and a type embedding layer. The initial multimodal model is used to perform preset tasks on historical medical record images contained in the training set to obtain the prediction results output by the initial multimodal model. This includes: converting multiple entities recorded in the ground truth labels into a first vector in the text box embedding layer; converting historical medical record images into a second vector in the image embedding layer; storing the location information of each entity in the location embedding layer; and converting the image type of historical medical record images into a third vector in the type embedding layer. The first vector, second vector, and third vector, along with the location information, are fused into a feature vector. Different preset tasks are performed on the feature vector in different task processing layers to obtain the prediction results output by the task processing layers. The prediction results output by different task processing layers are different.

[0008] Optionally, the initial loss function is iteratively updated based on the ground truth labels and multiple prediction results. Each iteration includes: for each image type, classifying the ground truth labels and multiple prediction results corresponding to the image type into datasets, resulting in multiple datasets corresponding to multiple image types; determining the prediction accuracy of the initial multimodal model for each image type based on the datasets, resulting in multiple prediction accuracies corresponding to multiple image types; determining multiple weights and multiple modulation factors corresponding to each image type, and determining the update result of the initial loss function based on the multiple prediction accuracies, multiple weights, and multiple modulation factors. Each image type includes multiple different entity categories, with different entity categories corresponding to different weights and different entity categories corresponding to different modulation factors. The modulation factors are used to adjust the contribution of entity categories in the process of updating the initial loss function, and the entity categories are used to indicate the type of information recorded in historical medical record images.

[0009] Optionally, the prediction accuracy of the initial multimodal model for each image type is determined based on the dataset, including: for each prediction result, determining the actual result corresponding to the prediction result in the ground truth labels, wherein the actual result corresponding to the entity prediction result is an entity, and the actual results corresponding to the relation head entity and relation tail entity are relations; determining the probability that the prediction result is the same as the actual result corresponding to the prediction result, wherein the probability is the prediction accuracy.

[0010] Optionally, the target connected subgraph is constructed as follows: For each image type, the final prediction result output by the initial multimodal model during the last iteration of updating the loss function is obtained, wherein the final prediction result includes: the final entity prediction result and the final relationship prediction result. The final relationship prediction result includes at least one of the following: entity relationship and text relationship. The text relationship is used to indicate the association between the text content contained in multiple training text boxes generated based on the historical medical record image. Each training text box is used to select different text content contained in the historical medical record image. The target connected subgraph is generated by connecting multiple entities in the final entity prediction result according to the entity relationship, wherein each entity in the final entity prediction result is a point in the target connected subgraph, and the straight line used to connect multiple entities with entity relationships is an edge in the target connected subgraph. The target connected subgraph includes: a first type of connected subgraph containing entities of different categories and a second type of connected subgraph containing entities of the same category.

[0011] Optionally, the preliminary information extraction result of the image to be processed is determined based on the processing result output by the image-text multimodal model and the target connected subgraph, including: for each target entity contained in the processing result, querying the target entity in multiple target connected subgraphs, and querying multiple other entities located on the same edge as the target entity; converting the target entity and multiple other entities into corresponding fields, and generating a data structure based on the multiple fields, wherein the data structure is the information extraction result.

[0012] Optionally, determining the final information extraction result of the image to be processed based on the preliminary information extraction result includes: determining the length of the target entity included in the preliminary information extraction result, and determining whether to perform long text segmentation processing on the target entity based on the length of the target entity, wherein long text segmentation processing indicates dividing the target entity into multiple sub-entities of equal length; if it is determined not to perform long text segmentation processing, the preliminary information extraction result is determined as the final information extraction result; if it is determined to perform long text segmentation processing, the final information extraction result is determined based on the multiple sub-entities and the target connected subgraph.

[0013] Optionally, determining whether to perform long text segmentation on the target entity based on its length includes: if the length of the target entity is greater than a preset length threshold, determining multiple segmentation text boxes based on the target entity, and sorting the multiple segmentation text boxes according to the position information of each segmentation text box contained in the target entity in the initial image to obtain a sorting sequence, wherein the sorting sequence records the identifiers of multiple segmentation text boxes, and different segmentation text boxes are used to select different sub-entities contained in the target entity; sequentially accumulating the length of each segmentation text box according to the order indicated by the sorting sequence, and after each accumulation, determining the comparison result between the accumulation result and the preset length threshold; and in accordance with the previous comparison result... If the accumulated result is less than or equal to a preset length threshold and the next comparison result indicates that the accumulated result is greater than the preset length threshold, the multiple segmented text boxes corresponding to the previous comparison result are combined into an initial long text box, and the accumulation continues until all segmented text boxes contained in the sorted sequence have been accumulated, and multiple initial long text boxes are output. For each initial long text box, if the length of the initial long text box is less than the preset length threshold, the initial long text box is padded with padding characters until the length of the initial long text box reaches the preset length threshold, then the padding stops, and the child entity is output. If the length of the initial long text box is equal to the preset length threshold, the initial long text box is output as a child entity.

[0014] According to another aspect of the embodiments of this application, an apparatus for extracting image information is also provided, comprising: an acquisition module, configured to acquire an image to be processed and preprocess the image to be processed to obtain an initial image; a text box generation module, configured to perform text analysis processing based on the initial image to obtain a text analysis processing result, wherein the text analysis processing result includes: multiple text boxes and related information for each text box, different text boxes being used to select different text content contained in the initial image, and the related information including: the position information of the text box in the initial image, and the text content located within the text box; a processing module, configured to determine the image type of the image to be processed based on the image to be processed and the text analysis processing result; and to process the image to be processed, the text analysis processing result, and the image type using a multimodal graph model to obtain a processing result output by the multimodal graph model, wherein the processing result includes: multiple targets The relationship between entities and target entities is defined as follows: the text content contained in each text box generates a target entity. The image-text multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images, each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The first determination module is used to determine the target connected subgraph corresponding to the image to be processed based on the image type of the image to be processed, and to determine the preliminary information extraction result of the image to be processed based on the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting entities as edges. The second determination module is used to determine the final information extraction result of the image to be processed based on the preliminary information extraction result.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, which stores a computer program, wherein the device where the non-volatile storage medium is located executes the above-described method for extracting image information by running the computer program.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described method for extracting image information through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the steps of the above-described method for extracting image information.

[0018] In this embodiment, the process involves acquiring an image to be processed and preprocessing it to obtain an initial image. Text analysis is then performed on the initial image to obtain text analysis results. These results include multiple text boxes and related information for each text box. Different text boxes are used to select different text content within the initial image. The related information includes the position of the text box within the initial image and the text content within the text box. The image type of the image to be processed is determined based on the image to be processed and the text analysis results. A multimodal graph model is then used to process the image to be processed, the text analysis results, and the image type to obtain the processing results output by the multimodal graph model. These processing results include multiple target entities and the relationships between these target entities. Each text box contains text content that generates a target entity. The multimodal graph model is trained using training data and multiple loss functions on an initial multimodal model based on a Transformer architecture. The training data includes various types of historical medical record images, and each type of historical medical record image contains... This approach includes different entity categories, each corresponding to a different loss function. It determines the target connected subgraph for the image to be processed based on its image type, and then determines the preliminary information extraction result based on the processing result output by the multimodal graph model and the target connected subgraph. Each target connected subgraph is constructed using multiple entities as nodes and lines connecting the entities as edges. The final information extraction result is determined based on the preliminary information extraction result. By correcting the tilt of the medical record image during preprocessing, information extraction errors caused by image tilt are avoided, thus improving the accuracy of image information. By employing a multimodal graph model with an improved model architecture and loss function for information extraction from medical record images, the efficiency and accuracy of information extraction are improved, achieving automated extraction of medical record image information. This solves the technical problem of incomplete information extraction and low accuracy in medical record image information extraction caused by related technologies that can only extract structured data from medical record images. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of the steps of a method for extracting image information according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating the generation of multiple text boxes based on an image to be processed, according to an embodiment of this application.

[0022] Figure 3 This is a model architecture diagram of a graphic multimodal model according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a target connected subgraph according to an embodiment of this application;

[0024] Figure 5 This is the entity extraction result of a medical record image of an ultrasound report type according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating the format conversion of an image information extraction result according to an embodiment of this application;

[0026] Figure 7 This is a structural diagram of an image information extraction device according to an embodiment of this application;

[0027] Figure 8 This is a flowchart of an image information extraction device according to an embodiment of this application;

[0028] Figure 9 This is a flowchart illustrating an image information extraction device for a patient A's uploaded image, according to an embodiment of this application.

[0029] Figure 10 This is a flowchart illustrating the tracking of blood lipid measurement results for patient B in a chronic disease management system according to an embodiment of this application;

[0030] Figure 11 This is a hardware structure block diagram of a computer terminal for implementing a method for extracting image information according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] In related technologies, the medical record image to be processed is assumed to be a complete and clear image. However, when extracting information from the image, the issue of image tilt is not taken into account. Therefore, in related technologies, information extraction from medical record images is often misaligned due to image tilt, resulting in information omissions. To address this problem, this application provides a related solution, which is described in detail below.

[0034] According to an embodiment of this application, a method for extracting image information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] Figure 1 This is a flowchart of the steps of the method for extracting image information according to the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step S102: Obtain the image to be processed and preprocess it to obtain the initial image.

[0037] This application provides a method for extracting information from images, particularly applicable to various types of medical record images, such as ultrasound images, laboratory reports, electrocardiograms, admission and discharge diagnoses, surgical records, and diagnostic certificates. The method provided in this application allows for information extraction from different types of images. To avoid incomplete information extraction due to image tilt, the method introduced in this application incorporates tilt correction technology. In step S102, the first step in extracting information from the input image using the method provided in this application is to perform preprocessing on the medical record image, including tilt correction. During preprocessing, at least the received image needs to be tilt corrected. Furthermore, images can be cleaned and filtered before tilt correction as needed. It should be noted that even without tilt correction, the method provided in this application can still be executed smoothly, but tilt correction avoids misalignment when extracting correlations, improving the accuracy of information extraction results.

[0038] In this embodiment, tilt correction of the image to be processed is to adjust the image that is not parallel to the horizontal line to a state parallel to the horizontal line. For example, for the image to be processed at a 90-degree or 180-degree angle to the horizontal line, it is rotated to a state parallel to the horizontal line. Specifically, the tilt correction of the image to be processed can be performed according to the following steps: Step 1, when there is a color image in the image to be processed, the color image is processed into grayscale, converting the color image into a grayscale image (composed of multiple pixels, with different grayscale regions corresponding to different pixel values); if the image to be processed is entirely grayscale, step 1 is not required; Step 2, the image to be processed after being converted into a grayscale image is binarized, converting the grayscale image into a binary image (a black and white image, containing only the pixel values ​​corresponding to black and white, with the black area being the area where the text box and text content are located, and the white area being other areas of the image to be processed); Step 3, the binary image... Step 4: Perform edge detection to determine the direction of text lines in the image to be processed (in step 3, the image to be processed is equivalent to a binary image); Step 5: Based on the direction of the text lines determined in step 3, determine the boundary of the text region to be processed along the direction of the text lines, and determine the target region containing text information in the image to be processed; Step 6: Determine the angle of the border line of the target region relative to the horizontal line, and the result is the tilt angle of the image to be processed; Step 7: Based on the tilt angle determined in step 5, rotate the image to be processed, rotate the image to be processed in the target direction, the rotation angle value is the same as the tilt angle value, and the target direction is the opposite direction to the tilt angle.

[0039] Step S104: Perform text analysis processing on the initial image to obtain the text analysis processing result. The text analysis processing result includes: multiple text boxes and related information for each text box. Different text boxes are used to select different text content contained in the initial image. The related information includes: the position information of the text box in the initial image and the text content located in the text box.

[0040] Further, after completing the preprocessing of the image to be processed, in step S104, the text content in the image to be processed is identified. Specifically, the preprocessed image to be processed (i.e., the initial image) is subjected to text content recognition and analysis: multiple text boxes are used to mark the text content recorded in the image to be processed, and the position of each text box in the initial image is determined. Finally, the text information of the image to be processed (i.e., the text analysis and processing result) is output. The text information obtained from the text analysis and processing (i.e., the text analysis and processing result) includes multiple text boxes that select the text content of the image to be processed, the position information of each text box, and the specific text content recorded in each text box. The position information of the text boxes can be obtained by... The position of a text box can be represented using coordinates. For example, the coordinates of its top-left and bottom-right corners can be used to represent its position. These coordinates can be defined in pixels. For instance, if the image to be processed is an 800x600 pixel image, and the top-left corner of the text box is located in the top-left region of the image, specifically at a horizontal position of 150 pixels (from left to right) and a vertical position of 200 pixels (from top to bottom), then the coordinates of the top-left corner can be represented as (X1=150, Y1=200). The bottom-right corner of the text box is located at a horizontal position of 450 pixels and a vertical position of 250 pixels in the image, then the coordinates of the bottom-right corner can be represented as (X2=450, Y2=250). In this embodiment, the position information of the text box can also be represented as (Xc, Yc, W, H) using center point coordinates (Xc, Yc), width, and height. Here, the center point coordinates are the coordinates of the center point of the text box, which can also be defined by pixels. For example, if the center point of the text box is 300 pixels horizontally and 100 pixels vertically in the image, then the center point coordinates of the text box are (Xc=300, Yc=100). W represents the width of the text box, and H represents the height of the text box. In step S104, Optical Character Recognition (OCR) technology can be used to divide the text content in the image to be processed into multiple text boxes. After processing the text content in the image using OCR technology, multiple text boxes, the text content selected in each text box, and the position of each text box are output. Figure 2 This is a diagram illustrating the generation of multiple text boxes from the image to be processed, such as... Figure 2As shown, the image to be processed is a laboratory report. After tilt correction of the report, a text box is used to select the text content contained within the report. The method provided in this embodiment can visualize the processed image (such as a laboratory report) through a connected terminal or display interface. Figure 2 As shown, in the visualization, different types of text boxes are displayed in different colors, and, as... Figure 2 As shown, directed line segments can also be used to connect text boxes that have a relationship.

[0041] Step S106: Determine the image type of the image to be processed based on the image to be processed and the text analysis processing results; and use a text-image multimodal model to process the image to be processed, the text analysis processing results, and the image type to obtain the processing results output by the text-image multimodal model. The processing results include: multiple target entities, the relationships between target entities, and the text content contained in each text box generates a target entity. The text-image multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images, each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions.

[0042] In this embodiment, the image to be processed, its image type, and the text analysis results (multiple text boxes and related information of the text boxes) obtained from text analysis of the image to be processed are used as input information for the graph-text multimodal model. The graph-text multimodal model processes and analyzes the above input information: converting the text content contained in each text box into a corresponding entity (i.e., a target entity), and outputting multiple target entities determined based on the input information, as well as the relationships between different target entities; for example... Figure 2In the above input information, "WBC" corresponds to one (target) entity, and "7.06" corresponds to another (target) entity. The image type of the image to be processed included in the above input information is determined jointly based on the image to be processed and the text analysis processing results after the image to be processed is processed. The image types include ultrasound images, laboratory reports, electrocardiograms, admission and discharge diagnoses, surgical records, and diagnostic certificates. In this embodiment, the image-text multimodal model determines the image type of the image to be processed in the following way: based on the labeled image type, the image is classified at the pixel level by an image classification model to classify the images to be processed that contain text and those that do not; then, combined with the text content, the image to be processed containing text is further classified by the encoder network (BERT) in the image-text multimodal model. The image classification model can be a convolutional neural network (CNN) model that can perform image classification. The image classification model can be included in the image-text multimodal model, or it can be a model that exists independently of the image-text multimodal model but can interact with it. The images to be processed that contain text may be ultrasound images, laboratory reports, electrocardiograms, admission and discharge diagnoses, surgical records, diagnostic certificates, etc. The text-image multimodal model used in this embodiment is a modified multimodal model (i.e., the initial multimodal model) used as the training object after modifying the model architecture of commonly used multimodal models in related technologies. Multiple types of historical medical record images are used as training data, each type including different entity categories. The model is trained using multiple different loss functions corresponding to these entity categories. In this embodiment, the information extraction results of the historical medical record images used as training data are known, and these results are used to verify the accuracy of the information extraction results during the training process of the initial multimodal model. As can be seen from the above, the text-image multimodal model is at least used to process text content, converting text into entities. That is, the text-image multimodal model needs to have the function of establishing a relationship between the input text (i.e., text box) and the processing result (i.e., target entity). To achieve this function, the model selected as the training object in this embodiment is a Transformer-based model, and the resulting text-image multimodal model is naturally also based on the Transformer architecture.

[0043] In step S106, the graph-text multimodal model can be loaded into memory. For example, the raw data of the graph-text multimodal model (such as the initial multimodal model) can be loaded from non-volatile memory into volatile memory so that the processor can run the graph-text multimodal model. The raw data of the graph-text multimodal model (such as the initial multimodal model) refers to unprocessed data, which typically includes the parameters and structural data of the graph-text multimodal model. The structural data can be the computational relationships based on the parameters, such as the forward propagation computational relationships between intermediate layers and between neurons. Specifically, the structural data can include the structure-related code of the graph-text multimodal model, such as code used to perform related calculations between intermediate layers and between neurons.

[0044] In one implementation, a region can be partitioned in memory for loading the text-image multimodal model, which may include a structure data storage area and a parameter storage area. The structure data storage area stores structure-related code, and the parameters referenced by it can be accessed via pointers to the addresses of specific parameters in the parameter storage area. During the initial training of the multimodal model, frequent parameter updates may be required; in this case, updating the parameter values ​​in the parameter storage area is sufficient.

[0045] Optionally, the training data also includes: multiple ground truth labels, where each ground truth label corresponds to a historical medical record image. Each ground truth label records the image type of the historical medical record image, multiple entities generated from the historical medical record image, multiple relationships, and the location information of each entity. Multiple entities of the same category contain multiple training text boxes. Each relationship includes: the association between multiple entities and the association between multiple training text boxes containing entities of the same category. Each training text box is used to select the text content contained in the historical medical record image; different training text boxes select different text content. The location information indicates the position of the training text box in the historical medical record image. The image-text multimodal model is trained using the following method: combining historical medical record images belonging to the same image type with the ground truth labels of the historical medical record images... The real labels are determined as the training set corresponding to the image type. For each image type, an initial multimodal model is used to perform a preset task on the historical medical record images contained in the training set to obtain the prediction results output by the initial multimodal model. The initial multimodal model outputs different prediction results when performing different preset tasks. The preset tasks include: entity prediction task and relationship prediction task. The prediction results include: entity prediction results, relationship head entities and relationship tail entities. Relationship head entities and relationship tail entities jointly indicate the relationship between entities. The initial loss function is iteratively updated based on the real labels and multiple prediction results until the change in the initial loss function is less than a preset threshold to obtain the image-text multimodal model. The change is the difference between the latest update result of the initial loss function and the previous update result of the latest update result.

[0046] As mentioned in the above embodiments, during the training of the initial multimodal model, the accuracy of the initial multimodal model is verified using the information extraction results of known historical medical record images. That is, in this embodiment, the training data used when training the initial multimodal model also includes the information extraction results of historical medical record images. These information extraction results include the following: multiple entities generated from the text content in the historical medical record images (each category of entity contains at least one training text box (a training text box refers to a text box generated from a historical medical record image)), location information for each entity (indicating the position of each entity in its corresponding historical medical record image), and multiple relationships determined based on the historical medical record images. These relationships include associations between different entities (such as...). Figure 2 The relationship between "WBC" and "7.06" in the text), and the association between multiple training text boxes contained in multiple entities of the same category (e.g., Figure 2 The relationship between 7.06 and 4.43 in [the original text]. Multiple entities, multiple relationships, and the location information of each entity generated from historical medical record images are included in the ground truth labels. Furthermore, each category of entities generated from historical medical record images contains at least one text box (i.e., a training text box) generated from the historical medical record images; however, the text content recorded in the multiple training text boxes containing multiple entities of the same category is all used to describe information of the same category, for example, Figure 2In the diagram, the yellow training text boxes describe English abbreviations used for testing items and belong to the "English abbreviations" category. The purple training text boxes describe reference units used for testing items and belong to the "reference range" category. Generating multiple entities based on text content in historical medical record images includes receiving annotation information (real labels) and generating entities based on the annotation information. The annotation information can be generated based on manually added annotations on historical medical record images. In this embodiment, when training the initial modal model using training data containing the above information, the initial multimodal model performs entity prediction and relationship prediction tasks on the training set generated from the training data. After performing the entity prediction and relationship prediction tasks, the initial multimodal model outputs corresponding prediction results. The prediction results output by performing the entity prediction task (i.e., entity prediction results) are multiple entities generated from historical medical record images in the training data. The prediction results output by performing the relationship prediction task are the relationships between entities, which are jointly represented by two entities with a relationship (i.e., the head entity and the tail entity). During training, the accuracy of the initial multimodal model in performing preset tasks (including entity prediction and relationship prediction) is determined using the real labels in the training data. This involves comparing the real labels with the prediction results to obtain the accuracy. If the accuracy is lower than the preset accuracy, the initial multimodal model and the preset initial loss function are iteratively updated until the change in the initial loss function is less than a preset threshold. This confirms that the initial multimodal model training is complete, and the initial multimodal model updated in the last iteration is output as the image-text multimodal model used in the application. The change in the initial loss function is used to measure the degree of update of the loss function. When the difference between the latest updated result of the loss function and the previous updated result (the change in the initial loss function) is less than the preset threshold, it indicates that continuing to update the loss function has little impact on the model's accuracy, and no further updates are necessary.

[0047] According to some optional embodiments of this application, the initial multimodal model includes multiple input layers and multiple task processing layers. The input layers include: a text box embedding layer, an image embedding layer, a location embedding layer, and a type embedding layer. The initial multimodal model is used to perform preset tasks on historical medical record images contained in the training set to obtain the prediction results output by the initial multimodal model. This includes: converting multiple entities recorded in the real labels into a first vector in the text box embedding layer; converting historical medical record images into a second vector in the image embedding layer; storing the location information of each entity in the location embedding layer; and converting the image type of the historical medical record images into a third vector in the type embedding layer. The first vector, second vector, and third vector, along with the location information, are fused into a feature vector. Different preset tasks are performed on the feature vector in different task processing layers to obtain the prediction results output by the task processing layers. The prediction results output by different task processing layers are different.

[0048] Figure 3 This is a model architecture diagram of a text-image multimodal model. As mentioned in the above embodiments, the initial multimodal model used in this application embodiment is a model with an adjusted model architecture. In this embodiment, the original transformer structure of the initial multimodal model is adjusted. The adjusted initial multimodal model contains multiple input layers and multiple task processing layers (used to execute different preset tasks mentioned in the above embodiments). In the task processing layer, a multilayer perceptron (MLP) can be used to process the preset tasks. Furthermore, since the initial multimodal model in this application embodiment is based on a transformer architecture, then... Figure 3 As shown, the text-image multimodal model also includes general architectures based on the transformer architecture, such as the multimodal transformer architecture and the output layer. Figure 3As shown, in this embodiment, the input layer of the model includes, in addition to a text box embedding layer (word / patch embedding) for receiving multiple text boxes generated from medical record images (including historical medical record images, images to be processed, and any image from which information is to be extracted), an image embedding layer (image embedding) for receiving medical record images, and a position embedding layer (position embedding) for receiving and storing the position information of entities, a class embedding layer (class embedding) for receiving the image type of the medical record images. During model training, the initial multimodal model converts multiple entities recorded by multiple real labels into vectors (i.e., the first vector) in the text box embedding layer (word / patch embedding), converts historical medical record images used as training data into vectors (i.e., the second vector) in the image embedding layer (image embedding), converts the image type of historical medical record images recorded in the real labels into vectors (i.e., the third vector) in the class embedding layer (class embedding), and stores the position information of each entity in the position embedding layer. Furthermore, in the multimodal transformer architecture, multiple vectors output from the input layer are fused with positional information to form a feature vector. For example, the first, second, and third vectors, along with the positional information, are concatenated sequentially into a single combined vector (i.e., a feature vector). Different preset tasks are then performed on the fused feature vector at each task processing layer. For example… Figure 3 As shown, the number of task processing layers (MLPHead) corresponds to the number of image types. Therefore, in this embodiment, the initial multimodal model includes at least a task processing layer for performing entity prediction tasks and a task processing layer for performing relation prediction tasks. After each of the above task processing layers performs its corresponding preset task, it outputs the task execution result (i.e., the prediction result) to the output layer. Figure 3 As shown, the prediction results received by the output layer include: the entity output by the entity prediction task (i.e., Output A Entity) and the relation output by the relation prediction task. Here, the relation indicates the relationship between entities; therefore, the output relation is represented in the following form: head entity - tail entity. For example, the output relation should contain... Figure 3 The relation head entity (Output BRelationHead) and relation tail entity (Output CRelation Tail) in the table.

[0049] Optionally, the initial loss function is iteratively updated based on the ground truth labels and multiple prediction results. Each iteration includes: for each image type, classifying the ground truth labels and multiple prediction results corresponding to the image type into datasets, resulting in multiple datasets corresponding to multiple image types; determining the prediction accuracy of the initial multimodal model for each image type based on the datasets, resulting in multiple prediction accuracies corresponding to multiple image types; determining multiple weights and multiple modulation factors corresponding to each image type, and determining the update result of the initial loss function based on the multiple prediction accuracies, multiple weights, and multiple modulation factors. Each image type includes multiple different entity categories, with different entity categories corresponding to different weights and different entity categories corresponding to different modulation factors. The modulation factors are used to adjust the contribution of entity categories in the process of updating the initial loss function, and the entity categories are used to indicate the type of information recorded in historical medical record images.

[0050] This application's embodiments introduce the concept of joint training. On one hand, it combines entity recognition and entity relationship recognition; on the other hand, it jointly trains different types of entity and relationship recognition tasks. By adding classembedding to the model input layer to optimize the model architecture, and by setting corresponding loss functions for different types of medical record images, the training process is optimized. Therefore, the image-text multimodal model in this application's embodiments can more accurately distinguish different types of medical record images and extract information more accurately. In this embodiment, adjusting the model architecture loss function can be achieved by adjusting the loss weights for different image types (i.e., types of medical record images). For example, the loss function weight ratio can be optimized based on the number of entities included in the image type. This embodiment uses CrossEntropy Loss as the basic loss function, which takes the following form: The base loss function with added loss weights is used as the initial loss function. The initial loss function for each image type is expressed in the following form: During the training process, by adjusting The value yields different numerical loss functions corresponding to different image types. In the above formula, c represents the c-th entity category, K represents the total number of entity categories, n represents the type of medical record image, N is the total number of samples, x represents a sample, and y represents the label corresponding to the sample. This represents the i-th input sample. Let be the vector representing the true label corresponding to the i-th sample in the c-th entity category. This represents the unnormalized raw score output by the model. These are model parameters. The activation function (sigmoid function) and the loss function correspond to which image type. The historical medical records of the image type are the samples. For example, when updating the loss function for the image type "ultrasound image", n represents the ultrasound image type, and the historical medical records of the ultrasound image type are the samples. This refers to the weights of entity category c in sample type n, specifically the weights corresponding to entity category c in image type n. Based on these different weights, a loss function is calculated for each entity category in image type n. The loss functions for each entity category are then summed to obtain the loss function for image type n. Note that the entity categories contained in historical medical record images differ for each image type; for example... Figure 2 In the laboratory report, the entity categories include four types: "English abbreviation", "test item", "result", and "reference range". This represents the proportion of the c-th entity category in sample type n. Entity category refers to the type of information recorded in the (historical) medical record image. For example, if the image type of the dataset being processed is a lab report, the entity categories include four types: "English abbreviation," "Test Item," "Result," and "Reference Range." When c represents "English abbreviation," The proportion of entities representing the "English abbreviations" category in all entities contained in historical medical record images (samples) of the lab report type; It is the modulation factor, which is set in this embodiment. The value range of is [0,5]. As can be seen from the above formula, by adjusting... The value can be adjusted. The degree of influence on the overall value of the loss function; this is achieved by setting [the appropriate parameters] in the loss function. ,when When the value approaches 1, it indicates that the proportion of all entities in entity category c within a certain sample type is 1. At this point, entity category c is an easily separable sample. Therefore, by adjusting... The value makes When the value of c approaches 0, the contribution of entity class c to the loss function decreases, which in turn increases the contribution of hard-to-distinguish samples (i.e., samples where the proportion of entity class c among all entities contained in the sample approaches 0) to the loss function. This vector represents the true label corresponding to the i-th sample in the c-th entity category. The true category is recorded in the true label. If the true label records the sample... If the category is c, then the sample The true category is c. =1; if the actual label records the sample The category is not c, sample The true category is not c. It is 0. This represents the unnormalized raw score output by the model. The i-th sample is an entity generated from historical medical record images in the dataset. During training, each entity... The entity category is known, therefore, calculation When, c is The actual entity category, therefore, in this embodiment, Also known as prediction accuracy; These are the parameters of the model, and their values ​​are preset values. Additionally, in this embodiment, the activation function is based on the formula... The described method is Activate, in the process of When activating, replace z in the formula with , where e is the base of the natural logarithm, and its value is 2.71828. In this embodiment, by introducing and The parameter settings indirectly affect the loss calculation for recognizing different entity categories, thereby achieving the goal of setting different loss functions for different entity categories. For example, when a certain type of medical record image contains two entity categories, K=2, and this type of loss function can be expressed as follows:

[0051] ,

[0052] At this point, substitute the values ​​determined based on the actual input and output of the model. , and This allows us to determine the loss function corresponding to this type. The method in this embodiment introduces weights. This results in different loss weights for different image types. Loss is calculated for each individual class, using the sigmoid function to calculate the probability of each class separately, ensuring that different classes do not affect each other. When K=2, it's equivalent to treating each class as a binary classification problem; when K>2, it's equivalent to treating each class as a multi-class classification problem. Simultaneously, a modulation factor is introduced. This can effectively alleviate the problem of sample imbalance when performing preset tasks.

[0053] According to some optional embodiments of this application, determining the prediction accuracy of the initial multimodal model for each image type based on the dataset includes: for each prediction result, determining the actual result corresponding to the prediction result in the ground truth labels, wherein the actual result corresponding to the entity prediction result is an entity, and the actual results corresponding to the relation head entity and relation tail entity are relations; determining the probability that the prediction result is the same as the actual result corresponding to the prediction result, wherein the probability is the prediction accuracy.

[0054] As mentioned in the previous embodiment, during model training, the entity category of each entity generated based on the medical record image is known. Therefore, in this embodiment, the prediction probability during the training process... It can also be referred to as prediction accuracy; during the training process. The following method described in this embodiment can be used to determine that the initial multimodal model outputs corresponding prediction results when performing different preset tasks, while the real labels record the actual results (i.e., real results) of these preset tasks. Therefore, during training, for a preset task, the prediction accuracy of the initial multimodal model for that preset task can be determined by determining the probability that its corresponding prediction result is the same as its corresponding actual result. For example, when the initial multimodal model performs an entity prediction task, it outputs entity prediction results, which are multiple entities generated based on the input historical medical record images. The actual result of the entity prediction task is the entity generated based on each medical record image recorded in the real labels. For each historical medical record image, the probability that its corresponding entity prediction result is the same as the entity recorded in its real label is determined. This probability is the prediction accuracy of a single historical image, while the prediction accuracy of the entity prediction task is the average of the prediction accuracies of multiple single historical images. When the initial multimodal model performs the relationship prediction task, it outputs a relationship prediction result, which is two entities with an association (i.e., a relationship head entity and a relationship tail entity). The actual result of the relationship prediction task is the relationship between historical medical record images recorded in the ground truth labels. For each historical medical record image, the probability that its corresponding relationship prediction result is the same as the relationship recorded in its corresponding ground truth label is determined. This probability is the prediction accuracy of a single historical medical record image. The prediction accuracy of the relationship prediction task is the ratio of the number C of historical medical record images with accurate relationship predictions to the total number D of medical record images in the dataset. The historical medical record images with accurate relationship predictions are those whose corresponding relationship prediction results are the same as the actual relationship recorded in the ground truth labels. Figure 2Taking a lab report as an example, the initial multimodal model performs entity prediction on the lab report and predicts "WBC". This prediction corresponds to the entity "WBC" in the true label, indicating accuracy. Therefore, the prediction accuracy for the entity category "WBC" is accurate. If all entities in the lab report are predicted, the initial multimodal model's prediction accuracy on this lab report is 100%. If only 80% of the entities are predicted, the initial multimodal model's prediction accuracy on this lab report is 80%. If there are 10 lab reports in the dataset, with 2 having a prediction accuracy of 100%, 2 having 80%, 2 having 60%, 2 having 50%, and 2 having 30%, then the initial multimodal model's prediction accuracy for the lab report type is (100%×2 + 80%×2 + 60%×2 + 50%×2 + 30%×2) / 10 = 64%.

[0055] Step S108: Determine the target connected subgraph corresponding to the image to be processed based on the image type of the image to be processed, and determine the preliminary information extraction result of the image to be processed based on the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting the entities as edges.

[0056] In step S108, a target connected subgraph corresponding to the image to be processed is determined based on the image type of the image to be processed. Based on the target connected subgraph and the processing result of text analysis of the image to be processed, information extraction is performed on the image to be processed, resulting in a preliminary information extraction result. In this embodiment, each image type of medical record image has an associated target connected subgraph. These target connected subgraphs are constructed synchronously during the initial multimodal modeling process. Each target connected subgraph associated with each image type includes multiple points and edges connecting those points. Each point in the target connected subgraph represents an entity, which is generated based on the medical record image whose image type corresponds to the image type of the target connected subgraph. The edges of the target connected subgraph are straight lines connecting two points (i.e., two entities) that have an association relationship. For example... Figure 2 In the lab report, multiple entities are generated from multiple text boxes under "Test Items," "English Abbreviations," "Results," and "Reference Range." These entities will serve as points in the target connected subgraph corresponding to this lab report type. Figure 2 The lines connecting some of these entities shown in the diagram will serve as edges in the target connected subgraph corresponding to the test report type.

[0057] Optionally, the target connected subgraph is constructed as follows: For each image type, the final prediction result output by the initial multimodal model during the last iteration of updating the loss function is obtained, wherein the final prediction result includes: the final entity prediction result and the final relationship prediction result. The final relationship prediction result includes at least one of the following: entity relationship and text relationship. The text relationship is used to indicate the association between the text content contained in multiple training text boxes generated based on the historical medical record image. Each training text box is used to select different text content contained in the historical medical record image. The target connected subgraph is generated by connecting multiple entities in the final entity prediction result according to the entity relationship, wherein each entity in the final entity prediction result is a point in the target connected subgraph, and the straight line used to connect multiple entities with entity relationships is an edge in the target connected subgraph. The target connected subgraph includes: a first type of connected subgraph containing entities of different categories and a second type of connected subgraph containing entities of the same category.

[0058] In this embodiment, each image type has a corresponding (target) connected subgraph. In this embodiment, the connected subgraph that corresponds to the image type of the image to be processed, which is found in multiple connected subgraph queries, is called the target connected subgraph. Figure 4 This is a schematic diagram of the target connected subgraph, such as... Figure 4 As shown in this embodiment, the target connected subgraph corresponding to each image type is a graph constructed after determining the points and edges of the graph based on entities and the relationships between entities. The entities and relationships extracted by the initial multimodal model for the medical record images are not exactly the same in each training round. That is to say, the target connected subgraph corresponding to each image type is updated with the iteration update of the loss function. Therefore, in this embodiment, the prediction result output by the initial multimodal model when performing the last iteration update for each image type (i.e., the final prediction result) is used to construct the target connected subgraph for each image type. The aforementioned final prediction result includes the final entity prediction result and the final relationship prediction result. Since each entity category contains multiple training text boxes, and different training text boxes record different text content from the image, the final relation prediction result includes not only the relationships between entities (i.e., entity relationships) but also the relationships between the text content recorded in multiple training text boxes under each entity category (i.e., text relationships). Therefore, when generating a target connected subgraph for a medical record image of a certain image type, there are actually two types of target connected subgraphs: one is a connected subgraph where the entities represented by the included nodes do not completely belong to the same category (i.e., the first type of connected subgraph), and the other is a connected subgraph where the entities represented by the included nodes are of the exact same type (i.e., the second type of connected subgraph). That is, the first type of connected subgraph contains entities of different categories, for example... Figure 4 China Unicom Figure 2The second type of connected subgraph contains only entities of the same category, for example... Figure 4 China Unicom Figure 1 Helianzi Figure 3 .by Figure 2 Taking a lab report as an example, let's illustrate the process of generating a target connected subgraph for this type of lab report. For a lab report, its corresponding connected subgraph contains a subgraph that connects entities such as English abbreviations, results, reference ranges, and other information that may exist in the lab report (e.g., indicator name, indicator unit, whether there are any abnormal markers) related to the same indicator (i.e., the first type of connected subgraph). The form of this subgraph is similar to... Figure 4 China Unicom Figure 2 Similarity refers to a connected subgraph composed of entities of different categories. Furthermore, these entities with different metrics can also construct multiple independent subgraphs, for example... Figure 4 China Unicom Figure 1 (Entity 1-Entity 1-Entity 2-Entity 3-Entity 4); and Connecting Sub-entities Figure 2 (Entity 2-Entity 3-Entity 4) are two independent subgraphs; among them, the connected subgraphs... Figure 1 Entity 1 and Entity 1 are different entities under the same entity category. For example, the English abbreviations in the laboratory report are entity categories. In this case, “WBC”, “NEU#”, “LYM#”, etc. are all entity 1. Figure 5 This is the entity extraction result of ultrasound report-type medical record images, with Figure 5 The example ultrasound report shown illustrates the construction of a target connectivity graph corresponding to this type of ultrasound report. Entities of the same category seen in ultrasound may have multiple text boxes, for example... Figure 5 The ultrasound report contains two entity categories: "Ultrasound Description" and "Ultrasound Diagnosis." Each category contains multiple text boxes. Therefore, when generating the target connected subgraph of the ultrasound report, a second type of connected subgraph will be generated based on the multiple text boxes under the same entity category (e.g., ...). Figure 4 China Unicom Figure 3 ).like Figure 5 As shown, the entity category "Ultrasound Diagnosis" is composed of multiple entities corresponding to the following text boxes: "Ultrasound Diagnosis," "Myocardial Involvement Disorders," "Left Ventricular Enlargement," "Left Ventricular Wall Motion Abnormalities," "Bicuspid Aortic Valve," "Mild to Moderate Aortic Stenosis with Mild to Moderate Regurgitation," and "Left Ventricular Function Reduction." Each text box corresponds to one entity (e.g., entity m). The connections between these entities are generated according to the positional coordinates of the text boxes (i.e., the word order of the content recorded in the text boxes in the original text). Based on the connected subgraph technique, the final output will be the linking result of the subgraph constructed from the same ultrasound-seen entities, such as the output... Figure 4 The second type of connected subgraph in the form of entity m-entity m-entity m.

[0059] According to some optional embodiments of this application, the preliminary information extraction result of the image to be processed is determined based on the processing result output by the image-text multimodal model and the target connected subgraph, including: for each target entity contained in the processing result, querying the target entity in multiple target connected subgraphs, and querying multiple other entities located on the same edge as the target entity; converting the target entity and multiple other entities into corresponding fields, and generating a data structure based on the multiple fields, wherein the data structure is the information extraction result.

[0060] In step S108, after determining the target connected subgraph corresponding to the image to be processed, the preliminary information extraction result of the image to be processed is determined by combining the target connected subgraph and the processing result (target entities and relations) output by the graph-text multimodal model for the image to be processed. Specifically, in this embodiment, the partial subgraph associated with the target entity can be determined in the target connected subgraph by querying the target entities contained in the processing result. This partial subgraph is the information extraction result of the target entity. The preliminary information extraction result of the image to be processed contains multiple partial subgraphs associated with multiple target entities. The partial subgraph corresponding to each target entity is composed of the target entity and multiple other entities located on the same edge as the target entity in the target connected subgraph corresponding to the image to be processed. In this embodiment of the application, the final output information extraction result is structured data. Therefore, after determining the preliminary information extraction result of the image to be processed according to the above method, it is also necessary to convert the multiple entities (including target entities and other entities) contained in some sub-images into corresponding fields according to their corresponding text content, and then generate the corresponding data structure according to these fields. The generated data structure is the information extraction result of the image to be processed. Figure 6 This is a schematic diagram illustrating the format conversion of the image information extraction results, such as... Figure 6 As shown, with Figure 2 Taking the lab report shown as an example, the image-text multimodal model processes and analyzes the multiple text boxes generated from the lab report and their related information, outputting the following (target) entities: "WBC", "White blood cell count", "7.06", "3.5-9.5", "10^9 / L", "NEU#", "Neutral cell count", "4.43", "1.8-6.3", and "10^9 / L". Multiple entities with relationships are output as a single line. Information is extracted and transformed from the target connected subgraph corresponding to the lab report to generate... Figure 6 The extraction results, such as Figure 6As shown, when converting the preliminary information extraction results into the output results of the data structure format, "WBC" and "NEU#" are stored in the item code field of the data structure, "white blood cell count" and "neutrophil count" are stored in the item name field of the data structure, "7.06" and "4.43" are stored in the item result field of the data structure, "3.5--9.5" and "1.8--6.3" are stored in the reference field of the data structure, and "10^9 / L" is stored in the item unit field of the data structure. In fact, after each medical record image to be processed is processed by the method provided in this application embodiment, the output information extraction results all follow the same data format, which is to extract the information recorded in the image to be processed and fill it into the corresponding fields in the data structure. The specific format of the data structure can be predefined, and can be defined as follows: Figure 6 The data structure shown here, which includes the itemcode, itemname, itemresult, reference, and itemunit fields, can also be defined as other data structures.

[0061] Step S110: Determine the final information extraction result of the image to be processed based on the preliminary information extraction result.

[0062] After obtaining a preliminary information extraction result from the image to be processed in step S108, step S110 generates a final information extraction result for the image to be processed based on the obtained preliminary information result. This final information extraction result may be the preliminary information extraction result obtained in step S108, or it may be another information extraction result determined by further processing the text boxes and combining the target connected subgraph corresponding to the image to be processed with the further processed text boxes. In this embodiment, the extracted final information result can be used to assist doctors in managing and issuing early warnings for patients with chronic diseases.

[0063] Optionally, determining the final information extraction result of the image to be processed based on the preliminary information extraction result includes: determining the length of the target entity included in the preliminary information extraction result, and determining whether to perform long text segmentation processing on the target entity based on the length of the target entity, wherein long text segmentation processing indicates dividing the target entity into multiple sub-entities of equal length; if it is determined not to perform long text segmentation processing, the preliminary information extraction result is determined as the final information extraction result; if it is determined to perform long text segmentation processing, the final information extraction result is determined based on the multiple sub-entities and the target connected subgraph.

[0064] When generating multiple text boxes from an image to be processed, there may be instances where the text boxes are incorrectly segmented (a single text box contains multiple entities); for example, when... Figure 2 In the example lab report image, the reference value and unit are divided into a text box. At this point, it's impossible to simultaneously identify these two entities and their corresponding links using only a multimodal image-text model. However, further text entity relationship extraction based on the multimodal recognition results can effectively solve this problem. Therefore, the long text segmentation method provided in this embodiment, after obtaining the preliminary information extraction result according to the method provided in the above embodiment, determines whether further text entity information extraction is needed based on the length of the (target) entity contained in the preliminary information extraction result. Since long text constitutes a large proportion of medical record images, to avoid information loss due to excessive text length, in this embodiment, when the target entity length is too long (e.g., the target entity length is greater than the predefined entity length), a long text segmentation method is performed using a text segmentation model (included in the multimodal image-text model) to optimize the long text. Related long text segmentation methods divide the input data into fixed-size blocks, but directly defining the length N may result in a meaningful word being segmented into two different text blocks. For example, if a conventional long text segmentation method is performed, when the length N is defined as 512, it will... Figure 2 The phrase "neutrophil count" in the lab report is separated into two parts. Therefore, this embodiment provides an optimized long text segmentation method. In this embodiment, text boxes are introduced into the long text segmentation process. The preceding text box with a length less than the text length threshold (N) is predefined as the segmentation boundary. Then, the corresponding text blocks (i.e., target entities) are filled to ensure that all text blocks (i.e., target entities) have the same length, ensuring that the same word is segmented into the same text block, thus improving the accuracy of relation extraction. In the method provided in the application embodiment, after performing the long text segmentation processing in this application embodiment, the result of long text processing is obtained. The result of long text processing (multiple sub-entities) is further combined with the target connected subgraph corresponding to the image to be processed for information extraction to obtain the secondary information extraction result. In the case where long text segmentation is required, the secondary information extraction result is converted into a data structure before it becomes the final information extraction result of the image to be processed. In the case where long text segmentation is not required, the preliminary information extraction result is converted into a data structure format to obtain the final information extraction result of the image to be processed.

[0065] According to some optional embodiments of this application, determining whether to perform long text segmentation processing on the target entity based on its length includes: when the length of the target entity is greater than a preset length threshold, determining multiple segmentation text boxes based on the target entity, and sorting the multiple segmentation text boxes according to the position information of each segmentation text box contained in the target entity in the initial image to obtain a sorting sequence, wherein the sorting sequence records the identifiers of multiple segmentation text boxes, and different segmentation text boxes are used to select different sub-entities contained in the target entity; sequentially accumulating the length of each segmentation text box according to the order indicated by the sorting sequence, and after each accumulation, determining the comparison result between the accumulation result and the preset length threshold; in the previous If the comparison result indicates that the accumulated result is less than or equal to the preset length threshold and the next comparison result indicates that the accumulated result is greater than the preset length threshold, the multiple segmented text boxes corresponding to the previous comparison result are combined into an initial long text box, and the accumulation continues until all segmented text boxes contained in the sorted sequence have been accumulated, and multiple initial long text boxes are output. For each initial long text box, if the length of the initial long text box is less than the preset length threshold, the initial long text box is padded with padding characters until the length of the initial long text box reaches the preset length threshold, then the padding stops, and the child entity is output. If the length of the initial long text box is equal to the preset length threshold, the initial long text box is output as a child entity.

[0066] As mentioned in the previous embodiment, the specific method for determining whether to perform long text segmentation based on the length of the target entity is as follows: when the length of the entity (i.e., the target entity) in the preliminary information extraction result is greater than the text box of the preset length threshold (i.e., the text length threshold (N)), it is determined that long text segmentation needs to be performed. The preset length threshold can be the length of the longest text content in the image to be processed, or it can be other custom values. In this embodiment, before performing long text segmentation on the multiple target entities generated from the image to be processed, multiple segmentation text boxes are determined based on the target entities. Each segmentation text box selects one entity contained within the target entity. Further, the multiple segmentation text boxes are arranged into one or more sorting sequences. Specifically, the segmentation text boxes are arranged to generate a sorting sequence based on the position information (such as position coordinates) of each segmentation text box. The sorting sequence can record either the segmentation text box or its identifier. In the text segmentation model of the image-text multimodal model, when performing long text segmentation, the segmentation point is selected in the sorted sequence. To avoid segmenting a complete word into different text boxes, in this embodiment, the preceding text box with a length less than a preset length threshold (N) is selected as the segmentation boundary. Specifically, the segmentation point can be determined by accumulating the length of the text boxes according to the order of their arrangement: starting from the first segmented text box, the length of the text box is accumulated one by one according to the order indicated by the sorted sequence. After each accumulation, the accumulated result is compared with the preset length threshold (N). This continues until the comparison result obtained from the previous comparison of the accumulated result and the preset length threshold (N) indicates that the accumulated result is less than or equal to N, and the comparison result obtained from the next comparison of the accumulated result and the preset length threshold (N) indicates that the accumulated result is greater than N. Then, the multiple text boxes corresponding to the accumulated result of the previous comparison are combined into an initial long text box. The last position of the last text box among the multiple text boxes contained in the initial long text box is the segmentation point. After completing one of the above-described cuts, the accumulation starts again from the next text box after the cut point. The cut points are determined according to the method described in the above embodiments until all text boxes in the sorted sequence have undergone the above accumulation process. After the above processing, multiple initial long text boxes can be generated based on a sorted sequence. To ensure that the lengths of the generated initial long text boxes are consistent, this embodiment further fills the multiple initial long text boxes with lengths less than a preset length threshold using meaningless characters as filler characters (such as empty characters) to fill the initial long text boxes with lengths less than N, making each initial long text box equal to N. The long text cutting is then stopped, and the filling results (i.e., child entities) of multiple initial long text boxes with lengths equal to the preset length threshold are output. For multiple initial long text boxes with lengths equal to the preset length threshold, there is no need to use meaningless characters to fill them; the initial long text boxes are directly output as child entities.The long text segmentation method provided in this application ensures that words with complete semantics are segmented into a single text box, thereby improving the accuracy of relation extraction results.

[0067] Through the above steps, a multimodal image-text model trained based on an optimized loss function can be used to extract multimodal image-text information from the corrected image to be processed, avoiding the problem of entity and relation misalignment and improving the accuracy of image information extraction. Specifically, in step S102, the image to be processed is preprocessed to correct tilt, initially correcting the image skew and providing a regular and clear input image for step S104, improving the accuracy of text analysis in step S104. In step S104, the text content of the image to be processed after tilt correction is segmented, dividing the text content in the image to be processed into multiple text boxes to form structured text units, providing key text semantics and spatial location information for step S106, enabling the multimodal image-text model to accurately identify entity categories. Step S106 uses an image type and entity category-aware Transformer architecture and a category-weighted loss function to achieve high-precision entity recognition and entity relation extraction, laying the foundation for building the multimodal image-text model in step S108. Connectivity subgraphs provide accurate information, which helps improve the accuracy of the final information extraction results. In step S108, the connected subgraph algorithm is used to cluster the scattered entities and entity relationships output by the multimodal graph model into a connected graph, forming semantically complete information. This provides clear contextual boundaries and related information for subsequent text extraction, reducing the problems of erroneous merging and missed extraction. In step S110, the length of the preliminary information extraction results output in step S108 is determined to determine whether to perform fine-grained text processing, thereby segmenting long texts. This ensures that meaningful words are not segmented during the long text segmentation process, requiring only lightweight text processing and avoiding global re-parsing of long texts after processing the image. This not only improves the accuracy of the final information extraction results but also avoids wasting computational resources. In summary, the method provided in this application embodiment achieves the technical effect of improving the accuracy of information extraction from case images through the collaborative closed loop of steps S101 to S110.

[0068] Figure 7 This is a structural diagram of an image information extraction device according to an embodiment of this application, such as... Figure 7As shown, the device for extracting image information includes: an acquisition module 70, used to acquire the image to be processed and preprocess the image to obtain an initial image; a text box generation module 72, used to perform text analysis processing based on the initial image to obtain text analysis processing results, wherein the text analysis processing results include: multiple text boxes and related information for each text box, different text boxes are used to select different text content contained in the initial image, and the related information includes: the position information of the text box in the initial image, and the text content located within the text box; a processing module 74, used to determine the image type of the image to be processed based on the image to be processed and the text analysis processing results; and to process the image to be processed, the text analysis processing results, and the image type using a multimodal graph model to obtain the processing results output by the multimodal graph model, wherein the processing results include: multiple target entities, and the relationships between the target entities. The relationship between the text boxes is defined as follows: each text box contains a text content that generates a target entity. The image-text multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images, each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The first determining module 76 is used to determine the target connected subgraph corresponding to the image to be processed according to the image type of the image to be processed, and to determine the preliminary information extraction result of the image to be processed according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting entities as edges. The second determining module 78 is used to determine the final information extraction result of the image to be processed according to the preliminary information extraction result.

[0069] Figure 8 This is a flowchart of the process of a device for extracting image information, such as... Figure 8As shown, when the image information extraction device extracts information from the image to be processed, it first inputs an image to be estimated into the acquisition module 70. The acquisition module 70 preprocesses the image to be processed, for example, by performing preliminary image tilt correction. Then, the text box generation module 72 performs OCR processing on the tilt-corrected image to obtain the OCR result (i.e., the text box). Based on the input image to be processed and the OCR result, image classification is performed. The classified image, image category, and OCR result (i.e., the text box) are transmitted to the processing module 74. The processing module 74 inputs the classified image, image category, and OCR result (i.e., the text box) into the trained image-text multimodal model. The output of the image-text multimodal model is used as the input of the first determination module 76. The first determination module 76 processes the image-text multimodal model output. (e.g., multiple target entities) are input into the constructed image entity connected subgraph (i.e., target connected subgraph), and the preliminary result of image information extraction (i.e., preliminary image information extraction result) is output. The second determining module 78 determines the final information extraction result of the image to be processed based on the preliminary image information extraction result. For example, the second determining module 78 determines whether there are target entities that need further text entity information extraction based on the length of the target entities. If not, the preliminary information extraction result is directly output as the final information extraction result; if so, the preliminary information extraction result is input into the trained text information extraction model, and the final information extraction result is output based on the constructed text entity connected subgraph (i.e., target connected subgraph). In this embodiment, the final information extraction result includes: structured entities and all entities associated with them.

[0070] It should be noted that, Figure 7 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.

[0071] Figure 9 This is a flowchart illustrating the process of extracting information from images uploaded by patient A using a device for extracting image information. Figure 9As shown, patient A actively uploads their completed laboratory test results, i.e., the image to be processed is in the form of a laboratory test report. The image information extraction device first judges whether the image is qualified based on its clarity. If the image is clear and readable, it is confirmed as qualified and proceeds to the next step; otherwise, if the image is blurry and the text cannot be extracted, it is judged as unqualified and returned to patient A. The acquisition module 70 performs tilt correction on the received image and classifies it into a laboratory test report by recognizing the text content in the received image. Further, the text box generation module 72 uses OCR technology to recognize the text content on the laboratory test report, including indicator names, values, units, and reference ranges, and generates multiple text boxes based on the text content. The processing module 74 uses a multimodal image-text model to perform entity recognition and relation extraction based on the classification and OCR results (i.e., text boxes), such as recognizing indicators such as "WBC (white blood cell count)" and "NEU# (neutrophil count)" and their corresponding results and abnormal markers. The first determining module 76 performs correlation analysis based on the results output by the processing module 74 and the target connected subgraph (including the patient's age, gender, past medical history, and current health status) generated based on the patient A's data during model training, and outputs preliminary information extraction results; the second determining module 78 outputs the final information extraction results of the image input by patient A based on the preliminary information extraction results. Figure 9 As shown, after obtaining the final information extraction results, the system determines whether the indicators on the laboratory report are abnormal based on the analysis results and the preset chronic disease management strategy. If an abnormality is found (such as a white blood cell count exceeding the reference range), the system will immediately send an alert to patient A and their doctor, indicating the need for attention and further action. If the indicators on the laboratory report are within the normal range, the system will update the patient's health record and enrich their health information.

[0072] The image information extraction method provided in this application embodiment can also be executed by a chronic disease management system that includes a data acquisition module, a data preprocessing module, an image classification module, a multimodal information extraction module, a text information extraction module, a chronic disease indicator management module, and a chronic disease auxiliary decision-making and early warning module. The data acquisition module serves as the entry point for raw medical record image data uploaded by patients or medical staff, receiving the images to be processed. The data preprocessing module performs preliminary image preprocessing based on the uploaded images, such as image tilt correction, low-quality image filtering, and duplicate image removal. The OCR module uses OCR technology to acquire image text content, text boxes, and position coordinate information. The image classification module first performs an image classification model based on image pixels to classify images into categories such as wounds, blood pressure monitors, and medicine boxes. Then, it performs text classification based on the image text content acquired by the OCR module, further distinguishing medical record images such as lab reports, ultrasounds, electrocardiograms, and surgical records. The multimodal information extraction module calls upon a text-image multimodal model, combines image classification results, and integrates text-image multimodal information. It then performs multi-task joint learning through cross-modal alignment tasks, finally outputting preliminary entity relationship information extraction results based on a connected subgraph. The text information extraction module further extracts entity relationships based on text relation entity information extraction techniques, ultimately outputting the final entity relationship results (i.e., the final information extraction results) based on the connected subgraph. The chronic disease indicator management module provides structured output of indicators of interest to physicians for different chronic diseases, combining original labeled anomaly indicators from images with medical expert knowledge bases to mark corresponding anomalies. The chronic disease auxiliary decision-making and early warning module displays indicators of interest to physicians for different chronic diseases within the system, supporting corresponding reminders and warnings based on strategies developed by medical staff.

[0073] Figure 10 This is a flowchart of the chronic disease management system tracking the blood lipid measurement results of patient B, such as... Figure 10 As shown, Patient B uploaded medical record images to the chronic disease management system and underwent information extraction. The information extraction results provided an early warning for Patient B's blood lipids. Therefore, the chronic disease management system manages Patient B's chronic disease. Figure 10As shown, the system prompts Patient B that they have not uploaded their blood lipid results within the specified period. The system will remind the patient via SMS, telephone, etc. If the patient still fails to upload after a preset number of reminders (e.g., N times), the system will notify Patient B's family to monitor and upload the relevant indicators in a timely manner. After Patient B or their family uploads the results, the chronic disease management system executes the image information extraction process: image classification (to confirm it is a blood lipid measurement report) and OCR technology to identify the text content and its corresponding location information in the image. Next, based on the results of image classification and OCR recognition, the system extracts entities and their relationships using a previously trained image-text multimodal model, such as identifying the names, values, units, and reference ranges of key indicators such as "total cholesterol" and "low-density lipoprotein". Further, the system integrates and analyzes the extracted entity information with Patient B's basic information (such as age, past medical history, etc.) and previous medical records to determine whether the blood lipid measurement results are abnormal or require special attention. At the same time, based on the analysis results and the preset chronic disease management strategy, the system determines whether Patient B's blood lipid indicators trigger the warning strategy. If any indicators are abnormal (such as total cholesterol exceeding the reference range), the system will send an alert to Patient B and their doctor or caregiver, indicating the need for further medical examination or adjustment of the treatment plan. If no abnormalities are found, or if the abnormalities have been properly managed, the system will store the structured results of this blood lipid measurement in Patient B's personal health record, providing updated data support for long-term chronic disease management and health status tracking.

[0074] In another aspect of this application, a non-volatile storage medium is provided, which stores a computer program, wherein the device where the non-volatile storage medium is located executes the above-described method for extracting image information by running the computer program.

[0075] The aforementioned non-volatile storage medium is used to store a program that performs the following functions: acquiring an image to be processed and preprocessing the image to obtain an initial image; performing text analysis processing on the initial image to obtain text analysis processing results, wherein the text analysis processing results include: multiple text boxes and related information for each text box, different text boxes are used to select different text content contained in the initial image, and the related information includes: the position information of the text box in the initial image, and the text content within the text box; determining the image type of the image to be processed based on the image to be processed and the text analysis processing results; and using a graph-text multimodal model to process the image to be processed, the text analysis processing results, and the image type to obtain the processing results output by the graph-text multimodal model, wherein the processing results include: multiple target entities, and the relationships between target entities. The relationship between the text boxes is defined as follows: each text box contains text content that generates a target entity. The image-text multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images, each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The target connected subgraph corresponding to the image to be processed is determined according to the image type of the image to be processed. The preliminary information extraction result of the image to be processed is determined according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting the entities as edges. The final information extraction result of the image to be processed is determined according to the preliminary information extraction result.

[0076] In another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the above-described method for extracting image information through the computer program.

[0077] The processor in the aforementioned electronic device is used to run a program that performs the following functions: acquires an image to be processed and preprocesses the image to obtain an initial image; performs text analysis processing on the initial image to obtain text analysis processing results, wherein the text analysis processing results include: multiple text boxes and related information for each text box, different text boxes are used to select different text content contained in the initial image, and the related information includes: the position information of the text box in the initial image, and the text content within the text box; determines the image type of the image to be processed based on the image to be processed and the text analysis processing results; and uses a graph-text multimodal model to process the image to be processed, the text analysis processing results, and the image type to obtain the processing results output by the graph-text multimodal model, wherein the processing results include: multiple target entities, target entities The relationship between the text and images is defined as follows: each text box contains text content that generates a target entity. The image-text multimodal model is trained using training data and multiple loss functions on an initial multimodal model based on the Transformer architecture. The training data includes various types of historical medical record images, each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The target connected subgraph corresponding to the image to be processed is determined according to the image type of the image to be processed. The preliminary information extraction result of the image to be processed is determined according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting the entities as edges. The final information extraction result of the image to be processed is determined according to the preliminary information extraction result.

[0078] In another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the steps of the above-described method for extracting image information.

[0079] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 11 A hardware block diagram of a computer terminal for implementing a method for extracting image information is shown. Figure 11 As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 11The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.

[0080] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0081] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image information extraction method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned image information extraction method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0083] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0084] It should be noted that each module in the above-mentioned image information extraction device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0085] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0086] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0091] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for extracting image information, characterized in that, include: Acquire the image to be processed and preprocess the image to obtain an initial image; Text analysis processing is performed on the initial image to obtain text analysis processing results, wherein the text analysis processing results include: multiple text boxes and related information for each text box, different text boxes are used to select different text content contained in the initial image, and the related information includes: the position information of the text box in the initial image, and the text content located in the text box; The image type of the image to be processed is determined based on the image to be processed and the text analysis processing result; and a text-image multimodal model is used to process the image to be processed, the text analysis processing result, and the image type to obtain the processing result output by the text-image multimodal model. The processing result includes: multiple target entities and the relationships between the target entities. Each text box contains text content that generates a target entity. The text-image multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images. Each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The target connected subgraph corresponding to the image to be processed is determined according to the image type of the image to be processed, and the preliminary information extraction result of the image to be processed is determined according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting the entities as edges. The final information extraction result of the image to be processed is determined based on the preliminary information extraction result.

2. The method according to claim 1, characterized in that, The training data also includes: multiple ground truth labels, wherein each ground truth label corresponds to one historical medical record image, each ground truth label is used to record the image type of the historical medical record image, multiple entities generated based on the historical medical record image, multiple relationships, and the location information of each entity, wherein multiple entities of the same category contain multiple training text boxes, each relationship includes: the association relationship between multiple entities, the association relationship between multiple training text boxes contained in entities of the same category, each training text box is used to select the text content contained in the historical medical record image, different training text boxes select different text content, and the location information is used to indicate the position of the training text box in the historical medical record image; The image-text multimodal model was trained using the following method: The historical medical record images belonging to the same image type and their true labels are determined as the training set corresponding to the image type. For each image type, the initial multimodal model is used to perform a preset task on the historical medical record images included in the training set to obtain the prediction result output by the initial multimodal model. The prediction result output by the initial multimodal model is different when performing different preset tasks. The preset tasks include: entity prediction task and relationship prediction task. The prediction result includes: entity prediction result, relationship head entity and relationship tail entity. The relationship head entity and the relationship tail entity jointly indicate the relationship between the entities. The initial loss function is iteratively updated based on the true labels and multiple prediction results until the change in the initial loss function is less than a preset threshold, thereby obtaining the image-text multimodal model. The change is the difference between the latest update result of the initial loss function and the previous update result of the latest update result.

3. The method according to claim 2, characterized in that, The initial multimodal model includes multiple input layers and multiple task processing layers. The input layers include: a text box embedding layer, an image embedding layer, a location embedding layer, and a type embedding layer. The initial multimodal model is used to perform a preset task on the historical medical record images included in the training set to obtain the prediction results output by the initial multimodal model, including: In the text box embedding layer, the multiple entities recorded in the real label are converted into a first vector; in the image embedding layer, the historical medical record image is converted into a second vector; in the location embedding layer, the location information of each entity is stored; and in the type embedding layer, the image type of the historical medical record image is converted into a third vector. The first vector, the second vector, the third vector, and the position information are fused into a feature vector; Different preset tasks are performed on the feature vector at different task processing layers to obtain the prediction results output by the task processing layer, wherein the prediction results output by different task processing layers are different.

4. The method according to claim 2, characterized in that, The initial loss function is iteratively updated based on the true labels and multiple prediction results, wherein each iteration update includes: For each image type, the true label and multiple prediction results corresponding to the image type are classified into a dataset to obtain multiple datasets corresponding to multiple image types; Based on the dataset, the prediction accuracy of the initial multimodal model for each image type is determined, resulting in multiple prediction accuracies corresponding to multiple image types; Multiple weights and multiple modulation factors are determined for each image type, and the update result of the initial loss function is determined based on multiple prediction accuracies, multiple weights and multiple modulation factors. Each image type includes multiple different entity categories, different entity categories correspond to different weights, and different entity categories correspond to different modulation factors. The modulation factors are used to adjust the contribution of the entity category in the process of updating the initial loss function, and the entity category is used to indicate the type of information recorded in the historical medical record image.

5. The method according to claim 4, characterized in that, Determining the prediction accuracy of the initial multimodal model for each image type based on the dataset includes: For each prediction result, the actual result corresponding to the prediction result is determined in the real label, wherein the actual result corresponding to the entity prediction result is the entity, and the actual results corresponding to the relation head entity and relation tail entity are the relations; Determine the probability that the predicted result is the same as the actual result corresponding to the predicted result, wherein the probability is the prediction accuracy.

6. The method according to claim 1, characterized in that, The target connected subgraph is constructed using the following method: For each image type, the final prediction result output by the initial multimodal model during the last iteration of updating the loss function is obtained, wherein the final prediction result includes: final entity prediction result and final relationship prediction result, wherein the final relationship prediction result includes at least one of the following: entity relationship and text relationship, wherein the text relationship is used to indicate the association between the text content contained in multiple training text boxes generated based on the historical medical record image, and each training text box is used to select different text content contained in the historical medical record image; A target connected subgraph is generated by connecting multiple entities in the final entity prediction result according to the entity relationship. Each entity in the final entity prediction result is a point in the target connected subgraph, and the straight line used to connect multiple entities with the entity relationship is an edge in the target connected subgraph. The target connected subgraph includes: a first type of connected subgraph containing entities of different categories, and a second type of connected subgraph containing entities of the same category.

7. The method according to claim 1, characterized in that, Based on the processing results output by the image-text multimodal model and the target connected subgraph, the preliminary information extraction results of the image to be processed are determined, including: For each target entity included in the processing result, the target entity is queried in the multiple target connected subgraphs, and multiple other entities located on the same edge as the target entity are queried. The target entity and the other entities are converted into corresponding fields, and a data structure is generated based on the fields, wherein the data structure is the information extraction result.

8. The method according to claim 1, characterized in that, The final information extraction result of the image to be processed is determined based on the preliminary information extraction result, including: The length of the target entity included in the preliminary information extraction result is determined, and whether to perform long text segmentation processing on the target entity is determined based on the length of the target entity, wherein the long text segmentation processing indicates dividing the target entity into multiple sub-entities of equal length; If it is determined that the long text segmentation process will not be performed, the preliminary information extraction result will be determined as the final information extraction result; If the long text segmentation process is to be performed, the final information extraction result is determined based on the multiple sub-entities and the target connected subgraph.

9. The method according to claim 8, characterized in that, Determining whether to perform long text segmentation on the target entity based on its length includes: When the length of the target entity is greater than a preset length threshold, multiple segmented text boxes are determined based on the target entity, and the multiple segmented text boxes are sorted according to the position information of each segmented text box contained in the target entity in the initial image to obtain a sorting sequence. The sorting sequence records the identifiers of the multiple segmented text boxes, and different segmented text boxes are used to select different sub-entities contained in the target entity. The length of each segmented text box is sequentially accumulated according to the order indicated by the sorting sequence. After each accumulation, the comparison result between the accumulated result and the preset length threshold is determined. If the previous comparison result indicates that the accumulated result is less than or equal to the preset length threshold and the next comparison result indicates that the accumulated result is greater than the preset length threshold, the multiple segmented text boxes corresponding to the previous comparison result are combined into an initial long text box, and the accumulation is continued until all the segmented text boxes included in the sorting sequence have been accumulated, and multiple initial long text boxes are output. For each of the initial long text boxes, if the length of the initial long text box is less than the preset length threshold, the initial long text box is filled with padding characters until the length of the initial long text box reaches the preset length threshold, then the filling stops and the sub-entity is output. If the length of the initial long text box is equal to the preset length threshold, the initial long text box will be output as the sub-entity.

10. An apparatus for extracting image information, characterized in that, include: The acquisition module is used to acquire the image to be processed and to preprocess the image to obtain an initial image; A text box generation module is used to perform text analysis processing based on the initial image to obtain text analysis processing results. The text analysis processing results include: multiple text boxes and related information for each text box. Different text boxes are used to select different text content contained in the initial image. The related information includes: the position information of the text box in the initial image and the text content located within the text box. The processing module is used to determine the image type of the image to be processed based on the image to be processed and the text analysis processing result; and to process the image to be processed, the text analysis processing result and the image type using a text-image multimodal model to obtain the processing result output by the text-image multimodal model. The processing result includes: multiple target entities and the relationships between the target entities. Each text box contains text content that generates a target entity. The text-image multimodal model is obtained by training an initial multimodal model based on the Transformer architecture using training data and multiple loss functions. The training data includes various types of historical medical record images. Each type of historical medical record image includes different entity categories, and different entity categories correspond to different loss functions. The first determining module is used to determine the target connected subgraph corresponding to the image to be processed according to the image type of the image to be processed, and to determine the preliminary information extraction result of the image to be processed according to the processing result output by the image-text multimodal model and the target connected subgraph. Each target connected subgraph is a graph constructed with multiple entities as nodes and straight lines connecting the entities as edges. The second determining module is used to determine the final information extraction result of the image to be processed based on the preliminary information extraction result.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the method for extracting image information according to any one of claims 1 to 9 by running the computer program.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method for extracting image information according to any one of claims 1 to 9 through the computer program.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method for extracting image information as described in any one of claims 1 to 9.

Images