Method and device for multimedia presentation of clinical diagnosis report based on global optimal matching

Abstract

The application provides a clinical diagnosis report multimedia display method and device based on global optimal matching, wherein the method comprises the following steps: acquiring report features of each report target in a clinical diagnosis report and image features of each image target in a medical image; determining the similarity between the report features of any report target and the image features of any image target based on the report features of any report target, the image features of any image target and the weights corresponding to multi-class lesion description features; and performing global matching on each report target and image target based on the similarity between each report target and each image target, so that the image target matched by each report target can be obtained, and each report target in the clinical diagnosis report can be accurately associated with the image target indicating the same lesion in the medical image; and then performing multimedia display based on all the report targets contained in the clinical diagnosis report and the image targets matched by each report target, so that a more intuitive, easier-to-understand and more comprehensive clinical diagnosis report display method can be provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a clinical diagnosis report multimedia display method and device based on global optimal matching. BACKGROUND

[0002] In clinical image examination, the radiologist will issue a text version of the clinical diagnosis report of the patient being examined and the medical image of the patient, and the patient can understand his own health status through the clinical diagnosis report.

[0003] However, on the one hand, the text version of the clinical diagnosis report issued by the radiologist is usually a professional description in text, and the description is generally simple. The information that an ordinary user can understand through the diagnosis report is very limited. In addition, the lesion in the diagnosis report is generally a fuzzy and general location description. Even if the medical image is provided, it is difficult for an ordinary user to directly identify and observe the lesion area from the medical image through the description in the diagnosis report. On the other hand, direct observation of the medical image requires certain imaging knowledge, and reading the medical image and locating the lung nodules and other lesions therein have a high technical threshold, which is very difficult for ordinary patients. Therefore, a more intuitive, easier to understand and more comprehensive clinical diagnosis report display method is needed to enable ordinary users to better observe and understand the condition of the lesion. SUMMARY

[0004] The present application provides a clinical diagnosis report multimedia display method and device based on global optimal matching, to solve the defects of insufficient information amount of the clinical diagnosis report alone, limited understanding, and high reading threshold and difficult understanding of the medical image alone in the prior art.

[0005] The present application provides a clinical diagnosis report multimedia display method based on global optimal matching, comprising:

[0006] Performing semantic analysis processing on the clinical diagnosis report to obtain report features of a plurality of report targets, and performing target recognition on the medical image corresponding to the clinical diagnosis report to obtain image features of a plurality of image targets; wherein the report features and the image features contain a plurality of different dimension lesion description features;

[0007] Based on the report features of any report target and the image features of any image target, and the weights corresponding to the plurality of lesion description features, determining the similarity between the any report target and the any image target;

[0008] Based on the similarity between any report target and any image target, performing global matching on the plurality of report targets and the plurality of image targets to obtain image targets matched by the plurality of report targets;

[0009] Multimedia presentation based on the aforementioned reporting targets and their matching image targets.

[0010] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided, wherein the weights corresponding to the multiple types of lesion description features are determined based on the feature extraction accuracy of the multiple types of lesion description features; the higher the feature extraction accuracy of any lesion description feature, the greater the weight corresponding to that lesion description feature.

[0011] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided, wherein the global matching of the plurality of reporting targets and the plurality of image targets based on the similarity between any reporting target and any image target specifically includes:

[0012] A cost matrix is ​​constructed based on the similarity between any reported target and any image target;

[0013] Subtract the minimum similarity of the corresponding row from the similarity of each row in the cost matrix, and then subtract the minimum similarity of the corresponding column from the similarity of each column in the cost matrix.

[0014] Global matching is performed based on the cost matrix to determine the minimum number of rows and columns containing all zero values ​​in the cost matrix, and the corresponding zero-containing rows and columns.

[0015] If the minimum number of rows and columns is less than the number of reported targets, then determine the minimum uncovered similarity outside the zero-containing rows and columns, subtract the minimum uncovered similarity from the similarity outside the zero-containing rows and columns, and add the minimum uncovered similarity to the similarity at the intersection of the zero-containing rows and columns. Then, perform global matching based on the updated cost matrix until the minimum number of rows and columns containing all zero values ​​in the cost matrix is ​​equal to the number of reported targets.

[0016] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided. The method for determining the similarity between any reporting target and any image target based on the reporting features of any reporting target, the image features of any image target, and the weights corresponding to multiple types of lesion description features specifically includes:

[0017] Numerical transformation is performed on the reporting features of the plurality of reporting targets and the image features of the plurality of image targets to obtain the reporting-normalized features of the plurality of reporting targets and the image-normalized features of the plurality of image targets;

[0018] The similarity between any reporting target and any image target is determined based on the report-standardized features of any reporting target and the image-standardized features of any image target, as well as the weights corresponding to the multi-type lesion description features.

[0019] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided, wherein semantic analysis processing of the clinical diagnostic reports is performed to obtain report features of several report targets, specifically including:

[0020] A correlation analysis is performed on the image description in the clinical diagnostic report and the diagnostic opinion in the clinical diagnostic report to obtain a second semantic unit in the diagnostic opinion that is associated with any first semantic unit of the image description;

[0021] And / or, determine historical diagnostic reports associated with the clinical diagnostic report, and obtain a third semantic unit associated with any first semantic unit of the image description in the historical diagnostic report that is associated with the clinical diagnostic report;

[0022] Based on the second semantic unit in the diagnostic opinion and / or the third semantic unit in the historical diagnostic report, as well as each of the first semantic units in the image description, semantic analysis is performed to obtain the reporting features of several reporting targets.

[0023] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided, wherein the multiple types of lesion description features of different dimensions include: the lung where the pulmonary nodule is located, the lung lobe where the pulmonary nodule is located, the lung segment where the pulmonary nodule is located, the size of the pulmonary nodule, and the type of pulmonary nodule.

[0024] According to the present invention, a multimedia display method for clinical diagnostic reports based on global optimal matching is provided, wherein the multimedia display based on the plurality of reporting targets and their matching image targets specifically includes:

[0025] Based on the coordinates of the image targets matched by the several reported targets in the corresponding medical images, the layers where the image targets matched by the several reported targets are located are selected in sequence, and the currently matched image target is highlighted.

[0026] Based on the image target layer matching the several report targets and the report features of the several report targets, a video stream of the clinical diagnosis report is generated;

[0027] An audio stream corresponding to the video stream is generated based on the report template, and the video stream and the audio stream are combined to obtain a multimedia display video of the clinical diagnosis report.

[0028] The present invention also provides a multimedia display device for clinical diagnostic reports based on global optimal matching, comprising:

[0029] The report image processing unit is used to perform semantic analysis on the clinical diagnosis report to obtain report features of several report targets, and to perform target recognition on the medical images corresponding to the clinical diagnosis report to obtain image features of multiple image targets; wherein, the report features and the image features include multiple types of lesion description features of different dimensions;

[0030] A similarity measurement unit is used to determine the similarity between any reported target and any image target based on the reporting features of any reported target, the image features of any image target, and the weights corresponding to multiple types of lesion description features;

[0031] A global matching unit is used to perform global matching on the plurality of reported targets and the plurality of image targets based on the similarity between any reported target and any image target, so as to obtain the image targets matched by the plurality of reported targets;

[0032] The multimedia display unit is used to perform multimedia display based on the aforementioned reporting targets and their matching image targets.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimedia display method for clinical diagnostic reports based on global optimal matching as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimedia display method for clinical diagnostic reports based on global optimal matching as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimedia display method for clinical diagnostic reports based on global optimal matching as described above.

[0036] The present invention provides a multimedia display method and apparatus for clinical diagnostic reports based on global optimal matching. This method acquires the report features of each report target in the clinical diagnostic report and the image features of each image target in the medical images. Based on the report features of any report target and the image features of any image target, as well as the weights corresponding to multiple types of lesion description features, the similarity between the two is determined. Then, based on the pairwise similarity between each report target and each image target, global matching is performed on each report target and image target to obtain the image target matched by each report target. This accurately associates each report target in the clinical diagnostic report with the image target in the medical images indicating the same lesion, overcoming the problems of brief lesion descriptions in clinical diagnostic reports and high false positive rates in automatically identified image targets in medical images. Furthermore, multimedia display is performed based on all report targets included in the clinical diagnostic report and the image targets matched by each report target, providing a more intuitive, easier-to-understand, and more comprehensive method for displaying clinical diagnostic reports, facilitating ordinary users to better observe and understand the condition of lesions. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of the multimedia display method for clinical diagnostic reports based on global optimal matching provided by the present invention;

[0039] Figure 2 This is the second flowchart of the multimedia display method for clinical diagnostic reports based on global optimal matching provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the multimedia display video provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the multimedia display device for clinical diagnostic reports based on global optimal matching provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Figure 1 This is a flowchart illustrating the multimedia display method for clinical diagnostic reports based on global optimal matching provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0045] Step 110: Perform semantic analysis on the clinical diagnosis report to obtain report features of several report targets, and perform target recognition on the medical images corresponding to the clinical diagnosis report to obtain image features of multiple image targets; wherein, the report features and the image features contain multiple types of lesion description features of different dimensions;

[0046] Step 120: Based on the reporting features of any reporting target and the image features of any image target, as well as the weights corresponding to the multiple types of lesion description features, determine the similarity between the any reporting target and the any image target;

[0047] Step 130: Based on the similarity between any reported target and any image target, perform global matching on the plurality of reported targets and the plurality of image targets to obtain the image targets matched by the plurality of reported targets;

[0048] Step 140: Perform multimedia display based on the aforementioned reported targets and their matching image targets.

[0049] Specifically, such as Figure 2 As shown, after obtaining any patient's clinical diagnosis report and its corresponding medical images, both the report and images can be processed separately to identify the targets and related information within these two media. The targets can indicate any type of lesion of interest to the user, such as lung nodules. Natural language processing (NLP) techniques can be used to perform semantic analysis on the clinical diagnosis report, identifying the descriptive text of the reported targets and determining the corresponding report features based on this text. For the medical images, computer vision algorithms, such as deep learning networks and other artificial intelligence algorithms, can be used to perform target recognition processing, obtaining the target regions within the images and determining the corresponding image features based on each target region.

[0050] Here, the reporting target and imaging target are the lesions obtained from the clinical diagnostic report and medical imaging, respectively, while the reporting features and imaging features are the lesion characteristics obtained from the clinical diagnostic report and medical imaging, respectively. Because the descriptions of lesions in the clinical diagnostic report are relatively brief and generalized, and lesions identified by artificial intelligence algorithms have a high false positive rate, in order to accurately associate the lesions described in the clinical diagnostic report with the actual lesions identified by the artificial intelligence algorithm from medical imaging, the aforementioned reporting features and imaging features can be constructed based on multiple types of lesion description features of different dimensions to comprehensively describe the corresponding lesions from different perspectives. For example, reporting features and imaging features can be constructed based on lesion description features such as the lung where the pulmonary nodule is located, the lung lobe where the pulmonary nodule is located, the lung segment where the pulmonary nodule is located, the size of the pulmonary nodule, and the type of the pulmonary nodule.

[0051] For any reported target and any imaging target, a similarity measure can be performed based on the reported features of the reported target and the imaging features of the imaging target to preliminarily assess the possibility that they correspond to the same lesion. Here, different weights can be assigned to multiple types of lesion descriptive features to differentiate the degree of influence of each type of lesion descriptive feature in confirming whether the reported target and the imaging target correspond to the same lesion, thereby improving the matching accuracy between the reported target and the imaging target. Based on the reported features of any reported target and the imaging features of any imaging target, as well as the weights corresponding to the multiple types of lesion descriptive features, the similarity between the reported target and the imaging target can be determined.

[0052] Taking five types of lesion description features—the lung where the pulmonary nodule is located, the lobe where the pulmonary nodule is located, the segment where the pulmonary nodule is located, the size of the pulmonary nodule, and the type of the pulmonary nodule—as an example, assuming that the reporting features of any reporting target obtained through natural language processing technology are (x1, x2, x3, x4, x5), and the image features of any image target obtained from medical images through artificial intelligence algorithms are (y1, y2, y3, y4, y5), where x1 and y1 both indicate the lung where the pulmonary nodule is located, x2 and y2 both indicate the lobe where the pulmonary nodule is located, x3 and y3 both indicate the segment where the pulmonary nodule is located, x4 and y4 both indicate the size of the pulmonary nodule, and x5 and y5 both indicate the type of pulmonary nodule, we can calculate the feature similarity between (x1, y1), (x2, y2), (x3, y3), (x4, y4), and (x5, y5) respectively, and then combine the weights of each type of lesion description feature to calculate the similarity between the reporting target and the image target.

[0053] Given that clinical diagnostic reports often provide brief and generalized descriptions of lesions, and that lesions identified by artificial intelligence algorithms have a high false positive rate, directly determining the matching result between reported targets and imaging targets based on their similarity can lead to inaccurate matching. Furthermore, directly determining the matching result between individual reported targets and individual imaging targets based on their similarity is affected by the matching order, resulting in non-unique matching results and insufficient accuracy. To address this, this invention utilizes a global matching algorithm. Based on the similarity between any reported target and any imaging target, it performs a global match between each reported target and each imaging target, obtaining a globally unique matching result, thereby obtaining the imaging target matched by each reported target.

[0054] Based on all reporting objectives included in the clinical diagnostic report and the corresponding imaging objectives, structured information about lesions can be generated. This structured information can then be displayed using multimedia to provide a more intuitive, easier-to-understand, and comprehensive method for presenting clinical diagnostic reports, facilitating better observation and understanding of lesions by ordinary users. For example, the imaging objectives corresponding to each reporting objective can be outlined in the relevant medical images, and the corresponding reporting features of the reporting objective can be displayed near the imaging objective. The presentation can be a text and image sequence, or corresponding audio narration can be inserted into the text and image sequence to create a video report.

[0055] The method provided in this invention obtains the report features of each report target in a clinical diagnostic report and the image features of each image target in a medical image. Based on the report features of any report target and the image features of any image target, as well as the weights corresponding to multiple types of lesion description features, the similarity between the two is determined. Based on the pairwise similarity between each report target and each image target, global matching is performed on each report target and image target to obtain the image target matched by each report target. This can accurately associate each report target in the clinical diagnostic report with the image target in the medical image indicating the same lesion, overcoming the problems of brief lesion descriptions in clinical diagnostic reports and high false positive rates of image targets automatically identified by medical images. Furthermore, based on all report targets included in the clinical diagnostic report and the image targets matched by each report target, multimedia display is performed, providing a more intuitive, easier-to-understand, and more comprehensive method for displaying clinical diagnostic reports, making it easier for ordinary users to observe and understand the condition of lesions.

[0056] Based on the above embodiments, the weights corresponding to the multiple lesion description features are determined based on the feature extraction accuracy of the multiple lesion description features; the higher the feature extraction accuracy of any lesion description feature, the greater the weight corresponding to that lesion description feature.

[0057] Specifically, when using natural language processing technology to extract report features from clinical diagnostic reports and artificial intelligence algorithms to extract image features from medical images, numerous factors can interfere, such as the quality of the recognition carriers (clinical diagnostic reports and medical images), the quality of the processing algorithms, and the difficulty of extracting lesion description features. Therefore, the feature extraction accuracy of various lesion description features obtained through these methods varies. Furthermore, the degree of negative impact on the determination of whether the report target and the image target correspond to the same lesion differs depending on the feature extraction accuracy of the lesion description features. In particular, the lower the feature extraction accuracy of any type of lesion description feature, the larger the error between it and the true result, and the greater the negative impact of the error in that type of lesion description feature on subsequent matching results. Therefore, in order to improve the matching accuracy between the reported target and the image target, the weight of the corresponding lesion description feature can be determined based on the feature extraction accuracy of various lesion description features. The higher the feature extraction accuracy of any type of lesion description feature, the greater the weight of that type of lesion description feature. Conversely, the lower the feature extraction accuracy of any type of lesion description feature, the smaller the weight of that type of lesion description feature.

[0058] Based on any of the above embodiments, the step of performing global matching of the plurality of reported targets and the plurality of image targets based on the similarity between any reported target and any image target specifically includes:

[0059] A cost matrix is ​​constructed based on the similarity between any reported target and any image target;

[0060] Subtract the minimum similarity of the corresponding row from the similarity of each row in the cost matrix, and then subtract the minimum similarity of the corresponding column from the similarity of each column in the cost matrix.

[0061] Global matching is performed based on the cost matrix to determine the minimum number of rows and columns containing all zero values ​​in the cost matrix, and the corresponding zero-containing rows and columns.

[0062] If the minimum number of rows and columns is less than the number of reported targets, then determine the minimum uncovered similarity outside the zero-containing rows and columns, subtract the minimum uncovered similarity from the similarity outside the zero-containing rows and columns, and add the minimum uncovered similarity to the similarity at the intersection of the zero-containing rows and columns. Then, perform global matching based on the updated cost matrix until the minimum number of rows and columns containing all zero values ​​in the cost matrix is ​​equal to the number of reported targets.

[0063] Specifically, before performing global matching on each reported target and each image target, a cost matrix can be constructed based on the similarity between each reported target and each image target. The rows of the cost matrix correspond to each reported target, and the columns of the cost matrix correspond to each image target.

[0064] Next, the minimum similarity of each row in the cost matrix is ​​determined, and the minimum similarity of the corresponding row is subtracted from the similarity of each row in the cost matrix. Then, the minimum similarity of each column in the cost matrix is ​​determined, and the minimum similarity of the corresponding column is subtracted from the similarity of each column in the cost matrix.

[0065] Global matching is performed based on the cost matrix to determine the minimum number of rows and columns containing all zero values ​​in the cost matrix, as well as the corresponding zero-containing rows and columns. If the minimum number of rows and columns equals the number of reported targets, it indicates that a global best match exists. Therefore, the global matching result can be determined based on the pairing of reported targets and image targets corresponding to the zero values ​​in the current cost matrix.

[0066] If the minimum number of rows and columns equals the number of reported targets, it indicates that there is no globally optimal match. Therefore, we can determine the minimum uncovered similarity outside the zero-containing rows and columns (i.e., the minimum value among the similarities outside the zero-containing rows and columns), subtract the minimum uncovered similarity from the similarities outside the zero-containing rows and columns, and add the minimum uncovered similarity to the similarity at the intersection of the zero-containing rows and columns. Then, we perform global matching again based on the updated cost matrix. We repeat these steps until the minimum number of rows and columns containing all zero values ​​in the cost matrix equals the number of reported targets.

[0067] Based on any of the above embodiments, determining the similarity between any reported target and any image target based on the reporting features of any reported target, the image features of any image target, and the weights corresponding to multiple lesion description features specifically includes:

[0068] Numerical transformation is performed on the reporting features of the plurality of reporting targets and the image features of the plurality of image targets to obtain the reporting-normalized features of the plurality of reporting targets and the image-normalized features of the plurality of image targets;

[0069] The similarity between any reporting target and any image target is determined based on the report-standardized features of any reporting target and the image-standardized features of any image target, as well as the weights corresponding to the multi-type lesion description features.

[0070] Specifically, to facilitate the calculation of the similarity between reported targets and image targets, the reported features of the reported targets and the image features of the image targets can be numerically transformed to obtain the reported-standardized features of the reported targets and the image-standardized features of the image targets. Then, based on the reported-standardized features of any reported target and the image-standardized features of any image target, as well as the weights corresponding to multiple lesion description features, the similarity between the reported target and the image target is determined.

[0071] Based on any of the above embodiments, the semantic analysis processing of the clinical diagnostic report to obtain several reporting features for reporting targets specifically includes:

[0072] A correlation analysis is performed on the image description in the clinical diagnostic report and the diagnostic opinion in the clinical diagnostic report to obtain a second semantic unit in the diagnostic opinion that is associated with any first semantic unit of the image description;

[0073] And / or, determine historical diagnostic reports associated with the clinical diagnostic report, and obtain a third semantic unit associated with any first semantic unit of the image description in the historical diagnostic report that is associated with the clinical diagnostic report;

[0074] Based on the second semantic unit in the diagnostic opinion and / or the third semantic unit in the historical diagnostic report, as well as each of the first semantic units in the image description, semantic analysis is performed to obtain the reporting features of several reporting targets.

[0075] Specifically, when performing semantic analysis on clinical diagnostic reports, considering that the reports contain two parts—image description and diagnostic opinion—and both parts contain lesion description information, a correlation analysis can be performed on the image description and diagnostic opinion to obtain a second semantic unit associated with any first semantic unit in the image description. Here, a semantic unit can be a word segment or a comma-separated clause, etc., and this embodiment of the invention does not specifically limit this. A second semantic unit associated with any first semantic unit in the image description refers to a semantic unit whose described lesion may be the same as the lesion described by the first semantic unit. Subsequently, semantic analysis can be performed by combining each first semantic unit of the image description and the second semantic units associated with the first semantic units, thereby enriching the lesion description information and extracting more accurate report features.

[0076] Furthermore, some clinical diagnostic reports also have associated historical diagnostic reports; for example, some clinical diagnostic reports may mention "comparison with the image from [date]". Therefore, it is possible to obtain historical diagnostic reports associated with the current clinical diagnostic report, and then obtain the third semantic unit associated with any first semantic unit of the image description in that historical diagnostic report that is relevant to the current clinical diagnostic report. Subsequently, semantic analysis can be performed by combining each first semantic unit of the image description with the second and third semantic units associated with the first semantic units, thereby enriching the descriptive information of the lesion and extracting more accurate report features.

[0077] Based on any of the above embodiments, the above-mentioned multiple lesion description features of different dimensions include, but are not limited to: the lung where the lung nodule is located, the lung lobe where the lung nodule is located, the lung segment where the lung nodule is located, the size of the lung nodule, and the type of lung nodule.

[0078] Based on any of the above embodiments, the multimedia display based on the plurality of reported targets and their matching image targets specifically includes:

[0079] Based on the coordinates of the image targets matched by the several reported targets in the corresponding medical images, the layers where the image targets matched by the several reported targets are located are selected in sequence, and the currently matched image target is highlighted.

[0080] Based on the image target layer matching the several report targets and the report features of the several report targets, a video stream of the clinical diagnosis report is generated;

[0081] An audio stream corresponding to the video stream is generated based on the report template, and the video stream and the audio stream are combined to obtain a multimedia display video of the clinical diagnosis report.

[0082] Specifically, for any given reporting target, the coordinates of the matching image target within the medical image are determined. The layers containing the matching image targets for each reporting target are sequentially selected, and the currently matched image target is highlighted within the layer, for example, by adding red. Based on the layers containing the matching image targets for each reporting target and the corresponding reporting features, a video stream of the clinical diagnosis report is generated. A single frame of this video stream can be a text-image frame formed by the layer containing any matching image target and the reporting features of that target, such as... Figure 3 As shown, an audio narration stream corresponding to the video stream is generated based on the report template, and the video stream and the corresponding audio narration stream are combined to obtain a multimedia presentation video of the clinical diagnosis report.

[0083] The following describes the multimedia display device for clinical diagnostic reports based on global optimal matching provided by the present invention. The multimedia display device for clinical diagnostic reports based on global optimal matching described below and the multimedia display method for clinical diagnostic reports based on global optimal matching described above can be referred to in correspondence with each other.

[0084] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the multimedia display device for clinical diagnostic reports based on global optimal matching provided by the present invention, as shown below. Figure 4 As shown, the device includes: a report image processing unit 410, a similarity measurement unit 420, a global matching unit 430, and a multimedia display unit 440.

[0085] The report image processing unit 410 is used to perform semantic analysis processing on the clinical diagnosis report to obtain report features of several report targets, and to perform target recognition on the medical image corresponding to the clinical diagnosis report to obtain image features of multiple image targets; wherein, the report features and the image features include multiple types of lesion description features of different dimensions;

[0086] The similarity measurement unit 420 is used to determine the similarity between the reported target and the image target based on the reporting features of any reported target, the image features of any image target, and the weights corresponding to multiple lesion description features;

[0087] The global matching unit 430 is used to perform global matching on the plurality of reported targets and the plurality of image targets based on the similarity between any reported target and any image target, so as to obtain the image targets matched by the plurality of reported targets;

[0088] The multimedia display unit 440 is used to perform multimedia display based on the plurality of reporting targets and their matching image targets.

[0089] The apparatus provided in this invention acquires the report features of each report target in a clinical diagnostic report and the image features of each image target in a medical image. Based on the report features of any report target and the image features of any image target, as well as the weights corresponding to multiple types of lesion description features, it determines the similarity between the two. Based on the pairwise similarity between each report target and each image target, it performs global matching on each report target and image target to obtain the image target matched by each report target. This can accurately associate each report target in the clinical diagnostic report with the image target in the medical image indicating the same lesion, overcoming the problems of brief lesion descriptions in clinical diagnostic reports and high false positive rates of image targets automatically identified by medical images. Furthermore, based on all report targets included in the clinical diagnostic report and the image targets matched by each report target, it performs multimedia display, providing a more intuitive, easier-to-understand, and more comprehensive method for displaying clinical diagnostic reports, making it easier for ordinary users to observe and understand the condition of lesions.

[0090] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a memory 520, a communication interface 530, and a communication bus 540, wherein the processor 510, the memory 520, and the communication interface 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 520 to execute a multimedia display method for clinical diagnostic reports based on global optimal matching. This method includes: performing semantic analysis on the clinical diagnostic report to obtain report features of several report targets, and performing target recognition on the medical images corresponding to the clinical diagnostic report to obtain image features of multiple image targets; wherein the report features and the image features include multiple types of lesion description features of different dimensions; determining the similarity between any report target and any image target based on the report features of any report target and the image features of any image target, as well as the weights corresponding to the multiple types of lesion description features; performing global matching on the several report targets and the multiple image targets based on the similarity between any report target and any image target to obtain image targets matched by the several report targets; and performing multimedia display based on the several report targets and their matched image targets.

[0091] Furthermore, the logical instructions in the aforementioned memory 520 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the multimedia display method for clinical diagnostic reports based on global optimal matching provided by the above methods, the method comprising: performing semantic analysis processing on the clinical diagnostic report to obtain report features of several report targets, and performing target recognition on the medical images corresponding to the clinical diagnostic report to obtain image features of multiple image targets; wherein the report features and the image features include multiple types of lesion description features of different dimensions; determining the similarity between any report target and any image target based on the report features of any report target and the image features of any image target, and the weights corresponding to the multiple types of lesion description features; performing global matching on the several report targets and the multiple image targets based on the similarity between any report target and any image target to obtain image targets matched by the several report targets; and performing multimedia display based on the several report targets and their matched image targets.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the aforementioned multimedia display method for clinical diagnostic reports based on global optimal matching. This method includes: performing semantic analysis on the clinical diagnostic report to obtain report features of several report targets; and performing target recognition on the medical images corresponding to the clinical diagnostic report to obtain image features of multiple image targets; wherein the report features and the image features include multiple types of lesion description features of different dimensions; determining the similarity between any report target and any image target based on the report features of any report target and the image features of any image target, as well as the weights corresponding to the multiple types of lesion description features; performing global matching on the several report targets and the multiple image targets based on the similarity between any report target and any image target to obtain image targets matched by the several report targets; and performing multimedia display based on the several report targets and their matched image targets.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimedia display method for clinical diagnostic reports based on global optimal matching, characterized in that, include: Semantic analysis is performed on the clinical diagnostic report to obtain report features of several report targets, and target recognition is performed on the medical images corresponding to the clinical diagnostic report to obtain image features of multiple image targets; wherein, the report features and the image features contain multiple types of lesion description features of different dimensions; the report targets and image targets are lesions obtained from the clinical diagnostic report and medical images, respectively; Based on the reporting features of any reporting target and the image features of any image target, as well as the weights corresponding to multiple lesion description features, the similarity between the reporting target and the image target is determined. The weights corresponding to the multiple lesion description features are determined based on the feature extraction accuracy of the multiple lesion description features. The higher the feature extraction accuracy of any lesion description feature, the greater the weight corresponding to the lesion description feature. Based on the similarity between any reported target and any image target, a cost matrix is ​​constructed, where the rows of the cost matrix correspond to each reported target and the columns correspond to each image target. The minimum similarity of the corresponding row is subtracted from the similarity of each row in the cost matrix, and the minimum similarity of the corresponding column is subtracted from the similarity of each column. Global matching is performed based on the cost matrix to determine the minimum number of rows and columns containing all zero values ​​in the cost matrix, and the corresponding zero-containing rows and columns. If the minimum number of rows and columns is less than the number of reported targets, the minimum uncovered similarity outside the zero-containing rows and columns is determined. The minimum uncovered similarity is subtracted from the similarity outside the zero-containing rows and columns, and the minimum uncovered similarity is added to the similarity at the intersection of the zero-containing rows and columns. Global matching is then performed based on the updated cost matrix until the minimum number of rows and columns containing all zero values ​​in the cost matrix equals the number of reported targets, thus obtaining the image targets matched by the reported targets. Multimedia presentation based on the aforementioned reporting targets and their matching image targets.

2. The multimedia display method for clinical diagnostic reports based on global optimal matching according to claim 1, characterized in that, The determination of the similarity between any reported target and any image target based on the reporting features of any reported target, the image features of any image target, and the weights corresponding to multiple lesion description features specifically includes: Numerical transformation is performed on the reporting features of the plurality of reporting targets and the image features of the plurality of image targets to obtain the reporting-normalized features of the plurality of reporting targets and the image-normalized features of the plurality of image targets; The similarity between any reporting target and any image target is determined based on the report-standardized features of any reporting target and the image-standardized features of any image target, as well as the weights corresponding to the multi-type lesion description features.

3. The multimedia display method for clinical diagnostic reports based on global optimal matching according to claim 1, characterized in that, The semantic analysis of clinical diagnostic reports yields several reporting features for reporting objectives, including: A correlation analysis is performed on the image description in the clinical diagnostic report and the diagnostic opinion in the clinical diagnostic report to obtain a second semantic unit in the diagnostic opinion that is associated with any first semantic unit of the image description; And / or, determine historical diagnostic reports associated with the clinical diagnostic report, and obtain a third semantic unit associated with any first semantic unit of the image description in the historical diagnostic report that is associated with the clinical diagnostic report; Based on the second semantic unit in the diagnostic opinion and / or the third semantic unit in the historical diagnostic report, as well as each of the first semantic units in the image description, semantic analysis is performed to obtain the reporting features of several reporting targets.

4. The multimedia display method for clinical diagnostic reports based on global optimal matching according to claim 1, characterized in that, The various lesion description features include: the lung in which the lung nodule is located, the lung lobe in which the lung nodule is located, the lung segment in which the lung nodule is located, the size of the lung nodule, and the type of lung nodule.

5. The multimedia display method for clinical diagnostic reports based on global optimal matching according to any one of claims 1 to 4, characterized in that, The multimedia display based on the aforementioned reported targets and their matching image targets specifically includes: Based on the coordinates of the image targets matched by the several reported targets in the corresponding medical images, the layers where the image targets matched by the several reported targets are located are selected in sequence, and the currently matched image target is highlighted. Based on the image target layer matching the several report targets and the report features of the several report targets, a video stream of the clinical diagnosis report is generated; An audio stream corresponding to the video stream is generated based on the report template, and the video stream and the audio stream are combined to obtain a multimedia display video of the clinical diagnosis report.

6. A multimedia display device for clinical diagnostic reports based on global optimal matching, characterized in that, include: The report image processing unit is used to perform semantic analysis on the clinical diagnosis report to obtain report features of several report targets, and to perform target recognition on the medical images corresponding to the clinical diagnosis report to obtain image features of multiple image targets; wherein, the report features and the image features include multiple types of lesion description features of different dimensions; the report targets and image targets are lesions obtained from the clinical diagnosis report and the medical images, respectively; A similarity measurement unit is used to determine the similarity between any reported target and any image target based on the reporting features of any reported target, the image features of any image target, and the weights corresponding to multiple lesion description features. The weights corresponding to the multiple lesion description features are determined based on the feature extraction accuracy of the multiple lesion description features. The higher the feature extraction accuracy of any lesion description feature, the greater the weight of the any lesion description feature. A global matching unit is used to construct a cost matrix based on the similarity between any reported target and any image target. The rows of the cost matrix correspond to each reported target, and the columns correspond to each image target. The unit subtracts the minimum similarity of the corresponding row from the similarity of each row in the cost matrix, and subtracts the minimum similarity of the corresponding column from the similarity of each column. Global matching is performed based on the cost matrix to determine the minimum number of rows and columns containing all zero values ​​in the cost matrix, and the corresponding zero-containing rows and columns. If the minimum number of rows and columns is less than the number of reported targets, the unit determines the minimum uncovered similarity outside the zero-containing rows and columns. The unit subtracts the minimum uncovered similarity from the similarity outside the zero-containing rows and columns, and adds the minimum uncovered similarity to the similarity at the intersection of the zero-containing rows and columns. Global matching is then performed based on the updated cost matrix until the minimum number of rows and columns containing all zero values ​​in the cost matrix equals the number of reported targets, thus obtaining the image targets matched by the reported targets. The multimedia display unit is used to perform multimedia display based on the aforementioned reporting targets and their matching image targets.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimedia display method for clinical diagnostic reports based on global optimal matching as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimedia display method for clinical diagnostic reports based on global optimal matching as described in any one of claims 1 to 5.

Images