A method, device and storage medium for generating an image diagnosis report

Abstract

The application discloses an image diagnosis report generation method and device and a storage medium. The method comprises the following steps: searching a target template matched with a diagnosis text input by a user based on the diagnosis text, through three different levels of matching degrees, i.e. a spatial matching degree between the diagnosis text and the candidate template at an anatomical site level, a semantic matching degree between the diagnosis text and the candidate template related to pathological information, and a clinical correlation degree between the diagnosis text and the candidate template, and further optimizing the search through a time decay factor, a clinical priority and a conflict degree between regions of interest, improving the accuracy of searching the required image diagnosis template for the user, and thus solving the technical problem that the keyword matching effect of the traditional keyword matching algorithm in the prior art is poor in a medical scene, and the accuracy of searching a correct template is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical informatization, in particular to a method and device for generating an image diagnosis report and a storage medium. BACKGROUND

[0002] Currently, the generation of an image diagnosis report is usually performed by a doctor who manually inputs keywords or sentences in a template library of an image diagnosis report according to an image result, searches for a required image diagnosis report template, and modifies the image diagnosis report template to obtain a complete image diagnosis report.

[0003] In the prior art, an image diagnosis report generation system usually performs simple semantic matching between keywords (or sentences) and information related to an image diagnosis report template. For example, a pre-trained language model (such as a BERT model and its variants) is used to calculate the semantic similarity between sentences or words for matching, thereby searching for a corresponding image diagnosis report template and providing the template to a doctor. However, the prior art has the following problems:

[0004] The semantic matching effect of a traditional semantic matching algorithm in a medical scenario is poor. For example, “miliary nodules” and “micro-nodular clusters” have the same meaning in a medical scenario, but the traditional semantic matching algorithm usually cannot match the two words together. Therefore, this approach reduces the accuracy of searching for a correct template.

[0005] In view of the technical problem of poor accuracy of searching for an image diagnosis report template in the prior art, no effective solution has been proposed so far. SUMMARY

[0006] Embodiments of the present disclosure provide a method and device for generating an image diagnosis report and a storage medium to at least solve the technical problem of poor accuracy of searching for an image diagnosis report template in the prior art.

[0007] According to an aspect of an embodiment of the present disclosure, a method for generating an image diagnosis report is provided, which includes: receiving, from a terminal device of a user, a diagnosis text corresponding to a medical image input by the user; determining a spatial matching degree between the diagnosis text and a candidate template on an anatomical site level, a semantic matching degree between the diagnosis text and the candidate template related to pathological information, and a clinical correlation degree between the diagnosis text and the candidate template; determining a first matching degree between the candidate template and the diagnosis text according to the spatial matching degree, the semantic matching degree, and the clinical correlation degree; determining a target template from the candidate template according to the first matching degree; and sending the target template to the terminal device and receiving an image diagnosis report corresponding to the target template from the terminal device.

[0008] According to another aspect of the embodiments of the present disclosure, a storage medium is also provided, which includes a stored program, wherein the program, when executed by a processor, performs the above method.

[0009] According to another aspect of the embodiments of the present disclosure, a device for generating an image diagnosis report is also provided, which includes: a receiving module configured to receive, from a terminal device of a user, a diagnosis text corresponding to a medical image and input by the user; a three-level matching module configured to determine a spatial matching degree between the diagnosis text and a candidate template at an anatomical site level, a semantic matching degree between the diagnosis text and the candidate template related to pathological information, and a clinical correlation degree between the diagnosis text and the candidate template; a matching degree determining module configured to determine a first matching degree between the candidate template and the diagnosis text according to the spatial matching degree, the semantic matching degree, and the clinical correlation degree; a selecting module configured to determine a target template from the candidate templates according to the first matching degree; and a sending module configured to send the target template to the terminal device and receive an image diagnosis report corresponding to the target template from the terminal device.

[0010] According to another aspect of the embodiments of the present disclosure, a device for generating an image diagnosis report is also provided, which includes: a processor; and a memory connected with the processor and configured to provide the processor with instructions for processing the following processing steps: receiving, from a terminal device of a user, a diagnosis text corresponding to a medical image and input by the user; determining a spatial matching degree between the diagnosis text and a candidate template at an anatomical site level, a semantic matching degree between the diagnosis text and the candidate template related to pathological information, and a clinical correlation degree between the diagnosis text and the candidate template; determining a first matching degree between the candidate template and the diagnosis text according to the spatial matching degree, the semantic matching degree, and the clinical correlation degree; determining a target template from the candidate templates according to the first matching degree; and sending the target template to the terminal device and receiving an image diagnosis report corresponding to the target template from the terminal device.

[0011] In the embodiments of the present disclosure, according to the technical solution of the present embodiments, the target template matched with the diagnosis text input by the user can be searched based on the diagnosis text input by the user according to the matching degrees at three different levels, i.e., the spatial matching degree between the diagnosis text and the candidate template at the anatomical site level, the semantic matching degree between the diagnosis text and the candidate template related to the pathological information, and the clinical correlation degree between the diagnosis text and the candidate template, thereby improving the accuracy of searching the required image diagnosis template for the user, and thereby solving the technical problem that the keyword matching effect of the traditional keyword matching algorithm in the medical field is poor and the accuracy of searching the correct template is reduced in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application, illustrate embodiments of the present disclosure and together with the description serve to explain the present disclosure. In the drawings:

[0013] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0014] Figure 2 is a schematic diagram of an image diagnosis report generation system according to Embodiment 1 of the present disclosure;

[0015] Figure 3 is a flowchart of an image diagnosis report generation method according to the first aspect of Embodiment 1 of the present disclosure;

[0016] Figure 4 is a structural diagram for training a linear transformation matrix according to the first aspect of Embodiment 1 of the present disclosure;

[0017] Figure 5 is a flowchart of a feature extraction process provided by Embodiment 1 of the present disclosure, which extracts features by combining a first feature extraction model with a self-attention mechanism;

[0018] Figure 6 is a path diagram according to the first aspect of Embodiment 1 of the present disclosure;

[0019] Figure 7 is a schematic diagram of an image diagnosis report generation device according to Embodiment 2 of the present disclosure; and

[0020] Figure 8 is a schematic diagram of an image diagnosis report generation device according to Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable persons skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present disclosure.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] According to this embodiment, a method embodiment for generating an image diagnostic report is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] The method embodiments provided in this example can be executed on a computer terminal, server, or similar computing device. Figure 1 A hardware block diagram of a computing device for implementing an image diagnostic report generation method is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interface. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] It should be noted that the one or more processors and / or other data processing circuitry described above can be referred to herein generically as "data processing circuitry." The data processing circuitry can be embodied as, include or otherwise be associated with software, hardware, firmware, or any combination thereof. Moreover, the data processing circuitry can be a single independent processing module or any combination of plural processing modules, all of which are incorporated in whole or in part within other elements of the computing device. As referred to in the embodiments of the present disclosure, the data processing circuitry functions as a processor to control, for example, selection of the variable resistance terminal path connected to the interface.

[0027] The memory can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the image diagnosis report generation method in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, i.e., implements the image diagnosis report generation method of the application program described above. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the computing device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0028] The transmission device is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0029] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computing device.

[0030] It should be noted that in some optional embodiments, the above-mentioned Figure 1 The computing device shown can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or combinations of both hardware and software elements. It should be noted that in some embodiments, the functions described herein can be implemented as part of an operating system or a specific application, component, method, or process Figure 1 is merely one example of a particular implementation and is intended to provide an example of the types of components that can be present in the computing device described above.

[0031] Figure 2is a schematic diagram of an image diagnosis report generation system according to the embodiment. Referring to Figure 2 As shown in the figure, the system includes a terminal device 100 and a server 200. The terminal device 100 is in communication connection with the server 200 through a network. A physician in the imaging department can input a diagnosis text (a template for searching a medical image diagnosis report) corresponding to a medical image on the terminal device 100. The terminal device 100 sends the diagnosis text to the server 200. The server 200 determines a target template according to the diagnosis text and sends the target template to the terminal device 100. The physician completes an image diagnosis report based on the target template through the terminal device 100.

[0032] It should be noted that the terminal device 100 and the server 200 in the system can both adopt the hardware structure described above.

[0033] Under the above operating environment, according to a first aspect of the embodiment, an image diagnosis report generation method is provided. The method is implemented by the server 200 shown in Figure 2 . Figure 4 A flowchart of the method is shown in Figure 3 , and the method includes the following steps.

[0034] S302: Receiving a diagnosis text corresponding to a medical image input by a user from a terminal device of the user;

[0035] S304: Determining a spatial matching degree between the diagnosis text and a candidate template on an anatomical site level, a semantic matching degree between the diagnosis text and the candidate template related to pathological information, and a clinical correlation degree between the diagnosis text and the candidate template;

[0036] S306: Determining a first matching degree between the candidate template and the diagnosis text according to the spatial matching degree, the semantic matching degree, and the clinical correlation degree; and

[0037] S308: Determining a target template from the candidate template according to the first matching degree; and

[0038] S310: Sending the target template to the terminal device and receiving an image diagnosis report corresponding to the target template from the terminal device.

[0039] Specifically, in the present embodiment, a user (a physician) can input a diagnosis text corresponding to a medical image through the terminal device 100, the diagnosis text being used by the user to search for an image diagnosis report template, the diagnosis text can be a short text used to describe the diagnosis content for the medical image. The terminal device 100 can send the diagnosis text to the server 200. The server 200 receives the diagnosis text corresponding to the medical image input by the user from the terminal device 100 of the user (S302). It should be noted that the diagnosis text can be input by the user by voice.

[0040] Then, the server 200 needs to retrieve the image diagnosis report template required by the user according to the diagnosis text. Therefore, the server 200 determines the spatial matching degree between the diagnosis text and the candidate template at the level of anatomical site, the semantic matching degree between the diagnosis text and the candidate template related to pathological information, and the clinical association degree between the diagnosis text and the candidate template (S304). And according to the spatial matching degree, the semantic matching degree and the clinical association degree, the first matching degree between the candidate template and the diagnosis text is determined (S306). Wherein, the candidate template mentioned here can be each image diagnosis report template in the template library.

[0041] Wherein, the spatial matching degree focuses on the level of anatomical site, that is, the correlation between the anatomical site involved in the diagnosis text and the anatomical site involved in the candidate template; the semantic matching degree focuses on the semantic correlation related to pathological information between the diagnosis text and the candidate template at the level of text; the clinical association degree can be determined by a knowledge graph related to medical treatment (which will be described in detail later), and the clinical association degree represents the association between the diagnosis text and the candidate template at the clinical level. The specific way of determining the spatial matching degree, the semantic matching degree and the clinical association degree will be described in detail later.

[0042] That is, by determining the matching degrees at three different levels (the spatial matching degree, the semantic matching degree and the clinical association degree), the above-mentioned first matching degree integrates various matching degrees between the diagnosis text and the candidate template, and the image diagnosis report template required by the user can be selected by the first matching degree.

[0043] Therefore, then the server 200 can determine the target template from the candidate template according to the first matching degree (S308), and send the target template to the terminal device 100, and the user edits the target template through the terminal device 100 to complete a complete image diagnosis report, and the terminal device 100 sends the image diagnosis report to the server 200, so that the server 200 receives the image diagnosis report corresponding to the target template from the terminal device 100 (S310).

[0044] As described in the background, in the prior art, the image diagnosis report generation system usually performs simple semantic matching between keywords and information related to the image diagnosis report template, so as to search for the corresponding image diagnosis report template and provide it to the doctor. However, the prior art has the following problems: the keyword matching effect of the traditional keyword matching algorithm (such as TF-IDF) in the medical scene is poor. For example, "miliary nodules" and "micro-nodular clusters" have the same meaning in the medical scene, but the traditional keyword matching algorithm usually cannot match these two words together. Therefore, using this method will reduce the accuracy of searching for the correct template.

[0045] Therefore, according to the technical scheme of the present embodiment, the matching degree on three different levels, i.e. the spatial matching degree between the diagnostic text and the candidate template on the anatomical site level, the semantic matching degree between the diagnostic text and the candidate template related to the pathological information, and the clinical correlation degree between the diagnostic text and the candidate template, can be used to search for the target template matching the diagnostic text input by the user, thereby improving the accuracy of searching for the required image diagnosis template for the user, and solving the technical problems of the prior art, i.e. the poor keyword matching effect of the traditional keyword matching algorithm in the medical scene, and the reduced accuracy of searching for the correct template.

[0046] Optionally, the operation of determining the spatial matching degree between the diagnostic text and the candidate template on the anatomical site level comprises: determining first anatomical site information in the candidate template and second anatomical site information in the diagnostic text; determining a site similarity according to the first anatomical site information and the second anatomical site information; determining a spatial position adjacency according to the adjacent relationship of the anatomical sites corresponding to the first anatomical site information and the second anatomical site information in the medical image; and determining the spatial matching degree according to the site similarity and the spatial position adjacency.

[0047] The first anatomical site information represents the anatomical site mentioned in the candidate template, and the second anatomical site information represents the anatomical site mentioned in the diagnostic text. The site similarity can represent the repetition degree between the anatomical site mentioned in the candidate template and the anatomical site mentioned in the diagnostic text, which can be calculated by the BM25 algorithm, which will be described in detail below. The spatial position adjacency mentioned above is used to add a new dimension on the basis of the site similarity to determine the spatial matching degree between the diagnostic text and the candidate template.

[0048] Optionally, according to the adjacent relationship of the anatomical sites corresponding to the first anatomical site information and the second anatomical site information in the medical image, the operation of determining the spatial position adjacency degree comprises: determining first coordinate information of the anatomical site corresponding to the first anatomical site information in the medical image, and determining second coordinate information of the anatomical site corresponding to the second anatomical site information in the medical image; determining a first coordinate feature corresponding to the first coordinate information and a second coordinate feature corresponding to the second coordinate information; and determining the spatial position adjacency degree according to the first coordinate feature and the second coordinate feature.

[0049] That is, the spatial position adjacency degree represents the adjacent relationship of the anatomical sites mentioned in the candidate template and the anatomical sites mentioned in the diagnosis text in the medical image. The position of each anatomical site is represented by its three-dimensional coordinates (x, y, z) in the medical image, and in this way, the first coordinate information and the second coordinate information mentioned above can be obtained. The first coordinate information and the second coordinate information can be standardized and mapped to a high-dimensional vector space, and the expression ability of the position information can be enhanced through linear transformation, so as to obtain the corresponding first coordinate feature and the second coordinate feature.

[0050] Specifically, the first coordinate information and the second coordinate information can be standardized by using, for example, Z-score, to obtain the first standardized coordinate corresponding to the first coordinate information and the second standardized coordinate corresponding to the second coordinate information. For example, the first coordinate information is , and the second coordinate information is The Z-score can be used to standardize each dimension of the x-axis, y-axis and z-axis, and the formula for standardizing the x-axis by using the Z-score is as follows:

[0051]

[0052] Thus, the first standardized coordinate corresponding to the first coordinate information is , and the second standardized coordinate corresponding to the second coordinate information is

[0053] The three-dimensional coordinates are mapped to a high-dimensional vector space through a linear transformation matrix ∈R d×3 , and the formula is as follows:

[0054]

[0055]

[0056] wherein, is the first standardized coordinate, is the second standardized coordinate, and i.e. the mapped high-dimensional vector (assuming d = 128, then and are 128-dimensional vectors), is the first coordinate feature, is the second coordinate feature. The linear transformation matrix can be obtained by training.

[0057] Specifically, to train the linear transformation matrix, a training sample set composed of standardized coordinate p'-anatomical site training samples can be constructed, which is specifically shown as follows:

[0058] Table 1

[0059]

[0060] Then, the architecture for training the linear transformation matrix is shown as Figure 4 , which includes a linear transformation matrix W, a neural network, and a softmax classifier. The linear transformation matrix W is used to map the standardized coordinates to high-dimensional vectors; the neural network is used to extract features from the high-dimensional vectors; and the dimension of the softmax classifier corresponds to the number of anatomical sites, and the softmax classifier respectively outputs probability values corresponding to each anatomical site for the input standardized coordinates.

[0061] Then, the neural network and the transformation matrix W shown in Figure 4 are trained using the above sample set, so as to obtain the linear transformation matrix W described above. Thus, the high-dimensional and can be obtained using the linear transformation matrix W.

[0062] Then, the cosine similarity between and can be determined as the spatial position adjacency, and the formula can be as follows:

[0063]

[0064] wherein, is the determined spatial position adjacency.

[0065] In determining the spatial position adjacency, the anatomical site mentioned in the candidate template and the anatomical site mentioned in the diagnosis text can be judged to be adjacent (or coincident) in anatomy according to the first coordinate feature and the second coordinate feature, and if so, the spatial position adjacency can be determined as (wherein a and b represent the corresponding anatomical sites in the candidate template and the diagnosis text, respectively), and if not, the spatial position adjacency can be set to 0.

[0066] If the adjacency between the first coordinate feature and the second coordinate feature is greater than the preset reference adjacency, it can be determined that the anatomical site mentioned in the candidate template and the anatomical site mentioned in the diagnosis text are anatomically adjacent (or coincident).

[0067] The formula for determining the spatial matching degree is as follows:

[0068]

[0069] wherein A represents the diagnosis text, B represents the candidate template, represents the spatial matching degree, represents the site similarity of the diagnosis text and the candidate template, which can be determined by the BM25 algorithm. β is the weight corresponding to the spatial position adjacency, which can be set by a person in advance, and can be specifically set to 0.15. is an adjacency function, used to indicate the spatial position adjacency between the anatomical site mentioned in the candidate template and the corresponding anatomical site mentioned in the diagnosis text. a j and b j are the corresponding anatomical sites in the candidate template and the diagnosis text, respectively.

[0070] The way of determining the site similarity by the BM25 algorithm can be specifically as shown in the following formula:

[0071]

[0072] wherein D represents the first anatomical site information in the candidate template, Q represents the second anatomical site information in the diagnosis text, may represent the i-th anatomical site mentioned in the second anatomical site information. represents the corresponding inverse document frequency, represents the average number of anatomical sites contained in the first anatomical site information and the second anatomical site information, represents the number of anatomical sites contained in the first anatomical site information, represents the frequency of occurrence in the first anatomical site information, represents and may be a preset parameter.

[0073] ​For example, the diagnostic text is “a 2cm solid nodule with irregular margin and lobulation, adjacent to the pleura traction is found in the right upper lobe of the patient's lung, the imaging performance is highly suspected of malignant lesions, and the puncture biopsy is recommended”, the anatomical site information (second anatomical site information) extracted from the diagnostic text is “right upper lobe of the lung”. The anatomical site mentioned in the candidate template A is the right upper lobe of the lung, the anatomical site mentioned in the candidate template B is the right lower lobe of the lung, the anatomical site mentioned in the candidate template C is the double lung, and the anatomical site mentioned in the candidate template D is the right upper lobe of the lung and the pleura. The examples of the site similarity, the spatial position adjacent degree and the spatial matching degree between the diagnostic text and each candidate template are shown in Table 2.

[0074] Table 2

[0075]

[0076] In the candidate template D, “right upper lobe of the lung” and “pleura” are mentioned, and the site similarity between the candidate template D and the diagnostic text is 0.90. In the candidate template D, “right upper lobe of the lung” and “pleura” are mentioned, and both need to be considered when calculating the spatial position adjacent degree. Since the two anatomical sites mentioned in the candidate template D are adjacent (or coincide) to the anatomical site mentioned in the diagnostic text, the determined spatial position adjacent degree is 0.30 (0.15x2).

[0077] Optionally, the operation of determining the semantic matching degree related to the pathological information between the candidate template and the diagnostic text comprises: performing feature extraction on the description information of the candidate template and the diagnostic text based on a first feature extraction model to obtain first semantic features corresponding to the description information and second semantic features corresponding to the diagnostic text, wherein the first feature extraction model is obtained by fine-tuning a preset semantic feature extraction model based on training samples related to a medical scene; and determining the semantic matching degree related to the pathological information between the candidate template and the diagnostic text according to the first semantic features and the second semantic features.

[0078] That is, the above-mentioned semantic feature extraction model is a semantic model used in a common scene, for example, the semantic feature extraction model can be a BERT model trained by a large number of text samples. The above-mentioned first feature extraction model is obtained by fine-tuning the semantic feature extraction model based on training samples related to a medical scene, and the MIMIC-III dataset can be used for fine-tuning. The training samples can be constructed by medical record texts, diagnosis reports and related radiology descriptions to fine-tune the semantic feature extraction model, so that the fine-tuned semantic feature extraction model (i.e., the first feature extraction model) is more suitable for a specific medical scene.

[0079] In addition, preferably, the semantic feature extraction model proposed in the present application can be based on the BERT model based on the self-attention mechanism, and further adopt a weighted attention mechanism based on the entity relationship between words for feature extraction, which is different from the self-attention mechanism, that is, the first feature extraction model can include a fine-tuned semantic feature extraction model, a named entity recognition module, a relationship extraction module, and a weighted attention model. The following is described by taking the diagnostic text as an example, and those skilled in the art should know that the description information of the candidate template can also be extracted in the same way. Figure 5 A flowchart of feature extraction by the first feature extraction model combined with the self-attention mechanism is shown.

[0080] Figure 5 A flowchart of feature extraction by the first feature extraction model combined with the self-attention mechanism is shown.

[0081] According to the technical solution of the present application, the entity types corresponding to the medical terms and diagnostic conclusions are defined in advance. Table 3 below shows some examples of entity types:

[0082] Table 3

[0083]

[0084] Therefore, after the server 200 performs text cleaning, word segmentation, and word embedding on the diagnostic text, it determines the word vectors corresponding to each word through the BERT model, and determines the entity types corresponding to each word of the diagnostic text through the named entity recognition module (which can be a named entity recognition model). For example:

[0085] The entity type of "right lung" is "anatomical site entity"; the entity type of "discovery" is "qualitative description entity"; "2" is "numerical entity"; "cm" is "unit entity"; "solid nodule" is "sign description entity"; and so on.

[0086] In addition, according to the technical solution of the present application, the relationship types used to describe the relationship between entities are also defined in advance. Table 4 below shows some examples of relationship types:

[0087] Table 4

[0088]

[0089] Thus, the server 200 can further determine the relationship between the entities through the relationship extraction module (which can be a relationship extraction model for extracting the relationship between entities). For example, "right lung" and "upper lobe" are directional relationships; "right lung" and "solid nodule" are positive sign relationships; "edge" and "irregular" are positive sign relationships; and so on.

[0090] In addition, further referring to Table 3, in the technical solution of the present application, according to the different relationship types, the weight coefficients corresponding to different relationship types are defined, that is, for two words, the entities corresponding to the two words can be determined, and then the relationship between the corresponding entities can be determined, and the weight coefficient corresponding to the relationship type of the relationship can be taken as the weight coefficient between the two words. The weight coefficient can be determined according to experience and manually adjusted in the actual training process. As can be seen from Table 3, the weight coefficient can reflect the relationship between the words related to medical diagnosis. For example, when the relationship between the words is a positive sign relationship or a negative sign relationship, the weight coefficient between the words is larger, which is 2.0. For example, when the relationship between the words is a state expression relationship, the weight coefficient between the words is smaller, which is 1.3. For example, when there is no entity relationship between the words, the weight coefficient is 1.0.

[0091] Then the server 200 can perform weighted attention-based feature extraction based on the weight coefficient on the basis of the word vectors output by the BERT model through the weighted attention model, thereby determining the semantic features (i.e., the second semantic features) corresponding to each word (i.e., the first semantic features when the first feature extraction model extracts features from the description information of the candidate template). That is, the weight coefficients corresponding to different relationship types are fused in the self-attention mechanism to realize weighted attention different from ordinary self-attention mechanism. The internal calculation process of the weighted attention model can be as follows:

[0092] Determine the semantic features f1~f n corresponding to each word vector x1~x n , query vector q1~q n , and value vector v1~v n . And determine the semantic features f1~f n

[0093] (1)

[0094] (2)

[0095] (3)

[0096] where f iis a semantic feature corresponding to each word in the diagnostic text (i.e., the second semantic feature, and the first semantic feature when the semantic feature extraction is performed on the description information of the candidate template), is a weight coefficient between each word determined based on Table 3, i.e., the weight coefficient between each word determined by the weight coefficient corresponding to different relationship types.

[0097] The above is an example of a diagnostic text. The semantic features (i.e., the first semantic features) of the candidate template can also be extracted based on the above method.

[0098] In this way, based on the weighted attention mechanism, the attention can be weighted based on the entity relationship between the words related to the medical diagnosis. In this way, the semantic features determined in this way can strengthen the more important information for matching the diagnostic text and the candidate template, thereby improving the accuracy of the matching.

[0099] In determining the semantic matching degree between the candidate template and the diagnostic text related to the pathological information according to the first semantic feature and the second semantic feature, the similarity between the first semantic feature and the second semantic feature, such as the Euclidean distance or the cosine similarity, can be determined to determine the semantic matching degree.

[0100] Further, according to the first semantic feature and the second semantic feature, the semantic matching degree between the candidate template and the diagnostic text related to the pathological information is determined, including: determining a first max-pooling result and a first average-pooling result corresponding to the first semantic feature, weighting and fusing the first max-pooling result and the first average-pooling result to obtain a first fusion feature corresponding to the description information; determining a second max-pooling result and a second average-pooling result corresponding to the second semantic feature, weighting and fusing the second max-pooling result and the second average-pooling result to obtain a second fusion feature corresponding to the diagnostic text; and determining the semantic matching degree between the candidate template and the diagnostic text related to the pathological information according to the first fusion feature and the second fusion feature.

[0101] That is, the above process is to better extract the global information and key information in the description information of the candidate template and the diagnostic text after the semantic feature extraction of the description information of the candidate template and the diagnostic text. Specifically, the server 200 respectively performs max-pooling and average-pooling on the first semantic feature corresponding to the description information of the candidate template and the second semantic feature corresponding to the diagnostic text after feature extraction by the first feature extraction model, to obtain the first max-pooling result and the first average-pooling result corresponding to the description information of the candidate template, and the second max-pooling result and the second average-pooling result corresponding to the diagnostic text.

[0102] After the maximum pooling result and the average pooling result of the description information and the diagnosis text are determined respectively, the maximum pooling result and the average pooling result can be fused by weighting, and the specific formula is as follows.

[0103]

[0104] wherein E is a word embedding matrix (i.e., the first semantic feature or the second semantic feature), and a can be a dynamic weight determined by an attention mechanism or a hyperparameter adjustment. When E is the first semantic feature, HybridVec is the first fusion feature, and when E is the second semantic feature, HybridVec is the second fusion feature.

[0105] Then, the server 200 can directly determine the semantic matching degree between the candidate template and the diagnosis text according to the first fusion feature and the second fusion feature, so that the semantic matching degree between the candidate template and the diagnosis text can be more accurately determined based on the first semantic feature and the second semantic feature.

[0106] Continuing the above example, the diagnosis text and the candidate templates remain unchanged, and the semantic correlation degree between the description information of each candidate template and the diagnosis text is shown in Table 5 below.

[0107] Table 5

[0108]

[0109] Optionally, the operation of determining the clinical correlation degree between the candidate template and the diagnosis text comprises: determining a first keyword in the candidate template and a second keyword in the diagnosis text; querying a node matched with the first keyword as a first starting node and a node matched with the second keyword as a second starting node based on a preset medical knowledge graph, the medical knowledge graph being used to represent the correlation relationship between radiological features, pathological types and anatomical sites; querying a first path from the first starting node and a second path from the second starting node according to the medical knowledge graph; and determining the clinical correlation degree between the candidate template and the diagnosis text according to the first path and the second path.

[0110] The medical knowledge graph mentioned above can include a radiological feature node (representing a radiological feature such as a nodule, a mass, etc.), a pathological type node (representing a pathological type such as canceration, inflammation, etc.), and an anatomical site node (representing an anatomical site such as the lung, the liver, etc.). The medical knowledge graph containing a large number of nodes can be constructed in advance, and the edges between the nodes in the medical knowledge graph represent the correlation relationship between the nodes, and the edge weight can represent the correlation degree between the nodes connected by the edge, for example, the correlation degree between “lung nodule” and “adenocarcinoma”. The edge weight can be determined by statistical analysis and expert annotation.

[0111] The degree of clinical relevance needs to be determined based on the medical knowledge graph mentioned above. First, the server 200 can determine the first keywords in the candidate template and the second keywords in the diagnosis text, wherein the keywords representing the radiological features, pathological types and anatomical sites can be extracted from the candidate template and the diagnosis text, so as to obtain the first keywords corresponding to the candidate template and the second keywords corresponding to the diagnosis text. The first keywords and the second keywords can each contain more than one word.

[0112] Then, the server 200 can query the node corresponding to the first keyword (referred to as the first starting node in the foregoing) and the node corresponding to the second keyword (referreded to as the second starting node in the foregoing) in the medical knowledge graph. No matter the first starting node or the second starting node, it can be taken as a starting point to start querying in the medical knowledge graph from the first starting node (or the second starting node) as a starting point to obtain a path from the first starting node (or the second starting node) as a starting point, so as to obtain the first path or the second path. The length of the path can be a fixed preset length, which can be set by human beings in advance.

[0113] Among them, the corresponding path can be determined by the multi-hop reasoning method. For example, the first starting node (or the second starting node) can be taken as a starting point, and a target neighbor node (hereinafter referred to as a first neighbor node) can be selected from the neighbor nodes of the starting point as the second node in the path. Then, a target neighbor node (hereinafter referred to as a second neighbor node) can be selected from the neighbor nodes of the first neighbor node as the third node in the path, and so on, until the length of the path reaches the preset length.

[0114] Among them, the way of selecting the target neighbor node can be: for a node, when selecting the target neighbor node of the node, the node type to which the node belongs (for example, is a radiological feature node, a pathological type node or an anatomical site node) is determined, and according to the node type, the target node type associated therewith (for example, the target node type associated with the radiological feature node type can be the pathological type node type, and the target node type associated with the pathological type node can be the anatomical site node type. At least one target node type associated with a certain node type can be preset). Then, from the neighbor nodes of the target node type, the neighbor nodes with edge weights exceeding a preset threshold are selected, and the nodes with the association degrees between the first keyword (or the second keyword) exceeding a preset association degree are selected from the neighbor nodes as the target neighbor nodes.

[0115] The association degree between the nodes can be determined by a graph neural network (such as Graphsage) to determine the node features of each node, and the association degree between the node features is used as the association degree between the nodes. In addition, the preset threshold can be different for different types of edge weights. For example, the edge weight between the radiology feature node and the pathology type node and the edge weight between the pathology type node and the anatomical site node are different types of edge weights, and different preset thresholds can be set accordingly.

[0116] Continuing with the above example, the diagnostic text in the above example is used as an example, and nodes related to the keywords "right lung upper lobe", "solid nodule", and "malignant" in the medical knowledge graph can be queried based on the diagnostic text. For example, starting from the "nodule" node corresponding to the "solid nodule", a path "nodule -> 1 hop: adenocarcinoma (weight 0.9) -> 2 hop: pleural invasion (weight 0.85)" starting from the "nodule" node can be queried, as shown in Figure 6

[0117] Figure 6 is a path diagram according to the first aspect of embodiment 1 of the present disclosure.

[0118] Figure 6 The above-mentioned path is shown in the diagram, starting from the nodule node, selecting the pathology type node "adenocarcinoma" node from the neighbor nodes, and after reaching the "adenocarcinoma" node, selecting the pathology type node "pleural invasion" node from its neighbor nodes, thereby forming a path. Table 6 shows an example of the clinical association degree determined based on the first path corresponding to the candidate template and the second path corresponding to the diagnostic text.

[0119] Table 6

[0120]

[0121] Specifically, the similarity between the first path corresponding to the candidate template and the second path corresponding to the diagnostic text can be used as the clinical association degree between the candidate template and the diagnostic text.

[0122] ​Then, the server 200 can determine a first matching degree between the candidate templates and the diagnosis text according to the spatial matching degree, the semantic matching degree and the clinical correlation degree. Specifically, the corresponding weights of the spatial matching degree, the semantic matching degree and the clinical correlation degree can be preset. For example, the weight of the spatial matching degree can be set to 40%, the weight of the semantic matching degree can be set to 35%, and the weight of the clinical correlation degree can be set to 25%. Then, the spatial matching degree, the semantic matching degree and the clinical correlation degree are weighted and summed according to the corresponding weights of the spatial matching degree, the semantic matching degree and the clinical correlation degree, respectively, to obtain the first matching degree.

[0123] Continuing the above example, Table 7 shows an example of the first matching degree of each candidate template. It should be noted that the values shown in each table in the present specification are for the purpose of illustrating the flow of the method, and are only examples and do not represent the actual calculated values.

[0124] Table 7

[0125]

[0126] Optionally, the operation of determining the target template from the candidate templates according to the first matching degree comprises: correcting the first matching degree according to the creation time of the candidate template, the lesion type corresponding to the candidate template and the conflict degree of the candidate template and the diagnosis text in the region of interest, to determine a second matching degree after correction; and determining the target template from the candidate templates according to the second matching degree.

[0127] Wherein, after the first matching degree is determined, the templates that meet the preset conditions can be selected from the candidate templates according to the first matching degree as the pending templates. For example, the templates with a first matching degree not less than a first preset threshold can be selected as the pending templates, or the top-k templates (k can be a preset value) can be selected according to the first matching degree from large to small as the pending templates. Then, the first matching degree of the pending templates can be corrected in the above manner to obtain the second matching degree. Finally, the target template to be recommended to the user is selected through the second matching degree.

[0128] Specifically, how to correct the first matching degree through the creation time, the lesion type and the conflict degree will be described in detail below.

[0129] Optionally, the operation of correcting the first matching degree according to the creation time of the candidate template, the lesion type corresponding to the candidate template, and the conflict degree of the candidate template and the diagnostic text in the region of interest includes: determining a time decay factor corresponding to the candidate template according to the creation time; determining a clinical priority corresponding to the candidate template according to the lesion type; and correcting the first matching degree according to the time decay factor, the clinical priority, and the conflict degree to obtain a second matching degree.

[0130] Specifically, the time decay factor can be determined by a time interval between the creation time and the current time. The larger the time interval, the smaller the time decay factor (the smaller the priority of the template). Specifically, the time decay factor can be set as: For example, γ can be set as 0.9, The time interval (months) between the creation time and the current time. The clinical priority corresponding to different lesion types can also be set. For example, the clinical priority corresponding to malignant lesions can be set as 1.2, the clinical priority corresponding to acute lesions can be set as 1.1, and the clinical priority corresponding to chronic lesions can be set as 0.9. The conflict degree of the candidate template and the diagnostic text in the region of interest mentioned above refers to the difference degree between the interested part (anatomical part) mentioned in the candidate template and the interested part (anatomical part) mentioned in the diagnostic text. Specifically, the intersection-over-union between the interested part (anatomical part) mentioned in the candidate template and the interested part (anatomical part) mentioned in the diagnostic text can be determined, and then the conflict degree can be obtained by 1-intersection-over-union.

[0131] The following is an example of correcting the first matching degree to obtain the second matching degree. As shown in Table 8 and Table 9, it is assumed that candidate template A and candidate template D are selected as the pending templates by the first matching degree, and the second matching degree corresponding to the two templates needs to be determined according to the clinical priority, the time decay factor, and the conflict degree.

[0132] Table 8

[0133]

[0134] Table 9

[0135]

[0136] As can be seen from Table 6, the second matching degree can be determined by the following formula.

[0137]

[0138] Then, the server 200 can select a target template according to the second matching degree. For example, templates with a second matching degree not less than a first preset threshold can be selected as target templates, or templates can be sorted according to the second matching degree from large to small, and templates of top-a (a can be a preset value) can be selected as target templates. By sending the target template to the terminal 100, the user can complete the corresponding image diagnosis report according to the target template.

[0139] In addition, referring to Figure 1 According to a second aspect of the embodiment, a storage medium is provided. The storage medium includes a stored program, wherein the program is executed by a processor to perform the method of any one of the above.

[0140] According to the technical solution of the embodiment, the first matching degree fused with multiple types of information can be determined by combining the spatial matching degree between the candidate template and the diagnosis text, the semantic matching degree between the candidate template and the diagnosis text related to the pathological information, and the clinical correlation degree between the candidate template and the diagnosis text determined by the medical knowledge graph. Moreover, the target template is determined not only according to the first matching degree, but also by combining the time decay factor, the clinical priority, and the conflict degree between the regions of interest, so that the first matching degree is further corrected. The second matching degree obtained by correcting the first matching degree is used to determine the target template, so that the template required by the user to complete the image diagnosis report can be more accurately searched out, and the efficiency of the user in completing the image diagnosis report can be further improved.

[0141] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0142] Those skilled in the art can clearly understand the method according to the above-mentioned embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0143] Example 2

[0144] Figure 7 An image diagnostic report generation apparatus 700 according to this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. (See reference...) Figure 7 As shown, the device 700 includes: a receiving module 710 for receiving diagnostic text corresponding to medical images input by a user from a user's terminal device; a three-level matching module 720 for determining the spatial matching degree between the diagnostic text and candidate templates at the anatomical location level, the semantic matching degree between the diagnostic text and candidate templates related to pathological information, and the clinical relevance between the diagnostic text and candidate templates; a matching degree determination module 730 for determining a first matching degree between the candidate templates and the diagnostic text based on the spatial matching degree, the semantic matching degree, and the clinical relevance; a selection module 740 for determining a target template from the candidate templates based on the first matching degree; and a sending module 750 for sending the target template to the terminal device and receiving an image diagnostic report corresponding to the target template from the terminal device.

[0145] Optionally, the three-level matching module 720 is specifically used to: determine the first anatomical location information in the candidate template and the second anatomical location information in the diagnostic text; determine the location similarity based on the first anatomical location information and the second anatomical location information; determine the spatial proximity based on the proximity relationship between the anatomical locations corresponding to the first anatomical location information and the second anatomical location information in the medical image; and determine the spatial matching degree based on the location similarity and the spatial proximity.

[0146] Optionally, the three-level matching module 720 is specifically used to: determine the first coordinate information of the anatomical site corresponding to the first anatomical site information in the medical image, and determine the second coordinate information of the anatomical site corresponding to the second anatomical site information in the medical image; perform feature embedding on the first coordinate information and the second coordinate information to obtain the first coordinate feature corresponding to the first coordinate information and the second coordinate feature corresponding to the second coordinate information; and determine the spatial proximity degree based on the first coordinate feature and the second coordinate feature.

[0147] Optionally, the three-level matching module 720 is specifically used to: extract features from the descriptive information of the candidate template and extract features from the diagnostic text based on the first feature extraction model, to obtain the first semantic feature corresponding to the descriptive information and the second semantic feature corresponding to the diagnostic text. The first feature extraction model is obtained by fine-tuning a preset semantic feature extraction model based on training samples related to the medical scenario. Based on the first and second semantic features, the degree of semantic matching between the candidate template and the diagnostic text related to pathological information is determined.

[0148] Optionally, the third matching module 720 is specifically configured to determine the first max-pooling result and the first average-pooling result corresponding to the first semantic feature, perform weighted fusion on the first max-pooling result and the first average-pooling result to obtain the first fusion feature corresponding to the description information, determine the second max-pooling result and the second average-pooling result corresponding to the second semantic feature, perform weighted fusion on the second max-pooling result and the second average-pooling result to obtain the second fusion feature corresponding to the diagnosis text, and determine the semantic matching degree between the candidate template and the diagnosis text related to the pathological information according to the first fusion feature and the second fusion feature.

[0149] Optionally, the third matching module 720 is specifically configured to determine the first keyword in the candidate template and the second keyword in the diagnosis text, query a node matched with the first keyword as a first starting node and a node matched with the second keyword as a second starting node based on a preset medical knowledge graph, the medical knowledge graph being used to represent the association relationship between the radiology features, the pathological types and the anatomical sites, query a first path with the first starting node as a starting point and a second path with the second starting node as a starting point according to the medical knowledge graph, and determine the clinical association degree between the candidate template and the diagnosis text according to the first path and the second path.

[0150] Optionally, the selection module 740 is specifically configured to correct the first matching degree according to the creation time of the candidate template, the lesion type corresponding to the candidate template and the conflict degree of the candidate template and the diagnosis text in the region of interest, determine the second matching degree after correction, and determine the target template from the candidate template according to the second matching degree.

[0151] Optionally, the selection module 740 is specifically configured to determine a time attenuation factor corresponding to the candidate template according to the creation time, determine a clinical priority corresponding to the candidate template according to the lesion type, and correct the first matching degree according to the time attenuation factor, the clinical priority and the conflict degree to obtain the second matching degree.

[0152] Therefore, according to the technical solution of the embodiment, the spatial matching degree between the candidate template and the diagnostic text, the semantic matching degree between the candidate template and the diagnostic text related to the pathological information, and the clinical correlation degree between the candidate template and the diagnostic text determined by the medical knowledge graph are combined to determine the first matching degree fused with multiple information. Moreover, the target template is determined not only according to the first matching degree, but also in combination with the time decay factor, the clinical priority, and the conflict degree between the regions of interest, so that the first matching degree is further corrected. The target template is determined by the second matching degree obtained by correcting the first matching degree, so that the template required by the user for completing the image diagnosis report is more accurately searched out, and the efficiency of the user in completing the image diagnosis report is further improved

[0153] Embodiment 3

[0154] Figure 8 An image diagnosis report generation apparatus 800 according to the embodiment is shown. The apparatus 800 corresponds to the method according to the first aspect of the embodiment 1. Referring to FIG. 8, the apparatus 800 includes a processor 810 and a memory 820 connected to the processor 810, for providing the processor 810 with instructions to process the following processing steps: receiving, from a terminal device of a user, a diagnostic text corresponding to a medical image input by the user; determining a spatial matching degree between the diagnostic text and a candidate template at an anatomical site level, a semantic matching degree between the diagnostic text and the candidate template related to pathological information, and a clinical correlation degree between the diagnostic text and the candidate template; determining a first matching degree between the candidate template and the diagnostic text according to the spatial matching degree, the semantic matching degree, and the clinical correlation degree; determining a target template from the candidate template according to the first matching degree; and sending the target template to the terminal device and receiving an image diagnosis report corresponding to the target template from the terminal device. Figure 8

[0155] Optionally, the operation of determining the spatial matching degree between the diagnostic text and the candidate template at the anatomical site level includes: determining first anatomical site information in the candidate template and second anatomical site information in the diagnostic text; determining a site similarity according to the first anatomical site information and the second anatomical site information; determining a spatial position adjacency according to the adjacent relationship of the anatomical sites corresponding to the first anatomical site information and the second anatomical site information in the medical image; and determining the spatial matching degree according to the site similarity and the spatial position adjacency.

[0156] ​Optionally, the operation of determining the spatial position adjacency degree according to the proximity relationship of the anatomical sites corresponding to the first anatomical site information and the second anatomical site information in the medical image comprises: determining first coordinate information of the anatomical site corresponding to the first anatomical site information in the medical image, and determining second coordinate information of the anatomical site corresponding to the second anatomical site information in the medical image; performing feature embedding on the first coordinate information and the second coordinate information to obtain first coordinate features corresponding to the first coordinate information and second coordinate features corresponding to the second coordinate information; and determining the spatial position adjacency degree according to the first coordinate features and the second coordinate features.

[0157] Optionally, the operation of determining the semantic matching degree between the candidate template and the diagnosis text related to the pathological information comprises: performing feature extraction on the description information of the candidate template and the diagnosis text based on a first feature extraction model, to obtain first semantic features corresponding to the description information and second semantic features corresponding to the diagnosis text, wherein the first feature extraction model is obtained by fine-tuning a preset semantic feature extraction model based on training samples related to a medical scene; and determining the semantic matching degree between the candidate template and the diagnosis text related to the pathological information according to the first semantic features and the second semantic features.

[0158] Optionally, the operation of determining the semantic matching degree between the candidate template and the diagnosis text related to the pathological information according to the first semantic features and the second semantic features comprises: determining first maximum pooling results and first average pooling results corresponding to the first semantic features, performing weighted fusion on the first maximum pooling results and the first average pooling results to obtain first fusion features corresponding to the description information; determining second maximum pooling results and second average pooling results corresponding to the second semantic features, performing weighted fusion on the second maximum pooling results and the second average pooling results to obtain second fusion features corresponding to the diagnosis text; and determining the semantic matching degree between the candidate template and the diagnosis text related to the pathological information according to the first fusion features and the second fusion features.

[0159] Optionally, the operation of determining the clinical correlation degree between the candidate template and the diagnosis text comprises: determining a first keyword in the candidate template and a second keyword in the diagnosis text; querying a node matching the first keyword as a first starting node and a node matching the second keyword as a second starting node based on a preset medical knowledge graph, the medical knowledge graph being used to represent the correlation relationship between radiology features, pathological types and anatomical sites; querying a first path from the first starting node and a second path from the second starting node according to the medical knowledge graph; and determining the clinical correlation degree between the candidate template and the diagnosis text according to the first path and the second path.

[0160] Optionally, the operation of determining the target template from the candidate templates according to the first matching degree comprises: correcting the first matching degree according to a creation time of the candidate template, a lesion type corresponding to the candidate template, and a conflict degree of the candidate template and the diagnostic text in the region of interest, determining a second matching degree after correction; and determining the target template from the candidate templates according to the second matching degree.

[0161] Optionally, the operation of correcting the first matching degree according to the creation time of the candidate template, the lesion type corresponding to the candidate template, and the conflict degree of the candidate template and the diagnostic text in the region of interest comprises: determining a time attenuation factor corresponding to the candidate template according to the creation time; determining a clinical priority corresponding to the candidate template according to the lesion type; and correcting the first matching degree according to the time attenuation factor, the clinical priority, and the conflict degree, to obtain a second matching degree.

[0162] Therefore, according to the technical solution of the embodiment, the first matching degree that fuses multiple types of information can be determined according to the spatial matching degree between the candidate template and the diagnostic text, the semantic matching degree between the candidate template and the diagnostic text related to the pathological information, and the clinical correlation degree between the candidate template and the diagnostic text determined by the medical knowledge graph, and the target template can be determined according to not only the first matching degree but also the time attenuation factor, the clinical priority, and the conflict degree in the region of interest, so that the first matching degree is further corrected, the second matching degree obtained by correcting the first matching degree is used to determine the target template, and the template required by the user to complete the image diagnosis report can be more accurately searched, thereby improving the efficiency of the user in completing the image diagnosis report.

[0163] The above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0164] In the above-mentioned embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0165] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division way, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0167] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0168] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0169] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for generating an image diagnostic report, characterized in that, include: Receive diagnostic text corresponding to medical images input by the user from the user's terminal device; The degree of spatial matching between the diagnostic text and the candidate template at the anatomical location level, the degree of semantic matching between the diagnostic text and the candidate template related to pathological information, and the degree of clinical relevance between the diagnostic text and the candidate template are determined. Based on the spatial matching degree, the semantic matching degree, and the clinical relevance degree, a first matching degree between the candidate template and the diagnostic text is determined; Based on the first matching degree, a target template is determined from the candidate templates; as well as Sending the target template to the terminal device and receiving an image diagnostic report corresponding to the target template from the terminal device, wherein the operation of determining the degree of clinical relevance between the candidate template and the diagnostic text includes: Determine the first keyword in the candidate template and the second keyword in the diagnostic text; Based on a preset medical knowledge graph, nodes matching the first keyword are queried as first starting nodes, and nodes matching the second keyword are queried as second starting nodes. The medical knowledge graph is used to represent the relationship between radiological features, pathological types, and anatomical locations. Based on the medical knowledge graph, a first path is retrieved starting from the first starting node, and a second path is retrieved starting from the second starting node; and Based on the first path and the second path, the degree of clinical relevance between the candidate template and the diagnostic text is determined.

2. The method according to claim 1, characterized in that, The operation of determining the degree of spatial matching between the diagnostic text and the candidate template at the anatomical site level includes: The first anatomical location information in the candidate template and the second anatomical location information in the diagnostic text are determined. Based on the first anatomical location information and the second anatomical location information, the similarity of the locations is determined; Based on the proximity relationship of the anatomical sites corresponding to the first anatomical site information and the second anatomical site information in medical images, the spatial proximity is determined. The degree of spatial matching is determined based on the similarity of the said parts and the adjacency of the said spatial locations.

3. The method according to claim 2, characterized in that, The operation of determining spatial proximity based on the proximity relationships of the anatomical sites corresponding to the first and second anatomical site information in medical images includes: Determine the first coordinate information of the anatomical site corresponding to the first anatomical site information in the medical image, and determine the second coordinate information of the anatomical site corresponding to the second anatomical site information in the medical image; Feature embedding is performed on the first coordinate information and the second coordinate information to obtain a first coordinate feature corresponding to the first coordinate information and a second coordinate feature corresponding to the second coordinate information; The spatial proximity is determined based on the first coordinate feature and the second coordinate feature.

4. The method according to claim 1, characterized in that, The operation of determining the degree of semantic matching related to pathological information between the candidate template and the diagnostic text includes: Based on the first feature extraction model, feature extraction is performed on the description information of the candidate template and on the diagnostic text to obtain a first semantic feature corresponding to the description information and a second semantic feature corresponding to the diagnostic text. The first feature extraction model is obtained by fine-tuning a preset semantic feature extraction model based on training samples related to medical scenarios. Based on the first semantic feature and the second semantic feature, the degree of semantic matching between the candidate template and the diagnostic text related to pathological information is determined.

5. The method according to claim 4, characterized in that, The operation of determining the degree of semantic matching related to pathological information between the candidate template and the diagnostic text based on the first semantic feature and the second semantic feature includes: Determine the first max pooling result and the first average pooling result corresponding to the first semantic feature, and perform weighted fusion on the first max pooling result and the first average pooling result to obtain the first fused feature corresponding to the description information; Determine the second max pooling result and the second average pooling result corresponding to the second semantic feature, and perform a weighted fusion of the second max pooling result and the second average pooling result to obtain the second fused feature corresponding to the diagnostic text; and Based on the first fusion feature and the second fusion feature, the degree of semantic matching between the candidate template and the diagnostic text related to pathological information is determined.

6. The method according to claim 1, characterized in that, The operation of determining the target template from the candidate templates based on the first matching degree includes: Based on the creation time of the candidate template, the lesion type corresponding to the candidate template, and the degree of conflict between the candidate template and the diagnostic text in the region of interest, the first matching degree is corrected to determine the corrected second matching degree; and Based on the second matching degree, the target template is determined from the candidate templates, and wherein... The operation of refining the first matching degree based on the creation time of the candidate template, the lesion type corresponding to the candidate template, and the degree of conflict between the candidate template and the diagnostic text in the region of interest, and determining the refined second matching degree, includes: Based on the creation time, determine the time decay factor corresponding to the candidate template; Based on the lesion type, determine the clinical priority corresponding to the candidate template; and The first matching degree is modified based on the time decay factor, the clinical priority, and the degree of conflict to obtain the second matching degree.

7. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, a processor performs the method according to any one of claims 1 to 6.

8. An image diagnostic report generation device, characterized in that, include: The receiving module is used to receive diagnostic text corresponding to medical images input by the user from the user's terminal device; A three-level matching module is used to determine the spatial matching degree between the diagnostic text and the candidate template at the anatomical location level, the semantic matching degree between the diagnostic text and the candidate template related to pathological information, and the clinical relevance between the diagnostic text and the candidate template. Specifically, the three-level matching module is used to: determine a first keyword in the candidate template and a second keyword in the diagnostic text; query nodes matching the first keyword as first starting nodes based on a preset medical knowledge graph, and query nodes matching the second keyword as second starting nodes; the medical knowledge graph is used to represent the relationships between radiological features, pathological types, and anatomical locations; query a first path starting from the first starting node and a second path starting from the second starting node based on the medical knowledge graph; and determine the clinical relevance between the candidate template and the diagnostic text based on the first and second paths. The matching degree determination module is used to determine a first matching degree between the candidate template and the diagnostic text based on the spatial matching degree, the semantic matching degree, and the clinical relevance degree. The selection module is configured to determine a target template from the candidate templates based on the first matching degree; and The sending module is used to send the target template to the terminal device and receive the image diagnostic report corresponding to the target template from the terminal device.

9. An image diagnostic report generation device, characterized in that, include: processor; as well as A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: Receive diagnostic text corresponding to medical images input by the user from the user's terminal device; The degree of spatial matching between the diagnostic text and the candidate template at the anatomical location level, the degree of semantic matching between the diagnostic text and the candidate template related to pathological information, and the degree of clinical relevance between the diagnostic text and the candidate template are determined. Based on the spatial matching degree, the semantic matching degree, and the clinical relevance degree, a first matching degree between the candidate template and the diagnostic text is determined; Based on the first matching degree, a target template is determined from the candidate templates; as well as Sending the target template to the terminal device and receiving an image diagnostic report corresponding to the target template from the terminal device, wherein the operation of determining the degree of clinical relevance between the candidate template and the diagnostic text includes: Determine the first keyword in the candidate template and the second keyword in the diagnostic text; Based on a preset medical knowledge graph, nodes matching the first keyword are queried as first starting nodes, and nodes matching the second keyword are queried as second starting nodes. The medical knowledge graph is used to represent the relationship between radiological features, pathological types, and anatomical locations. Based on the medical knowledge graph, a first path is retrieved starting from the first starting node, and a second path is retrieved starting from the second starting node; and Based on the first path and the second path, the degree of clinical relevance between the candidate template and the diagnostic text is determined.

Images