Key structure positioning and structured report output method and device for CT (Computed Tomography) image

Abstract

The invention discloses a key structure positioning and structured report output method and device for a CT image, and belongs to the technical field of data processing.The key structure positioning and structured report output method for the CT image comprises the steps that the 3D CT image is acquired; inputting the CT image into a trained visual encoder to obtain a high-order visual feature vector; based on the constructed vector database, matching similar images of the high-order visual feature vectors to obtain matched similar images and original structured report texts; inputting the high-order visual feature vector, the matched similar image and the original structured report text into a trained large language model decoder to obtain a preliminary structured report; and inputting the preliminary structured report into the trained key structure positioning model to obtain a structured report with position coordinates. According to the method, the output precision and the positioning precision of the model on the structured text can be effectively improved, and index measurement of a subsequent key structure is facilitated.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a method and device for locating key structures of CT images and outputting structured reports. Background Art

[0002] In recent years, significant progress has been made in lesion localization and organ segmentation technologies based on CT images. These advances are mainly due to the rapid development of deep learning technology, especially convolutional neural networks (CNN). In terms of lesion localization, advanced target detection algorithms, such as Faster R-CNN and YOLO (You Only Look Once), have been widely used in the automatic detection of lesions such as lung nodules and liver tumors. These algorithms not only improve the accuracy of lesion detection, but also significantly reduce the missed detection rate, making it possible to detect early diseases.

[0003] However, the existing key structure positioning and recognition based on CT images and structured text output models still have the following technical problems:

[0004] Existing technologies cannot combine multiple tasks such as CT image-based key structure localization and fluent structured text output;

[0005] Existing techniques make positioning or reporting inaccuracies when reasoning about new data. Summary of the invention

[0006] The purpose of this application is to provide a method and device for locating key structures of CT images and outputting structured reports to achieve the combination of key structure positioning based on CT images and smooth structured text output, and ensure the accuracy of positioning and reporting.

[0007] According to a first aspect of an embodiment of the present application, a method for locating key structures of a CT image and outputting a structured report is provided, and the method may include:

[0008] Acquire 3D CT images;

[0009] Input the CT image into the trained visual encoder to obtain a high-order visual feature vector;

[0010] Based on the constructed vector database, similar images of high-order visual feature vectors are matched to obtain matching similar images and original structured report text;

[0011] Input the high-order visual feature vector, matching similar images and the original structured report text into the trained large language model decoder to obtain a preliminary structured report;

[0012] The preliminary structured report is input into the trained key structure positioning model to obtain a structured report with location coordinates.

[0013] In some optional embodiments of the present application, the trained visual encoder is trained by the following method:

[0014] Construct an autoregressive encoder based on the original CT image;

[0015] Based on the trained large language model and the structured report of the original CT image, the autoregressive encoder is trained by contrastive learning so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, and obtain a primary visual encoder;

[0016] The loss function is fused to align the text feature vector and the visual feature vector to obtain the trained visual encoder.

[0017] In some optional embodiments of the present application, an autoregressive encoder is constructed based on the original CT image, including:

[0018] Obtaining original 3D CT images;

[0019] Cut the original 3D CT image into multiple 3D CT image blocks;

[0020] Each CT image block is mapped to a vector space of fixed dimension through a linear projection layer to obtain the corresponding image feature vector, and a position code is added to each CT image block;

[0021] After each CT image block is output by the encoder, a high-dimensional feature vector sequence containing spatial context information is obtained and sent to the decoder. The output of the decoder is the original 3D CT image.

[0022] The encoder is trained using the mean square error as the training loss function to obtain an autoregressive encoder.

[0023] In some optional embodiments of the present application, based on the trained large language model and the structured report of the original CT image, the autoregressive encoder is trained by contrastive learning so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, and obtain a primary visual encoder, including:

[0024] Input the structured report of the original CT image into the trained large language model to obtain text vector features;

[0025] Input the original CT image into the autoregressive encoder to obtain the visual feature vector;

[0026] Contrastive learning is used to bring similar samples closer and dissimilar samples farther apart, so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, and obtain a primary visual encoder.

[0027] In some optional embodiments of the present application, the loss function is fused to align the text feature vector and the visual feature vector to obtain a trained visual encoder, including:

[0028] Different weights are given to cosine similarity loss, Euclidean distance loss, and Manhattan distance loss so that the primary visual encoder learns feature vectors from two modalities;

[0029] The fusion loss is calculated as follows:

[0030]

[0031] Among them, L f is the fusion loss; L c is the cosine similarity loss; L d is the Euclidean distance loss; L m is the Manhattan distance loss.

[0032] In some optional embodiments of the present application, the vector database is constructed by the following method:

[0033] Obtain CT image data and its structured report;

[0034] Input the CT image data into the trained visual encoder to obtain the visual feature vector, which is stored in an encoded manner;

[0035] Input the structured report into the trained large language model encoder to obtain the text feature vector;

[0036] The text feature vector and the structured report text content are stored together in a vector database.

[0037] In some optional embodiments of the present application, the trained key structure positioning model is obtained by training using the following method:

[0038] Obtain CT image training data and its corresponding structured report;

[0039] Input the CT image training data into the frozen visual encoder to obtain the visual feature vector;

[0040] Input the structured report into the large language model encoder to obtain a text feature vector;

[0041] Set the key of the visual feature vector matrix to image and the key of the textual feature vector matrix to report;

[0042] Combine images and reports into json files and store them in text form to form the target text;

[0043] Input the target text into the new large language model and build LoRA. The output of the new large language model is the 3D spatial position coordinates of the key structures mentioned in the structured report, and training is performed.

[0044] The new large language model converts the coordinates into the original CT space and draws a rectangular box to display it in the CT image training data to obtain a trained key structure positioning model.

[0045] According to a second aspect of an embodiment of the present application, a device for locating key structures of a CT image and outputting a structured report is provided, and the device may include:

[0046] An acquisition module, used for acquiring 3D CT images;

[0047] The visual feature extraction module is used to input the CT image into the trained visual encoder to obtain a high-order visual feature vector;

[0048] A matching module is used to match similar images of high-order visual feature vectors based on the constructed vector database to obtain matching similar images and original structured report text;

[0049] The structured report output module is used to input the high-order visual feature vector, matching similar images and the original structured report text into the trained large language model decoder to obtain a preliminary structured report;

[0050] The positioning module is used to input the preliminary structured report into the trained key structure positioning model to obtain a structured report with location coordinates.

[0051] According to a third aspect of an embodiment of the present application, an electronic device is provided, which may include:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to execute instructions to implement the key structure positioning and structured report output method of the CT image as shown in any one of the embodiments of the first aspect.

[0055] According to a fourth aspect of the embodiments of the present application, a storage medium is provided. When the instructions in the storage medium are executed by a processor of an information processing device or a server, the information processing device or the server implements the key structure positioning and structured report output method of a CT image as shown in any one of the embodiments of the first aspect.

[0056] The above technical solution of the present application has the following beneficial technical effects:

[0057] The method of the embodiment of the present application can effectively improve the output accuracy and positioning accuracy of the model for structured text by utilizing a vector database to provide "memory" and "reference" for CT images and structured text output, and facilitate the subsequent measurement of key structure indicators; and the method utilizes visual models and large language models to achieve semantic 3D spatial positioning of key structures in CT images, thereby playing a role in accurate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of a method for locating key structures of CT images and outputting structured reports in an exemplary embodiment of the present application;

[0059] Figure 2 It is a flowchart of a method for locating key structures of CT images and outputting structured reports in a specific embodiment of the present application;

[0060] Figure 3 is a flowchart of an image-text alignment process in an exemplary embodiment of the present application;

[0061] Figure 4 is a flow chart of a method for constructing a vector database in an exemplary embodiment of the present application;

[0062] Figure 5 is a schematic diagram of a process for locating a key structure in an exemplary embodiment of the present application;

[0063] Figure 6 It is a schematic diagram of the structure of a device for locating key structures of CT images and outputting structured reports in an exemplary embodiment of the present application;

[0064] Figure 7 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present application;

[0065] Figure 8 It is a schematic diagram of the hardware structure of an electronic device in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with specific implementations and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concepts of the present application.

[0067] The accompanying drawings show schematic diagrams of layer structures according to embodiments of the present application. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clarity. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0068] Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0069] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0070] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0071] The following, in conjunction with the accompanying drawings, describes in detail the key structure positioning and structured report output method and device of CT images provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0072] like Figure 1 As shown, in a first aspect of an embodiment of the present application, a method for locating key structures of a CT image and outputting a structured report is provided, and the method may include:

[0073] S110: Acquire a 3D CT image;

[0074] S120: input the CT image into the trained visual encoder to obtain a high-order visual feature vector;

[0075] S130: Based on the constructed vector database, matching similar images of the high-order visual feature vectors to obtain matching similar images and original structured report text;

[0076] S140: inputting the high-order visual feature vector, the matched similar image and the original structured report text into the trained large language model decoder to obtain a preliminary structured report;

[0077] S150: Input the preliminary structured report into the trained key structure positioning model to obtain a structured report with position coordinates.

[0078] The method of this embodiment uses a vector database to provide "memory" and "reference" for CT images and structured text output, which can effectively improve the output accuracy and positioning accuracy of the model for structured text, and facilitate the subsequent measurement of key structure indicators; and the method uses visual models and large language models to achieve semantic 3D spatial positioning of key structures in CT images, thereby playing a role in accurate recognition.

[0079] This implementation designs a multimodal network and uses a vector database to achieve key structure positioning and structured report output based on CT images. The specific flow chart is as follows Figure 2 As shown in the figure, the process mainly consists of several core parts, namely visual encoder, vector database, large language model decoder and key structure positioning model.

[0080] During training, a 3D CT image is input, and then the high-order visual feature vector of the CT is obtained through the pre-trained and frozen visual encoder. The vector is then sent to the established vector database for rapid matching of similar images and finding the corresponding structured report text vector and original text. At the same time, the CT visual feature vector and the matching CT visual vector and text jointly use the LoRA (Low-Rank Adaptation of Large Language Models) technology to fine-tune the large language model, and then a preliminary structured report is obtained through the frozen large language model decoder. In order to locate the key structure, the key structure positioning model takes the structured report as input and returns the corresponding position coordinates of the original CT image to mark the corresponding area on the CT. During inference, the large language model encoder fine-tuned by LoRA is connected to the matching image and structured text and the feature vector of the input CT, and the back-end output can be performed.

[0081] In some embodiments, the trained visual encoder is trained by the following method:

[0082] Construct an autoregressive encoder based on the original CT image;

[0083] Based on the trained large language model and the structured report of the original CT image, the autoregressive encoder is trained by contrastive learning so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, and obtain a primary visual encoder;

[0084] The loss function is fused to align the text feature vector and the visual feature vector to obtain the trained visual encoder.

[0085] This embodiment introduces the construction method of the visual encoder. Figure 3As shown, the encoder uses a "two-stage" training approach to align the structured text of the training data and its corresponding CT, so that the visual encoder can effectively incorporate text priors and fully extract the image feature vectors of the CT.

[0086] In the first stage, the Transformer architecture is used to build an autoregressive encoder and decoder.

[0087] Specifically, the input of the autoregressive encoder is a CT image. During preprocessing, the original 3D CT image is cut into multiple 3D CT image blocks of fixed size. Each CT image block is mapped to a vector space of fixed dimension through a linear projection layer to obtain the corresponding image feature vector. At the same time, position encoding is added to each CT image block to retain the spatial position information. The encoding layer is mainly composed of a multi-head self-attention mechanism, a feedforward network, layer normalization, and a residual connection. After the multiple CT image blocks are output by the encoder, a high-dimensional feature vector sequence containing spatial context information is obtained to be fed into the decoder. The decoder mainly includes self-attention, feedforward network, layer normalization, and residual connection under the autoregressive mechanism, and finally outputs the original image.

[0088] The purpose of network training is to obtain the same CT image at both the input and output ends by training the autoregressive encoder and the autoregressive decoder. During training, the network input is the original CT image, and the output is also the CT image. The mean square error is selected as the loss function for training. After training, the autoregressive encoder has the basic understanding ability and semantic abstraction ability of CT images. The autoregressive encoder is then used as a pre-training model for the visual encoder. In the second stage, the paired matching CT images and structured reports are sent to the visual encoder and the large language model encoder respectively to obtain the visual feature vector and text feature vector of the CT image, respectively.

[0089] The second phase of training is divided into two steps.

[0090] The first step uses a contrastive learning strategy to enable the model to understand the relationship between similar images and reports and dissimilar images and reports.

[0091] Specifically, we first freeze the visual encoder to extract the visual feature vectors of all CT images in the training set, and use cosine similarity to calculate the similarity between different CT images. That is, the higher the cosine similarity, the more similar the two CT images are, and the lower the similarity, the less similar the two CT images are. Similarly, we pass the structured reports of all CT images in the training set through the large language model encoder to obtain text feature vectors, and then use cosine similarity to calculate the similarity between different text feature vectors. That is, the larger the value, the more similar they are, and vice versa.

[0092] Suppose the similarity between a CT image and any other CT image is s i, the similarity between the text of a CT image and the text of any other CT image is s t , the final similarity is s f =0.5*s i +0.5*s t When s f If the value is lower than 0.5, the pair of samples is considered dissimilar, and if it is higher than 0.5, the pair of samples is considered similar. During training, the visual encoder is unfrozen and can be trained. Similar and dissimilar CT pairs are input to train the visual encoder, and the loss function is contrast loss. Through contrast learning, similar samples can be pulled closer and dissimilar samples can be pulled farther away, so that the visual encoder can initially recognize the characteristics of different types of images and texts.

[0093] In the second step, the text feature vector and the visual feature vector are aligned through the fusion loss function, so that the network can understand the multimodal relationship between the CT and the corresponding structured report. The fusion loss function considers cosine similarity loss, Euclidean distance loss, and Manhattan distance loss and gives different weights to comprehensively learn the feature vectors from the two modalities. That is, let the fusion loss be L f , the cosine similarity loss is L c , the Euclidean distance loss is L d , Manhattan distance loss is L m , then the loss function of the model in the second step of training is in

[0094] In some embodiments, constructing an autoregressive encoder based on the original CT image includes:

[0095] Obtaining original 3D CT images;

[0096] Cut the original 3D CT image into multiple 3D CT image blocks;

[0097] Each CT image block is mapped to a vector space of fixed dimension through a linear projection layer to obtain the corresponding image feature vector, and a position code is added to each CT image block;

[0098] After each CT image block is output by the encoder, a high-dimensional feature vector sequence containing spatial context information is obtained and sent to the decoder. The output of the decoder is the original 3D CT image.

[0099] The encoder is trained using the mean square error as the training loss function to obtain an autoregressive encoder.

[0100] In some embodiments, based on the trained large language model and the structured report of the original CT image, the autoregressive encoder is trained by contrastive learning so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, thereby obtaining a primary visual encoder, including:

[0101] Input the structured report of the original CT image into the trained large language model to obtain text vector features;

[0102] Input the original CT image into the autoregressive encoder to obtain the visual feature vector;

[0103] Contrastive learning is used to bring similar samples closer and dissimilar samples farther apart, so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, and obtain a primary visual encoder.

[0104] In some embodiments, the loss function is integrated to align the text feature vector and the visual feature vector to obtain a trained visual encoder, including:

[0105] Different weights are given to cosine similarity loss, Euclidean distance loss, and Manhattan distance loss so that the primary visual encoder learns feature vectors from two modalities;

[0106] The fusion loss is calculated as follows:

[0107]

[0108] Among them, L f is the fusion loss; L c is the cosine similarity loss; L d is the Euclidean distance loss; L m is the Manhattan distance loss.

[0109] In some embodiments, the vector database is constructed by the following method:

[0110] Obtain CT image data and its structured report;

[0111] Input the CT image data into the trained visual encoder to obtain the visual feature vector, which is stored in an encoded manner;

[0112] Input the structured report into the trained large language model encoder to obtain the text feature vector;

[0113] The text feature vector and the structured report text content are stored together in a vector database.

[0114] This embodiment shows how to construct a vector database. Figure 4As shown in the figure, the database is mainly used to store visual feature vectors from CT images, text feature vectors from structured reports, and original text information. Taking a single paired CT image and structured report as an example, the CT image is first passed through a visual encoder to obtain a visual feature vector, which is stored in an encoded manner. At the same time, the structured report matching the CT image is passed through a large language model encoder to obtain a text feature vector, and then the text feature vector and the structured report text content are stored together in the vector database. The vector database has an efficient vector storage and retrieval mechanism, as well as an adaptive vector compression algorithm.

[0115] Specifically, efficient compression in vector storage is achieved through an autoencoder algorithm, which compresses and decompresses vectors by constructing multiple linear encoders. In the compression stage, the autoencoder first learns a low-dimensional representation of the original vector to capture the main features in the data. By reducing redundant information, the compressed vector retains important semantic features while greatly reducing its size. In the decompression stage, the decoder restores these low-dimensional representations back to a high-dimensional space close to the original vector. This compression method not only reduces storage requirements, but also reduces the bandwidth occupied by data transmission in distributed storage systems, thereby effectively improving retrieval performance. Vector storage adopts a distributed vector storage method to distribute vector data on multiple storage nodes, which can balance the load and improve the concurrent performance of retrieval. When searching for vectors, CT and text data in different modalities are mapped to a shared vector space, allowing users to use queries in different modalities to search for target data, so that they have comparable identical vector spaces. Through efficient vector compression and retrieval mechanisms, corresponding matching vectors in multiple modalities can be effectively queried.

[0116] In some embodiments, the trained key structure positioning model is obtained by training using the following method:

[0117] Obtain CT image training data and its corresponding structured report;

[0118] Input the CT image training data into the frozen visual encoder to obtain the visual feature vector;

[0119] Input the structured report into the large language model encoder to obtain a text feature vector;

[0120] Set the key of the visual feature vector matrix to image and the key of the textual feature vector matrix to report;

[0121] Combine images and reports into json files and store them in text form to form the target text;

[0122] Input the target text into the new large language model and build LoRA. The output of the new large language model is the 3D spatial position coordinates of the key structures mentioned in the structured report, and training is performed.

[0123] The new large language model converts the coordinates into the original CT space and draws a rectangular box to display it in the CT image training data to obtain a trained key structure positioning model.

[0124] This embodiment shows the key structure positioning process. Specifically, Figure 5 As shown in Figure 1, the process is to feed paired CTs and matched structured reports into the frozen visual encoder and the large language model encoder, respectively.

[0125] First, the CT image is passed through the frozen visual encoder to obtain the corresponding visual feature vector, which is a 1*512 feature matrix. At the same time, the structured report is passed through the large language model encoder to obtain the corresponding text feature vector, which is also a 1*512 feature matrix. The key of the visual feature vector matrix is ​​set to "image", and the key of the text feature vector matrix is ​​set to "report". The two are combined into a json file and stored in text form.

[0126] The text is then used as the input of a new large language model, and LoRA is constructed. The output of the new large language model is the 3D spatial position coordinates of the key structures mentioned in the structured report, and training is performed. After training, the model finally converts the coordinates to the original CT space and draws a rectangular frame to display it in the original image, thereby realizing the positioning function of the key structure based on the large language model. Different structures store the corresponding coordinates in the form of dictionaries, and then distinguish them by rectangular frames of different colors.

[0127] The above-mentioned embodiment method can realize the structured text output and quantitative value calculation function of key parts based on CT images and vector databases by using "reference learning". This method uses a "two-stage" training strategy and autoregressive coding technology to effectively extract high-order abstract features of CT images; introduces the concept of "reference learning", that is, using the vector database to provide "memory" and "reference" for CT images and structured text output, which can effectively improve the output accuracy and positioning accuracy of the model for structured text, and facilitate the subsequent measurement of key structure indicators; uses visual models and large language models to achieve semantic 3D spatial positioning of key structures in CT images, which plays a role in accurate recognition.

[0128] The above-mentioned embodiment method is based on actual clinical scenarios, and specifically optimizes and solves the semantic alignment problem of CT images and structured reports, which will assist clinicians in quickly and accurately making disease judgments, saving a lot of manpower and material costs for subsequent rapid disposal and treatment; and by designing a multimodal graphic output framework, it can achieve smooth output of CT image structured reports, and at the same time can locate key structures and measure key indicators, enhancing the interpretability of the framework, which will save diagnosis time for patients and improve the diagnosis efficiency of doctors.

[0129] It should be noted that the method for locating key structures of CT images and outputting structured reports provided in the embodiments of the present application may be executed by a device for locating key structures of CT images and outputting structured reports, or a control module in the device for locating key structures of CT images and outputting structured reports for executing the method. In the embodiments of the present application, the method for locating key structures of CT images and outputting structured reports by the device for locating key structures of CT images and outputting structured reports is taken as an example to illustrate the device for locating key structures of CT images and outputting structured reports provided in the embodiments of the present application.

[0130] like Figure 6 As shown, in a second aspect of the embodiment of the present application, a device for locating key structures of a CT image and outputting a structured report is provided, and the device may include:

[0131] An acquisition module 610 is used to acquire a 3D CT image;

[0132] A visual feature extraction module 620 is used to input the CT image into a trained visual encoder to obtain a high-order visual feature vector;

[0133] A matching module 630 is used to match similar images of high-order visual feature vectors based on the constructed vector database to obtain matching similar images and original structured report text;

[0134] The structured report output module 640 is used to input the high-order visual feature vector, the matched similar images and the original structured report text into the trained large language model decoder to obtain a preliminary structured report;

[0135] The positioning module 650 is used to input the preliminary structured report into the trained key structure positioning model to obtain a structured report with position coordinates.

[0136] The key structure positioning and structured report output device of the CT image in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0137] The key structure positioning and structured report output device of the CT image in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0138] The key structure positioning and structured report output device of CT images provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0139] Alternatively, if Figure 7 As shown, the embodiment of the present application also provides an electronic device 700, including a first processor 701, a first memory 702, and a program or instruction stored in the first memory 702 and executable on the first processor 701. When the program or instruction is executed by the first processor 701, each process of the embodiment of the key structure positioning and structured report output method of the above-mentioned CT image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0140] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0141] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0142] The hardware structure 800 of the electronic device includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a second memory 809, and a second processor 810 and other components.

[0143] Those skilled in the art will appreciate that the hardware structure 800 of the electronic device may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the second processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0144] It should be understood that in the embodiment of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts, a touch detection device and a touch controller. Other input devices 8072 may include but are not limited to physical keyboards, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, and joysticks, which are not repeated here. The second memory 809 can be used to store software programs and various data, including but not limited to applications and operating systems. The second processor 810 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and applications, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the second processor 810.

[0145] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned CT image key structure positioning and structured report output method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0147] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned CT image key structure positioning and structured report output method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0149] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0151] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for locating key structures of CT images and outputting structured reports, characterized in that: include: Acquire 3D CT images; Inputting the CT image into a trained visual encoder to obtain a high-order visual feature vector; Based on the constructed vector database, similar images of the high-order visual feature vectors are matched to obtain matching similar images and original structured report text; Inputting the high-order visual feature vector, the matched similar image and the original structured report text into the trained large language model decoder to obtain a preliminary structured report; The preliminary structured report is input into the trained key structure positioning model to obtain a structured report with position coordinates.

2. The method for locating key structures of CT images and outputting structured reports according to claim 1, characterized in that: The trained visual encoder is trained by the following method: Construct an autoregressive encoder based on the original CT image; Based on the trained large language model and the structured report of the original CT image, the autoregressive encoder is trained by contrastive learning so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, thereby obtaining a primary visual encoder; The loss function is fused to align the text feature vector and the visual feature vector to obtain the trained visual encoder.

3. The method for locating key structures of CT images and outputting structured reports according to claim 2, characterized in that: The method of constructing an autoregressive encoder based on the original CT image comprises: Obtaining original 3D CT images; Cutting the original 3D CT image into a plurality of 3D CT image blocks; Mapping each of the CT image blocks to a vector space of fixed dimension through a linear projection layer to obtain a corresponding image feature vector, and adding a position code to each of the CT image blocks; After each of the CT image blocks is output by the encoder, a high-dimensional feature vector sequence containing spatial context information is obtained, and the high-dimensional feature vector sequence is sent to the decoder, and the output of the decoder is the original 3D CT image; The encoder is trained using mean square error as a training loss function to obtain an autoregressive encoder.

4. The method for locating key structures of CT images and outputting structured reports according to claim 2, characterized in that: The structured report based on the trained large language model and the original CT image is trained by contrastive learning to enable the autoregressive encoder to initially recognize the characteristics of different types of images and texts, thereby obtaining a primary visual encoder, including: Inputting the structured report of the original CT image into the trained large language model to obtain text vector features; Inputting the original CT image into the autoregressive encoder to obtain a visual feature vector; Contrastive learning is used to bring similar samples closer and dissimilar samples farther apart, so that the autoregressive encoder can initially recognize the characteristics of different types of images and texts, thereby obtaining a primary visual encoder.

5. The method for locating key structures of CT images and outputting structured reports according to claim 2, characterized in that: The fusion loss function is used to align the text feature vector and the visual feature vector to obtain a trained visual encoder, including: Giving different weights to cosine similarity loss, Euclidean distance loss and Manhattan distance loss so that the primary visual encoder learns feature vectors from two modalities; The fusion loss is calculated as follows: Among them, L f is the fusion loss; L c is the cosine similarity loss; L d is the Euclidean distance loss; L m is the Manhattan distance loss.

6. The method for locating key structures of CT images and outputting structured reports according to claim 1, characterized in that: The vector database is constructed by the following method: Obtain CT image data and its structured report; Inputting the CT image data into a trained visual encoder to obtain a visual feature vector, and storing it in an encoded manner; Inputting the structured report into a trained large language model encoder to obtain a text feature vector; The text feature vector and the structured report text content are stored together in a vector database.

7. The method for locating key structures of CT images and outputting structured reports according to claim 1, characterized in that: The trained key structure positioning model is obtained by training using the following method: Obtain CT image training data and its corresponding structured report; Inputting the CT image training data into a frozen visual encoder to obtain a visual feature vector; Inputting the structured report into a large language model encoder to obtain a text feature vector; Set the key of the visual feature vector matrix to image and the key of the textual feature vector matrix to report; Combining the image and the report into a json file and storing them in text form to form a target text; Input the target text into a new large language model and construct LoRA, wherein the output of the new large language model is the 3D spatial position coordinates of the key structures mentioned in the structured report, and training is performed; The new large language model converts the coordinates into the original CT space, and draws a rectangular frame to display it in the CT image training data, thereby obtaining the trained key structure positioning model.

8. A device for locating key structures of CT images and outputting structured reports, characterized in that: include: An acquisition module, used for acquiring 3D CT images; A visual feature extraction module, used for inputting the CT image into a trained visual encoder to obtain a high-order visual feature vector; A matching module, used for matching similar images of the high-order visual feature vector based on the constructed vector database, to obtain matching similar images and original structured report text; A structured report output module, used for inputting the high-order visual feature vector, the matched similar image and the original structured report text into the trained large language model decoder to obtain a preliminary structured report; The positioning module is used to input the preliminary structured report into the trained key structure positioning model to obtain a structured report with position coordinates.

9. An electronic device, characterized in that: include: A processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for locating key structures of CT images and outputting structured reports as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for locating key structures of CT images and outputting structured reports as described in any one of claims 1 to 7 are implemented.

Images