A data analysis method, device and related products of medical CT images

Abstract

The present application provides a data analysis method, device, equipment and storage medium for medical CT images. By analyzing the multimodal information corresponding to the CT image to be analyzed, the data analysis result of the CT image to be analyzed is obtained. Among them, the multimodal information at least includes the full-area image of the target part, the text to be analyzed and the image of the marked area, and the marked area image includes the location information of the marked area in the CT image to be analyzed. In this way, by using the multimodal information, the hidden characteristics of the CT image to be analyzed are excavated from multiple angles such as the overall structure and shape of the target part, the details of the text description, and the morphology, size, and position of the lesion area.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a medical CT image data analysis method, device and related products. Background Art

[0002] As an advanced medical imaging technology, CT examination has been promoted and used by many medical institutions in recent years. At present, the data analysis of CT images mainly relies on the user's naked eye observation of the image surface. For example, the user observes whether there is a large amount of ascites, obvious peritoneal thickening and other image appearances on the CT image, and analyzes the data characteristics of the image. However, this method is difficult to analyze the hidden features in the CT image. Therefore, it is very likely to affect the doctor's diagnosis and treatment decision, and even delay the precious diagnosis and treatment cycle. Summary of the invention

[0003] The present application provides a data analysis method, device and related products for medical CT images, aiming to accurately analyze the hidden features in CT image data.

[0004] In a first aspect, the present application provides a method for analyzing medical CT image data, the method comprising:

[0005] Acquire the CT image to be analyzed of the target part;

[0006] Acquire multimodal information corresponding to the CT image to be analyzed, wherein the multimodal information at least includes a full-region image of a target part corresponding to the CT image to be analyzed, text to be analyzed, and an image of an annotated region, wherein the image of the annotated region includes position information of an annotated region in the CT image to be analyzed;

[0007] Performing feature extraction on the multimodal information to obtain multimodal features corresponding to the multimodal information, the multimodal features at least including a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the labeled area image;

[0008] The multimodal features are fused to obtain fused features, and the data analysis results of the CT image to be analyzed are obtained by analyzing the fused features.

[0009] Optionally, obtaining a marked region image corresponding to the CT image to be analyzed includes:

[0010] Acquire a position vector of the marked region in the CT image to be analyzed, wherein the position vector includes position information of the marked region in the CT image to be analyzed, and a ratio of the marked region to the entire region of the CT image to be analyzed;

[0011] Cut out a cutout region including the marked region from the CT image to be analyzed to obtain a preliminary marked region image;

[0012] The position vector and the image features of the preliminary marked area image are spliced, and the image corresponding to the spliced ​​image features is used as the marked area image.

[0013] Optionally, fusing the multimodal features to obtain fused features includes:

[0014] The multimodal features are spliced ​​according to a preset sequence rule to obtain a multimodal input sequence; the multimodal input sequence is a sequence formed by the spliced ​​multimodal features;

[0015] The multimodal input sequence is input into a pre-trained Bert model to obtain the fused features; the Bert model is used to fuse the spliced ​​multimodal features and output the fused features.

[0016] Optionally, the method further comprises:

[0017] Constructing an image-text matching loss function for the image feature and the text feature in the multimodal feature;

[0018] Based on the image-text matching loss function, the Bert model is trained.

[0019] Exemplarily, the image-text matching loss function is:

[0020] L ITM (θ)=-E (X,H):D [y log S θ (X,H)+(1-y)log(1-S θ (X,H))]

[0021] Wherein, E represents expectation, X represents the text feature, H represents the first image feature, D represents the multimodal input sequence, S θ represents the predicted value, θ is the model parameter, and y represents whether the full-area image matches the text to be analyzed.

[0022] Optionally, the image-text matching loss function is:

[0023] L MLM (θ)=-E (X,H,S):D log P θ (x m |X \m ,H,S)

[0024] Wherein, E represents expectation, X represents the text feature, H represents the first image feature, S represents the second image feature, D represents the multimodal input sequence, and x m represents the masked text vector in the text feature, X \m Indicates that X is divided by x m The elements of , θ are model parameters.

[0025] Optionally, the method further comprises:

[0026] Marking the CT image to be analyzed using an identifier corresponding to the data analysis result, where different data analysis results correspond to different identifiers;

[0027] The marked CT image to be analyzed is output.

[0028] In a second aspect, the present application provides a medical CT image data analysis device, comprising:

[0029] An acquisition unit, used for acquiring a CT image to be analyzed of a target part;

[0030] The acquisition unit is further used to acquire multimodal information corresponding to the CT image to be analyzed, wherein the multimodal information at least includes a full-region image of a target part corresponding to the CT image to be analyzed, a text to be analyzed, and an image of an annotated region, wherein the image of the annotated region includes position information of an annotated region in the CT image to be analyzed;

[0031] A feature extraction unit, configured to extract features from the multimodal information to obtain multimodal features corresponding to the multimodal information, wherein the multimodal features at least include a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the labeled area image;

[0032] A fusion unit, used for fusing the multimodal features to obtain fusion features;

[0033] The analyzing unit is used to obtain the data analysis result of the CT image to be analyzed by analyzing the fusion feature.

[0034] In a third aspect, the present application provides a data analysis device for medical CT images, characterized in that it includes: a memory and a processor;

[0035] a memory having a computer program stored thereon;

[0036] A processor is used to execute the computer program in the memory to implement the steps of the medical CT image data analysis method described in any one of the first aspects.

[0037] In a fourth aspect, an embodiment of the present application provides a computer storage medium, in which a code is stored. When the code is executed, a device executing the code implements any method described in the first aspect.

[0038] Compared with the prior art, this application has the following beneficial effects:

[0039] The technical solution of the present application provides a data analysis method, device and related products for medical CT images. In this solution, the multimodal information corresponding to the CT image to be analyzed is used as the processing object, and the data analysis results of the CT image to be analyzed are obtained by extracting features from the multimodal information, fusing features, and analyzing the fused features. The multimodal information at least includes the full-region image of the target part, the text to be analyzed and the labeled regional image, and the full-region image of the target part includes the complete information of the target part. By analyzing the full-region image of the target part to be analyzed, the hidden features of the CT image to be analyzed can be mined from the overall structure and shape of the target part. The text to be analyzed covers a lot of detailed information. By analyzing the text to be analyzed of the CT image to be analyzed, the hidden features of the CT image to be analyzed can be mined from more microscopic structural features. The labeled regional image includes the morphology, size, position, etc. of the labeled region. By analyzing the labeled regional image, the hidden features of the CT image to be analyzed can be mined from the morphology, size, position, etc. of the lesion area. In general, the technical solution of the present application utilizes the multimodal information of the CT image to be analyzed to perform data analysis on the CT image to be analyzed, and can accurately analyze the hidden characteristics of the CT image to be analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a medical CT image data analysis method provided in an embodiment of the present application;

[0042] Figure 2A It is a schematic diagram of an abdominal CT image and the corresponding image report;

[0043] Figure 2B A schematic diagram of a method for obtaining an image of a marked area is provided for an embodiment of the present application;

[0044] Figure 3 A schematic diagram of obtaining multimodal features provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of an extraction of a first Faster RCNN backbone network extraction model provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a method for analyzing abdominal CT images provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a medical CT image data analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] As an advanced medical imaging technology, CT examination has been promoted and used by many medical institutions in recent years. CT images are multi-layer images obtained by CT examination tools to scan the patient's lesion location and are a common reference for CT examinations. Analyzing the image characteristics of CT images is helpful for doctors' diagnosis and treatment and decision-making.

[0049] At present, data analysis of CT images mainly relies on users observing the image surface with their naked eyes. For example, users observe whether there is a large amount of ascites, obvious peritoneal thickening, and other image phenomena on the CT image, and analyze the data characteristics of the image. However, this method is difficult to analyze hidden features in CT images, such as whether there are tumor features in the abdominal cavity. For example, before gastric cancer surgery, doctors did not find the characteristics of gastric cancer tumors in the abdominal cavity through CT images. Doctors will make wrong diagnosis and treatment decisions for gastric cancer patients, resulting in delays in precious diagnosis and treatment cycles.

[0050] In view of the above problems, the embodiment of the present application provides a data analysis method for medical CT images, which obtains the data analysis results of the CT images to be analyzed by analyzing the multimodal information corresponding to the CT images to be analyzed. Among them, the multimodal information at least includes the full-area image of the target part, the text to be analyzed, and the image of the marked area, and the marked area image includes the location information of the marked area in the CT image to be analyzed. In this way, by using the multimodal information, the hidden characteristics of the CT image to be analyzed are excavated from the aspects of the overall structure and shape of the target part, the details of the text description, and the morphology, size, and position of the lesion area.

[0051] Next, the data analysis method of medical CT images provided in the embodiment of the present application is introduced in combination with the drawings and specific implementation methods.

[0052] See also Figure 1 , is a flow chart of a medical CT image data analysis method provided in an embodiment of the present application. Figure 1 The data analysis method comprises:

[0053] S1: Obtain a CT image of the target area to be analyzed.

[0054] In actual applications, after a CT examination is performed on a user, the obtained CT examination data, such as CT images, image reports, etc., will be saved in a specified image database. The image database is connected to the terminal device. When an identity identifier that uniquely indicates the identity of the user is input into the terminal device, the CT image to be analyzed of the target part can be directly obtained from the image database. For example, user A performs an abdominal CT examination, and the obtained multiple abdominal CT images, as well as the image reports of the abdominal CT images, are saved in a specified image database. When the identity identifier A that uniquely indicates the identity of the user is input into the terminal device, the abdominal CT image can be directly obtained. This abdominal CT image is the CT image to be analyzed of the target part.

[0055] In the embodiment of the present application, the CT images to be analyzed of the target part may have hidden features, which may affect the doctor's diagnosis and treatment. For example, the abdominal CT images may contain hidden tumor information, such as hidden peritoneal metastasis of gastric cancer. If the hidden tumor information is not analyzed before surgery, the doctor may make an incorrect treatment decision.

[0056] It should be noted that the CT image to be analyzed of the target part provided in the embodiment of the present application may be a CT image including all the target parts, or may be a CT image including part of the target parts, and the embodiment of the present application does not make any specific limitation.

[0057] S2: Acquire multimodal information corresponding to the CT image to be analyzed.

[0058] Multimodal information is used to characterize the description information of different angles of the CT image to be analyzed. In practical applications, the description information of different angles of the CT image to be analyzed includes image angle description information and text angle description information. Among them, the full area image of the target part and the labeled area image describe the CT image to be analyzed from the image angle. The text to be analyzed describes the CT image to be analyzed from the text angle.

[0059] It can be understood that the full-area image of the target part is a CT image including the entire area of ​​the target part, that is, a CT image covering the complete structural information of the entire target part. By analyzing the CT image to be analyzed using the full-area image, the hidden features of the CT image to be analyzed can be mined from the overall structure and shape of the target part.

[0060] Exemplarily, the full-region image of the target part can be obtained according to the image area of ​​the CT image to be analyzed. Specifically: if the image area of ​​the CT image to be analyzed includes the full region of the target part, the CT image to be analyzed is used as the full-region image of the target part. If the influence area of ​​the CT image to be analyzed only includes part of the CT image of the target part, at this time, the full-region image of the target part can be called from the specified image database storing the CT image.

[0061] Exemplarily, the full-area image of the target part can be obtained based on the user identifier of the CT image to be analyzed. For example, the user identifier of the CT image to be analyzed can be obtained first. The user identifier can be a sequence identifier composed of the user identity information corresponding to the CT image to be analyzed. The user identity information can be a sequence composed of the consultation number and the clinic number. Exemplary explanation: If the consultation number is 10 and the clinic number is 03928843, the user identifier can be 1003928843. The user identifier can also be other identifiers used to uniquely represent the user identity information, such as a sequence identifier composed of an identity card number and a CT examination time. This application does not specifically limit the form and content of the user identifier.

[0062] Then, according to the user identification and the calling condition, the full-area image of the target part can be searched and called from the image database storing the CT images. It can be understood that the CT images stored in the image database include full-area images of the target part and partial-area images of the target part of different users. By using the preset calling condition and user identification, such as the calling condition being the full-area feature of the target part, the full-area image of the target part can be searched from the image database.

[0063] In the embodiment of the present application, the text to be analyzed is a text description of the CT image to be analyzed. In practical applications, the text to be analyzed may be image report data of the CT image to be analyzed. For example, Figure 2A The above is a schematic diagram of an abdominal CT image and a corresponding image report.

[0064] contrast Figure 2A By analyzing the abdominal CT images and image reports, we can find that: compared with abdominal CT images, image reports describe the CT images to be analyzed from more microscopic structural features, such as "thickening of the gastric antrum wall", "uneven mucosal surface", "narrowing of the antrum, unclear boundary between thickened gastric wall and normal gastric wall, rough gastric wall serosal surface" and other more microscopic structural features that cannot be observed from abdominal CT images. Therefore, by analyzing the text to be analyzed of the CT images to be analyzed, we can dig out the hidden features of the CT images to be analyzed from a more microscopic structural level.

[0065] During a CT scan, the lesion area will be marked, so the lesion area is also called the marked area or the region of interest. The shape, size, and location of the lesion area also imply many features. By analyzing the images of the marked area, we can dig out the hidden features of the CT image to be analyzed from the aspects of the shape, size, and location of the lesion area.

[0066] The labeled region image includes the position information of the labeled region in the CT image to be analyzed. The embodiment of the present application can obtain the labeled region image in a variety of ways.

[0067] In one achievable method, first obtain the position vector of the marked area in the CT image to be analyzed. The position vector includes the position information L of the marked area in the CT image to be analyzed, and the proportion r of the marked area to the entire area of ​​the CT image to be analyzed. Exemplary description: The position vector can be (L, r).

[0068] A cropping region including the marked region is cropped from the CT image to be analyzed to obtain a preliminary marked region image. For example, the cropping region may be a circumscribed quadrilateral, a circumscribed circle, etc. of the marked region.

[0069] The position vector and the image feature S of the preliminary labeled region image are concatenated, and the image corresponding to the concatenated image feature is used as the labeled region image. Exemplary description: The concatenated image feature is A = concat (S, p), where p is the position vector (L, r). The image corresponding to A is the labeled region image.

[0070] In another possible implementation, see Figure 2B , a schematic diagram of a method for obtaining an image of a marked area is provided for an embodiment of the present application. First, a position coordinate system XOY of the CT image to be analyzed is created. The width value W and the height value H of the CT image to be analyzed are obtained, with the width being parallel to the X-axis and the height being parallel to the Y-axis. The minimum horizontal coordinate value x1, the maximum horizontal coordinate value x2, the minimum vertical coordinate value y1, and the maximum vertical coordinate value y1 of the marked area are obtained.

[0071] Then, according to the minimum x1, maximum x2, minimum y1 and maximum y1 of the marked area, the cropping area 202 is obtained. Specifically, draw straight lines x=x1, x=x2, y=y1 and y=y2, and the area formed by the four straight lines is the cropping area 202. Obviously, the cropping area 202 is the circumscribed rectangle of the marked area 201, and the length of the rectangle is x2-x1 and the width is y2-y1.

[0072] In the embodiment of the present application, the ratio of the cropped area to the entire area of ​​the CT image to be analyzed can be used as the ratio of the marked area to the entire area of ​​the CT image to be analyzed.

[0073] Constructing the position vector p for:

[0074]

[0075] The first four elements are the relative positions of the marked areas in the CT image to be analyzed, and the last element is the size ratio of the marked area to the overall area of ​​the image. In this way, the positional relationship and size relationship between the marked area and the overall area can be obtained.

[0076] The CT image to be analyzed is cropped according to the cropping region to obtain a preliminary marked region image 203. The image features S and position vectors of the preliminary region image are spliced ​​to obtain a marked region image.

[0077] It should be noted that, in addition to the above-mentioned text to be analyzed, full-area images of the target area, and images of the labeled areas, the multimodal information may also include description information from other angles. This application does not specifically limit the multimodal information.

[0078] S3: Extract the information of the multimodal information to obtain multimodal features corresponding to the multimodal information.

[0079] Multimodal features refer to information features of multimodal information. In the embodiment of the present application, the multimodal features at least include a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the impression of the marked area.

[0080] In the embodiment of the present application, multimodal features can be obtained in the following manner.

[0081] Based on the first image feature extraction model, image features are extracted from the full-area image of the target part to obtain the first image feature. Based on the text feature extraction model, text features are extracted from the text to be analyzed to obtain text features corresponding to the text to be analyzed. Based on the second image feature extraction model, image features are extracted from the labeled area image to obtain the second image feature.

[0082] It should be noted that the above extraction operations can be performed simultaneously or sequentially, and those skilled in the art can adjust them as needed. The above first image feature extraction model and the second image feature extraction model can be feature extraction models of the same type or feature extraction models of different types, which is not limited in this application.

[0083] Exemplarily, the first image feature extraction model and the second image feature extraction model are both Faster-RCNN models, and the text feature extraction model is a Bert model.

[0084] See also Figure 3, which is a schematic diagram of obtaining multimodal features provided by an embodiment of the present application. The full-area image 301 of the target part, the image report 302 corresponding to the CT image to be analyzed, and the marked area image 303 are input into the feature extraction module to extract multimodal features.

[0085] The feature extraction module includes a Bert model 304 , a first Faster-RCNN model 305 and a second Faster-RCNN model 306 .

[0086] The image report 302 is input into the Bert model 304 of the feature extraction module, that is, the image report is embedded and each character is vectorized. For example, if the input text is X = {lung window: multiple small nodules are seen in both lungs, ...}, after the Bert model calculation, the vector identification of each character can be obtained, that is:

[0087] x1=bert(lung), x2=bert(window), and so on, we get the text feature X={x1,x2,...,x N}, N is the text length, that is, the number of characters.

[0088] The full-area image 301 of the target part is input into the first Faster-RCNN model 305 to obtain the first image feature H = {h1, h2, ..., h m}, m is the number of feature points extracted from the full-area image 301 of the target part.

[0089] See also Figure 4 , is a schematic diagram of the extraction of a first Faster-RCNN model provided in an embodiment of the present application. The first Faster RCNN model includes 13 convolutional layers, 13 relu activation layers and 4 pooling layers, and the acquisition method is as follows:

[0090] The j-th feature map matrix of the l-th layer It can be obtained by weighted convolution of several feature maps in the previous layer:

[0091]

[0092] Among them, f(·) represents the rule activation function, N j Represents the input feature map combination, * represents convolution, for The weight value connected to neuron j, represents the i-th feature map of the previous layer, Represents the bias term.

[0093] Use maximum pooling to process the feature map matrix Extract features:

[0094]

[0095] maxpooling(·) represents the maximum pooling operation. Finally, the first image feature H = {h1, h2, ..., h m}, and a feature map of the first image feature.

[0096] The labeled region image 303 is input into the second Faster-RCNN model 306 to obtain the second image feature S = {s1, s2, ..., s k}, k is the number of feature points extracted from the labeled area image. For example, the second Faster-RCNN model is the same as the first Faster-RCNN model, both of which contain 13 convolutional layers, 13 relu activation layers, and 4 pooling layers. The specific acquisition method is the same as Figure 4 The same, I won't go into details here.

[0097] It should be noted that the above-mentioned first image feature extraction model, second image feature extraction model and text feature extraction model may also be other types of extraction models, and this application does not make specific limitations.

[0098] S4: Fusion of multimodal features to obtain fusion features.

[0099] In the embodiment of the present application, the multimodal features may be fused based on a feature fusion model to obtain fused features. Exemplarily, the feature fusion model may be a Bert model.

[0100] The Bert model is often used to represent characters in natural language processing tasks, and is rarely used in other fields such as images. This application can use the Bert model as a feature fusion model to fuse multimodal features. The specific fusion method is as follows:

[0101] The multimodal features are spliced ​​according to the preset sequence rules to obtain a multimodal input sequence. The multimodal input sequence is a sequence formed by the spliced ​​multimodal features. For example, the multimodal feature is a text feature X = {x1, x2, ..., x N}, the first image feature H = {h1, h2, ..., h m}, and the second image feature S = {s1, s2, ..., s k The default sequence rule is the Bert input sequence format, and the multimodal input sequence obtained after splicing is: {[CLS],x1,...,x n ,[SEP],h1,...,h m ,[SEP],s1,...s k}

[0102] Input the multimodal input sequence into the pre-trained Bert model to obtain fusion features. Specifically, the Bert model can be trained using the image-text matching loss function of the image features and text features in the pre-built multimodal features. The specific training method is:

[0103] Step 1: Input the sample set of multimodal features into the Bert model and calculate the output prediction value.

[0104] The sample set of multimodal features includes a plurality of multimodal feature samples, and each multimodal feature sample includes at least a sample of a first image feature, a sample of a text feature, and a sample of a second image feature.

[0105] Step 2: Use the pre-built image-text matching loss function to calculate the error between the predicted value and the true value.

[0106] Step 3: When the error is greater than the preset error threshold, correct the weight values ​​in the forward Bert model calculation along the direction of the image text matching loss function gradient direction with the minimum error. When the error is not greater than the preset error threshold, the Bert model training is completed.

[0107] Exemplarily, the image-text matching loss function may be:

[0108] L ITM (θ)=-E (X,H):D [y log S θ (X,H)+(1-y)log(1-S θ (X,H))] (Formula 4)

[0109] Where E represents expectation, X represents text feature, H represents the first image feature, D represents the multimodal input sequence, S θ represents the predicted value, θ is the model parameter, and y represents whether the full-area image of the target part matches the text to be analyzed.

[0110] This loss function makes full use of the information consistency between the full-area image of the target part and the text to be analyzed. The matched first image features and text features are used as positive samples, and the randomly shuffled first image features and text features are used as negative samples for training. The image-text matching loss function constructed using this consistency information is used to train the Bert model, and the Bert model results obtained after training are more accurate.

[0111] Exemplarily, the image-text matching loss function may also be:

[0112] L MLM (θ)=-E (X,H,S):D log P θ (x m |X\m ,H,S) (Formula 5)

[0113] Among them, E represents expectation, X represents text features, H represents the first image features, S represents the second image features, D represents the multimodal input sequence, and x m Represents the masked text vector in the text feature, X \m Indicates that X is divided by x m The elements of , θ are model parameters.

[0114] This loss function randomly masks the text elements of the text to be analyzed, and uses the unmasked text elements of the text to be analyzed to predict the masked text elements. For example, the text elements can be randomly masked with a probability of 15%, and the unmasked text elements of the text to be analyzed can be used to predict the masked text elements.

[0115] S5: By analyzing the fusion features, a data analysis result of the CT image to be analyzed is obtained.

[0116] The embodiment of the present application can input the fusion features into a pre-trained data analysis model to obtain a data analysis result. The data analysis result is the hidden features of the CT image to be analyzed, such as hidden tumor features.

[0117] In one achievable method, the data analysis model consists of a fully connected layer and a classifier. The fused features are input into the fully connected layer, and then the data analysis results can be obtained through the classifier.

[0118] Exemplarily, the embodiment of the present application may also mark the identifier corresponding to the data analysis result on the CT image to be analyzed. Different data analysis results have different identifiers. For example: if the data analysis result is that occult peritoneal metastasis occurs, the CT image to be analyzed is marked with a "1" identifier. If the data analysis result is that occult peritoneal metastasis does not occur, the CT image to be analyzed is marked with a "0" identifier.

[0119] The data analysis model used in the embodiment of the present application is a pre-trained data analysis model. The specific training method is: based on the cross loss function, the data analysis model is trained. The cross loss function is:

[0120]

[0121] Among them, cross_entropy(·) represents the cross entropy loss function, Y represents the true value, Represents the predicted value.

[0122] In order to enable those skilled in the art to better understand the embodiments of the present application, the following description is made by taking the abdominal CT image as an example in which the specific CT image to be analyzed is an abdominal CT image.

[0123] See also Figure 5 , is a schematic diagram of a data analysis method for abdominal CT images provided in an embodiment of the present application. The present application example is used to analyze the hidden features existing in abdominal CT images - hidden tumor features, to help doctors diagnose whether hidden peritoneal metastasis of gastric cancer occurs.

[0124] Figure 5 (a) is a portal venous phase abdominal CT image, which includes the entire abdominal area and the marked area. The multimodal information of the abdominal CT image is obtained by using the above, and the multimodal information includes the entire abdominal area image, the image report of the CT image and the marked area image.

[0125] The labeled region image includes a preliminary labeled region image and a position vector. The position vector is specifically shown in formula (1).

[0126] The whole abdominal area image is input into the first Faster-RCNN model to obtain the first image feature H = {h1, h2, ..., h m}, m is the number of feature points in the full area image of the target part.

[0127] The image report of CT image is input into the Bert model to obtain the text feature X = {x1, x2, ..., x N}, N is the length of the text.

[0128] The labeled region image is input into the second Faster-RCNN model to obtain the second image feature S = {s1, s2, ..., s k}, k is the number of feature points extracted from the labeled area image.

[0129] The first image feature, the text feature and the second image feature are input into the Bert model to perform feature fusion to obtain a fused feature. The specific fusion method is shown in step S4 and will not be described here.

[0130] The fusion feature is input into the fully connected layer, the fully connected layer compresses the fusion feature, and determines the preliminary hidden feature of the portal venous phase abdominal CT image based on the compressed feature, and outputs it to the classifier. The classifier analyzes the hidden feature (i.e., the data analysis result) based on the preliminary hidden feature. Exemplarily, the hidden feature includes two types, the presence of hidden tumor features and the absence of hidden tumor features.

[0131] The classifier may be a Sigmoid classifier, and the preliminary occult features are input into the Sigmoid classifier, and the obtained classification result is the data analysis result of the portal venous phase abdominal CT image. The data analysis result is output. For example, 1 indicates the presence of occult tumor features, and 0 indicates the absence of occult tumor features.

[0132] The embodiment of the present application provides a data analysis method for medical CT images, which obtains the data analysis results of the CT images to be analyzed by analyzing the multimodal information corresponding to the CT images to be analyzed. Among them, the multimodal information at least includes the full-area image of the target part, the text to be analyzed, and the image of the marked area, and the marked area image includes the position information of the marked area in the CT image to be analyzed. In this way, the hidden characteristics of the CT image to be analyzed are accurately analyzed by using the diagnosis and treatment information from the image angle, the diagnosis and treatment information from the text angle, the diagnosis and treatment information of the marked area, and the complementary relationship between the multimodal information.

[0133] Based on the medical CT image data analysis method provided in the aforementioned embodiment, the present application also provides a medical CT image data analysis device. Figure 6 This is a schematic diagram of the structure of a medical CT image data analysis device provided in an embodiment of the present application. Figure 6 As shown, the data analysis device for medical CT images includes:

[0134] An acquisition unit 601 is used to acquire a CT image to be analyzed of a target part;

[0135] The acquisition unit 601 is further used to acquire multimodal information corresponding to the CT image to be analyzed, the multimodal information at least including the full-region image of the target part corresponding to the CT image to be analyzed, the text to be analyzed and the marked region image, the marked region image including the position information of the marked region in the CT image to be analyzed;

[0136] A feature extraction unit 602 is used to extract features from the multimodal information to obtain multimodal features corresponding to the multimodal information, wherein the multimodal features at least include a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the labeled area image;

[0137] A fusion unit 603, used for fusing multi-modal features to obtain fusion features;

[0138] The analyzing unit 604 is used to obtain the data analysis result of the CT image to be analyzed by analyzing the fusion features.

[0139] The acquisition unit 601 is further used for:

[0140] Obtaining a position vector of a marked region in the CT image to be analyzed, where the position vector includes position information of the marked region in the CT image to be analyzed, and a ratio of the marked region to the total region of the CT image to be analyzed;

[0141] Cut out a cutout region including the marked region from the CT image to be analyzed, and obtain a preliminary marked region image;

[0142] The position vector and the image features of the preliminary marked area image are spliced, and the image corresponding to the spliced ​​image features is used as the marked area image.

[0143] The fusion unit 603 is specifically used for:

[0144] The multimodal features are spliced ​​according to a preset sequence rule to obtain a multimodal input sequence; the multimodal input sequence is a sequence formed by the spliced ​​multimodal features;

[0145] The multimodal input sequence is input into the pre-trained Bert model to obtain fused features; the Bert model is used to fuse the spliced ​​multimodal features and output the fused features.

[0146] The data analysis device for medical CT images also includes:

[0147] The training unit is used to construct an image-text matching loss function for image features and text features in multimodal features; and to train the Bert model based on the image-text matching loss function.

[0148] Exemplarily, the image-text matching loss function is:

[0149] L ITM (θ)=-E (X,H):D [y log S θ (X,H)+(1-y)log(1-S θ (X,H))]

[0150] Among them, E represents expectation, X represents text features, H represents the first image features, D represents the multimodal input sequence, S θ represents the predicted value, θ is the model parameter, and y represents whether the image of the entire area matches the text to be analyzed.

[0151] Exemplarily, the image-text matching loss function is:

[0152] L MLM (θ)=-E (X,H,S):D log P θ (x m |X \m ,H,S)

[0153] Among them, E represents expectation, X represents text features, H represents the first image features, S represents the second image features, D represents the multimodal input sequence, and x m Represents the masked text vector in the text feature, X \m Indicates that X is divided by x m The elements of , θ are model parameters.

[0154] The data analysis device for medical CT images also includes: a marking unit, which is used to mark the CT image to be analyzed using an identifier corresponding to the data analysis result, and different data analysis results correspond to different identifiers; and output the marked CT image to be analyzed.

[0155] Exemplarily, the feature extraction unit 602 is specifically used for:

[0156] Based on the first image feature extraction model, image features are extracted from the full-area image of the target part to obtain a first image feature;

[0157] Based on the text feature extraction model, text features are extracted from the text to be analyzed to obtain text features corresponding to the text to be analyzed;

[0158] Based on the second image feature extraction model, image features are extracted from the labeled area image to obtain second image features.

[0159] The embodiment of the present application provides a data analysis device for medical CT images, wherein an acquisition unit acquires a CT image to be analyzed of a target part, and acquires multimodal information corresponding to the CT image to be analyzed. A feature extraction unit extracts features from the multimodal information to obtain multimodal features. A fusion unit fuses multimodal features to obtain fused features, and an analysis unit analyzes the fused features to obtain data analysis results. In this way, the hidden features of the CT image to be analyzed are accurately analyzed by utilizing the diagnosis and treatment information from the image angle, the diagnosis and treatment information from the text angle, the diagnosis and treatment information of the marked area, and the complementary relationship between the multimodal information.

[0160] The present application further provides a medical CT image data analysis device, which includes: a memory and a processor.

[0161] A memory having a computer program stored therein.

[0162] The processor is used to execute the computer program in the memory to implement some or all of the steps in the medical CT image data analysis method introduced in the above embodiment.

[0163] The present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements part or all of the steps in the medical CT image data analysis method introduced in the above-mentioned embodiment.

[0164] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and equipment embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for analyzing medical CT image data, characterized in that: The method comprises: Acquire the CT image to be analyzed of the target part; Acquire multimodal information corresponding to the CT image to be analyzed, the multimodal information at least including a full-area image of a target part corresponding to the CT image to be analyzed, a text to be analyzed, and an image of an annotated area, the image of the annotated area including position information of an annotated area in the CT image to be analyzed; wherein the image of the annotated area is an image of a lesion area, and is acquired in the following manner: acquire a position vector of the annotated area in the CT image to be analyzed, the position vector including position information of the annotated area in the CT image to be analyzed, and a proportion of the annotated area to the entire area of ​​the CT image to be analyzed; crop a cropped area including the annotated area from the CT image to be analyzed, and acquire a preliminary annotated area image; splice the image features of the position vector and the preliminary annotated area image, and use the image corresponding to the spliced ​​image features as the annotated area image; Performing feature extraction on the multimodal information to obtain multimodal features corresponding to the multimodal information, the multimodal features at least including a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the labeled area image; The multimodal features are spliced ​​according to a preset sequence rule to obtain a multimodal input sequence; the multimodal input sequence is a sequence formed by the spliced ​​multimodal features; The multimodal input sequence is input into a pre-trained Bert model to obtain fusion features; the Bert model is used to fuse the spliced ​​multimodal features and output the fused features; and the data analysis results of the CT image to be analyzed are obtained by analyzing the fusion features.

2. The method according to claim 1, characterized in that: The method further comprises: Constructing an image-text matching loss function for the image feature and the text feature in the multimodal feature; Based on the image-text matching loss function, the Bert model is trained.

3. The method according to claim 2, characterized in that: The image-text matching loss function is: L ITM (θ)=-E (X,H):D [y log S θ (X,H)+(1-y)log(1-S θ (X,H))] Wherein, E represents expectation, X represents the text feature, H represents the first image feature, D represents the multimodal input sequence, S θ represents the predicted value, θ is the model parameter, and y represents whether the full-area image matches the text to be analyzed.

4. The method according to claim 2, characterized in that: The image-text matching loss function is: L MLM (θ)=-E (X,H,S):D log P θ (x m |X \m ,H,S) Wherein, E represents expectation, X represents the text feature, H represents the first image feature, S represents the second image feature, D represents the multimodal input sequence, and x m represents the masked text vector in the text feature, X \m Indicates that X is divided by x m The elements of , θ are model parameters.

5. The method according to claim 1, characterized in that: The method further comprises: Marking the CT image to be analyzed using an identifier corresponding to the data analysis result, where different data analysis results correspond to different identifiers; The marked CT image to be analyzed is output.

6. A data analysis device for medical CT images, characterized in that: The device comprises: An acquisition unit, used for acquiring a CT image to be analyzed of a target part; The acquisition unit is further used to acquire multimodal information corresponding to the CT image to be analyzed, wherein the multimodal information at least includes a full-area image of a target part corresponding to the CT image to be analyzed, a text to be analyzed, and an image of an annotated area, wherein the image of the annotated area includes position information of an annotated area in the CT image to be analyzed; wherein the image of the annotated area is an image of a lesion area, and is acquired in the following manner: acquiring a position vector of the annotated area in the CT image to be analyzed, wherein the position vector includes position information of the annotated area in the CT image to be analyzed, and a proportion of the annotated area to the entire area of ​​the CT image to be analyzed; cropping a cropped area including the annotated area from the CT image to be analyzed to acquire a preliminary annotated area image; splicing the image features of the position vector and the preliminary annotated area image, and using the image corresponding to the spliced ​​image features as the annotated area image; A feature extraction unit, configured to extract features from the multimodal information to obtain multimodal features corresponding to the multimodal information, wherein the multimodal features at least include a first image feature corresponding to the full-area image of the target part, a text feature corresponding to the text to be analyzed, and a second image feature corresponding to the labeled area image; A fusion unit is used to splice the multimodal features according to a preset sequence rule to obtain a multimodal input sequence; the multimodal input sequence is a sequence formed by the spliced ​​multimodal features; the multimodal input sequence is input into a pre-trained Bert model to obtain a fused feature; the Bert model is used to fuse the spliced ​​multimodal features and output the fused features; The analyzing unit is used to obtain the data analysis result of the CT image to be analyzed by analyzing the fusion feature.

7. A data analysis device for medical CT images, characterized in that: include: Memory and processor; a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the medical CT image data analysis method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of data analysis of the medical CT images described in any one of claims 1-5 are implemented.

Images