Image processing methods, computer-aided diagnostic methods and equipment

Abstract

This application provides an image processing method, a computer-aided diagnostic method, and an apparatus. The image processing method includes: acquiring a medical image containing multiple parts of a target object; extracting features from the medical image to obtain part-level image features corresponding to each part; determining the detection result category information corresponding to each part, and extracting features from each detection result category information to obtain text features corresponding to each detection result category information; and predicting the detection results based on the part-level image features and text features corresponding to each part to obtain predicted detection results for multiple parts of the target object. This application enables multi-disease detection of multiple parts of a target object, improving the scalability of medical image interpretation and assisted diagnosis.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, a computer-aided diagnostic method, a device, a system, a computer storage medium, and a computer program product. Background Technology

[0002] Medical images are important tools used in clinical practice for the detection and diagnosis of diseases. The interpretation of medical images is a complex process that requires professionals with extensive experience and knowledge to complete.

[0003] With the continuous development of artificial intelligence technology, it has shown great potential in helping to improve the efficiency and accuracy of medical image interpretation and diagnosis.

[0004] However, traditional artificial intelligence models are usually single-disease models, that is, a corresponding diagnostic model is developed for a single disease. This traditional approach has a single model function, and therefore has the problem of limited scalability in interpretation and auxiliary diagnosis. Summary of the Invention

[0005] In view of the above, embodiments of this application provide an image processing method, a computer-aided diagnostic method, an apparatus, a system, a computer storage medium, and a computer program product to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of this application, an image processing method is provided, comprising: Acquire medical imaging images, wherein the medical imaging images contain multiple parts of the object to be detected; Feature extraction is performed on the medical images to obtain the site-level image features corresponding to each site. The detection result category information corresponding to each part is determined, and the features of each detection result category information are extracted to obtain the text features corresponding to each detection result category information; Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the predicted detection results of the multiple locations of the detected object.

[0007] According to a second aspect of the embodiments of this application, a computer-aided diagnostic method is provided, comprising: Acquire medical CT images, wherein the medical CT images contain multiple parts of the object to be detected; Feature extraction is performed on the medical CT images to obtain the location-level image features corresponding to each part; Determine the disease category information corresponding to each part, and extract features from each disease category information to obtain the text features corresponding to each disease category information; Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the disease diagnosis results for the multiple locations of the detected object.

[0008] According to a third aspect of the embodiments of this application, a computer-aided diagnostic method is provided, applied to a client device, the method comprising: Acquire medical CT images, wherein the medical CT images contain multiple parts of the object to be detected; The medical CT images are sent to a server device, which then extracts the site-level image features corresponding to each part of the medical CT images. Based on the site-level image features and text features, the detection results are predicted to obtain the disease diagnosis results for the multiple parts of the detected object. The text features are obtained by extracting features from the disease category information corresponding to each part. Receive disease diagnosis results returned by the server device; The disease diagnosis results are displayed on the interface.

[0009] According to a fourth aspect of the embodiments of this application, a computer-aided diagnostic method is provided, applied to a server device, the method comprising: Receive medical CT images sent by a client device, wherein the medical CT images contain multiple parts of the object being examined; Extract the location-level image features corresponding to each part in the medical CT images; Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the disease diagnosis results of the multiple locations of the detected object; the text features are obtained by feature extraction of the disease category information corresponding to each location. The disease diagnosis results are returned to the client device so that they can be displayed on the client device's screen.

[0010] According to a fifth aspect of the present application, a client device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the third aspect.

[0011] According to a sixth aspect of the present application, a server device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the fourth aspect.

[0012] According to a seventh aspect of the embodiments of this application, a computer-aided diagnostic system is provided, comprising: a client device as described in the fifth aspect above, and a server device as described in the sixth aspect above.

[0013] According to an eighth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of the first to fourth aspects.

[0014] The image processing scheme provided in this application pre-sets detection result category information at the part level for multiple parts of the detection object in text form, and obtains the part-level text features corresponding to the detection result category information at each part level. Correspondingly, after acquiring the medical image, the part-level image features of each part contained in the image are extracted. That is, during image feature extraction, the complete image features are broken down into more fine-grained part-level image features. Then, based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the predicted detection results of multiple parts of the detection object are obtained, thereby realizing the detection of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0016] Figure 1 This is a flowchart illustrating the steps of an image processing method according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the decomposition of the text image and the text detection results; Figure 3 A schematic diagram of the image processing flow in a multi-disease auxiliary diagnosis scenario; Figure 4 This is a schematic diagram illustrating the correction of image-text similarity tag information; Figure 5This is a flowchart illustrating the steps of a computer-aided diagnostic method according to an embodiment of this application. Figure 6 This is a schematic diagram of an exemplary computer-aided diagnostic system to which the computer-aided diagnostic method of the present application is applied; Figure 7 This is a flowchart illustrating the steps of another computer-aided diagnostic method according to an embodiment of this application; Figure 8 This is a flowchart illustrating the steps of another computer-aided diagnostic method according to an embodiment of this application; Figure 9 This is a structural block diagram of an image processing apparatus according to an embodiment of this application; Figure 10 This is a structural block diagram of a computer-aided diagnostic device according to an embodiment of this application; Figure 11 This is a structural block diagram of another computer-aided diagnostic device according to an embodiment of this application; Figure 12 This is a structural block diagram of another computer-aided diagnostic device according to an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a client device according to an embodiment of this application; Figure 14 This is a schematic diagram of the structure of a server device according to an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0018] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0019] Reference Figure 1 , Figure 1 This is a flowchart illustrating the steps of an image processing method according to an embodiment of this application. The image processing method provided in this application embodiment can be executed by any suitable device with image processing capabilities. Specifically, the image processing method provided in this embodiment may include the following steps: Step 102: Acquire medical imaging images, which contain multiple parts of the object being detected.

[0020] Specifically, the medical image obtained in this embodiment can be an image formed by detecting a preset area of ​​the object to be detected. There are no limitations on the detection content, or the form and content of the image, which can be customized according to actual conditions. For example, the medical image can be a CT image obtained from CT (Computed Tomography) imaging, an X-ray image obtained from X-ray imaging, an ultrasound image obtained from ultrasound imaging, an MRI image obtained from MRI (Magnetic Resonance Imaging), and so on.

[0021] Furthermore, the aforementioned preset area includes multiple parts of the test object. The parts of the test object included in the preset area can be the body components obtained after dividing the body of the test object according to a preset granularity. This application embodiment does not limit the preset area or the specific granularity of division; it can be customized according to actual conditions. For example, it can be divided according to the granularity of medical anatomy, thereby dividing the body into 104 body components such as organs, bones, muscles, and blood vessels, and then each of these individual body components can be considered as a part. Further, because the granularity of the aforementioned anatomical division is too detailed, clinical diagnostic reports cannot accurately locate the lesion area on the aforementioned fine-grained body components. For example, according to the granularity of medical anatomy, the human lung is usually divided into 5 different lobes (left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe), but diagnostic reports usually do not specifically locate which lobe has a lesion, but rather describe it broadly as "lung inflammation." Based on the above, using a fine-grained partitioning method might lead to a data imbalance during subsequent model training sample collection, resulting in an excessive amount of normal sample data and an insufficient amount of abnormal sample data. Therefore, after obtaining the various body components at the medical anatomical level, these components can be combined according to diagnostic conventions to determine the location within this application. For example, the left upper lobe, left lower lobe, right upper lobe, right lower lobe, right middle lobe, and right lower lobe can be combined to form the lung; the myocardium, left atrium, left ventricle, right atrium, and right ventricle can be combined to form the heart, and so on.

[0022] In summary, for example, taking a human as the detection subject, the preset area can be the chest region of a human, and correspondingly, multiple parts in the preset area can include: heart, lungs, esophagus, trachea, etc.; the preset area can also be the abdominal region of a human, and correspondingly, multiple parts in the preset area can include: stomach, small intestine, large intestine, liver, gallbladder, pancreas, spleen, kidney, etc.; the preset area can also be the chest and abdomen region of a human, and correspondingly, multiple parts in the preset area can include: heart, lungs, esophagus, trachea, stomach, small intestine, large intestine, liver, gallbladder, pancreas, spleen, kidney, etc.

[0023] Step 104: Extract features from the medical images to obtain the location-level image features corresponding to each part.

[0024] Specifically, the site-level image features corresponding to each site refer to deep image features that can reflect the image details of each site. Since medical images contain multiple sites, the number of site-level image features obtained through this step is also multiple, with one site corresponding to one site-level image feature.

[0025] During feature extraction, the medical image can be segmented into parts to obtain sub-images corresponding to each part, and then feature extraction can be performed on each sub-image to obtain the part-level image features corresponding to each part; alternatively, the entire medical image can be encoded to obtain the overall image features corresponding to the entire image, and then the part-level image features can be separated from the overall image features based on the segmentation mask.

[0026] Step 106: Determine the detection result category information corresponding to each part, and extract features from each detection result category information to obtain the text features corresponding to each detection result category information.

[0027] Specifically, the detection result category information corresponding to each part can be a text description of the detection result category corresponding to each part. This detection result category can be a pre-defined part-level detection result that may occur in each part. Further, taking a disease diagnosis scenario as an example, the detection result category information can be concretized as image report text, that is, a text description of the diagnostic result type corresponding to each part. This diagnostic result type can be a pre-defined part-level diagnostic result that may occur in each part. For example, for the liver, the corresponding diagnostic result types can include: healthy, cirrhosis, liver cancer, etc. As mentioned above, in this embodiment, the detection result category information can be individual words. Further, considering that the training samples in the model training phase are usually sentences or text paragraphs, in order to improve the accuracy of the prediction results, the detection result category information can also be sentences or text paragraphs composed of sentences. For example, part-level diagnostic results in the form of corresponding sentences or text paragraphs can be generated by professionals, or part-level diagnostic results in the form of single words can be modified using prompt templates to form part-level diagnostic results in the form of sentences or text paragraphs.

[0028] After obtaining the diagnostic result type information corresponding to each part, text feature extraction can be used to obtain the text features corresponding to each diagnostic result type information. Similar to image feature extraction, the text features obtained in this step are also part-level text features.

[0029] Step 108: Based on the corresponding part-level image features and text features, the detection results are predicted to obtain the predicted detection results of multiple parts of the detected object.

[0030] Specifically, the similarity between each part-level image feature and each text feature can be calculated separately, and then the predicted detection result for each part can be obtained based on the similarity calculation result. For example, for a certain part-level image feature, the detection result category corresponding to the text feature with a similarity greater than a preset similarity threshold can be determined as the detection result for the part corresponding to that part-level image feature. For example, assuming that for the liver part, the text feature with a high similarity to the part-level image feature corresponding to the liver is the text feature corresponding to the detection result category information "health"; and for the stomach part, the text feature with a high similarity to the part-level image feature corresponding to the stomach is the text feature corresponding to the detection result category information "inflammation", then the following predicted detection results can be obtained: the predicted detection result for the liver is "health", and the predicted detection result for the stomach is "gastritis".

[0031] The method provided in this application embodiment, after acquiring the medical image to be detected, eliminates the need for professional personnel to issue corresponding detection reports. Based on the site-level image features extracted from the medical image and the site-level text features extracted from pre-defined detection result category information describing possible disease types in each site, detection results for multiple sites contained in the medical image to be detected can be obtained. In other words, for the user, the method provided in this application embodiment only requires inputting the medical image to be detected, without the need for manual diagnosis of the detection report or the use of targeted manual diagnostic reports as input data, thus achieving automated diagnosis of medical images.

[0032] Furthermore, the image processing method provided in this application embodiment pre-embeds textual information to pre-define location-level detection result categories for multiple parts of the detection object, and obtains location-level text features corresponding to each location-level detection result category. Correspondingly, after acquiring the medical image, location-level image features of each part contained in the image are extracted. That is, during image feature extraction, the complete image features are broken down into finer-grained location-level image features. Based on the location-level image features corresponding to each part and the aforementioned location-level text features, the predicted detection results for multiple parts of the detection object are obtained, thereby realizing multi-disease detection for multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis.

[0033] Optionally, in some embodiments, the process of extracting features from medical images to obtain site-level image features corresponding to each site may specifically include: The medical image is subjected to site detection to obtain the segmentation mask of each site; the medical image is divided into blocks to obtain multiple image blocks; based on the segmentation mask of each site, the image block corresponding to each site is determined from the multiple image blocks; and feature extraction is performed on the image block corresponding to each site to obtain the site-level image features corresponding to each site.

[0034] Specifically, the process of obtaining segmentation masks for each part involves two parts: part detection and image segmentation. Part detection refers to identifying and locating each part in the medical image and determining its bounding box. Image segmentation refers to dividing the medical image into multiple sub-images, aiming to separate the parts within the image. The segmentation result is usually represented by a segmentation mask (image). The segmentation mask has the same size as the medical image, and the value of each pixel in the segmentation mask is typically used to indicate whether the pixel belongs to a specific part. In this embodiment, a separate segmentation mask can be obtained for each part. In this case, the segmentation mask is a binary mask, with pixel values ​​typically being 0 or 1, where 0 represents the background and 1 represents the foreground (the area where the part is located). Alternatively, the same segmentation mask can be generated for all parts. In this case, a multi-class segmentation mask is obtained, where the pixel values ​​of different parts are different, while the pixel values ​​of the same part are the same.

[0035] In the above embodiments of this application, on the one hand, segmentation masks corresponding to each part of the medical image are obtained; on the other hand, the medical image is divided into blocks to obtain multiple image blocks. Then, these multiple image blocks can be grouped according to the segmentation masks of each part, thereby obtaining image blocks corresponding to each part, so as to extract the part-level image features corresponding to each part from the image blocks corresponding to each part. Segmentation masks typically provide pixel-level precise information, thus enabling more accurate differentiation and segmentation of different parts in medical images. Therefore, the process of obtaining part-level image features corresponding to each part using segmentation masks can improve the accuracy of the final obtained part-level image features, thereby helping to improve the accuracy of prediction and detection results.

[0036] Optionally, in some embodiments, feature extraction is performed on the medical image to obtain site-level image features corresponding to each site, which may include: The medical image is input into the pre-trained prediction model, and the image encoding module in the prediction model encodes the medical image to obtain the site-level image features corresponding to each site. Correspondingly, feature extraction is performed on the information of each detection result category to obtain the text features corresponding to each detection result category information, which may include: The detection result category information is input into the pre-trained prediction model. The text encoding module in the prediction model encodes the detection result category information to obtain the text features corresponding to each detection result category information. Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the predicted detection results for multiple locations of the detected object, which may include: The prediction module in the prediction model predicts the detection results based on the corresponding part-level image features and text features, and outputs the predicted detection results for multiple parts of the detected object.

[0037] Specifically, in this embodiment, a prediction model containing multiple different modules can be used to perform corresponding processing operations on the acquired medical image and detection result category information to obtain the final predicted detection result. After acquiring the medical image and detection result category information, the image and text are input into the trained prediction model, and the model can output the predicted detection result. The operation process is relatively simple. Furthermore, since the prediction model is pre-trained and has corresponding prior knowledge, the output predicted detection result also has high accuracy.

[0038] Optionally, in some embodiments, the training process of the above-described prediction model may specifically include: Obtain training samples, which contain multiple text-image pairs; each text-image pair includes: a sample image, and the text detection result corresponding to the sample image; The text image is input into the prediction model, and the prediction model obtains the part-level sample image features corresponding to each part in the sample image, and the part-level text features corresponding to each part in the text detection result. Based on the location-level sample image features and location-level text features of each location, image-text similarity prediction information for each location is generated. Based on the image-text similarity prediction information of each part, and the pre-determined image-text similarity label information of each part, the loss value corresponding to each part is calculated. The loss values ​​corresponding to each part are fused to obtain the fused loss value; The model parameters are adjusted based on the fusion loss value to obtain the trained prediction model.

[0039] Specifically, during the training of a prediction model, the training samples in the same training batch (Batch) typically contain multiple text-image pairs. Each text-image pair can include a sample image and its corresponding document detection result. The sample image and document detection result are usually obtained for the same detection object. The text detection result corresponding to the sample image can be a detection result represented in text form using any suitable correlation technique. This detection result typically involves detection results for various body parts. For example, the text detection result could be a diagnostic report given by a professional (such as a doctor) based on the sample image.

[0040] The specific implementation of obtaining the part-level sample image features corresponding to each part in the sample image through the prediction model can be found in step 104 above, and will not be repeated here.

[0041] When obtaining the part-level text features corresponding to each part in the text detection results through the prediction model, the neural network sub-model in the text encoding module of the prediction model can be used to achieve this: the above neural network sub-model can be trained in advance to obtain the trained sub-model, and then after obtaining the text detection results, the part-level text features of each part can be obtained through the trained sub-model.

[0042] When using neural network sub-models to obtain text features at the part level, there are two main implementation methods: The first approach involves inputting the complete text detection results into a sub-model, which then outputs the location-level text features for each part. This method obtains location-level text features without requiring additional preprocessing of the text detection results, making the process simple and easy to implement.

[0043] The second approach involves decomposing the text detection results into multiple sub-text detection results before inputting them into the neural network sub-model. Each sub-text detection result corresponds to a specific part. Each sub-text detection result is then input into the neural network sub-model, where it is encoded to obtain part-level text features corresponding to each part. In this approach, instead of inputting the complete text detection result into the neural network sub-model, the entire result is preprocessed to break it down into sub-text detection results corresponding to different parts. Each sub-text detection result is then encoded to obtain part-level text features. Because the text detection results are pre-segmented at the part level, the text input to the neural network sub-model in a single step is highly correlated and corresponds to the same part, thus improving the accuracy of the final part-level text features.

[0044] Furthermore, text detection results typically include two parts: "image description" and "detection hints." The "image description" describes the visual state of each part in the medical image used as a sample image, while the "detection hints" infer the detection results for each part based on the information described in the "image description." Therefore, in the second approach described above, the process of decomposing the text detection results into multiple sub-text detection results can be as follows: Entity recognition is performed on both the "image description" and "detection hint" parts of the text detection results to identify the part information contained in both the "image description" and "detection hint" parts; then, the descriptive information corresponding to each part is extracted from both the "image description" and "detection hint" parts; finally, the descriptive information corresponding to the same part in both parts is integrated to obtain the sub-text detection results corresponding to each part in the text detection results.

[0045] Regarding the entity recognition process described above, as the granularity of the detected object can be flexibly set, the following situations may occur: the "image description" or "detection prompt" part of the text detection result does not contain a certain part, but uses more detailed descriptive information than a certain part. For example, for the part "small intestine", the "image description" or "detection prompt" part does not contain the entity "small intestine", but contains more granular body components such as "jejunum", "ileum" or "duodenum". In order to improve the recognition accuracy, the fine-grained depth components contained in each part can be pre-set, and then when the fine-grained depth components are identified, the corresponding part information is determined to be identified.

[0046] Additionally, it's possible that a certain area appears in one of the "Image Description" and "Detection Tips" sections but not in the other. In this case, to improve accuracy, special markers (such as empty characters) can be used to supplement the missing area information in the section where the area is not listed. Furthermore, it's also possible that a certain area is not listed in either the "Image Description" or "Detection Tips" sections. Based on clinical experience, in this situation, the detection result for the missing area can be assumed to be "healthy" or "no significant abnormalities detected."

[0047] See Figure 2 , Figure 2 This is a schematic diagram illustrating the decomposition of the text image and the text detection results. Figure 2 The left side of the image represents a schematic diagram of the segmentation results obtained after segmenting the medical CT image of the human chest and abdomen region. Different pixel values ​​are used in the segmentation result diagram to represent different parts of the human body. Figure 2 The right-hand portion represents the decomposition process of the diagnostic report corresponding to the medical CT image (corresponding to the text detection result in the embodiments of this application), from... Figure 2 As can be seen on the right side, by decomposing the diagnostic report, the diagnostic results for the four human body parts—liver, kidney, pancreas, and stomach—were finally obtained (corresponding to the subtext detection results in the embodiments of this application).

[0048] Furthermore, the image-text similarity prediction information generated based on the location-level sample image features and location-level text features of each location can characterize the degree of similarity between the location-level sample image features and location-level text features corresponding to the same location. Since the training samples in the same training batch contain multiple text-image pairs, multiple similarity prediction information can usually be obtained for a single part in the same training batch. Further, for ease of recording, the above image-text similarity prediction information can be presented in the form of a similarity prediction matrix.

[0049] The process of generating image-text similarity prediction information for each part based on part-level sample image features and part-level text features can be represented by the following formula: = , =

[0050] Where j represents the location index number. This represents the total number of image samples containing the complete j-th region; This represents the cosine similarity between vectors a and b. This represents a learnable temperature parameter; Indicates that for the first The first part, the first Image features at the region level to the first Predicted image-text similarity values ​​for text features at the part level; Indicates that for the first The first part, the first The first part-level text features to the first Image-text similarity prediction values ​​for image features at the part level; Indicates the first The nth sample image Location-level image features corresponding to each location; Indicates the first In the text detection results, the first one Each part corresponds to a part-level text feature.

[0051] Correspondingly, the final fusion loss value obtained by fusion. It can be expressed by the following formula: = ( ) in, H represents the total number of anatomical-level sites; H represents the cross-entropy loss. Indicates the first Image features at the region level to the first Image-text similarity label values ​​of text features at the part level; Indicates the first Image features at the region level to the first Predicted image-text similarity values ​​for text features at the part level; Indicates the first The first part-level text features to the first Image-text similarity label values ​​for image features at the part level; Indicates the first The first part-level text features to the first Image-text similarity prediction values ​​for image features at the part level.

[0052] In this embodiment, both the sample images and the corresponding text detection results undergo finer-grained part-level segmentation. This enables feature matching and alignment of the sample images and text detection results at the part-level during model training. Compared to traditional feature matching and alignment methods at the global coarse-grained level, the segmentation and fine-grained alignment scheme provided in this embodiment effectively improves alignment accuracy and enhances the interpretability of the prediction model's output. Furthermore, the fine-grained part-level segmentation of the sample images and corresponding text detection results effectively increases the number of training samples, thereby improving the prediction accuracy of the trained prediction model.

[0053] See Figure 3 , Figure 3 This is a schematic diagram of the image processing workflow in a multi-disease auxiliary diagnosis scenario. The following is combined with... Figure 3 The image processing process of this application embodiment in the scenario of multi-disease auxiliary diagnosis is explained: Acquiring medical CT images And, the corresponding text detection results On the one hand, regarding images The image is segmented into blocks to obtain the segmented processing results, which are then input into the image encoder in the image encoding module of the prediction model. Figure 3Taking ViT (Visual Transformer) as an example (this does not constitute a limitation on image encoders), it encodes each image block to obtain the encoding result corresponding to each image block; for the image... Part detection is performed to obtain segmentation masks for each part. Then, based on the segmentation masks, the encoding results for each component are determined from the encoding results corresponding to each image patch. An initial feature vector representing the overall image features of the part is added to the encoding results corresponding to each part. A self-attention mechanism is introduced within each part to obtain the part-level image features corresponding to each part. ,in, This represents the region-level image feature corresponding to the j-th region in the i-th sample image. On the other hand, the text detection results... The text is decomposed to obtain the sub-text detection results corresponding to each part. ; Detection results of each subtext (Represents the sub-text detection result corresponding to the j-th part in the i-th text detection result) Input the text encoder in the text encoding module of the prediction model ( Figure 3 Taking the BERT model as an example (this does not constitute a limitation on the text encoder), the region-level text features of each part are obtained. (This represents the part-level text feature corresponding to the j-th part in the i-th text detection result).

[0054] Then, based on the feature data of the sample images of each part, and location-level text features Generate image-text similarity prediction matrices for each part: (The image-text similarity prediction matrix corresponding to the j-th part); then calculate the loss value based on the image-text similarity prediction matrix of each part, and adjust the prediction model parameters until the trained prediction model is obtained.

[0055] Optionally, in some embodiments, the process of determining the image-text similarity label information for each part may include: For a single part: Obtain the initial image-text similarity label information for the single part. The initial image-text similarity label information is determined based on whether the part-level sample image features and part-level text features correspond to the same detection object. Obtain the similarity label correction information for the single part. If the part-level sample image features and part-level text features of a single part correspond to different detection objects, and the detection results of the single parts of different detection objects are all normal, then the part-level sample image features and part-level text features of the single part are determined to be similar. Fuse the initial image-text similarity label information and the similarity label correction information to obtain the image-text similarity label information for the single part.

[0056] Specifically, when determining the initial image-text similarity label information, for location-level sample image features and location-level text features corresponding to the same location, if the two features correspond to the same detection object, or in other words, the two features originate from the same text-image pair, then the two features can be determined to be similar, for example, the similarity between the two features can be determined to be 1. Conversely, if they are not similar, then the two features are determined to be dissimilar, for example, the similarity between the two features can be determined to be 0.

[0057] When determining the initial image-text similarity labels, location-level sample image features and location-level text features from the same text image pair are identified as positive samples (identified as similar), while location-level sample image features and location-level text features from different text image pairs are identified as negative samples (identified as dissimilar). However, in the medical field, for location-level sample image features and location-level text features from different text image pairs, there is often a certain semantic relationship between these two features. For example, for a certain location, two test subjects that are both diagnosed as normal usually have similar location-level sample image features and location-level text features.

[0058] Based on the above, if we take the initial image-text similarity label information as the final label information from the perspective of whether the two features originate from the same text-image pair, there will be a problem of false negative samples. That is, for two detection objects that are both diagnosed as normal, false negative samples are constructed between their part-level sample image features and part-level text features. These false negative samples should have been identified as similar, but they are mistakenly identified as dissimilar. As a result, during the pre-training process of the prediction model, the training prediction model incorrectly identifies this type of sample as dissimilar, thereby affecting the prediction accuracy of the trained prediction model.

[0059] In the embodiments described above, in addition to obtaining the initial image-text similarity label information, similarity label correction information for individual parts is also acquired. Specifically, when determining the similarity label correction information, if the part-level sample image features and part-level text features correspond to different detection objects, and the detection results for individual parts of different detection objects are all normal, then the part-level sample image features and part-level text features are determined to be similar. Subsequently, the initial image-text similarity label information is corrected and fused using the aforementioned similarity label correction information, thereby correcting the aforementioned false negative samples to positive samples. This avoids the problem of low prediction accuracy caused by the existence of false negative samples and improves the prediction accuracy of the prediction model.

[0060] Optionally, in some embodiments, before fusing the initial image-text similarity tag information and the similarity tag correction information, the above method further includes: Obtain similarity label reference information for individual parts generated by the auxiliary model for the training samples; the auxiliary model is different from the prediction model; The initial image-text similarity tag information and similarity tag correction information are integrated, including: By integrating the initial image-text similarity tag information, the similarity tag correction information, and the similarity tag reference information, the image-text similarity tag information for a single part is obtained.

[0061] Specifically, in the medical field, for two test subjects diagnosed with the same disease at a certain site, their site-level sample image features and site-level text features are usually similar. Given this, if the site-level sample image features and site-level text features of two test subjects diagnosed with the same disease are determined to be dissimilar, the aforementioned problem of false negative samples will also occur.

[0062] To address the aforementioned issues, this application introduces an auxiliary model. This auxiliary model generates similarity prediction results for the training samples and uses these results as reference information for similarity values. The initial image-text similarity label information is further corrected and fused, thereby training the prediction model as effectively as possible. This enables the prediction model to determine the similarity of the location-level image features and location-level text features of two detection objects diagnosed with the same disease, further improving the prediction accuracy of the prediction model.

[0063] Furthermore, in this embodiment, a collaborative teaching training framework can be adopted, setting up two prediction models to be trained: a first prediction model and a second prediction model. During the model training phase, the two models are trained alternately. That is, for the same training sample, the first prediction model and the second prediction model are used respectively to make predictions, obtaining prediction results. The prediction results obtained by the first prediction model (or the second prediction model) are used as the aforementioned similarity label reference information to correct and fuse the initial image-text similarity label information of the second prediction model (or the first prediction model), thereby assisting the training process of the second prediction model (or the first prediction model). The first prediction model and the second prediction model can be models with different structures, or they can be models with the same structure. When the first prediction model and the second prediction model are models with the same structure, the initialization parameters, training sample iteration order, etc., of the two models can be different.

[0064] The above image-text similarity tag information can be calculated using the following formula: ,

[0065] ,

[0066] in, As a preset parameter, it can be set to 0.5 for example; and These respectively represent the similarity label information from image features to text features obtained through the first prediction model, and the similarity label information from text features to image features obtained through the first prediction model. and These respectively represent the similarity label information from image features to text features obtained through the second prediction model, and the similarity label information from text features to image features obtained through the second prediction model. and These respectively represent the similarity prediction information from image features to text features obtained through the first prediction model, and the similarity prediction information from text features to image features obtained through the first prediction model. and These respectively represent the image feature to text feature similarity prediction information obtained through the second prediction model, and the text feature to image feature to similarity prediction information obtained through the second prediction model.

[0067] See Figure 4 , Figure 4 This is a schematic diagram illustrating the correction of image-text similarity tag information. The following is combined with... Figure 4 The process of correcting the image-text similarity tag information provided in the above embodiments of this application is briefly described as follows: like Figure 4 As shown, assuming that the training samples in the same training batch contain 4 text-image pairs, where the first text-image pair includes an image Subtext detection results targeting the pancreas The second text-image pair includes an image. Subtext detection results targeting the pancreas The third text-image pair includes an image. Subtext detection results targeting the pancreas The fourth text-image pair includes an image. Subtext detection results targeting the pancreas The image-text similarity prediction matrix for the pancreas region obtained through the prediction model is A, where, ... These are the location-level sample image features obtained from the corresponding text image pairs. ... The image-text similarity label matrix is ​​B, and the similarity label correction matrix is ​​C. Additionally, the similarity label reference matrix generated by the auxiliary model for the training samples is D. The final image-text similarity label matrix E corresponding to the pancreas region is equal to the fusion result of matrices B, C, and D. Further, after fusing the image-text similarity label matrix E, the loss value L corresponding to the pancreas region can be calculated based on the image-text similarity prediction matrix A and the image-text similarity label matrix E.

[0068] Furthermore, the aforementioned similarity label reference matrix D can be a matrix obtained by the first prediction model and the second prediction model through collaborative teaching.

[0069] Reference Figure 5 , Figure 5 This is a flowchart illustrating the steps of a computer-aided diagnostic method according to an embodiment of this application. The computer-aided diagnostic method provided in this application can be executed by any suitable device with image processing capabilities. Specifically, the computer-aided diagnostic method provided in this embodiment may include the following steps: Step 502: Acquire medical CT images, which contain multiple parts of the object being examined.

[0070] Specifically, the medical CT images obtained in this application embodiment can be images formed by performing CT scans on a preset area of ​​the object to be detected. The preset area includes multiple parts of the object to be detected. The parts of the object to be detected included in the preset area can be the body composition structure obtained after dividing the body of the object to be detected according to a preset granularity. This application embodiment does not limit the preset area or the specific granularity of division; it can be customized according to actual conditions.

[0071] Step 504: Extract features from the medical CT images to obtain the location-level image features corresponding to each part.

[0072] Specifically, the location-level image features corresponding to each location refer to the deep image features that can reflect the image details of each location. Since medical CT images contain multiple locations, the number of location-level image features obtained through this step is also multiple, with one location corresponding to one location-level image feature.

[0073] Step 506: Determine the disease category information corresponding to each part, and extract features from each disease category information to obtain the text features corresponding to each disease category information.

[0074] Specifically, the disease category information corresponding to each part indicates the disease category corresponding to each part in text form. The disease category can be a pre-defined site-level disease that may occur in each part. For example, for the liver, the corresponding disease categories may include: healthy, cirrhosis, liver cancer, etc.

[0075] After obtaining the disease category information corresponding to each body part, text feature extraction can be used to obtain the text features corresponding to each disease category. Similar to image feature extraction, the text features obtained in this step are also body part-level text features.

[0076] Step 508: Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the disease diagnosis results for multiple locations of the detected object.

[0077] The computer-aided diagnosis method provided in this application pre-defines disease category information at the site level for multiple parts of the test object in text form, and obtains the site-level text features corresponding to the disease category information at each site level. Correspondingly, after acquiring the medical CT image, the site-level image features of each part contained in the image are extracted. That is, during image feature extraction, the complete image features are broken down into finer-grained site-level image features. Then, based on the site-level image features corresponding to each part and the aforementioned site-level text features, the disease diagnosis results of multiple parts of the test object are obtained, thereby realizing the diagnosis of multiple diseases in multiple parts of the test object and improving the scalability of medical image interpretation and assisted diagnosis.

[0078] See Figure 6 , Figure 6 This is a schematic diagram of an exemplary computer-aided diagnostic system to which the computer-aided diagnostic method of the present application is applicable. For ease of understanding, it is first described in conjunction with... Figure 6 The application scenarios of the computer-aided diagnostic method provided in the embodiments of this application will be explained.

[0079] like Figure 6 As shown, the system 600 may include a server 602, a communication network 604, and / or one or more user devices (i.e., client devices) 606. Figure 6 The example in the text shows multiple user devices.

[0080] The server 602 can be any suitable device for performing multi-disease auxiliary diagnosis based on medical CT images containing multiple parts of the subject being examined, including but not limited to server clusters, cloud computing server clusters, etc. In some embodiments, the server 602 can perform any suitable function. For example, in some embodiments, the server 602 can be used to receive medical CT images sent by the user equipment 606, perform image processing based on the received medical CT images to obtain multi-disease diagnostic results for multiple parts of the subject being examined, and return the multi-disease diagnostic results to the user equipment 606.

[0081] In some embodiments, communication network 604 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 604 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 606 may be connected to communication network 604 via one or more communication links (e.g., communication link 612), and communication network 604 may be linked to server 602 via one or more communication links (e.g., communication link 614). Communication links may be any communication link suitable for transmitting data between user equipment 606 and server 602, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0082] User equipment 606 may include any one or more user devices. In some embodiments, user equipment 606 may include any suitable type of device. For example, in some embodiments, user equipment 606 may include a computer and / or any other suitable type of user device.

[0083] Reference Figure 7 , Figure 7 This is a flowchart illustrating the steps of another computer-aided diagnostic method according to an embodiment of this application. The computer-aided diagnostic method provided in this application can be... Figure 6 User equipment 606 in the system shown performs the operation. Specifically, the computer-aided diagnostic method provided in this embodiment includes the following steps: Step 702: Acquire medical CT images, which contain multiple parts of the object being examined.

[0084] Specifically, the medical CT images obtained in this application embodiment can be images formed by performing CT scans on a preset area of ​​the object to be detected. The preset area includes multiple parts of the object to be detected. The parts of the object to be detected included in the preset area can be the body composition structure obtained after dividing the body of the object to be detected according to a preset granularity. This application embodiment does not limit the preset area or the specific granularity of division; it can be customized according to actual conditions.

[0085] Step 704: The medical CT image is sent to the server device so that the server device can extract the site-level image features corresponding to each part in the medical CT image; based on the site-level image features and text features corresponding to each part, the detection result is predicted to obtain the disease diagnosis results of multiple parts of the detected object; each text feature is obtained by extracting the disease category information corresponding to each part.

[0086] Specifically, after acquiring medical CT images, the server device can extract the location-level image features of each part contained in the image. That is, during image feature extraction, the complete image features are broken down into more fine-grained location-level image features. Then, based on the location-level image features corresponding to each part and the above-mentioned location-level text features, the disease diagnosis results of multiple parts of the detected object are obtained.

[0087] Step 706: Receive the disease diagnosis results returned by the server device.

[0088] Step 708: Display the disease diagnosis results on the display interface.

[0089] The computer-aided diagnosis method provided in this application pre-defines disease category information at the site level for multiple parts of the test object in text form, and obtains the site-level text features corresponding to the disease category information at each site level through a server-side device that provides computer-aided diagnosis services. Correspondingly, after the client device obtains the medical CT image, it can send the image to the server-side device so that the server-side device can extract the site-level image features of each part contained in the image. That is, during image feature extraction, the complete image features are broken down into more fine-grained site-level image features. Then, based on the site-level image features corresponding to each part and the aforementioned site-level text features, the disease diagnosis results for multiple parts of the test object are obtained, thereby realizing the diagnosis of multiple diseases for multiple parts of the test object and improving the scalability of medical image interpretation and assisted diagnosis.

[0090] Reference Figure 8 , Figure 8 This is a flowchart illustrating the steps of another computer-aided diagnostic method according to an embodiment of this application. The computer-aided diagnostic method provided in this application can be... Figure 6 The server 602 in the system shown executes the procedure. Specifically, the computer-aided diagnostic method provided in this embodiment includes the following steps: Step 802: Receive medical CT images sent by the client device. The medical CT images contain multiple parts of the object being examined.

[0091] Specifically, the medical CT images obtained in this application embodiment can be images formed by performing CT scans on a preset area of ​​the object to be detected. The preset area includes multiple parts of the object to be detected. The parts of the object to be detected included in the preset area can be the body composition structure obtained after dividing the body of the object to be detected according to a preset granularity. This application embodiment does not limit the preset area or the specific granularity of division; it can be customized according to actual conditions.

[0092] Step 804: Extract the location-level image features corresponding to each part in the medical CT image.

[0093] Specifically, the location-level image features corresponding to each location refer to the deep image features that can reflect the image details of each location. Since medical CT images contain multiple locations, the number of location-level image features obtained through this step is also multiple, with one location corresponding to one location-level image feature.

[0094] Step 806: Based on the location-level image features and text features corresponding to each location, the detection results are predicted to obtain the disease diagnosis results of multiple locations of the detected object; each text feature is obtained by extracting features from the disease category information corresponding to each location.

[0095] Step 808: Return the disease diagnosis results to the client device so that the disease diagnosis results can be displayed through the client device's display interface.

[0096] The computer-aided diagnosis method provided in this application pre-defines disease category information at the site level for multiple parts of the test object in text form, and obtains the site-level text features corresponding to the disease category information at each site level through a server-side device that provides computer-aided diagnosis services. Correspondingly, after the client device obtains the medical CT image, it can send the image to the server-side device so that the server-side device can extract the site-level image features of each part contained in the image. That is, during image feature extraction, the complete image features are broken down into more fine-grained site-level image features. Then, based on the site-level image features corresponding to each part and the aforementioned site-level text features, the disease diagnosis results for multiple parts of the test object are obtained, thereby realizing the diagnosis of multiple diseases for multiple parts of the test object and improving the scalability of medical image interpretation and assisted diagnosis.

[0097] Figure 9 This is a structural block diagram of an image generation apparatus according to an embodiment of this application. The image generation apparatus provided in this application includes: Image acquisition module 902 is used to acquire medical image images, which contain multiple parts of the object to be detected. The image feature extraction module 904 is used to extract features from medical images and obtain the site-level image features corresponding to each site. The text feature extraction module 906 is used to determine the detection result category information corresponding to each part, and to extract features from each detection result category information to obtain the text features corresponding to each detection result category information. The resulting module 908 is used to predict the detection results based on the corresponding part-level image features and text features of each part, and to obtain the predicted detection results of multiple parts of the detected object.

[0098] Optionally, in some embodiments, the image feature extraction module 904 is specifically used for: Perform site detection on medical images to obtain segmentation masks for each site; Medical images are segmented into blocks to obtain multiple image blocks; Based on the segmentation mask of each part, the image block corresponding to each part is determined from multiple image blocks; and feature extraction is performed on the image block corresponding to each part to obtain the part-level image features corresponding to each part.

[0099] Optionally, in some embodiments, the result-obtaining module 908 is specifically used for: For a single part, the similarity between the part-level image features and each text feature corresponding to the single part is calculated, and the prediction and detection results of the single part are obtained based on the similarity calculation results.

[0100] Optionally, in some embodiments, the image feature extraction module 904 is specifically used for: The medical image is input into the pre-trained prediction model, and the image encoding module in the prediction model encodes the medical image to obtain the site-level image features corresponding to each site. The text feature extraction module 906, when performing the step of extracting features from each detection result category information to obtain the text features corresponding to each detection result category information, is specifically used to: input each detection result category information into the pre-trained prediction model, and encode each detection result category information into text through the text encoding module in the prediction model to obtain the text features corresponding to each detection result category information; The resulting module 908 is specifically used to: predict detection results based on the location-level image features and text features corresponding to each location through the prediction module in the prediction model, and output the predicted detection results of multiple locations of the detected object.

[0101] Optionally, in some embodiments, the image processing apparatus further includes: The model training module is used to acquire training samples, which contain multiple text-image pairs. Each text-image pair includes a sample image and the corresponding text detection result. The text-image pairs are input into the prediction model, which obtains the part-level sample image features corresponding to each part in the sample image and the part-level text features corresponding to each part in the text detection result. Based on the part-level sample image features and part-level text features, image-text similarity prediction information for each part is generated. Based on the image-text similarity prediction information for each part and the pre-determined image-text similarity label information for each part, the loss value corresponding to each part is calculated. The loss values ​​corresponding to each part are fused to obtain a fused loss value. The model parameters of the prediction model are adjusted based on the fused loss value to obtain the trained prediction model.

[0102] Optionally, in some embodiments, the model training module is further configured to decompose the text detection results into multiple sub-text detection results before executing the text image-to-input prediction model; wherein, one sub-text detection result corresponds to one part. Correspondingly, when the model training module executes the steps of inputting text images into the prediction model, obtaining part-level sample image features corresponding to each part in the sample image through the prediction model, and part-level text features corresponding to each part in the text detection result, it is specifically used for: The sample image and multiple sub-text detection results are input into the prediction model. The image encoding module in the prediction model encodes the sample image to obtain the part-level sample image features corresponding to each part in the sample image. The text encoding module in the prediction model encodes each sub-text detection result to obtain the part-level text features corresponding to each part.

[0103] Optionally, in some embodiments, the model training module is further used for: For a single part: The initial image-text similarity label information for a single part is obtained. The initial image-text similarity label information is determined based on whether the part-level sample image features and part-level text features correspond to the same detection object. Obtain similarity label correction information for a single part; where, if the part-level sample image features and part-level text features of a single part correspond to different detection objects, and the detection results of a single part of different detection objects are all normal, then the part-level sample image features and part-level text features of a single part are determined to be similar. By integrating the initial image-text similarity label information and the similarity label correction information, image-text similarity label information for a single part is obtained.

[0104] Optionally, in some embodiments, before performing the fusion of initial image-text similarity label information and similarity label correction information, the model training module is further configured to: obtain similarity label reference information for a single part generated by the auxiliary model for the training samples; the auxiliary model is different from the prediction model; Correspondingly, when the model training module performs the step of fusing initial image-text similarity label information and similarity label correction information, it is specifically used to: fuse initial image-text similarity label information, similarity label correction information, and similarity label reference information to obtain image-text similarity label information for a single part.

[0105] The image processing apparatus of this application embodiment is used to implement the methods in the foregoing image processing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each unit in the image processing apparatus of this application embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0106] Figure 10 This is a structural block diagram of a computer-aided diagnostic device according to an embodiment of this application. The computer-aided diagnostic device provided in this application includes: The CT image acquisition module 1002 is used to acquire medical CT images, which contain multiple parts of the object being detected. The CT image feature extraction module 1004 is used to extract features from medical CT images and obtain the corresponding site-level image features for each site. The disease category feature extraction module 1006 is used to determine the disease category information corresponding to each part, and to extract features from each disease category information to obtain the text features corresponding to each disease category information. The diagnostic result acquisition module 1008 is used to predict the detection results based on the corresponding site-level image features and text features of each site, and to obtain the disease diagnosis results of multiple sites of the detected object.

[0107] The computer-aided diagnostic apparatus of this application embodiment is used to implement the computer-aided diagnostic method in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each unit in the computer-aided diagnostic apparatus of this application embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0108] Figure 11 This is a structural block diagram of another computer-aided diagnostic device according to an embodiment of this application. The computer-aided diagnostic device provided in this application includes: The acquisition module 1102 is used to acquire medical CT images, which contain multiple parts of the object to be detected. The sending module 1104 is used to send medical CT images to the server device, so that the server device can extract the site-level image features corresponding to each part in the medical CT images; based on the site-level image features corresponding to each part and each text feature, the detection result is predicted to obtain the disease diagnosis results of multiple parts of the detected object; each text feature is obtained by feature extraction of the disease category information corresponding to each part; The receiving module 1106 is used to receive the disease diagnosis results returned by the server device; Display module 1108 is used to display disease diagnosis results in the display interface.

[0109] The computer-aided diagnostic apparatus of this application embodiment is used to implement the computer-aided diagnostic method in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each unit in the computer-aided diagnostic apparatus of this application embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0110] Figure 12 This is a structural block diagram of another computer-aided diagnostic device according to an embodiment of this application. The computer-aided diagnostic device provided in this application includes: The image receiving module 1202 is used to receive medical CT images sent by the client device, which contain multiple parts of the object being examined. Extraction module 1204 is used to extract location-level image features corresponding to each part in medical CT images; The diagnostic module 1206 is used to predict the detection results based on the corresponding site-level image features and text features, and to obtain the disease diagnosis results of multiple sites of the detected object; each text feature is obtained by feature extraction of the disease category information corresponding to each site. The result return module 1208 is used to return the disease diagnosis results to the client device so that the disease diagnosis results can be displayed through the display interface of the client device.

[0111] The computer-aided diagnostic apparatus of this application embodiment is used to implement the computer-aided diagnostic method in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each unit in the computer-aided diagnostic apparatus of this application embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0112] Reference Figure 13 This document illustrates a schematic diagram of a client device according to an embodiment of this application. The specific implementation of the client device is not limited by the specific embodiments of this application.

[0113] like Figure 13 As shown, the electronic device may include: a processor 1302, a communications interface 1304, a memory 1306, and a communications bus 1308.

[0114] in: The processor 1302, communication interface 1304, and memory 1306 communicate with each other via communication bus 1308.

[0115] Communication interface 1304 is used to communicate with other electronic devices or servers.

[0116] The processor 1302 is used to execute program 1310, specifically the relevant steps in the above method embodiments.

[0117] Specifically, program 1310 may include program code that includes computer operation instructions.

[0118] The processor 1302 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0119] Memory 1306 is used to store program 1410. Memory 1306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0120] Specifically, program 1310 can be used to cause processor 1402 to execute the relevant steps in the above method embodiments.

[0121] The specific implementation of each step in program 1310 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0122] Reference Figure 14 This document illustrates a structural schematic diagram of a server-side device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the server-side device.

[0123] like Figure 14 As shown, the electronic device may include: a processor 1402, a communications interface 1404, a memory 1406, and a communications bus 1408.

[0124] in: The processor 1402, communication interface 1404, and memory 1406 communicate with each other via communication bus 1408.

[0125] Communication interface 1404 is used to communicate with other electronic devices or servers.

[0126] The processor 1402 is used to execute program 1410, specifically to perform the relevant steps in the above method embodiments.

[0127] Specifically, program 1410 may include program code that includes computer operation instructions.

[0128] The processor 1402 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0129] Memory 1406 is used to store program 1410. Memory 1406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0130] Specifically, program 1410 can be used to cause processor 1402 to execute the relevant steps in the above method embodiments.

[0131] The specific implementation of each step in program 1410 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0132] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0133] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0134] Experimental data and results To verify the effectiveness of the method provided in this application, an evaluation was conducted using a dataset. Specifically: First, a large CT dataset was used for performance evaluation. See Table 1 below, which shows a comparison of prediction performance using different methods on the dataset. AUC (Area Under Curve), ACC (Accuracy), Spec (Specificity), and Sens (Sensitivity) were used as performance metrics for model performance evaluation.

[0135] Table 1

[0136] As shown in Table 1, compared with existing solutions, the methods provided in this application all achieve higher performance index values. That is, compared with other methods, the solutions provided in this application have better detection results.

[0137] See Table 2 below, which compares the performance of different methods in downstream report generation tasks. Table 2 uses AUC, Spec, Sens, and natural language generation metrics such as BLEU-4 as performance evaluation indicators when evaluating model performance.

[0138] Table 2

[0139] As shown in Table 2, compared with existing solutions, the method provided in this application embodiment achieves higher performance index values ​​regardless of whether the encoder is fine-tuned or frozen during pre-training. That is, compared with other methods, the solution provided in this application embodiment has better detection performance. Among these, CLIP (Contrastive Language-Image Pre-training) in existing solutions is a multimodal pre-trained neural network model; MAE (Masked Autoencoders) refers to Masked Autoencoders, a self-supervised learning method for computer vision.

[0140] Secondly, two additional datasets were used for performance evaluation. See Table 3 below, which shows a comparison of prediction performance after using different methods to predict the above datasets. AUC, ACC, F1 score, and accuracy were used as performance evaluation metrics.

[0141] Table 3

[0142] As shown in Table 3, compared with existing solutions, the method provided in this application embodiment achieves higher performance metrics for both dataset 1 and dataset 2. In other words, the solution provided in this application embodiment has better detection performance compared to other methods.

[0143] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0144] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0145] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0146] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0147] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An image processing method, comprising: Acquire medical imaging images, wherein the medical imaging images contain multiple parts of the object to be detected; The medical images are used to extract features through a prediction model, and the corresponding site-level image features are obtained for each site. The detection result category information corresponding to each part is determined, and the feature of each detection result category information is extracted through the prediction model to obtain the text features corresponding to each detection result category information; The prediction model is used to predict the detection results of the multiple parts of the object based on the corresponding part-level image features and text features. The step of extracting features from the medical image to obtain location-level image features corresponding to each location includes: performing location detection on the medical image to obtain segmentation masks for each location; performing block processing on the medical image to obtain multiple image blocks; determining the image block corresponding to each location from the multiple image blocks based on the segmentation masks for each location; and extracting features from the image blocks corresponding to each location to obtain location-level image features corresponding to each location. The prediction model is obtained by adjusting model parameters based on the fusion loss value. The fusion loss value is calculated based on the image-text similarity prediction information of each part and the pre-determined image-text similarity label information of each part. The loss values ​​corresponding to each part are then fused. The image-text similarity label information of each part is obtained based on the initial image-text similarity label information of a single part and the similarity label correction information of a single part. The similarity label correction information is used to indicate that if the part-level sample image features and part-level text features of a single part correspond to different detection objects, and the detection results of the single part of the different detection objects are all normal, then the part-level sample image features and part-level text features of the single part are determined to be similar.

2. The method according to claim 1, wherein, The method of predicting detection results based on the location-level image features and text features corresponding to each location to obtain the predicted detection results for the multiple locations of the detected object includes: For a single part, the similarity between the part-level image features and each text feature corresponding to the single part is calculated, and the prediction detection result of the single part is obtained based on the similarity calculation result.

3. The method according to claim 1, wherein, The process involves using a prediction model to extract features from the medical image, obtaining location-level image features for each part, including: The medical image is input into a pre-trained prediction model, and the image encoding module in the prediction model encodes the medical image to obtain the site-level image features corresponding to each site. The step of extracting features from each detection result category information using the prediction model to obtain the text features corresponding to each detection result category information includes: The detection result category information is input into the pre-trained prediction model. The text encoding module in the prediction model is used to encode the text of each detection result category information to obtain the text features corresponding to each detection result category information. The method involves using the prediction model to predict the detection results based on the corresponding part-level image features and text features of each part, thereby obtaining the predicted detection results for the multiple parts of the detected object, including: The prediction module in the prediction model predicts the detection results based on the corresponding part-level image features and text features, and outputs the predicted detection results of the multiple parts of the detected object.

4. The method according to claim 3, wherein, The training process of the prediction model includes: Obtain training samples, which contain multiple text-image pairs; each text-image pair includes a sample image and a text detection result corresponding to the sample image. The text image pair is input into the prediction model, and the prediction model obtains the part-level sample image features corresponding to each part in the sample image, and the part-level text features corresponding to each part in the text detection result; Based on the location-level sample image features and location-level text features of each location, image-text similarity prediction information for each location is generated. Based on the image-text similarity prediction information of each part and the pre-determined image-text similarity label information of each part, the loss value corresponding to each part is calculated. The loss values ​​corresponding to each part are fused to obtain the fused loss value; The model parameters of the prediction model are adjusted based on the fusion loss value to obtain the trained prediction model.

5. The method according to claim 4, wherein, Before inputting the text-image pair into the prediction model, the method further includes: The text detection results are decomposed to obtain multiple sub-text detection results; each sub-text detection result corresponds to a part. The step of inputting the text image pair into the prediction model, obtaining part-level sample image features corresponding to each part in the sample image through the prediction model, and part-level text features corresponding to each part in the text detection result, includes: The sample image and the detection results of the multiple sub-texts are input into the prediction model. The image encoding module in the prediction model performs image encoding on the sample image to obtain the part-level sample image features corresponding to each part in the sample image. The text encoding module in the prediction model performs text encoding on each sub-text detection result to obtain the part-level text features corresponding to each part.

6. The method according to claim 4 or 5, wherein, The process of determining the image-text similarity label information for each part includes: For a single part: The initial image-text similarity label information of the single part is obtained, wherein the initial image-text similarity label information is determined based on whether the part-level sample image features and part-level text features correspond to the same detection object; Obtain the similarity label correction information for the individual part; By integrating the initial image-text similarity tag information and the similarity tag correction information, the image-text similarity tag information for the individual part is obtained.

7. The method according to claim 6, wherein, Before fusing the initial image-text similarity tag information and the similarity tag correction information, the method further includes: Obtain similarity label reference information for the individual parts generated by the auxiliary model for the training samples; the auxiliary model is different from the prediction model; The process of fusing the initial image-text similarity tag information and the similarity tag correction information includes: By integrating the initial image-text similarity tag information, the similarity tag correction information, and the similarity tag reference information, the image-text similarity tag information for the individual part is obtained.

8. A computer-aided diagnostic method, comprising: Acquire medical CT images, wherein the medical CT images contain multiple parts of the object to be detected; The medical CT images are used to extract features from the predictive model to obtain the location-level image features corresponding to each part. The disease category information corresponding to each part is determined, and the prediction model is used to extract features from each disease category information to obtain the text features corresponding to each disease category information. The prediction model is used to predict the detection results based on the location-level image features and text features corresponding to each location, thereby obtaining the auxiliary disease diagnosis results for the multiple locations of the detected object. The step of extracting features from the medical CT image to obtain location-level image features corresponding to each location includes: performing location detection on the medical CT image to obtain segmentation masks for each location; performing block processing on the medical CT image to obtain multiple image blocks; determining the image block corresponding to each location from the multiple image blocks based on the segmentation masks for each location; and extracting features from the image blocks corresponding to each location to obtain location-level image features corresponding to each location. The prediction model is obtained by adjusting model parameters based on the fusion loss value. The fusion loss value is calculated based on the image-text similarity prediction information of each part and the pre-determined image-text similarity label information of each part. The loss values ​​corresponding to each part are then fused. The image-text similarity label information of each part is obtained based on the initial image-text similarity label information of a single part and the similarity label correction information of a single part. The similarity label correction information is used to indicate that if the part-level sample image features and part-level text features of a single part correspond to different detection objects, and the detection results of the single part of the different detection objects are all normal, then the part-level sample image features and part-level text features of the single part are determined to be similar.

9. A computer-aided diagnostic method applied to a client device, the method comprising: Acquire medical CT images, wherein the medical CT images contain multiple parts of the object to be detected; The medical CT images are sent to a server device, which then extracts site-level image features corresponding to each part of the medical CT images using a prediction model. The prediction model then predicts detection results based on the site-level image features and text features corresponding to each part, obtaining auxiliary disease diagnosis results for the multiple parts of the detected object. The text features are obtained by extracting features from the disease category information corresponding to each part. The server device extracts site-level image features corresponding to each part of the medical CT images in the following manner: performing site detection on the medical CT images to obtain segmentation masks for each part; dividing the medical CT images into blocks to obtain multiple image blocks; determining the image block corresponding to each part from the multiple image blocks based on the segmentation masks; and processing the image blocks corresponding to each part separately. Feature extraction is performed on the corresponding image blocks to obtain the corresponding part-level image features. The prediction model is obtained by adjusting model parameters based on the fusion loss value. The fusion loss value is calculated based on the image-text similarity prediction information of each part and the pre-determined image-text similarity label information of each part. The loss values ​​corresponding to each part are then fused. The image-text similarity label information of each part is obtained based on the initial image-text similarity label information and the similarity label correction information of a single part. The similarity label correction information indicates that if the part-level sample image features and part-level text features of a single part correspond to different detection objects, and the detection results of the single part of the different detection objects are all normal, then the part-level sample image features and part-level text features of the single part are determined to be similar. Receive auxiliary disease diagnosis results returned by the server device; The auxiliary disease diagnosis results are displayed in the display interface.

10. A computer-aided diagnostic method applied to a server-side device, the method comprising: Receive medical CT images sent by a client device, wherein the medical CT images contain multiple parts of the object being examined; The prediction model extracts location-level image features corresponding to each location in the medical CT image, including: performing location detection on the medical CT image to obtain the segmentation mask of each location; performing block processing on the medical CT image to obtain multiple image blocks; determining the image block corresponding to each location from the multiple image blocks based on the segmentation mask of each location; and extracting features from the image blocks corresponding to each location to obtain location-level image features corresponding to each location. The prediction model predicts detection results based on the corresponding site-level image features and text features of each site, thereby obtaining auxiliary disease diagnosis results for the multiple sites of the detected object. The text features are obtained by extracting features from the disease category information corresponding to each site. The prediction model is obtained by adjusting model parameters based on a fusion loss value. This fusion loss value is calculated based on the image-text similarity prediction information of each site and the pre-determined image-text similarity label information of each site. The loss values ​​of each site are then fused. The image-text similarity label information of each site is obtained based on the initial image-text similarity label information and the similarity label correction information of a single site. The similarity label correction information indicates that if the site-level sample image features and site-level text features of a single site correspond to different detected objects, and the detection results of the single sites of the different detected objects are all normal, then the site-level sample image features and site-level text features of the single site are determined to be similar. The auxiliary disease diagnosis results are returned to the client device so that they can be displayed on the client device's display interface.

11. A client device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in claim 9.

12. A server-side device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in claim 10.

13. A computer-aided diagnostic system, comprising: The client device as described in claim 11 and the server device as described in claim 12.

14. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-10.

15. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-10.

Images