Image processing method, computer-aided diagnosis method and equipment

Abstract

The embodiment of the invention provides an image processing method, a computer-aided diagnosis method and equipment. The image processing method comprises the following steps: acquiring a medical image, wherein the medical image comprises a plurality of parts of a detection object; feature extraction is carried out on the medical image, and part-level image features corresponding to all parts are obtained respectively; determining detection result category information corresponding to each part, and performing feature extraction on each piece of detection result category information to obtain text features corresponding to each piece of detection result category information; and performing detection result prediction based on the part-level image features corresponding to the parts and the text features to obtain prediction detection results of the multiple parts of the detection object. According to the embodiment of the invention, multi-disease detection of multiple parts of the detected object is realized, and the expansibility of medical image interpretation and auxiliary diagnosis is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image processing method, a computer-aided diagnosis method, a device, a system, a computer storage medium, and a computer program product. Background Art

[0002] Medical imaging images are important tools for detecting and diagnosing diseases in clinical practice. The interpretation of medical imaging images is a complex process that requires professionals with rich experience and knowledge to complete.

[0003] With the continuous development of artificial intelligence technology, artificial intelligence has shown great potential in helping to improve the efficiency and accuracy of medical image interpretation and diagnosis.

[0004] However, traditional artificial intelligence models are usually single-disease models, that is, a corresponding diagnostic model is developed for a single disease. With this traditional approach, the model has a single function and therefore has the problem of poor scalability in interpretation and auxiliary diagnosis. Summary of the invention

[0005] In view of this, embodiments of the present application provide an image processing method, a computer-aided diagnosis method, a device, a system, a computer storage medium, and a computer program product to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, there is provided an image processing method, comprising: Acquire a medical image, wherein the medical image contains multiple parts of a detection object; Extracting features from the medical image to obtain part-level image features corresponding to each part; Determine the detection result category information corresponding to each part, and perform feature extraction on each detection result category information to obtain text features corresponding to each detection result category information; The detection result prediction is performed based on the part-level image features and the text features corresponding to each part, so as to obtain the predicted detection results of the multiple parts of the detection object.

[0007] According to a second aspect of an embodiment of the present application, a computer-aided diagnosis method is provided, comprising: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; Extracting features from the medical CT image to obtain part-level image features corresponding to each part; Determine the disease category information corresponding to each part, and perform feature extraction on each disease category information to obtain text features corresponding to each disease category information; The detection result prediction is performed based on the part-level image features and the text features corresponding to each part, so as to obtain the disease diagnosis results of the multiple parts of the detection object.

[0008] According to a third aspect of an embodiment of the present application, a computer-aided diagnosis method is provided, which is applied to a client device, and the method includes: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; The medical CT image is sent to a server device so that the server device extracts part-level image features corresponding to each part in the medical CT image; based on the part-level image features corresponding to each part and each text feature, a detection result prediction is performed to obtain disease diagnosis results of the multiple parts of the detection object; the each text feature is obtained by extracting features of disease category information corresponding to each part; Receive the disease diagnosis results returned by the server device; The disease diagnosis result is displayed in the display interface.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-aided diagnosis method is provided, which is applied to a server device, and the method includes: Receiving a medical CT image sent by a client device, wherein the medical CT image includes multiple parts of a detection object; Extracting part-level image features corresponding to each part in the medical CT image; The detection result is predicted based on the part-level image features and text features corresponding to each part, so as to obtain the disease diagnosis results of the multiple parts of the detection object; the text features are obtained by extracting the disease category information corresponding to each part; The disease diagnosis result is returned to the client device so as to be displayed through the display interface of the client device.

[0010] According to the fifth aspect of an embodiment of the present application, a client device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the third aspect.

[0011] According to the sixth aspect of an embodiment of the present application, a server-side device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the fourth aspect.

[0012] According to a seventh aspect of an embodiment of the present application, a computer-aided diagnosis system is provided, comprising: a client device as described in the fifth aspect above, and a server device as described in the sixth aspect above.

[0013] According to an eighth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method as described in any one of the first to fourth aspects is implemented.

[0014] The image processing solution provided in the embodiment of the present application pre-adopts a text form, presets part-level detection result category information for multiple parts of the detection object, and obtains part-level text features corresponding to the detection result category information at each part level; correspondingly, after the medical image is acquired, the part-level image features of each part contained in the image are extracted respectively, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the predicted detection results of multiple parts of the detection object are obtained, thereby realizing the detection of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 is a flowchart of a method for image processing according to an embodiment of the present application; Figure 2 Schematic diagram of the decomposition of text images and text detection results; Figure 3 This is a schematic diagram of the image processing process in the scenario of multi-disease auxiliary diagnosis; Figure 4 A schematic diagram for correcting image-text similarity label information; Figure 5is a flowchart of the steps of a computer-aided diagnosis method according to an embodiment of the present application; Figure 6 A schematic diagram of an exemplary computer-aided diagnosis system applicable to the computer-aided diagnosis method of an embodiment of the present application; Figure 7 is a flowchart of another computer-aided diagnosis method according to an embodiment of the present application; Figure 8 is a flowchart of steps of another computer-aided diagnosis method according to an embodiment of the present application; Fig. 9 is a structural block diagram of an image processing device according to an embodiment of the present application; Fig.10 is a structural block diagram of a computer-aided diagnosis device according to an embodiment of the present application; Fig.11 is a structural block diagram of another computer-aided diagnosis device according to an embodiment of the present application; Fig.12 is a structural block diagram of another computer-aided diagnosis device according to an embodiment of the present application; Fig.13 is a schematic diagram of the structure of a client device according to an embodiment of the present application; Fig.14 It is a structural diagram of a server device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0018] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0019] Reference Figure 1 , Figure 1 1 is a flowchart of the steps of an image processing method according to an embodiment of the present application. The image processing method provided in the embodiment of the present application can be performed by any appropriate device with image processing capabilities. Specifically, the image processing method provided in this embodiment can include the following steps: Step 102: Acquire a medical image, wherein the medical image contains multiple parts of the detected object.

[0020] Specifically, the medical image obtained in the embodiment of the present application may be an image formed by detecting a preset area of ​​the detection object. There is no limitation on the detection content, the form and content of the image, and the settings may be customized according to the actual situation. For example, the medical image may be a CT image obtained by CT (Computed Tomography), an X-ray image obtained by X-ray imaging, an ultrasound image obtained by ultrasound imaging, an MRI image obtained by MRI (Magnetic Resonance Imaging), and the like.

[0021] In addition, the above-mentioned preset area includes multiple parts of the detection object. The parts of the detection object included in the preset area can be the body composition structure obtained after dividing the body of the detection object according to the preset division granularity. The embodiment of the present application does not limit the preset area and the specific division granularity, and can be customized according to the actual situation. For example: the body can be divided according to the division granularity of the medical anatomical level, so that the body is divided into 104 parts of the body composition structure such as organs, bones, muscles, and blood vessels, and then the above-mentioned single body composition structure can be regarded as a part. Further, because the above-mentioned anatomical level division granularity is too detailed, the clinical diagnosis report cannot accurately locate the lesion area on the above-mentioned fine-grained body composition structure. For example: according to the division granularity of the medical anatomical level, the human lungs are usually divided into 5 different lobes (left upper lobe, left lower lobe, right upper lobe, right middle lobe and right lower lobe), but the diagnosis report usually does not specifically locate which lobe has the lesion, but rather describes it more broadly as "lung inflammation". Based on the above situation, if a fine-grained division method is used, it may lead to an imbalance in the amount of sample data, with too much normal sample data and too little abnormal sample data, during the subsequent model training sample collection process. Therefore, after obtaining the various body components at the medical anatomical level, the various body components can also be combined according to diagnostic habits, etc., so that each combination result can be determined as the part in this application. For example: the left upper lobe, left lower lobe, right upper lobe, right lower lobe, right middle lobe and right lower lobe can be combined into the lung part; the myocardium, left atrium, left ventricle, right atrium, right ventricle, etc. can be combined into the heart part, and so on.

[0022] In summary, illustratively, taking the detection object as a person, the preset area can be the chest area of ​​the person, and accordingly, the multiple parts in the preset area can include: heart, lungs, esophagus, trachea, etc.; the preset area can also be the abdominal area of ​​the person, and accordingly, the multiple parts in the preset area can include: stomach, small intestine, large intestine, liver, gallbladder, pancreas, spleen, kidney, etc.; the preset area can also be the chest and abdominal area of ​​the person, and accordingly, the multiple parts in the preset area can include: heart, lungs, esophagus, trachea, stomach, small intestine, large intestine, liver, gallbladder, pancreas, spleen, kidney, etc.

[0023] Step 104 , extracting features from the medical image to obtain part-level image features corresponding to each part.

[0024] Specifically, the part-level image features corresponding to each part refer to deep image features that can reflect the image detail information of each part. Since the medical image contains multiple parts, the number of part-level image features obtained by this step is also multiple, where one part corresponds to one part-level image feature.

[0025] When extracting features, the medical image can be first segmented into parts to obtain sub-images corresponding to each part, and then features can be extracted from each sub-image to obtain part-level image features corresponding to each part; or the entire medical image can be encoded first to obtain overall image features corresponding to the entire image, and then the part-level image features can be separated from the above-mentioned overall image features according to the segmentation mask.

[0026] Step 106, determining the detection result category information corresponding to each part, and performing feature extraction on each detection result category information to obtain text features corresponding to each detection result category information.

[0027] Specifically, the detection result category information corresponding to each part can be the detection result category corresponding to each part described in text form, wherein the above detection result category can be the part-level detection result that may appear in the part pre-set for each part. Further, taking the disease diagnosis scenario as an example, the above detection result category information can be concretized as an imaging report text, that is: the diagnosis result type corresponding to each part described in text form. The above diagnosis result type can be the part-level diagnosis result that may appear in the part pre-set for each part. For example: for the liver part, the corresponding diagnosis result type may include: health, cirrhosis, liver cancer, etc. As described above, in the embodiment of the present application, the detection result category information can be a word. Further, considering that the training samples in the model training stage are usually sentences or text paragraphs, therefore, in order to improve the accuracy of the prediction results, the detection result category information can also be a sentence or a text paragraph composed of sentences, etc., for example: the corresponding part-level diagnosis result in the form of a sentence or text paragraph can be formed by a professional, or the part-level diagnosis result in the form of a single word can be transformed by a prompt template, etc., so as to form a part-level diagnosis result in the form of a sentence or text paragraph.

[0028] After obtaining the diagnosis result type information corresponding to each part, the text feature extraction method can be used to obtain the text features corresponding to each diagnosis result type information. Similar to the image feature extraction, the text features obtained in this step are also part-level text features.

[0029] Step 108 , predicting the detection results based on the part-level image features and the text features corresponding to each part, and obtaining predicted detection results of multiple parts of the detection object.

[0030] Specifically, the similarity between each part-level image feature and each text feature can be calculated respectively, and then the predicted detection result of each part can be obtained according to the similarity calculation result. For example: for a certain part-level image feature, the detection result category corresponding to the text feature whose similarity with the part-level image feature is greater than the preset similarity threshold can be determined as the detection result of the part corresponding to the part-level image feature. Exemplarily, it is assumed that for the liver part, the text feature with a higher similarity to the part-level image feature corresponding to the liver is the text feature corresponding to the detection result category information of "health"; for the stomach part, the text feature with a higher similarity to the part-level image feature corresponding to the stomach is the text feature corresponding to the detection result category information of "inflammation". Then the following predicted detection results can be obtained: the predicted detection result of the liver is "healthy", and the predicted detection result of the stomach is "gastritis".

[0031] Through the above method provided by the embodiment of the present application, after obtaining the medical image to be detected, there is no need for professionals to issue a corresponding test report for the above medical image to be detected. The embodiment of the present application is based on the part-level image features extracted from the above medical image, and the part-level text features extracted from the pre-set test result category information describing the possible disease types of each part, so as to obtain the test results for multiple parts contained in the above medical image to be detected. In other words, for the user, during the implementation of the method provided by the embodiment of the present application, it is sufficient to input the medical image to be detected, without the need for manual diagnosis of the test report, and without the need to use the targeted manual diagnosis report as the input data of the method, thereby realizing the automated diagnosis of medical images.

[0032] Furthermore, the image processing method provided in the embodiment of the present application pre-adopts text form, presets part-level detection result category information for multiple parts of the detection object, and obtains part-level text features corresponding to the detection result category information at each part level; correspondingly, after acquiring the medical image, the part-level image features of each part contained in the image are extracted respectively, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the predicted detection results of multiple parts of the detection object are obtained, thereby realizing the detection of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis.

[0033] Optionally, in some of the embodiments, the process of extracting features from the medical image to obtain part-level image features corresponding to each part may specifically include: Part detection is performed on the medical image to obtain the segmentation mask of each part; the medical image is divided into blocks to obtain multiple image blocks; according to the segmentation mask of each part, the image block corresponding to each part is determined from the multiple image blocks; and features are extracted from the image blocks corresponding to each part to obtain the part-level image features corresponding to each part.

[0034] Specifically, the above process of obtaining the segmentation mask of each part involves two parts: part detection and image segmentation. Among them, part detection refers to identifying and locating each part in the medical image and determining the bounding box of each part. Image segmentation refers to the process of dividing a medical image into multiple sub-images, with the purpose of separating each part in the image. The segmentation result can usually be represented by a segmentation mask (image). The segmentation mask image has the same size as the medical image, and the value of each pixel in the segmentation mask image is usually used to indicate whether the pixel belongs to a specific part. In the embodiment of the present application, a segmentation mask image can be obtained for each part. In this case, the segmentation mask image is a binary mask image, and the pixel value is usually 0 or 1, where 0 represents the background and 1 represents the foreground (the area where the part is located); the same segmentation mask image can also be generated for each part. In this case, a multi-category segmentation mask image is obtained. In this mask image, the pixel values ​​in the areas where different parts are located are different, and the pixel values ​​in the areas where the same parts are located are the same.

[0035] In the above embodiment of the present application, on the one hand, the segmentation mask of each part corresponding to the medical image is obtained, and on the other hand, the medical image is divided into blocks to obtain multiple image blocks; then, the above multiple image blocks can be grouped according to the segmentation mask of each part, so as to obtain the image blocks corresponding to each part, so as to extract the part-level image features corresponding to each part from the image blocks corresponding to each part. The segmentation mask map can usually provide accurate information at the pixel level, and thus can more accurately distinguish and segment different parts in the medical image. Therefore, the above process of obtaining the part-level image features corresponding to each part with the help of the segmentation mask map can improve the accuracy of the final part-level image features, and thus help to improve the accuracy of the predicted detection results.

[0036] Optionally, in some of the embodiments, feature extraction is performed on the medical image to obtain part-level image features corresponding to each part, which may include: The medical image is input into the pre-trained prediction model, and the image encoding module in the prediction model is used to encode the medical image to obtain the part-level image features corresponding to each part; Correspondingly, feature extraction is performed on each detection result category information to obtain text features corresponding to each detection result category information, which may include: Input each test result category information into a pre-trained prediction model, and perform text encoding on each test result category information through a text encoding module in the prediction model to obtain text features corresponding to each test result category information; The detection result prediction is performed based on the part-level image features and text features corresponding to each part, and the predicted detection results of multiple parts of the detection object are obtained, which may include: Through the prediction module in the prediction model, the detection result prediction is performed based on the part-level image features and text features corresponding to each part, and the predicted detection results of multiple parts of the detection object are output.

[0037] Specifically, in the embodiment of the present application, a prediction model including multiple different modules can be used to perform corresponding processing operations on the acquired medical image and the category information of the test result, so as to obtain the final predicted test result. After acquiring the medical image and the category information of the test result, the image and text are input into the trained prediction model, and the predicted test result can be output through the model, and the operation process is relatively simple. Moreover, since the prediction model is pre-trained and has the corresponding prediction prior knowledge, the output predicted test result also has a high accuracy.

[0038] Optionally, in some of the embodiments, the training process of the above prediction model may specifically include: Obtaining a training sample, wherein the training sample includes a plurality of text-image pairs; the text-image pairs include: a sample image, and a text detection result corresponding to the sample image; Input the text image pair into the prediction model, and obtain the part-level sample image features corresponding to each part in the sample image and the part-level text features corresponding to each part in the text detection result through the prediction model; Generate image-text similarity prediction information of each part based on part-level sample image features and part-level text features of each part; And according to the image-text similarity prediction information of each part and the predetermined image-text similarity label information of each part, the loss value corresponding to each part is calculated; The loss values ​​corresponding to each part are fused to obtain a fused loss value; The model parameters of the prediction model are adjusted according to the fusion loss value to obtain a trained prediction model.

[0039] Specifically, during the prediction model training process, the training samples of the same training batch (Batch) may generally include multiple text-image pairs, wherein a text-image pair may include a sample image and a file detection result corresponding to the sample image, and the sample image and the file detection result may generally be obtained for the same detection object. The text detection result corresponding to the sample image may be a detection result for the sample image represented in text form obtained by using any suitable related technology, and the detection result generally involves detection results for various parts. Exemplarily, the text detection result may be a diagnostic report given by a professional (such as a doctor) for the sample image.

[0040] The specific implementation method of obtaining the part-level sample image features corresponding to each part in the sample image through the prediction model can be referred to the above step 104, which will not be repeated here.

[0041] When the part-level text features corresponding to each part in the text detection results are obtained through the prediction model, it can be realized with the help of the neural network sub-model in the text encoding module of the prediction model: the above-mentioned neural network sub-model can be trained in advance to obtain a trained sub-model, and then after obtaining the text detection results, with the help of the above-mentioned trained sub-model, the part-level text features of each part can be obtained respectively through the sub-model.

[0042] When obtaining the text features of each part level with the help of the neural network sub-model, there are two specific implementation methods: The first method is to input the complete text detection results into a sub-model, and output the part-level text features of each part through the sub-model. In this method, the part-level text features can be obtained without performing additional preprocessing operations on the text detection results. Therefore, the processing process is simple and easy to implement.

[0043] The second method is that before inputting the text detection result into the neural network sub-model, the text detection result can be decomposed to obtain multiple sub-text detection results; wherein one sub-text detection result corresponds to one part; and then each sub-text detection result is input into the neural network sub-model respectively, and each sub-text detection result is encoded by the neural network sub-model respectively to obtain the part-level text features corresponding to each part. In this method, after obtaining the text detection result, the complete text detection result is not input into the neural network sub-model, but the text detection result is preprocessed first, so as to split it into sub-text detection results corresponding to different parts, and then each sub-text detection result is encoded to obtain the part-level text features of each part. Since the text detection result is pre-segmented at the part level, the text input into the neural network sub-model at a single time is all text with high correlation corresponding to the same part, which helps to improve the accuracy of the final part-level text features.

[0044] Furthermore, the text detection result usually includes two parts: "image description" and "detection prompt", wherein the "image description" part is used to describe the visual state presented by each part in the medical image as a sample image, and the "detection prompt" part is used to infer the detection results of each part based on the information described in the "image description" part. Therefore, with regard to the second method mentioned above, the process of decomposing the text detection result and obtaining multiple sub-text detection results can be: respectively performing entity recognition on the "image description" part and the "detection prompt" part in the text detection result, thereby identifying the part information contained in the "image description" part and the part information contained in the "detection prompt" part; then, extracting the description information corresponding to each part information from the "image description" part and the "detection prompt" part respectively; and then integrating the description information corresponding to the same part information in the "image description" part and the "detection prompt" part, so as to obtain the sub-text detection results corresponding to each part in the text detection result.

[0045] As for the above-mentioned entity recognition process, as mentioned above, since the division granularity of the detection object can be flexibly set, the following situation may exist: a certain part does not exist in the "image description" part or the "detection prompt" part in the text detection result, but a more detailed description information than a certain part is used, such as: for the part of "small intestine", the "small intestine" entity does not exist in the "image description" part or the "detection prompt" part, but there are finer-grained body components such as "jejunum", "ileum" or "duodenum". In view of the above situation, in order to improve the recognition accuracy, the fine-grained depth component structure contained in each part can be pre-set, and then when the fine-grained depth component structure is recognized, it is determined that the corresponding part information is recognized.

[0046] At the same time, there may be a situation where a certain part exists in one of the two parts, "Image Description" and "Detection Tips", but not in the other part. At this time, in order to improve the recognition accuracy, special marks (such as empty characters, etc.) can be used in the part where the part does not appear to supplement the missing part information. In addition, there may also be a situation where a certain part does not appear in both the "Image Description" and the "Detection Tips". According to clinical experience, in this case, the test result of the non-appearing part can be defaulted to "healthy" or "no significant abnormality".

[0047] See also Figure 2 , Figure 2 Schematic diagram of the decomposition of text images and text detection results. Figure 2 The left part of FIG. 1 is a schematic diagram of a segmentation result obtained after performing part segmentation on a medical CT image of a chest and abdomen region of a human body. Different pixel values ​​are used to represent different parts of the human body in the schematic diagram of the segmentation result. Figure 2 The right part of the figure represents the decomposition process of the diagnosis report corresponding to the medical CT image (corresponding to the text detection result in the embodiment of the present application), from Figure 2 As can be seen from the right part, by decomposing the diagnosis report, the diagnosis results corresponding to the four human body parts, namely the liver, kidney, pancreas and stomach, are finally obtained (corresponding to the sub-text detection results in the embodiment of the present application).

[0048] In addition, the image-text similarity prediction information generated based on the part-level sample image features and part-level text features of each part can represent the similarity between the part-level sample image features and part-level text features corresponding to the same part. Since the training samples of the same training batch contain multiple text-image pairs, for the training samples of the same training batch, multiple similarity prediction information can usually be obtained for a single part. Furthermore, for the convenience of recording, the image-text similarity prediction information can be presented in the form of a similarity prediction matrix.

[0049] The above process of generating the image-text similarity prediction information of each part based on the part-level sample image features and part-level text features of each part can be expressed by the following formula: = , =

[0050] Where j represents the part index number, Represents the total number of image samples containing the complete j-th part; Represents the cosine similarity between vector a and vector b; represents the learnable temperature parameter; Indicates that for Part, Part-level image features to The image-text similarity prediction value of each part-level text feature; Indicates that for Part, Part-level text features to The image-text similarity prediction value of each part-level image feature; Indicates In the sample image Part-level image features corresponding to each part; Indicates Among the text detection results The part-level text features corresponding to each part.

[0051] Correspondingly, the fusion loss value obtained by the final fusion is It can be expressed by the following formula: = ( ) in, represents the total number of anatomical parts; H represents the cross entropy loss; Indicates Part-level image features to The image-text similarity label value of each part-level text feature; Indicates Part-level image features to The image-text similarity prediction value of each part-level text feature; Indicates Part-level text features to The image-text similarity label value of each part-level image feature; Indicates Part-level text features to The image-text similarity prediction value of part-level image features.

[0052] In the embodiment of the present application, the sample image and the corresponding text detection results are subjected to more fine-grained part-level segmentation, and then in the model training process, feature matching and alignment of the sample image and text detection results at the part-level fine-grained level are achieved. Compared with the traditional feature matching and alignment of sample images and text detection results at the global coarse-grained level, the splitting and fine-grained alignment scheme provided by the embodiment of the present application can effectively improve the accuracy of alignment, and improve the interpretability of the output results of the prediction model. In addition, with the fine-grained part-level segmentation of the sample image and the corresponding text detection results, the number of training samples can be effectively increased, which helps to improve the prediction accuracy of the prediction model obtained by training.

[0053] See also Figure 3 , Figure 3 The following is a schematic diagram of the image processing flow in the scenario of multi-disease auxiliary diagnosis. Figure 3 The image processing process of the embodiment of the present application in the multi-disease auxiliary diagnosis scenario is explained: Obtain medical CT images , and the corresponding text detection results On the one hand, for images Perform block processing to obtain block processing results, and input the block processing results into the image encoder in the prediction model image encoding module ( Figure 3In the figure, ViT, Visual Transformer is taken as an example, which does not constitute a limitation on the image encoder), to encode each image block to obtain the encoding result corresponding to each image block; Perform part detection to obtain segmentation masks of each part; then determine the encoding results corresponding to each component from the encoding results corresponding to each image block according to the segmentation masks of each part; add an initial feature vector used to characterize the overall image features of the part to the encoding results corresponding to each part; introduce a self-attention mechanism within each part to obtain the part-level image features corresponding to each part ,in, represents the part-level image feature corresponding to the jth part in the i-th sample image. On the other hand, the text detection result Decompose and obtain the sub-text detection results corresponding to each part ; The detection results of each sub-text (represents the sub-text detection result corresponding to the jth part in the i-th text detection result) is input into the text encoder in the prediction model text encoding module ( Figure 3 The BERT model is used as an example, which does not constitute a limitation on the text encoder), thereby obtaining the part-level text features of each part (represents the part-level text feature corresponding to the j-th part in the i-th text detection result).

[0054] Afterwards, we can use the part-level sample image features of each part and part-level text features , generate the image-text similarity prediction matrix of each part: (the image-text similarity prediction matrix corresponding to the jth part); then, the loss value is calculated according to the image-text similarity prediction matrix of each part, and the prediction model parameters are adjusted until a trained prediction model is obtained.

[0055] Optionally, in some of the embodiments, the process of determining the image-text similarity label information of each part may include: For a single part: obtain initial image-text similarity label information of the single part, wherein the initial image-text similarity label information is based on whether the part-level sample image features and the part-level text features correspond to the same detection object for similarity judgment; obtain similarity label correction information of the single part; wherein, if the part-level sample image features and the part-level text features of the single part correspond to different detection objects respectively, and the detection results of the single parts of different detection objects are normal, then it is determined that the part-level sample image features and the part-level text features of the single part are similar; fuse the initial image-text similarity label information and the similarity label correction information to obtain the image-text similarity label information of the single part.

[0056] Specifically, when the initial image-text similarity label information is determined, for the part-level sample image features and part-level text features corresponding to the same part, if the two features correspond to the same detection object, or in other words, the two features are derived from the same text-image pair, the two features can be determined to be similar, for example, the similarity of the two features can be determined to be 1. Otherwise, the two features are determined to be dissimilar, for example, the similarity of the two features can be determined to be 0.

[0057] When determining the above-mentioned initial image-text similarity label information, the part-level sample image features and part-level text features derived from the same text image pair are determined as positive samples (determined to be similar), while the part-level sample image features and part-level text features not derived from the same text image pair are determined as negative samples (determined to be dissimilar). However, in the medical field, for part-level sample image features and part-level text features derived from different text image pairs, there is often a certain semantic association between these two features. For example: with respect to a certain part, for two test objects that are both diagnosed as normal, their part-level sample image features and part-level text features are usually similar.

[0058] Based on the above situation, if we use the above initial image-text similarity label information as the final label information from the perspective of whether the two features come from the same text-image pair, there will be a problem of pseudo-negative samples, that is: for two test objects that are both diagnosed as normal, a pseudo-negative sample is constructed between their part-level sample image features and part-level text features. The pseudo-negative sample should have been determined to be similar, but was mistakenly determined to be dissimilar. As a result, during the pre-training process of the prediction model, the trained prediction model mistakenly identifies this type of sample as dissimilar, thereby affecting the prediction accuracy of the trained prediction model.

[0059] In the above embodiment of the present application, in addition to obtaining the above initial image-text similarity label information, similarity label correction information of a single part is also obtained. Wherein, when determining the similarity label correction information, if the part-level sample image features and the part-level text features correspond to different detection objects respectively, and the detection results of the single parts of different detection objects are normal, then it is determined that the above part-level sample image features and the part-level text features are similar. Afterwards, the initial image-text similarity label information is corrected and fused through the above similarity label correction information, so that the above pseudo-negative samples are corrected to positive samples, thereby avoiding the problem of low prediction accuracy of the prediction model caused by the existence of pseudo-negative samples, and improving the prediction accuracy of the prediction model.

[0060] Optionally, in some of the embodiments, before fusing the initial image-text similarity label information and the similarity label correction information, the method further includes: Obtain similarity label reference information of a single part generated by the auxiliary model for the training sample; the auxiliary model is different from the prediction model; The initial image-text similarity label information and similarity label correction information are integrated, including: The initial image-text similarity label information, the similarity label correction information and the similarity label reference information are integrated to obtain the image-text similarity label information of a single part.

[0061] Specifically, in the medical field, for a certain part, the part-level sample image features and part-level text features of two test objects that are diagnosed with the same disease are usually similar. Based on this situation, if the part-level sample image features and part-level text features of two test objects that are diagnosed with the same disease are determined to be dissimilar, the above-mentioned problem of false negative samples will also occur.

[0062] To solve the above problems, an auxiliary model is introduced in an embodiment of the present application. The auxiliary model generates similarity prediction results for training samples, and uses the results as reference information for similarity values. The initial image and text similarity label information is further corrected and integrated, so as to train the prediction model as much as possible, so that the prediction model can determine the part-level sample image features and part-level text features of two test objects diagnosed with the same disease as similar, thereby further improving the prediction accuracy of the prediction model.

[0063] Furthermore, in an embodiment of the present application, a collaborative teaching training framework can also be used to set two prediction models to be trained: a first prediction model and a second prediction model, and then in the model training stage, the above two models are alternately trained, that is: for the same training sample, the first prediction model and the second prediction model are used to perform predictions, respectively, to obtain prediction results; and the prediction results obtained by the first prediction model (or the second prediction model) are used as the above-mentioned similarity label reference information to correct and merge the initial image-text similarity label information of the second prediction model (or the first prediction model), thereby assisting the training process of the second prediction model (or the first prediction model). Among them, the first prediction model and the second prediction model are models of different structures, or they can be models of the same structure. When the first prediction model and the second prediction model are models of the same structure, the initialization parameters of the two models, the iteration order of the training samples, etc. can be different.

[0064] Specifically, the above-mentioned image-text similarity label information can be calculated using the following formula: ,

[0065] ,

[0066] in, is a preset parameter, which can be set to 0.5 by way of example; and Respectively representing similarity label information from image features to text features obtained by the first prediction model, and similarity label information from text features to image features obtained by the first prediction model; and Respectively representing the similarity label information from the image feature to the text feature obtained by the second prediction model, and the similarity label information from the text feature to the image feature obtained by the second prediction model; and Respectively representing similarity prediction information from image features to text features obtained by the first prediction model, and similarity prediction information from text features to image features obtained by the first prediction model; and The similarity prediction information from the image feature to the text feature obtained by the second prediction model and the similarity prediction information from the text feature to the image feature obtained by the second prediction model are respectively represented.

[0067] See also Figure 4 , Figure 4 This is a schematic diagram of the correction of the image-text similarity label information. Figure 4 The correction process of the image-text similarity label information provided in the above embodiment of the present application is briefly described: like Figure 4 As shown, it is assumed that the training samples of the same training batch contain 4 text-image pairs, among which the first text-image pair includes image and subtext detection results for pancreas , the second text-image pair includes the image and subtext detection results for pancreas , the third text-image pair includes the image and subtext detection results for pancreas , the fourth text-image pair includes the image and subtext detection results for pancreas The image-text similarity prediction matrix of the pancreas obtained by the prediction model is A, where ,……, are the part-level sample image features obtained from the corresponding text image pairs, ,……, are the part-level text features obtained from the corresponding text image pairs. The initial image-text similarity label matrix is ​​B, the similarity label correction matrix is ​​C, and the similarity label reference matrix generated by the auxiliary model for the training sample is D. The image-text similarity label matrix E corresponding to the pancreas part is equal to the fusion result of the above matrices B, C and D. Further, after the image-text similarity label matrix E is obtained by fusion, the loss value L corresponding to the pancreas part can be calculated based on the image-text similarity prediction matrix A and the image-text similarity label matrix E.

[0068] Furthermore, the similarity label reference matrix D may be a matrix obtained by the first prediction model and the second prediction model through collaborative teaching.

[0069] Reference Figure 5 , Figure 5 The following is a flowchart of a computer-aided diagnosis method according to an embodiment of the present application. The computer-aided diagnosis method provided by the embodiment of the present application can be performed by any appropriate device with image processing capabilities. Specifically, the computer-aided diagnosis method provided by the embodiment can include the following steps: Step 502: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object.

[0070] Specifically, the medical CT image obtained in the embodiment of the present application may be an image formed by performing CT detection on a preset area of ​​the detection object. The above-mentioned preset area contains multiple parts of the detection object. The parts of the detection object contained in the preset area may be the body composition structure obtained after dividing the body of the detection object according to the preset division granularity. The embodiment of the present application does not limit the preset area and the specific division granularity, and can be customized according to actual conditions.

[0071] Step 504: extract features from the medical CT image to obtain part-level image features corresponding to each part.

[0072] Specifically, the part-level image features corresponding to each part refer to deep image features that can reflect the image detail information of each part. Since the medical CT image contains multiple parts, the number of part-level image features obtained by this step is also multiple, where one part corresponds to one part-level image feature.

[0073] Step 506, determining the disease category information corresponding to each part, and performing feature extraction on each disease category information to obtain text features corresponding to each disease category information.

[0074] Specifically, the disease category information corresponding to each part indicates the disease category corresponding to each part described in text form, wherein the above disease category may be a part-level disease that may occur in the part pre-set for each part. For example, for the liver, the corresponding disease category may include: healthy, cirrhosis, liver cancer, etc.

[0075] After obtaining the disease category information corresponding to each part, the text feature extraction method can be used to obtain the text features corresponding to each disease category information. Similar to the image feature extraction, the text features obtained in this step are also part-level text features.

[0076] Step 508 , predicting the detection results based on the part-level image features and the text features corresponding to each part, and obtaining the disease diagnosis results of multiple parts of the detection object.

[0077] The computer-aided diagnosis method provided in the embodiment of the present application pre-adopts text form, presets part-level disease category information for multiple parts of the detection object, and obtains part-level text features corresponding to each part-level disease category information; correspondingly, after obtaining the medical CT image, the part-level image features of each part contained in the image are extracted respectively, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the disease diagnosis results of multiple parts of the detection object are obtained, thereby realizing the diagnosis of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis.

[0078] See also Figure 6 , Figure 6 This is a schematic diagram of an exemplary computer-aided diagnosis system applicable to the computer-aided diagnosis method of the embodiment of the present application. For ease of understanding, first combine Figure 6 The application scenario of the computer-aided diagnosis method provided in the embodiment of the present application is explained.

[0079] like Figure 6 As shown, the system 600 may include a server 602, a communication network 604 and / or one or more user devices (ie, client devices) 606. Figure 6 The example in FIG. 1 is a plurality of user devices.

[0080] The server 602 may be any appropriate device for performing multi-disease auxiliary diagnosis based on medical CT images of multiple parts of the test object, including but not limited to server clusters, computing cloud server clusters, etc. In some embodiments, the server 602 may perform any appropriate function. For example, in some embodiments, the server 602 may be used to receive medical CT images sent by the user device 606, perform image processing based on the received medical CT images, thereby obtaining multi-disease diagnosis results for multiple parts of the test object, and return the multi-disease diagnosis results to the above-mentioned user device 606.

[0081] In some embodiments, the communication network 604 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 604 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The user device 606 can be connected to the communication network 604 via one or more communication links (e.g., communication link 612), and the communication network 604 can be linked to the service end 602 via one or more communication links (e.g., communication link 614). The communication link can be any communication link suitable for transmitting data between the user device 606 and the service end 602, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link or any suitable combination of such links.

[0082] User equipment 606 may include any one or more user equipment. In some embodiments, user equipment 606 may include any suitable type of equipment. For example, in some embodiments, user equipment 606 may include a computer and / or any other suitable type of user equipment.

[0083] Reference Figure 7 , Figure 7 FIG. 1 is a flowchart of another computer-aided diagnosis method according to an embodiment of the present application. The computer-aided diagnosis method provided in the embodiment of the present application can be Figure 6 The user equipment 606 in the system shown in the figure executes. Specifically, the computer-aided diagnosis method provided in this embodiment includes the following steps: Step 702: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object.

[0084] Specifically, the medical CT image obtained in the embodiment of the present application may be an image formed by performing CT detection on a preset area of ​​the detection object. The above-mentioned preset area contains multiple parts of the detection object. The parts of the detection object contained in the preset area may be the body composition structure obtained after dividing the body of the detection object according to the preset division granularity. The embodiment of the present application does not limit the preset area and the specific division granularity, and can be customized according to actual conditions.

[0085] Step 704, sending the medical CT image to the server device so that the server device extracts the part-level image features corresponding to each part in the medical CT image; based on the part-level image features corresponding to each part and each text feature, the detection result is predicted to obtain the disease diagnosis results of multiple parts of the detection object; each text feature is obtained by extracting the disease category information corresponding to each part.

[0086] Specifically, after acquiring the medical CT image, the server-side device can extract the part-level image features of each part contained in the image respectively, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the text features of the above-mentioned part levels, the disease diagnosis results of multiple parts of the detection object are obtained.

[0087] Step 706, receiving the disease diagnosis result returned by the server device.

[0088] Step 708, displaying the disease diagnosis result in the display interface.

[0089] The computer-aided diagnosis method provided in the embodiment of the present application pre-adopts text form, presets part-level disease category information for multiple parts of the detection object, and obtains part-level text features corresponding to the disease category information at each part level through a server device that provides computer-aided diagnosis services; correspondingly, after the client device obtains the medical CT image, the image can be sent to the server device, so that the part-level image features of each part contained in the image can be extracted by the server device, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the disease diagnosis results of multiple parts of the detection object are obtained, thereby realizing the diagnosis of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis.

[0090] Reference Figure 8 , Figure 8 FIG. 1 is a flowchart of another method for computer-aided diagnosis according to an embodiment of the present application. The method for computer-aided diagnosis provided by the embodiment of the present application can be performed by Figure 6 Specifically, the computer-aided diagnosis method provided in this embodiment includes the following steps: Step 802: receiving a medical CT image sent by a client device, wherein the medical CT image includes multiple parts of a detection object.

[0091] Specifically, the medical CT image obtained in the embodiment of the present application may be an image formed by performing CT detection on a preset area of ​​the detection object. The above-mentioned preset area contains multiple parts of the detection object. The parts of the detection object contained in the preset area may be the body composition structure obtained after dividing the body of the detection object according to the preset division granularity. The embodiment of the present application does not limit the preset area and the specific division granularity, and can be customized according to actual conditions.

[0092] Step 804: extracting part-level image features corresponding to each part in the medical CT image.

[0093] Specifically, the part-level image features corresponding to each part refer to deep image features that can reflect the image detail information of each part. Since the medical CT image contains multiple parts, the number of part-level image features obtained by this step is also multiple, where one part corresponds to one part-level image feature.

[0094] Step 806, predicting the detection results based on the part-level image features and text features corresponding to each part, and obtaining the disease diagnosis results of multiple parts of the detection object; each text feature is obtained by extracting the disease category information corresponding to each part.

[0095] Step 808, returning the disease diagnosis result to the client device so as to display the disease diagnosis result through the display interface of the client device.

[0096] The computer-aided diagnosis method provided in the embodiment of the present application pre-adopts text form, presets part-level disease category information for multiple parts of the detection object, and obtains part-level text features corresponding to the disease category information at each part level through a server device that provides computer-aided diagnosis services; correspondingly, after the client device obtains the medical CT image, the image can be sent to the server device, so that the part-level image features of each part contained in the image can be extracted by the server device, that is: when extracting image features, the complete image features are split into more fine-grained part-level image features, and then based on the part-level image features corresponding to each part and the above-mentioned part-level text features, the disease diagnosis results of multiple parts of the detection object are obtained, thereby realizing the diagnosis of multiple diseases in multiple parts of the detection object and improving the scalability of medical image interpretation and auxiliary diagnosis.

[0097] Fig. 9 : is a structural block diagram of an image generation device according to an embodiment of the present application. The image generation device provided by the embodiment of the present application includes: An image acquisition module 902 is used to acquire a medical image, wherein the medical image contains multiple parts of the detected object; An image feature extraction module 904 is used to extract features from medical images and obtain part-level image features corresponding to each part; The text feature extraction module 906 is used to determine the detection result category information corresponding to each part, and perform feature extraction on each detection result category information to obtain the text feature corresponding to each detection result category information; The result obtaining module 908 is used to predict the detection result based on the part-level image features and the text features corresponding to each part, and obtain the predicted detection results of multiple parts of the detection object.

[0098] Optionally, in some embodiments, the image feature extraction module 904 is specifically configured to: Perform part detection on medical images to obtain segmentation masks of each part; Perform block processing on the medical image to obtain multiple image blocks; According to the segmentation mask of each part, an image block corresponding to each part is determined from multiple image blocks; and features are extracted from the image blocks corresponding to each part respectively to obtain part-level image features corresponding to each part.

[0099] Optionally, in some of the embodiments, the result obtaining module 908 is specifically used to: For a single part, the similarity between the part-level image features corresponding to the single part and each text feature is calculated respectively, and the predicted detection result of the single part is obtained based on the similarity calculation result.

[0100] Optionally, in some embodiments, the image feature extraction module 904 is specifically configured to: The medical image is input into the pre-trained prediction model, and the image encoding module in the prediction model is used to encode the medical image to obtain the part-level image features corresponding to each part; The text feature extraction module 906, when performing the step of extracting features from each detection result category information to obtain text features corresponding to each detection result category information, is specifically used to: input each detection result category information into a pre-trained prediction model, perform text encoding on each detection result category information through a text encoding module in the prediction model, and obtain text features corresponding to each detection result category information; The result obtaining module 908 is specifically used to: predict the detection results based on the part-level image features and text features corresponding to each part through the prediction module in the prediction model, and output the predicted detection results of multiple parts of the detection object.

[0101] Optionally, in some of the embodiments, the image processing apparatus further comprises: A model training module is used to obtain training samples, which include multiple text-image pairs; the text-image pairs include: sample images, and text detection results corresponding to the sample images; the text-image pairs are input into a prediction model, and the part-level sample image features corresponding to each part in the sample image and the part-level text features corresponding to each part in the text detection results are obtained through the prediction model; based on the part-level sample image features and part-level text features of each part, image-text similarity prediction information of each part is generated; and based on the image-text similarity prediction information of each part and the predetermined image-text similarity label information of each part, the loss value corresponding to each part is calculated; the loss values ​​corresponding to each part are fused to obtain a fused loss value; the model parameters of the prediction model are adjusted according to the fused loss value to obtain a trained prediction model.

[0102] Optionally, in some of the embodiments, the model training module is further used to decompose the text detection result to obtain a plurality of sub-text detection results before inputting the text image pair into the prediction model; wherein one sub-text detection result corresponds to one part; Correspondingly, when the model training module performs the step of inputting the text image pair into the prediction model, obtaining the part-level sample image features corresponding to each part in the sample image through the prediction model, and the part-level text features corresponding to each part in the text detection result, it is specifically used to: The sample image and multiple sub-text detection results are input into the prediction model, and the sample image is encoded by the image encoding module in the prediction model to obtain the part-level sample image features corresponding to each part in the sample image; the sub-text detection results are respectively text-encoded by the text encoding module in the prediction model to obtain the part-level text features corresponding to each part.

[0103] Optionally, in some of the embodiments, the model training module is further used to: For a single part: Obtaining initial image-text similarity label information of a single part, wherein the initial image-text similarity label information is similarity determined based on whether the part-level sample image features and the part-level text features correspond to the same detection object; Obtain similarity label correction information for a single part; wherein, if the part-level sample image features and the part-level text features of a single part correspond to different detection objects respectively, and the detection results of the single parts of different detection objects are normal, then it is determined that the part-level sample image features and the part-level text features of the single part are similar; The initial image-text similarity label information and the similarity label correction information are integrated to obtain the image-text similarity label information of a single part.

[0104] Optionally, in some of the embodiments, before performing the fusion of the initial image-text similarity label information and the similarity label correction information, the model training module is further used to: obtain similarity label reference information of a single part generated by the auxiliary model for the training sample; the auxiliary model is different from the prediction model; Correspondingly, when the model training module executes the step of fusing the initial image-text similarity label information and the similarity label correction information, it is specifically used to: fuse the initial image-text similarity label information, the similarity label correction information and the similarity label reference information to obtain the image-text similarity label information of a single part.

[0105] The image processing device of the embodiment of the present application is used to implement the method in the aforementioned image processing method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each unit in the image processing device of the embodiment of the present application can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0106] Fig.10 The structure block diagram of a computer-aided diagnosis device according to an embodiment of the present application. The computer-aided diagnosis device provided by the embodiment of the present application includes: The CT image acquisition module 1002 is used to acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; The CT image feature extraction module 1004 is used to extract features from the medical CT image to obtain part-level image features corresponding to each part; The disease category feature extraction module 1006 is used to determine the disease category information corresponding to each part, and extract features from each disease category information to obtain text features corresponding to each disease category information; The diagnosis result obtaining module 1008 is used to predict the detection result based on the part-level image features and text features corresponding to each part, and obtain the disease diagnosis results of multiple parts of the detection object.

[0107] The computer-aided diagnosis device of the embodiment of the present application is used to implement the computer-aided diagnosis method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each unit in the computer-aided diagnosis device of the embodiment of the present application can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0108] Fig.11 1 is a block diagram of another computer-aided diagnosis device according to an embodiment of the present application. The computer-aided diagnosis device provided by the embodiment of the present application includes: An acquisition module 1102 is used to acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; The sending module 1104 is used to send the medical CT image to the server device, so that the server device extracts the part-level image features corresponding to each part in the medical CT image; based on the part-level image features corresponding to each part and each text feature, the detection result is predicted to obtain the disease diagnosis results of multiple parts of the detection object; each text feature is obtained by extracting the disease category information corresponding to each part; The receiving module 1106 is used to receive the disease diagnosis result returned by the server device; The display module 1108 is used to display the disease diagnosis result in the display interface.

[0109] The computer-aided diagnosis device of the embodiment of the present application is used to implement the computer-aided diagnosis method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each unit in the computer-aided diagnosis device of the embodiment of the present application can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0110] Fig.12 1 is a structural block diagram of another computer-aided diagnosis device according to an embodiment of the present application. The computer-aided diagnosis device provided in the embodiment of the present application includes: The image receiving module 1202 is used to receive a medical CT image sent by a client device, where the medical CT image contains multiple parts of the detected object; An extraction module 1204 is used to extract part-level image features corresponding to each part in the medical CT image; The diagnosis module 1206 is used to predict the detection result based on the part-level image features and text features corresponding to each part, and obtain the disease diagnosis results of multiple parts of the detection object; each text feature is obtained by extracting the disease category information corresponding to each part; The result returning module 1208 is used to return the disease diagnosis result to the client device so as to display the disease diagnosis result through the display interface of the client device.

[0111] The computer-aided diagnosis device of the embodiment of the present application is used to implement the computer-aided diagnosis method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each unit in the computer-aided diagnosis device of the embodiment of the present application can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0112] Reference Fig.13 , shows a schematic diagram of the structure of a client device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the client device.

[0113] like Fig.13 As shown, the electronic device may include: a processor (processor) 1302 , a communication interface (Communications Interface) 1304 , a memory (memory) 1306 , and a communication bus 1308 .

[0114] in: The processor 1302 , the communication interface 1304 , and the memory 1306 communicate with each other via a communication bus 1308 .

[0115] The communication interface 1304 is used to communicate with other electronic devices or servers.

[0116] The processor 1302 is used to execute the program 1310, and specifically can execute the relevant steps in the above method embodiment.

[0117] Specifically, the program 1310 may include program codes, which include computer operation instructions.

[0118] The processor 1302 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0119] The memory 1306 is used to store the program 1410. The memory 1306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0120] The program 1310 may be specifically used to enable the processor 1402 to execute the relevant steps in the above method embodiment.

[0121] The specific implementation of each step in program 1310 can refer to the corresponding description of the corresponding steps in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0122] Reference Fig.14 , shows a schematic diagram of the structure of a server device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the server device.

[0123] like Fig.14 As shown, the electronic device may include: a processor (processor) 1402 , a communication interface (Communications Interface) 1404 , a memory (memory) 1406 , and a communication bus 1408 .

[0124] in: The processor 1402 , the communication interface 1404 , and the memory 1406 communicate with each other via a communication bus 1408 .

[0125] The communication interface 1404 is used to communicate with other electronic devices or servers.

[0126] The processor 1402 is used to execute the program 1410, and specifically can execute the relevant steps in the above method embodiment.

[0127] Specifically, the program 1410 may include program codes, which include computer operation instructions.

[0128] The processor 1402 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0129] The memory 1406 is used to store the program 1410. The memory 1406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0130] The program 1410 may be specifically used to enable the processor 1402 to execute the relevant steps in the above method embodiment.

[0131] The specific implementation of each step in program 1410 can refer to the corresponding description of the corresponding steps in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0132] The present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned multiple method embodiments is implemented. The computer storage medium includes, but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0133] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0134] Experimental data and results In order to verify the effect that can be obtained by the method provided in the embodiment of the present application, the present application conducts effect evaluation through a data set. Specifically: First, a large CT dataset was used for effect evaluation. See Table 1 below, which shows the comparison data between the prediction performance after using different methods to predict the above dataset. When evaluating the model performance, AUC (Area Under Curve), ACC (Accuracy), Spec (Specificity), and Sens (Sensitivity) were used as performance evaluation indicators.

[0135] Table 1

[0136] It can be seen from Table 1 that, compared with the existing solutions, the methods provided in the embodiments of the present application have obtained higher performance index values. That is, compared with other methods, the solutions provided in the embodiments of the present application have better detection effects.

[0137] See Table 2 below, which is a performance comparison of different methods in the downstream report generation task. Table 2 When evaluating the model performance, AUC, Spec, Sens, and natural language generation indicators such as BLEU-4 are used as performance evaluation indicators.

[0138] Table 2

[0139] According to Table 2, compared with the existing solutions, regardless of whether the encoder is fine-tuned or frozen during the pre-training process, the method provided in the embodiment of the present application obtains a higher performance index value. That is to say: compared with other methods, the solution provided in the embodiment of the present application has a better detection effect. Among them, CLIP (Contrastive Language-Image Pre-training) in the existing solution is a multimodal pre-training neural network model; MAE (Masked Autoencoders) model refers to Masked Autoencoders, which is a self-supervised learning method for computer vision.

[0140] Secondly, two other data sets were used for effect evaluation. See Table 3 below, which shows the comparison data between the prediction performance after using different methods to predict the above data sets. When evaluating the model performance, AUC, ACC, F1 score, and accuracy were used as performance evaluation indicators.

[0141] Table 3

[0142] According to Table 3, compared with the existing solutions, the method provided by the embodiment of the present application obtains a higher performance index value for either data set 1 or data set 2. That is, compared with other methods, the solution provided by the embodiment of the present application has a better detection effect.

[0143] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0144] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0145] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses a code for implementing the method or method shown herein, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown herein.

[0146] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0147] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. An image processing method, comprising: Acquire a medical image, wherein the medical image contains multiple parts of a detection object; Extracting features from the medical image to obtain part-level image features corresponding to each part; Determine the detection result category information corresponding to each part, and perform feature extraction on each detection result category information to obtain text features corresponding to each detection result category information; The detection result prediction is performed based on the part-level image features and the text features corresponding to each part, so as to obtain the predicted detection results of the multiple parts of the detection object.

2. The method according to claim 1, wherein: The feature extraction of the medical image to obtain part-level image features corresponding to each part includes: Performing part detection on the medical image to obtain segmentation masks of various parts; Performing block processing on the medical image to obtain a plurality of image blocks; According to the segmentation mask of each part, an image block corresponding to each part is determined from the multiple image blocks; and features are extracted from the image blocks corresponding to each part respectively to obtain part-level image features corresponding to each part.

3. The method according to claim 1 or 2, wherein: The prediction of the detection results based on the part-level image features and the text features corresponding to the parts to obtain the predicted detection results of the multiple parts of the detection object includes: For a single part, the similarities between the part-level image features corresponding to the single part and each text feature are calculated respectively, and the predicted detection result of the single part is obtained based on the similarity calculation result.

4. The method according to claim 1, wherein: The feature extraction of the medical image to obtain part-level image features corresponding to each part includes: Inputting the medical image into a pre-trained prediction model, performing image encoding on the medical image through an image encoding module in the prediction model, and obtaining part-level image features corresponding to each part; The feature extraction of each detection result category information to obtain text features corresponding to each detection result category information includes: Inputting each detection result category information into a pre-trained prediction model, and performing text encoding on each detection result category information through a text encoding module in the prediction model to obtain text features corresponding to each detection result category information; The prediction of the detection results based on the part-level image features and the text features corresponding to the parts to obtain the predicted detection results of the multiple parts of the detection object includes: Through the prediction module in the prediction model, the detection result prediction is performed based on the part-level image features and the text features corresponding to each part, and the predicted detection results of the multiple parts of the detection object are output.

5. The method according to claim 4, wherein: The training process of the prediction model includes: Acquire a training sample, wherein the training sample includes a plurality of text-image pairs; the text-image pairs include: a sample image, and a text detection result corresponding to the sample image; Inputting the text image pair into the prediction model, obtaining the part-level sample image features corresponding to each part in the sample image and the part-level text features corresponding to each part in the text detection result through the prediction model; Generate image-text similarity prediction information of each part based on part-level sample image features and part-level text features of each part; According to the image-text similarity prediction information of each part and the predetermined image-text similarity label information of each part, the loss value corresponding to each part is calculated; The loss values ​​corresponding to each part are fused to obtain a fused loss value; The model parameters of the prediction model are adjusted according to the fusion loss value to obtain a trained prediction model.

6. The method according to claim 5, wherein: Before inputting the text-image pair into the prediction model, the method further includes: Decomposing the text detection result to obtain a plurality of sub-text detection results, wherein one sub-text detection result corresponds to one part; The step of inputting the text image pair into the prediction model and obtaining the part-level sample image features corresponding to each part in the sample image and the part-level text features corresponding to each part in the text detection result through the prediction model includes: The sample image and the multiple sub-text detection results are input into the prediction model, and the sample image is encoded by the image encoding module in the prediction model to obtain part-level sample image features corresponding to each part in the sample image; and the sub-text detection results are respectively text-encoded by the text encoding module in the prediction model to obtain part-level text features corresponding to each part.

7. The method according to claim 5 or 6, wherein: The process of determining the image-text similarity label information of each part includes: For a single part: Acquire initial image-text similarity label information of the single part, wherein the initial image-text similarity label information is similarity determined based on whether the part-level sample image features and the part-level text features correspond to the same detection object; Obtaining similarity label correction information of the single part; wherein, if the part-level sample image features and the part-level text features of the single part correspond to different detection objects respectively, and the detection results of the single part of the different detection objects are normal, then determining that the part-level sample image features and the part-level text features of the single part are similar; The initial image-text similarity label information and the similarity label correction information are integrated to obtain the image-text similarity label information of the single part.

8. The method according to claim 7, wherein: Before fusing the initial image-text similarity label information and the similarity label correction information, the method further includes: Acquire similarity label reference information of the single part generated by an auxiliary model for the training sample; the auxiliary model is different from the prediction model; The fusing of the initial image-text similarity label information and the similarity label correction information includes: The initial image-text similarity label information, the similarity label correction information and the similarity label reference information are integrated to obtain the image-text similarity label information of the single part.

9. A computer-aided diagnosis method comprising: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; Extracting features from the medical CT image to obtain part-level image features corresponding to each part; Determine the disease category information corresponding to each part, and perform feature extraction on each disease category information to obtain text features corresponding to each disease category information; The detection result prediction is performed based on the part-level image features and the text features corresponding to each part, so as to obtain the disease diagnosis results of the multiple parts of the detection object.

10. A computer-aided diagnosis method, applied to a client device, the method comprising: Acquire a medical CT image, wherein the medical CT image includes multiple parts of the detection object; Sending the medical CT image to a server device so that the server device extracts part-level image features corresponding to each part in the medical CT image; The detection result is predicted based on the part-level image features and text features corresponding to each part, so as to obtain the disease diagnosis results of the multiple parts of the detection object; the text features are obtained by extracting the disease category information corresponding to each part; Receive the disease diagnosis results returned by the server device; The disease diagnosis result is displayed in the display interface.

11. A computer-aided diagnosis method, applied to a server device, the method comprising: Receiving a medical CT image sent by a client device, wherein the medical CT image includes multiple parts of a detection object; Extracting part-level image features corresponding to each part in the medical CT image; The detection result is predicted based on the part-level image features and text features corresponding to each part, so as to obtain the disease diagnosis results of the multiple parts of the detection object; the text features are obtained by extracting the disease category information corresponding to each part; The disease diagnosis result is returned to the client device so as to be displayed through the display interface of the client device.

12. A client device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method according to claim 10.

13. A server device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method according to claim 11.

14. A computer-aided diagnosis system comprising: The client device as claimed in claim 12 and the server device as claimed in claim 13.

15. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

16. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the method according to any one of claims 1 to 11.

Images