Image processing method and apparatus
The image processing method automatically detects the metastasis of lymph nodes in CT images based on local and global feature detection, solving the problems of high detection difficulty and high data labeling cost in the prior art, and improving detection accuracy and efficiency.
Patent Information
- Application Number
- PCT/CN2024/096833
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
The prior art is difficult to accurately and automatically detect whether a single lymph node is metastasized in CT images, resulting in the inability to obtain less costly and more accurate training data, which in turn affects the accurate prediction of lymph node metastasis by deep learning models.
Through the image processing method, an image processing task is received, multiple target images of the target partition to be detected, input an image processing model, and detect based on local and global image features to realize automatic operation of whether there are abnormal objects to be detected in the target partition to be detected. Test.
It improves the accuracy and detection efficiency of the test results of the subjects treated, reduces the dependence on professional medical staff, and simplifies the prediction process of lymph node metastasis.
Smart Images

Figure CN2024096833_05122024_PF_FP_ABST
Abstract
Description
Image processing method and device
[0001] Cross-reference
[0002] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on May 31, 2023, with priority number 202310628573.8 and invention name “Image Processing Method and Device,” the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the field of computer technology, and in particular to an image processing method. Background Art
[0004] Lymph node metastasis is a common type of cancer, and usually requires professional doctors to determine the test results from the pathology report based on their experience.
[0005] However, currently, determining whether a single lymph node in a CT image has metastasized based on a pathology report requires considerable effort from professionals, making it difficult to implement clinically. This results in an inability to accurately label individual lymph nodes in a large number of CT images based on pathology reports, and an inability to obtain low-cost and more accurate training data. Therefore, it is difficult to automatically detect whether a single lymph node has metastasized.
[0006] Summary of the Invention
[0007] In view of this, embodiments of the present disclosure provide an image processing method. One or more embodiments of the present disclosure also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.
[0008] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:
[0009] receiving an image processing task, wherein the image processing task carries a plurality of target images corresponding to the target to-be-detected partition, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target to-be-detected partition;
[0010] Multiple target images are input into an image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0011] According to a second aspect of an embodiment of the present disclosure, a CT image processing method is provided, comprising:
[0012] receiving a CT image processing task, wherein the CT image processing task carries multiple CT images corresponding to the target partition to be detected, and the CT image processing task is configured to detect whether there are abnormal cells to be detected in the target partition to be detected;
[0013] Multiple CT images are input into a CT image processing model to obtain detection results corresponding to the target partition to be detected, wherein the CT image processing model determines the detection results based on local image features and global image features of each CT image.
[0014] According to a third aspect of an embodiment of the present disclosure, a method for training an image processing model is provided, which is applied to a cloud-side device and includes:
[0015] Obtaining a training sample pair and guidance information for the training sample pair, wherein the training sample pair includes a training sample and a sample label, the training sample includes a plurality of sample images corresponding to the target partition to be detected, and the sample label is set to identify a sample result of the target partition to be detected;
[0016] Input the training sample pairs and guidance information into the initial image processing model to obtain the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected;
[0017] Calculate the first loss value of the initial image processing model based on the sample prediction result and the sample result;
[0018] Calculate the second loss value of the initial image processing model based on the target attention information and the guidance information;
[0019] Adjusting the model parameters of the initial image processing model according to the first loss value and the second loss value until a preset stop condition is reached, thereby obtaining the model parameters of the image processing model;
[0020] Send the model parameters of the image processing model to the end device.
[0021] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image processing method, including:
[0022] Receive an image processing request sent by a user, wherein the image processing request includes an image processing task, the image processing task carries multiple target images corresponding to the target to-be-detected partition, and the image processing task is set to detect whether there is an abnormal to-be-detected object in the target to-be-detected partition;
[0023] Inputting multiple target images into an image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image;
[0024] Send the detection results corresponding to the target partition to be detected to the user.
[0025] According to a fifth aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:
[0026] a receiving module configured to receive an image processing task, wherein the image processing task carries a plurality of target images corresponding to the target to-be-detected partition, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target to-be-detected partition;
[0027] The detection module is configured to input multiple target images into an image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0028] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computing device, including:
[0029] memory and processor;
[0030] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.
[0031] According to a seventh aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions, and the computer-executable instructions can implement the steps of the above method when executed by a processor.
[0032] According to an eighth aspect of an embodiment of the present disclosure, a computer program is provided, wherein when the computer program is executed in a computer, the steps of the above method are implemented when the computer is executed.
[0033] According to a ninth aspect of the embodiments of the present disclosure, a computer-aided diagnosis method for cancer cell metastasis to lymph nodes is provided, comprising:
[0034] receiving a CT image processing task, wherein the CT image processing task carries a plurality of CT images corresponding to a target mediastinal lymph node to be detected, and the CT image processing task is configured to detect whether cancer cells in the target mediastinal lymph node to be detected have metastasized to lymph nodes;
[0035] The multiple CT images are input into a computer-aided diagnosis model to obtain a computer-aided diagnosis result corresponding to the target mediastinal lymph node to be detected, wherein the computer-aided diagnosis model determines the computer-aided diagnosis result based on local image features and global image features of each CT image.
[0036] According to a tenth aspect of an embodiment of the present disclosure, a computer-aided diagnosis system for cancer cell metastasis to lymph nodes is provided, comprising:
[0037] a first receiving module, configured to receive a CT image processing task, wherein the CT image processing task carries a plurality of CT images corresponding to a target mediastinal lymph node to be detected, and the CT image processing task is configured to detect whether cancer cells in the target mediastinal lymph node to be detected have metastasized to lymph nodes;
[0038] An input module is used to input the multiple CT images into a computer-aided diagnosis model to obtain a computer-aided diagnosis result corresponding to the target mediastinal lymph node to be detected, wherein the computer-aided diagnosis model determines the computer-aided diagnosis result based on the local image features and global image features of each CT image.
[0039] An image processing method provided by an embodiment of the present disclosure receives an image processing task, wherein the image processing task carries multiple target images corresponding to a target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected; the multiple target images are input into an image processing model to obtain detection results corresponding to the target partition to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image.
[0040] In this way, by carrying multiple target images corresponding to the target to-be-detected partition in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target to-be-detected partition output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target to-be-detected partition, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] FIG1 is an architecture diagram of an image processing system provided by one embodiment of the present disclosure;
[0042] FIG2 is a flowchart of an image processing method provided by one embodiment of the present disclosure;
[0043] FIG3 is a structural diagram of an image processing model provided by an embodiment of the present disclosure;
[0044] FIG4 is a structural diagram of an image processing model provided by an embodiment of the present disclosure;
[0045] FIG5 is a structural diagram of an image processing model provided by an embodiment of the present disclosure;
[0046] FIG6 is a structural diagram of an image processing model provided by one embodiment of the present disclosure;
[0047] FIG7 is a structural diagram of an image processing model provided by one embodiment of the present disclosure;
[0048] FIG8 is a structural diagram of an image processing model provided by one embodiment of the present disclosure;
[0049] FIG9 is a structural diagram of an image processing model provided by one embodiment of the present disclosure;
[0050] FIG10 is a flowchart of a CT image processing method provided by one embodiment of the present disclosure;
[0051] FIG11 is a flowchart of a method for training an image processing model provided by one embodiment of the present disclosure;
[0052] FIG12 is a flowchart of an image processing method provided by one embodiment of the present disclosure;
[0053] FIG13 is a flowchart of a processing process of an image processing method provided by one embodiment of the present disclosure;
[0054] FIG14 is a schematic diagram of training an image processing model provided by one embodiment of the present disclosure;
[0055] FIG15 is a flow chart of a method for detecting esophageal cancer provided by one embodiment of the present disclosure;
[0056] FIG16 is a schematic structural diagram of an image processing device provided by one embodiment of the present disclosure;
[0057] FIG17 is a structural block diagram of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0058] The following description sets forth many specific details to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.
[0059] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a," "the," and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.
[0060] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0061] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0062] First, the terms involved in one or more embodiments of the present disclosure are explained.
[0063] CT (Computed Tomography): Computerized tomography uses a precisely collimated X-ray beam and a highly sensitive detector to perform cross-sectional scans around a certain part of the human body. It has the characteristics of fast scanning time and clear images, and can be used to detect a variety of diseases.
[0064] EC (esophageal cancer): Esophageal cancer is a highly lethal cancer with a low 5-year survival rate according to incomplete statistics. However, early detection of resectable / curable esophageal cancer can significantly reduce the mortality rate. Lymph node metastasis is a common and typical symptom.
[0065] Lymph nodes (LNs) are lymph nodes. Their primary functions include filtering lymph, removing bacteria and foreign matter, and producing lymphocytes and antibodies. Lymphatic vessels from various parts of the body or organs generally converge at nearby local lymph nodes. When a part of the body or organ becomes diseased or inflamed, foreign matter such as bacteria and toxins can spread through the lymph vessels to nearby lymph nodes. These local lymph nodes intercept and remove these foreign matter, acting as a defensive barrier to prevent the spread of disease. If these local lymph nodes fail to intercept and remove these bacteria or toxins, the disease can spread to distant sites along the efferent ducts of the local lymph nodes.
[0066] LNS (lymph node station): Within a human organ or tissue structure with a lymphatic system, each zone containing lymph nodes is divided into zones within the organ or tissue structure based on actual needs. For example, dividing the mediastinum into zones yields nine mediastinal lymph nodes. Different lymph nodes can have different zone numbers. For example, the nine mediastinal lymph nodes could be numbered: S1 (left + right), S2 (left + right), S3 anterior, S3 posterior, S4 (left + right), S5, S6, S7, and S8.
[0067] RECIST (response evaluation criteria in solid tumor) is a standard for evaluating the clinical efficacy of solid tumors. It determines the size and number of measurable lesions at baseline, standardizes measurement methods, and evaluates efficacy by changes in target lesions during treatment and follow-up.
[0068] CAD (computer aided diagnosis): Computer-aided diagnosis refers to the use of imaging, medical image processing technology and other possible physiological and biochemical means, combined with computer analysis and calculation, to assist in the discovery of lesions and improve the accuracy of diagnosis.
[0069] Deep-Stationing model: a model that can accurately segment lymph nodes.
[0070] 2.5D Mask R-CNN (2.5D Deep Learning Detection Framework): A two-stage framework. The first stage scans the image and generates regions that are likely to contain an object. The second stage classifies the region and generates a bounding box and mask. Mask R-CNN is a popular framework for instance segmentation.
[0071] AdamW optimizer: A model training optimizer in PyTorch that is set to gradient update the parameters of the neural network to minimize the loss function.
[0072] PyTorch: It is an environment that provides a runtime for building and training models.
[0073] Predicting lymph node (LN) metastasis on computed tomography (CT) scans is crucial for esophageal cancer staging, treatment planning, and prognosis. However, determining whether a single LN has metastasized based on CT images requires significant effort and expense from specialized medical personnel, and relies heavily on experience, which can lead to inaccurate positioning and potentially devastating consequences. Currently, RECIST and other criteria, such as morphological and textural features, can be used to predict LN metastasis. However, research reports indicate that the accuracy of these predictions remains low, making them inadequate for accurate prediction of LN metastasis.
[0074] With the continuous development of computer technology, various learning models have gradually become applicable to outcome prediction. With the remarkable success of deep learning in various medical imaging computer-aided diagnosis (CAD) tasks, the application of deep learning models to the prediction of lymph node metastasis has been proposed. However, deep learning models require training with large amounts of precisely labeled data to achieve accurate predictions. To predict lymph node metastasis, professional medical staff must determine whether individual lymph nodes have metastasized from various medical images (such as CT images and MRIs) based on pathology reports and label whether individual lymph nodes in these medical images are abnormal.
[0075] However, in current surgical practice for lymph node dissection, pathology reports only indicate the number of lymph nodes removed in each lymph node station and the number of metastatic lymph nodes found in the corresponding lymph node station (LNS). Therefore, even for experienced medical experts, establishing a one-to-one pairing between the lymph nodes observed in CT images and the individual metastatic lymph nodes indicated in the pathology report is extremely difficult. As the dataset expands, classifying individual lymph node metastases using pathologically confirmed labels may become unscalable. Therefore, deep learning models struggle to obtain large-scale, accurately labeled training data, resulting in an inability to output relatively accurate prediction results, making them difficult to apply in practical applications for the prediction of lymph node metastasis.
[0076] Based on this, an embodiment of the present disclosure provides an image processing method, which receives an image processing task, wherein the image processing task carries multiple target images corresponding to the target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected; the multiple target images are input into an image processing model to obtain detection results corresponding to the target partition to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0077] In this way, by carrying multiple target images corresponding to the target to-be-detected partition in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target to-be-detected partition output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target to-be-detected partition, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0078] Referring to FIG1 , FIG1 shows an architecture diagram of an image processing system provided by an embodiment of the present disclosure. The image processing system may include a client 100 and a server 200;
[0079] The client 100 is configured to send an image processing task to the server 200, wherein the image processing task carries multiple target images corresponding to the target partition to be detected, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target partition to be detected.
[0080] The server 200 is configured to receive image processing tasks, input multiple target images into the image processing model, and obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0081] The client 100 is further configured to receive the detection result of the image processing task sent by the server 200.
[0082] Applying the solution of the embodiment of the present disclosure, an image processing task is received, wherein the image processing task carries multiple target images corresponding to the target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected; the multiple target images are input into the image processing model to obtain the detection results corresponding to the target partition to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0083] In this way, by carrying multiple target images corresponding to the target to-be-detected partition in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target to-be-detected partition output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target to-be-detected partition, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0084] In practical applications, the image processing system may include multiple clients 100 and a server 200. Multiple clients 100 can establish communication connections through the server 200. In image processing scenarios, the server 200 is used to provide information extraction services between multiple clients 100. Multiple clients 100 can act as senders or receivers, respectively, and communicate through the server 200.
[0085] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In an image processing scenario, a user can publish a data stream to the server 200 through the client 100, and the server 200 generates detection results based on the data stream and pushes the detection results to other clients with which communication has been established.
[0086] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.
[0087] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, Hypertext Markup Language 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, personal computer or other end-side device. Various other types of applications can usually be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0088] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server (cloud-side device) that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.
[0089] It is worth noting that the image processing methods provided in the embodiments of the present disclosure are generally executed by the server. However, in other embodiments of the present disclosure, the client may also have similar functions to the server to execute the image processing methods provided in the embodiments of the present disclosure. In other embodiments, the image processing methods provided in the embodiments of the present disclosure may also be executed jointly by the client and the server.
[0090] In the present disclosure, an image processing method is provided. The present disclosure also relates to an image processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0091] Referring to FIG. 2 , FIG. 2 shows a flow chart of an image processing method provided by an embodiment of the present disclosure, which specifically includes the following steps:
[0092] Step 202: receiving an image processing task, wherein the image processing task carries a plurality of target images corresponding to the target to-be-detected partition, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target to-be-detected partition.
[0093] In actual applications, the image processing tasks sent by users can be received through the server or the client.
[0094] Specifically, the image processing task is used to detect whether there are abnormal objects to be detected in the target partition to be detected. The image processing task carries multiple target images corresponding to the target partition to be detected. Furthermore, the target partition to be detected can be understood as a partition used to predict whether there are abnormal objects to be detected. The objects to be detected can be in two states: normal or abnormal. That is, when the human body is healthy, the objects to be detected should be in a normal state. When there are symptoms in the target partition to be detected, the objects to be detected may be in an abnormal state. By predicting whether there are abnormal objects to be detected in the target partition to be detected, the state of the objects to be detected can be further judged based on the prediction results, thereby providing assistance for the positioning and precise treatment of abnormal objects to be detected.
[0095] It should be noted that in one or more embodiments of the present disclosure, the image processing task can be applied to the recognition of various types of medical images, and the determination of whether there are abnormal objects to be detected in the target partition to be detected in the medical image based on the image features. For example, in the application scenario of treating fractures, it is possible to predict whether a fracture has occurred in the target partition to be detected based on the medical image of the target partition to be detected, thereby locating the fracture site. In the application scenario of lymph node metastasis, it is possible to predict whether lymph node metastasis has occurred in the target partition to be detected based on the medical image of the target partition to be detected, thereby helping doctors to accurately locate and treat abnormal lymph nodes.
[0096] In one or more embodiments of the present disclosure, in order to improve the accuracy of the prediction results of the target partition to be detected and to make the prediction and positioning of abnormal objects to be detected more accurate, before receiving the image processing task, a specific area to be detected can also be partitioned. Specifically, the area to be detected can be understood as an organ, tissue, or structure of the human body, such as the esophagus, aortic arch, ascending aorta, heart, spine, mediastinum, etc. The area to be detected can be partitioned according to actual conditions, and the area to be detected can be divided into multiple partitions to be detected, and one of the multiple partitions to be detected can be determined as the target partition to be detected. It should be noted that determining a target partition to be detected from multiple partitions to be detected can be randomly determined or specified according to the needs of actual applications, and the present disclosure does not impose any restrictions on this.
[0097] For example, in the application scenario of predicting lymph node metastasis, assuming that the area to be detected is the mediastinal tissue of the human body, a CT image of the patient's entire mediastinal tissue can be collected. According to the 9 lymph nodes divided into the mediastinal tissue, one of the lymph nodes is selected as the target partition to be detected, and multiple target images corresponding to the target partition to be detected are carried into the image processing task for input, thereby realizing the prediction of whether there are abnormal objects to be detected in the target partition to be detected.
[0098] In practical applications, the object to be detected can be a certain type of cell, a certain type of tissue structure, etc. For example, depending on the actual application scenario, it can be a fracture area, a single lymph node, a single cancer cell, etc.
[0099] By receiving the image processing task, multiple target images corresponding to the target to-be-detected partition carried in the image processing task can be used as input and set to detect whether there is an abnormal to-be-detected object in the target to-be-detected partition.
[0100] Step 204: Input multiple target images into the image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0101] In practical applications, when receiving an image processing task, multiple target images carried by the image processing task can be obtained, and the multiple target images can be input into the image processing model to obtain the detection results corresponding to the target partitions to be detected output by the image processing model.
[0102] Specifically, the image processing model can extract the local image features and global image features of each target image based on the input multiple target images, and determine the detection results of the target partition to be detected based on the local image features and global image features of each target image. Furthermore, the detection results of the target partition to be detected can be understood as the prediction results of whether there are abnormal objects to be detected in the target partition to be detected. The detection result can be a probability value output by the image processing model. In practical applications, it can be specified as a value from 0 to 1. The closer the value output by the image processing model is to 0, the lower the probability of the presence of an abnormal object to be detected in the partition to be detected. The closer the value output by the image processing model is to 1, the higher the probability of the presence of an abnormal object to be detected in the partition to be detected.
[0103] In order to improve the task execution effect of the image processing model in performing the image processing task and improve the accuracy of the output detection results, in one or more embodiments of the present disclosure, the image processing model may include a feature extraction layer, a feature fusion layer and an output layer, wherein the feature fusion layer may be configured to fuse local image features and global image features of each target image;
[0104] Refer to Figure 3, which shows a structural diagram of an image processing model provided by an embodiment of the present disclosure. The model structure of the image processing model may include a feature extraction layer 302, a feature fusion layer 304 and an output layer 306. The feature extraction layer 302, the feature fusion layer 304 and the output layer 306 can be connected in sequence. Multiple target images can be first input into the feature extraction layer 302 to obtain the results processed by the feature extraction layer 302, and then the feature extraction layer 302 inputs the processed results into the feature fusion layer 304 to obtain the results processed by the feature fusion layer 304, and the feature fusion layer 304 inputs the processed results into the output layer 306 to obtain the detection results of the target partition to be detected that are finally output by the output layer.
[0105] Accordingly, inputting multiple target images into the image processing model to obtain detection results corresponding to the target partitions to be detected may include the following steps S3002 to S3006:
[0106] S3002: Input multiple target images into the feature extraction layer to obtain image feature information corresponding to each target image.
[0107] In practical applications, multiple target images can be input into the feature extraction layer to obtain image feature information corresponding to each target image.
[0108] Specifically, the feature extraction layer can be understood as a structure for extracting image feature information from the target image. Furthermore, the feature extraction layer can include one or more full convolution blocks, which are configured to map the image information in each target image through convolution to obtain image feature information corresponding to each target image.
[0109] Optionally, the feature extraction layer may include a full convolution block. Inputting multiple target images into the feature extraction layer to obtain image feature information corresponding to each target image may include the following steps:
[0110] A target image to be processed is determined from a plurality of target images;
[0111] The target image to be processed is input into the full convolution block to obtain image feature information.
[0112] Specifically, the target image to be processed can be any one of multiple target images. The multiple target images are input into the full convolution block of the feature extraction layer. The convolution structure in the full convolution block can map the image information and extract the image feature information corresponding to the multiple target images.
[0113] Optionally, in order to further reduce the number of network parameters, enhance feature expressiveness, and obtain better model performance, the feature extraction layer may include n sequentially connected full convolution blocks, where n is a positive integer greater than or equal to 2.
[0114] Accordingly, inputting multiple target images into the feature extraction layer to obtain image feature information corresponding to each target image may further include the following steps:
[0115] determining a target image to be processed among a plurality of target images;
[0116] Input the target image to be processed into the first full convolution block to obtain the first image feature information;
[0117] Input the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information, where 2≤i≤n;
[0118] Determine whether i is equal to n;
[0119] If not, then i is incremented by 1, and the operation of inputting the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information is continued;
[0120] If so, the i-th image feature information is determined as the image feature information corresponding to the target image to be processed.
[0121] Referring to FIG. 4 , FIG. 4 shows a structural diagram of an image processing model provided by an embodiment of the present disclosure.
[0122] Specifically, the feature extraction layer may include a plurality of full convolution blocks 400 connected in sequence, determine a target image to be processed among a plurality of target images, input the target image to be processed into the first full convolution block 400, and obtain the first image feature information output by the first full convolution block 400; input the first image feature information into the second full convolution block 400, and obtain the second image feature information output by the second full convolution block 400... and so on, until the n-1th image feature information is input into the nth full convolution block 400, and the nth image feature information output by the nth full convolution block 400 is obtained, and the nth image feature information is determined as the image feature information corresponding to the target image to be processed.
[0123] In practical applications, the image feature information corresponding to multiple target images can be obtained according to the above method.
[0124] By inputting multiple target images into one or more full convolution blocks in the feature extraction layer, the image feature information corresponding to the multiple target images can be obtained for subsequent feature processing to obtain the prediction of the detection results of the target partition to be detected.
[0125] In practical applications, image feature information corresponding to multiple target images is obtained, and each image feature information can be input into a feature fusion layer to obtain image fusion information corresponding to each target image.
[0126] S3004: Input the feature information of each image into the feature fusion layer to obtain the image fusion information corresponding to each target image.
[0127] Specifically, the feature fusion layer is configured to fuse the local image features and global image features of each target image. By inputting the feature information of each image into the feature fusion layer, the local features and global features in each image feature information can be extracted separately, thereby obtaining local-global fusion information based on the extracted local and global features.
[0128] Since the local feature information of the image extracted by the convolution layer is likely to lose information at a fine scale during the process of encoding the global feature information by the codec, in order to encode the feature information of the image without losing the local feature information, in one or more embodiments of the present disclosure, the feature information of each image is input into the feature fusion layer, the local feature information and the global feature information in each image feature information are extracted respectively, and the local feature information and the global feature information are fused to obtain the image fusion information corresponding to each target image, thereby improving the accuracy of the feature information and further improving the accuracy of the model output results.
[0129] Furthermore, in one or more embodiments of the present disclosure, the feature fusion layer may include one or more local-global fusion blocks to extract and fuse local feature information and global feature information in the image feature information respectively.
[0130] Optionally, the feature fusion layer may include a local-global fusion block, and inputting the feature information of each image into the feature fusion layer to obtain the image fusion information corresponding to each target image may include the following steps:
[0131] Determining target image feature information from each image feature information;
[0132] The target image feature information is input into the local-global fusion block to obtain image fusion information.
[0133] Specifically, the target image feature information is any one of the image feature information. The target image feature information is input into the local-global fusion block, and the obtained image fusion information is more accurate.
[0134] Optionally, the feature fusion layer may include m local-global fusion blocks connected sequentially, where m is a positive integer greater than or equal to 2.
[0135] Accordingly, inputting the feature information of each image into the feature fusion layer to obtain the image fusion information corresponding to each target image may include the following steps:
[0136] Determining target image feature information from each image feature information;
[0137] Inputting the target image feature information into the first local-global fusion block to obtain the first image fusion information;
[0138] Input the j-1th image fusion information into the jth local-global fusion block to obtain the jth image fusion information, where 2≤j≤m;
[0139] Determine whether j is equal to m;
[0140] If not, then j is incremented by 1, and the operation of inputting the j-1th image fusion information into the jth local-global fusion block to obtain the jth image fusion information is continued;
[0141] If so, the j-th image fusion information is determined as the image fusion information corresponding to the target image feature information.
[0142] Referring to FIG5 , FIG5 shows a structural diagram of an image processing model provided by an embodiment of the present disclosure;
[0143] Specifically, the feature fusion layer may include a plurality of local-global fusion blocks 500 connected in sequence, determine the target image feature information from the feature information of each image, input the target image feature information into the first local-global fusion block 500, and obtain the first image fusion information output by the first local-global fusion block 500; input the first image fusion information into the second local-global fusion block 500, and obtain the second image fusion information output by the second local-global fusion block 500... and so on, until the m-1th image fusion information is input into the mth local-global fusion block 500, and obtain the mth image fusion information output by the mth local-global fusion block 500, and determine the mth image fusion information as the image fusion information corresponding to the target image feature information.
[0144] In practical applications, the image fusion information corresponding to each image feature information can be obtained according to the above method.
[0145] By inputting the feature information of each image into one or more local-global fusion blocks in the feature fusion layer, the image fusion information corresponding to multiple target images can be obtained. The global features in the image feature information can be encoded while ensuring the accuracy of the local feature information, which can be used for subsequent feature processing to obtain the prediction of the detection results of the target partition to be detected, thereby improving the accuracy of the output results of the image processing model.
[0146] In one or more embodiments of the present disclosure, the local-global fusion block may include at least one local-global feature fusion layer, and the local-global feature fusion layer may include a local feature extraction sublayer, a global feature extraction sublayer, and a fusion layer.
[0147] Specifically, the local-global feature fusion layer is configured to extract local feature information and global feature information from the feature information of each image, and fuse the local feature information and the global feature information to obtain image fusion information corresponding to each target image. Furthermore, the local feature extraction sublayer is configured to extract local feature information corresponding to each target image, the global feature extraction sublayer is configured to extract global feature information corresponding to each target image, and the fusion layer is configured to fuse the extracted local feature information and global feature information to obtain image fusion information corresponding to each target image.
[0148] Optionally, a local-global fusion block may include a local-global feature fusion layer. In order to improve the accuracy of feature information and train a more accurate image processing model, a local-global fusion block may also include multiple local-global feature fusion layers connected in sequence.
[0149] Accordingly, for the t-th local-global feature fusion layer, the following steps may be included:
[0150] Acquire feature information of the image to be processed, wherein the feature information of the image to be processed includes feature information of the target image or fusion information of the t-1th image;
[0151] Input the feature information of the image to be processed into the local feature extraction sublayer to obtain the local feature information corresponding to the feature information of the image to be processed;
[0152] Input the feature information of the image to be processed into the global feature extraction sublayer to obtain the global feature information corresponding to the feature information of the image to be processed;
[0153] The local feature information and the global feature information are input into the fusion layer to obtain the image fusion information corresponding to the t-th local-global feature fusion layer.
[0154] Specifically, the image feature information to be processed includes target image feature information or the t-1th image fusion information, wherein the target image feature information is the image feature information output by the feature extraction layer and input to the feature fusion layer; the t-1th image fusion information is the image fusion information output by the t-1th local-global feature fusion layer.
[0155] 6 , which shows a structural diagram of an image processing model provided by an embodiment of the present disclosure, wherein the local-global feature fusion layer 600 may include a local feature extraction sublayer 6002 , a global feature extraction sublayer 6004 , and a fusion layer 6006 .
[0156] The local-global feature fusion layer 600 can receive feature information X of the image to be processed, and input the feature information X of the image to be processed into the local feature extraction sublayer 6002 and the global feature extraction sublayer 6004 respectively, and extract the local feature information of the feature information X of the image to be processed through the local feature extraction sublayer 6002; extract the global feature information of the feature information X of the image to be processed through the global feature extraction sublayer 6004, and then input the local feature information and the global feature information into the fusion layer 6006, and obtain the local-global fusion information of the feature information X of the image to be processed through the fusion layer 6006.
[0157] In practical applications, by inputting the feature information of the image to be processed into the local-global feature fusion layer within the local-global fusion block, local and global feature information can be extracted from the image to be processed. These information is then fused to produce the fused image information output by the local-global feature fusion layer. By extracting and fusing the local and global feature information separately, a global feature encoding can be obtained without losing fine-scale information, thereby improving the accuracy of the image processing model.
[0158] In one or more embodiments of the present disclosure, the local feature extraction sublayer may include a convolution processing unit.
[0159] Inputting the feature information of the image to be processed into the local feature extraction sublayer to obtain the local feature information corresponding to the feature information of the image to be processed may include the following steps:
[0160] Inputting the feature information of the image to be processed into the convolution processing unit, and extracting the initial local feature information corresponding to the feature information of the image to be processed;
[0161] Perform feature mapping on the initial local feature information to obtain local feature information corresponding to the feature information of the image to be processed.
[0162] Referring to FIG. 7 , FIG. 7 shows a structural diagram of an image processing model provided by an embodiment of the present disclosure.
[0163] Specifically, the local feature extraction sublayer 700 may include a convolution processing unit 7002 and a first convolution kernel 7004. The convolution processing unit 7002 may be understood as a convolution kernel for extracting local feature information. The feature information of the image to be processed is input into the convolution processing unit 7002, which extracts and outputs initial local feature information corresponding to the feature information of the image to be processed. The initial local feature information may be understood as feature information of the same dimension as the feature information of the image to be processed.
[0164] In order to facilitate data classification of local feature information and obtain feature information with a finer scale, in one or more embodiments of the present disclosure, a first convolution kernel 7004 may be connected after the convolution processing unit 7002. The first convolution kernel 7004 is used to perform feature mapping on the initial local feature information to obtain local feature information corresponding to the feature information of the image to be processed.
[0165] Optionally, the local feature information may be feature information of a higher dimension than the initial local feature information. For example, the dimension of the feature information X to be processed is c-dimensional, and the dimension of the initial local feature information obtained by the convolution processing unit 7002 is also c-dimensional. After feature mapping is performed by the first convolution kernel 7004, the dimension of the obtained local feature information may be d-dimensional.
[0166] Optionally, the convolution processing unit 7002 may be a Conv3*3*3 convolution unit, and the first convolution kernel 7004 may be a Conv1*1*1 convolution kernel.
[0167] By inputting the feature information of the image to be processed into the local feature extraction sublayer, and obtaining the local feature information corresponding to the feature information of the image to be processed through the convolution processing unit and the first convolution kernel, the fineness of the image feature information extraction can be improved, and more accurate image processing results can be obtained, thereby improving the accuracy of the image processing model.
[0168] In one or more embodiments of the present disclosure, the global feature extraction sublayer may include a codec image processing unit.
[0169] Inputting the feature information of the image to be processed into the global feature extraction sublayer to obtain the global feature information corresponding to the feature information of the image to be processed may include the following steps:
[0170] Perform feature mapping on the feature information of the image to be processed to obtain the feature information of the target image to be processed corresponding to the feature information of the image to be processed;
[0171] The target image to be processed feature information is input into the encoding and decoding image processing unit to obtain global feature information corresponding to the target image to be processed feature information.
[0172] In practical applications, in order to facilitate the splicing and fusion of local feature information and global feature information, before extracting the global feature information through the codec image processing unit, the feature information of the image to be processed can be mapped to feature information with the same dimension as the local feature information.
[0173] Specifically, the target image feature information to be processed is feature information of a higher dimension than the feature information of the image to be processed obtained after feature mapping. The codec image processing unit has a coding-decoding structure and can extract global feature information from the target image feature information to be processed.
[0174] Referring to FIG8 , FIG8 shows a structural diagram of an image processing model provided by an embodiment of the present disclosure.
[0175] Specifically, the global feature extraction sublayer 800 may include a second convolution kernel 8002 and a codec image processing unit 8004. A second convolution kernel 8002 may be connected before the codec image processing unit 8004. After the feature information of the image to be processed is input into the global feature extraction sublayer, it may first be mapped into target image feature information of a higher dimension to be processed through the second convolution kernel 8002, and then the target image feature information to be processed is input into the codec image processing unit 8004. The encoder and decoder in the codec image processing unit 8004 extract the global feature information from the target image feature information to be processed, thereby obtaining the global feature information of the image feature information to be processed.
[0176] Optionally, the second convolution kernel 8002 can be a Conv1*1*1 convolution kernel, and the codec image processing unit 8004 can be a transformer processing unit, which can extract global feature information from the target image feature information through the Unfold->transformer->Fold method.
[0177] By inputting the feature information of the image to be processed into the global feature extraction sublayer, and obtaining the global feature information corresponding to the feature information of the image to be processed through the second convolution kernel and the codec image processing unit, the integrity of the image feature information extraction can be improved, and more comprehensive image feature information can be obtained, thereby improving the reliability and accuracy of the image processing model.
[0178] In practical applications, after obtaining local feature information and global feature information, the local feature information and the global feature information can be input into the fusion layer to obtain image fusion information.
[0179] In one or more embodiments of the present disclosure, inputting local feature information and global feature information into a fusion layer to obtain image fusion information corresponding to the t-th local-global feature fusion layer may include the following steps:
[0180] Splice local feature information and global feature information to obtain initial image fusion information;
[0181] Perform feature mapping on the initial image fusion information to obtain the image fusion information corresponding to the t-th local-global feature fusion layer.
[0182] Specifically, the initial image fusion information is the feature information in the same dimension as the local feature information and the global feature information obtained by splicing the local feature information and the global feature information. The image fusion information can be understood as feature mapping the initial image fusion information to obtain feature information with the same dimension as the feature information of the image to be processed.
[0183] Referring to FIG9 , FIG9 shows a structural diagram of an image processing model provided by an embodiment of the present disclosure;
[0184] Specifically, fusion layer 900 includes a splicing unit 9002 and a third convolution kernel 9004. Splicing unit 9002 is configured to receive local feature information and global feature information output by the local feature extraction sublayer and the global feature extraction sublayer, and splice the local feature information and the global feature information to obtain initial image fusion information. Third convolution kernel 9004 receives the initial image fusion information output by splicing unit 9002, performs feature mapping on the initial image fusion information, and obtains image fusion information corresponding to the t-th local-global feature fusion layer. Optionally, third convolution kernel 9004 can be a Conv1*1*1 convolution kernel.
[0185] By inputting local feature information and global feature information into the fusion layer, splicing them through the fusion layer to obtain initial image fusion information, and performing feature mapping on the initial image fusion information to obtain image fusion information, the local information of the target image at a finer scale and the more complete global information can be fused to obtain more accurate image feature information, which can make the prediction results output by the image processing model more accurate.
[0186] In practical applications, the image fusion information corresponding to each target image is obtained, and the image fusion information can be input into the output layer to obtain the detection result corresponding to the target partition to be detected output by the output layer.
[0187] S3006: Input the image fusion information into the output layer to obtain the detection result corresponding to the target partition to be detected.
[0188] Specifically, the output layer can calculate and output the detection results corresponding to the target partition to be detected based on the image fusion information corresponding to each target image. Furthermore, the output layer can classify the image fusion information through a softmax (normalization) operation to obtain the predicted probability of the presence of an abnormal object to be detected in the target partition to be detected, and output it as the detection result corresponding to the target partition to be detected.
[0189] Multiple target images are input into the image processing model, and the input multiple target images are processed into image feature information through the feature extraction layer of the image processing model. The image feature information is processed into image fusion information through the feature fusion layer. The image fusion information is then processed through the output layer to output the detection result corresponding to the target to-be-detected partition. This can improve the accuracy of predicting whether there are abnormal objects to be detected in the target to-be-detected partition, avoid inaccurate judgment results caused by human factors, and improve the detection efficiency of the target to-be-detected partition and the objects to be detected, thereby providing assistance for further medical diagnosis and treatment of the objects to be detected.
[0190] With the continuous development of computer technology, deep learning has gradually been applied to various medical imaging computer-aided diagnosis (CAD) tasks. However, deep learning models rely on large-scale, accurately labeled training samples and sample labels as training data for model training. For image processing application scenarios where it is difficult to match the abnormal states of the objects to be detected one by one from medical images, obtaining large amounts of accurate training data requires a high cost, making model training more difficult.
[0191] Based on this, in one or more embodiments of the present disclosure, the image processing model is obtained through the following training S20402-S20410:
[0192] S20402: Obtain training sample pairs and guidance information of the training sample pairs, wherein the training sample pairs include training samples and sample labels, the training samples include multiple sample images corresponding to the target partition to be detected, and the sample labels are set to identify sample results of the target partition to be detected.
[0193] Specifically, the training sample pair includes a training sample and a sample label. The training sample can be understood as a plurality of sample images corresponding to the target partition to be detected, and the sample label is set to identify the sample result of the target partition to be detected.
[0194] In practical applications, to increase the amount of training data for model training, reduce annotation costs, and improve annotation accuracy, the model training method for the image processing model provided in one or more embodiments of the present disclosure can annotate whether an abnormal object to be detected exists in the target partition to be detected in the medical image based on the pathology report. In other words, the sample result can be understood as whether an abnormal object to be detected exists in the target partition to be detected. The sample label of the sample image with an abnormal object to be detected can be set to 1, and the sample label of the sample image without an abnormal object to be detected can be set to 0.
[0195] Since in the training sample pair, the training sample is the target partition to be detected for anomaly detection, and the sample label is the probability of whether there is an abnormal object to be detected in the target partition to be detected, and the target partition to be detected may also contain other human cells or tissues and structures other than the object to be detected, which may interfere with the model training, the model training can be implicitly or explicitly guided by guidance information. The guidance information of the training sample pair can be understood as information that guides the attention of the image processing model in the process of model training based on the training sample pair. Furthermore, the guidance information can be understood as the position information of the object to be detected in the target partition to be detected. Through the guidance information, the model's attention can be focused on the object to be detected in the target partition to be detected, preventing the model from being interfered with by other objects in the target partition to be detected other than the object to be detected, resulting in inaccurate training results.
[0196] In order to obtain more accurate guidance information and thus improve the accuracy of model training, in one or more real-time examples of the present disclosure, obtaining guidance information for training sample pairs may include the following steps:
[0197] Determining the position information of the object to be detected in the target to be detected area of each sample image;
[0198] The position information of the object to be detected is used as the guidance information corresponding to each sample image.
[0199] It should be noted that in order to focus the model's attention on the objects to be detected in the target detection partition of each sample image and improve the accuracy of model training, the position information of the objects to be detected in the target detection partition can be determined based on the medical image, and the position information of the objects to be detected can be used as the guidance information corresponding to each sample image. Alternatively, a depth detection model corresponding to the objects to be detected can be pre-trained. Using the depth detection model of the objects to be detected, all visible objects to be detected in the target detection partition of the medical image can be automatically detected and segmented to obtain mask information corresponding to the objects to be detected. The mask information obtained by the above method can be used as the guidance information corresponding to each sample image.
[0200] S20404: Input the training sample pairs and guidance information into the initial image processing model to obtain the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected.
[0201] In practical applications, after obtaining the guidance information of the training sample pairs, the training sample pairs and the guidance information can be input into the initial image processing model. After processing by the initial image processing model, the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected can be obtained.
[0202] Specifically, the initial image processing model can be understood as an image processing model that has not yet completed training, the sample prediction result can be understood as the predicted probability that there are abnormal objects to be detected in the training samples output by the initial image processing model based on the input training sample pairs and guidance information, and the target attention information can be understood as the location information of the object to be detected in the target partition to be detected.
[0203] In one or more embodiments of the present disclosure, the image processing model may include a feature extraction layer, a feature fusion layer, and an output layer.
[0204] Accordingly, the training sample pairs and the guidance information are input into the initial image processing model to obtain the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected, which may include the following steps:
[0205] Splicing training samples and guidance information to obtain target splicing information;
[0206] Input the target splicing information into the feature extraction layer to obtain the feature extraction information output by the feature extraction layer;
[0207] Inputting the feature extraction information into the feature fusion layer to obtain feature fusion information output by the feature fusion layer and at least one attention feature map;
[0208] Input the feature fusion information into the output layer to obtain the sample prediction results corresponding to the target partition to be detected;
[0209] Aggregate at least one attention feature map to obtain target attention information corresponding to the target partition to be detected.
[0210] Specifically, the image processing model can splice the training samples and guidance information in the training sample pair to obtain target splicing information. The target splicing information can be understood as the training sample that has been integrated with the position information of the object to be detected. Feature extraction information can be understood as the feature information obtained by the feature extraction layer performing feature extraction based on the input target splicing information. Feature fusion information can be understood as the feature information obtained by the feature fusion layer performing local feature information extraction, global feature information extraction, and fusing local feature information and global feature information based on the input feature extraction information. The attention feature map can be understood as the intermediate product output by each local-global fusion block in the feature fusion layer during the process of processing feature information. The attention feature map can be used to represent the position information of the object to be detected in the target partition to be detected.
[0211] It should be noted that each local-global fusion block in the feature fusion layer may include at least one local-global feature fusion layer. In order to improve the accuracy of training and obtain a more accurate attention feature map, in one or more embodiments of the present disclosure, the attention feature map output by the last local-global feature fusion layer in each local-global fusion block can be obtained.
[0212] In practical applications, after obtaining the feature fusion information and at least one attention feature map output by the feature fusion layer, the feature fusion information can be input into the output layer to obtain the sample prediction result corresponding to the target partition to be detected; and at least one attention feature map can be aggregated to obtain the target attention information corresponding to the target partition to be detected.
[0213] Specifically, the sample prediction result is the prediction result output by the initial image processing model on whether there is an abnormal object to be detected in the target partition to be detected. The prediction result can be set as: the higher the probability of detecting that there is an abnormal object to be detected in the target partition to be detected, the closer the prediction result is to 1; the lower the probability of detecting that there is an abnormal object to be detected in the target partition to be detected, the closer the prediction result is to 0.
[0214] In practical applications, aggregating at least one attention feature map to obtain the target attention information corresponding to the target partition to be detected can be achieved by interpolating each attention feature map to the same size as the attention feature map output by the first local-global fusion block, thereby realizing attention aggregation of at least one attention feature map and obtaining the target attention information of the target partition to be detected.
[0215] By splicing training samples and guidance information, the target splicing information is input into the initial image processing model, and the guidance information is used to explicitly guide the model training process, thereby improving the accuracy of model training, thereby improving the accuracy of the detection results output by the model, and improving the task processing effect of the model when performing image processing tasks. By obtaining the attention feature map output by the feature fusion layer during training and aggregating the attention feature map to obtain the target attention information corresponding to the target partition to be detected, it can be used to subsequently calculate the model loss value through comparison of the target attention information and the guidance information, adjust the model parameters, and achieve implicit guidance of the model training process.
[0216] S20406: Calculate a first loss value of the initial image processing model based on the sample prediction result and the sample result.
[0217] Specifically, the first loss value can be understood as the error between the model task execution result and the actual result obtained by comparing the prediction result output by the initial image processing model with the sample label.
[0218] In practical applications, the sample prediction results and sample results can be compared, and the first loss value of the initial image processing model can be calculated through methods such as cross entropy calculation.
[0219] S20408: Calculate the second loss value of the initial image processing model based on the target attention information and the guidance information.
[0220] Specifically, the second loss value can be understood as the model error obtained by comparing the attention position information of the object to be detected in the target detection partition with the position information that the model should actually pay attention to during the model training process based on the initial image processing model.
[0221] In practical applications, the target attention information and the guidance information can be compared, and the second loss value of the initial image processing model can be calculated through methods such as cross entropy calculation.
[0222] S20410: Adjust the model parameters of the initial image processing model according to the first loss value and the second loss value until a preset stop condition is reached to obtain the image processing model.
[0223] Specifically, the preset stopping condition can be the preset thresholds corresponding to the first loss value and the second loss value respectively, or it can be the preset number of model cycle training. It can be determined according to actual conditions, and this disclosure does not impose any restrictions on this.
[0224] In actual applications, the first loss value and the second loss value of the initial image processing model are calculated, and the parameters of the initial image processing model can be adjusted according to the first loss value and the second loss value. The parameter adjustment method can be, for example, reverse transfer parameter adjustment, which can be determined according to the situation in the actual application. This disclosure does not impose any restrictions on this.
[0225] By splicing the guidance information with the training samples and calculating the second loss value through the guidance information and the target attention information, it is possible to achieve explicit and implicit guidance of the model training through the guidance information, thereby avoiding the model from being interfered with by other objects other than the objects to be detected in the target to be detected partition, improving the accuracy of model training, and thus improving the accuracy of the output prediction results when the image processing model performs image processing tasks.
[0226] An image processing method provided by an embodiment of the present disclosure receives an image processing task, wherein the image processing task carries multiple target images corresponding to a target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected; the multiple target images are input into an image processing model to obtain detection results corresponding to the target partition to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image.
[0227] In this way, by carrying multiple target images corresponding to the target to-be-detected partition in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target to-be-detected partition output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target to-be-detected partition, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0228] Referring to FIG10 , FIG10 shows a flow chart of a CT image processing method provided by an embodiment of the present disclosure, which specifically includes the following steps:
[0229] Step 1002: Receive a CT image processing task, wherein the CT image processing task carries multiple CT images corresponding to the target partition to be detected, and the CT image processing task is set to detect whether there are abnormal cells to be detected in the target partition to be detected.
[0230] Step 1004: Input multiple CT images into a CT image processing model to obtain detection results corresponding to the target partition to be detected, wherein the CT image processing model determines the detection results based on local image features and global image features of each CT image.
[0231] It should be noted that the implementation of steps 1002 to 1004 is the same as the implementation of steps 202 to 204 described above, and will not be described in detail in this embodiment of the present disclosure.
[0232] For example, assuming that the CT image processing task is to determine whether there is lymph node metastasis in the esophageal cancer (EC) lymph node station, the above-mentioned CT image processing method can be used to determine the probability of lymph node metastasis in the target lymph node station to be detected.
[0233] By applying the solution of the embodiment of the present disclosure, by carrying multiple target images corresponding to the target partition to be detected in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target partition to be detected output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target partition to be detected, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0234] Referring to FIG11 , FIG11 shows a flowchart of a method for training an image processing model provided by an embodiment of the present disclosure, which is applied to a cloud-side device and specifically includes the following steps:
[0235] Step 1102: Obtain training sample pairs and guidance information of the training sample pairs, wherein the training sample pairs include training samples and sample labels, the training samples include multiple sample images corresponding to the target partition to be detected, and the sample labels are set to identify the sample results of the target partition to be detected.
[0236] Step 1104: Input the training sample pairs and guidance information into the initial image processing model to obtain the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected.
[0237] Step 1106: Calculate the first loss value of the initial image processing model based on the sample prediction result and the sample result.
[0238] Step 1108: Calculate the second loss value of the initial image processing model based on the target attention information and the guidance information.
[0239] Step 1110: Adjust the model parameters of the initial image processing model according to the first loss value and the second loss value until a preset stop condition is reached, thereby obtaining the model parameters of the image processing model.
[0240] Step 1112: Send the model parameters of the image processing model to the end-side device.
[0241] It should be noted that the implementation method of steps 1102 to 1110 is the same as that of the above-mentioned S20402 to S20410, and will not be repeated in this embodiment of the present disclosure.
[0242] In actual applications, after the cloud-side device sends the model parameters of the image processing model to the end-side device, the end-side device can build the image processing model locally according to the model parameters of the image processing model, and further use the image processing model to perform image processing.
[0243] By applying the solution of the embodiment of the present disclosure, by obtaining training sample pairs and guidance information of the training sample pairs, the training samples and the guidance information are input into the initial image processing model, and after splicing, feature extraction and fusion of local and global feature information are performed, and the sample prediction results are output. The first loss value of the initial image processing model is calculated based on the comparison of the sample prediction results and the sample results. The second loss value of the initial image processing model is calculated by comparing the intermediate product target attention information output by the initial image processing model with the guidance information. The model parameters are adjusted according to the first loss value and the second loss value, which can achieve implicit and explicit guidance of the model through the guidance information, and can make the final image processing model more accurate.
[0244] Referring to FIG12 , FIG12 shows a flowchart of an image processing method provided by an embodiment of the present disclosure, which specifically includes the following steps:
[0245] Step 1202: Receive an image processing request sent by a user, wherein the image processing request includes an image processing task, the image processing task carries multiple target images corresponding to the target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected.
[0246] Step 1204: Input multiple target images into the image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on the local image features and global image features of each target image.
[0247] Step 1206: Send the detection result corresponding to the target partition to be detected to the user.
[0248] It should be noted that the specific implementation of steps 1202 to 1204 is the same as the implementation of steps 202 to 204 described above, and will not be described in detail in this embodiment of the present disclosure.
[0249] By applying the solution of the embodiment of the present disclosure, by carrying multiple target images corresponding to the target partition to be detected in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target partition to be detected output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target partition to be detected, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0250] The following further illustrates the image processing method provided by the present disclosure using the application of the image processing method in detecting abnormal lymph node stations in esophageal cancer as an example, with reference to FIG13 . FIG13 shows a flowchart of the processing process of an image processing method provided by one embodiment of the present disclosure, which specifically includes the following steps:
[0251] Step 1302: Receive a mediastinal lymph node CT image processing task, wherein the mediastinal lymph node CT image processing task carries multiple CT images corresponding to the target mediastinal lymph node to be detected, and the mediastinal lymph node CT image processing task is set to detect whether there is lymph node metastasis in the target mediastinal lymph node to be detected.
[0252] In practical applications, a CT image processing task of a mediastinal lymph node station is received, and multiple CT images corresponding to the target mediastinal lymph node station to be detected are input into the image processing model. The image processing model can then obtain a detection result of whether lymph node metastasis occurs in the target mediastinal lymph node station to be detected based on the output of the multiple CT images.
[0253] Step 1304: Input the multiple CT images into the feature extraction layer included in the image processing model to obtain image feature information corresponding to each CT image, wherein the feature extraction layer includes two fully convolutional blocks connected in sequence.
[0254] Exemplarily, as shown in FIG14 , the image processing model includes a feature extraction layer, a feature fusion layer, and an output layer, wherein the feature extraction layer, the feature fusion layer, and the output layer are sequentially connected. Inputting multiple CT images into the feature extraction layer may include the following steps: inputting the multiple CT images into full convolution block 1, obtaining image feature information 1 output by full convolution block 1, inputting image feature information 1 into full convolution block 2, obtaining image feature information 2 output by full convolution block 2, and determining image feature information 2 as the image feature information output by the feature extraction layer.
[0255] Step 1306: Input the feature information of each image into the feature fusion layer included in the image processing model to obtain image fusion information corresponding to each CT image, wherein the feature fusion layer includes 3 local-global fusion blocks connected in sequence, and each local-global fusion block includes at least one local-global feature fusion layer.
[0256] For example, as shown in Figure 14, the feature fusion layer includes three sequentially connected local-global fusion blocks: 1, 2, and 3. Each local-global fusion block includes at least one local-global feature fusion layer. Each local-global feature fusion layer includes a local feature extraction sublayer, a global feature extraction sublayer, and a fusion layer. Furthermore, the local feature extraction sublayer includes a convolution subunit, and the global feature extraction sublayer includes a codec image processing unit.
[0257] For example, assuming that local-global fusion block 1 includes two local-global feature fusion layers, each image feature information is input into local-global feature fusion layer 1 within local-global fusion block 1. Each image feature information is then input into the local feature extraction sublayer and the global feature extraction sublayer within local-global feature fusion layer 1, respectively. Local feature information is obtained using the convolution subunit within the local feature extraction sublayer. Global feature information is obtained using the codec image processing unit within the global feature extraction sublayer. The local feature information and global feature information are fused through the fusion layer to obtain image fusion information. The local-global fusion information is then input into local-global feature fusion layer 2. Image fusion information 1 is then output from local-global fusion block 1 in the same manner as described above. After obtaining image fusion information 1 output from local-global fusion block 1, image fusion information 1 can be input into local-global fusion block 2 to obtain image fusion information 2 output from local-global fusion block 2. Image fusion information 2 is then input into local-global fusion block 3 to obtain image fusion information 3 output from local-global fusion block 3. Image fusion information 3 is then determined as the image fusion information output from the feature fusion layer.
[0258] Step 1308: Input the image fusion information into the output layer of the image processing model to obtain the detection results corresponding to the target mediastinal lymph node station to be detected.
[0259] In practical applications, the image fusion information is input into the output layer of the image processing model, and the image fusion information can be softmax (normalized) processed through the output layer to obtain the detection results corresponding to the target mediastinal lymph node station to be detected output by the image processing model.
[0260] Specifically, the closer the result to be detected is to 0, the lower the possibility that lymph node metastasis exists in the target mediastinal lymph node station to be detected, and the closer the result to be detected is to 1, the higher the possibility that lymph node metastasis exists in the target mediastinal lymph node station to be detected.
[0261] By applying the solution of the embodiment of the present disclosure, by carrying multiple target images corresponding to the target partition to be detected in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target partition to be detected output by the image processing model can be obtained, thereby enabling automatic detection of whether there are abnormal objects to be detected in the target partition to be detected, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0262] The following further illustrates the image processing model training method provided by the present disclosure, using the application of the image processing model training method in esophageal cancer lymph node abnormality detection as an example, with reference to FIG14 . FIG14 shows a schematic diagram of an image processing model training method provided by one embodiment of the present disclosure.
[0263] CT images and pathology reports of the target sites of esophageal cancer patients are obtained. The target mediastinal lymph node stations are determined from the CT images of the target sites. The multiple CT images corresponding to the target mediastinal lymph node stations are used as training samples for the image processing model. Sample labels are annotated for each CT image based on the pathology reports. The sample label corresponding to the training sample with lymph node metastasis in the target mediastinal lymph node station is 1, and the sample label corresponding to the training sample without lymph node metastasis in the target mediastinal lymph node station is 0.
[0264] Through the pre-trained deep lymph node detection model, all visible lymph nodes (short diameter ≥ 5mm) in the above CT images are automatically detected and segmented. The lymph node masks obtained by these automatic segmentations are used as the guidance information of the attention prior to guide the model learning.
[0265] The constructed training sample pairs and the guidance information corresponding to the training sample pairs are used as the input of the image processing model and input into the initial image processing model to be trained. The initial image processing model receives the training sample pairs and the guidance information, first splices the training samples and the guidance information to obtain sample splicing information, and then inputs the sample splicing information into two sequentially connected full convolution blocks (full convolution block 1, full convolution block 2) to obtain the image feature information output by the second full convolution block; then the image feature information is input into three sequentially connected local-global fusion blocks (local-global fusion block 1, local-global fusion block 2, local-global fusion block 3) to obtain the image fusion information output by the third local-global fusion block, wherein the local-global fusion block 1 includes 2 local-global feature fusion layers, the local-global fusion block 2 includes 4 local-global feature fusion layers, and the local-global fusion block 3 includes 3 local-global feature fusion layers.
[0266] In the process of obtaining the initial image processing model to process the image feature information through three local-global fusion blocks, the attention feature map output by the last local-global feature fusion layer of each local-global fusion block is interpolated to the same size, and the target attention information corresponding to the three attention feature maps is aggregated.
[0267] The image fusion information output by the third local-global fusion block is output to the output layer to obtain the probability of lymph node metastasis in the mediastinal lymph node station to be detected output by the initial image processing model, and this probability value is determined as the sample prediction result.
[0268] Based on the sample prediction results and sample labels, a first loss value of the initial image processing model is calculated. Based on the target attention information and guidance information, a second loss value of the initial image processing model is calculated. The model parameters of the initial image model are adjusted based on the first and second loss values until a preset stopping condition is met, thereby obtaining an image processing model.
[0269] By applying the solution of the embodiment of the present disclosure, by obtaining training sample pairs and guidance information of the training sample pairs, the training samples and the guidance information are input into the initial image processing model, and after splicing, feature extraction and fusion of local and global feature information are performed, and the sample prediction results are output. The first loss value of the initial image processing model is calculated based on the comparison of the sample prediction results and the sample results. The second loss value of the initial image processing model is calculated by comparing the intermediate product target attention information output by the initial image processing model with the guidance information. The model parameters are adjusted according to the first loss value and the second loss value, which can achieve implicit and explicit guidance of the model through the guidance information, and can make the final image processing model more accurate.
[0270] Referring to FIG15 , FIG15 shows a flowchart of a computer-aided diagnosis method for cancer cell metastasis to lymph nodes provided by one embodiment of the present disclosure, which specifically includes the following steps:
[0271] Step 1502: Train the image detection model.
[0272] Specifically, multiple esophageal cancer CT images can be obtained, and the lymph nodes in the CT images can be segmented using the Deep-Stationing model to obtain multiple lymph nodes in different locations. Using the pathology report of the esophageal cancer patient, the presence of esophageal cancer lymph node metastasis in each lymph node is annotated to obtain a training sample pair, wherein the lymph node with lymph node metastasis is labeled as 1, and the lymph node without lymph node metastasis is labeled as 0. Furthermore, the training sample pair includes a training sample and a sample label. The training sample includes multiple sample images corresponding to the target lymph node to be detected, and the sample label is set to identify the sample result of the target lymph node to be detected. At the same time, since the pathology report can only determine whether lymph node metastasis has occurred in the lymph node station, but cannot accurately mark whether a single cell in the lymph node station has undergone lymph node metastasis, in order to improve the detection accuracy and help doctors locate the location of abnormal cells for subsequent diagnosis and treatment, one or more embodiments of the present disclosure also use an LN detection algorithm to segment the LN instance to obtain an LN mask. By using the LN mask as guidance information for the training sample pairs and inputting it into the initial image detection model along with the training sample pairs, we can obtain the sample prediction results output by the initial image detection model, as well as the target attention information for the target lymph node to be detected. By explicitly guiding the detection results of the image detection model using the sample results and implicitly guiding the detection results of the image detection model using the guidance information, the image detection model can focus its attention on the target lymph node, improving the accuracy of its location and, therefore, the accuracy of the image detection model's output results.
[0273] Specifically, the above-mentioned CT images include plain CT (Non-contrast enhanced CT, NECT) images and enhanced CT (Contrast-enhanced CT, CECT) images.
[0274] It should be noted that the LN detection algorithm is based on the 2.5D Mask R-CNN framework, which takes 9 consecutive axial CT slices as input, fuses 2D feature images to obtain 3D context information, and predicts the position and mask of LN instances in the CT slices to obtain accurate LN location information.
[0275] Based on the sample prediction results output by the initial image detection model and the sample results corresponding to the sample labels, the first loss value of the initial image detection model can be calculated; based on the target attention information output by the initial image detection model and the guidance information corresponding to the training sample pairs, the second loss value of the initial image detection model can be calculated; based on the first loss value and the second loss value, the model parameters of the initial image detection model are adjusted together, and then the adjusted initial image detection model is used to continue to predict whether cancer cells exist in multiple sample images corresponding to the target lymph node station to be detected, until the preset stopping condition is reached, and a trained image detection model is obtained.
[0276] Step 1504: Receive an esophageal cancer CT image detection task, wherein the esophageal cancer CT image detection task carries multiple target lymph node station CT images corresponding to the target lymph node to be detected, and the esophageal cancer CT image detection task is set to detect whether lymph node metastasis occurs in the target lymph node to be detected.
[0277] In practical applications, when receiving the task of detecting esophageal cancer CT images, the lymph nodes in the esophageal cancer CT images can be segmented in advance through the Deep-Stationing model to determine the target lymph nodes to be detected. Multiple target lymph node CT images corresponding to the target lymph nodes to be detected are input into the trained image detection model to detect whether there are cancer cells in the lymph nodes.
[0278] Step 1506: Input multiple target lymph node CT images into the image processing model to obtain detection results corresponding to the target lymph node to be detected, wherein the image detection model determines the detection results based on the local image features and global image features of the CT images of each target lymph node.
[0279] In practical applications, through the image processing model, multiple lymph node CT images can be first input into the feature extraction layer of the image detection model to obtain image feature information corresponding to each lymph node CT image. The feature extraction layer includes two sequentially connected full convolution blocks. Specifically, multiple lymph node CT images are input into full convolution block 1 to obtain image feature information 1 output by full convolution block 1. This image feature information 1 is then input into full convolution block 2 to obtain image feature information 2 output by full convolution block 2. Image feature information 2 is then determined as the image feature information output by the feature extraction layer.
[0280] Then, the feature information of each image can be input into the feature fusion layer included in the image processing model to obtain the image fusion information corresponding to the CT image of each lymph node station, wherein the feature fusion layer includes three local-global fusion blocks connected in sequence, each local-global fusion block includes at least one local-global feature fusion layer, each local-global feature fusion layer includes a local feature extraction sublayer, a global feature extraction sublayer and a fusion layer, the local feature extraction sublayer includes a convolution subunit, and the global feature extraction sublayer includes a codec image processing unit.
[0281] Finally, the image fusion information can be input into the output layer of the image processing model to obtain the detection result corresponding to the target lymph node to be detected. In practical applications, the image fusion information is input into the output layer of the image detection model, and the output layer can perform softmax (normalization) processing on the image fusion information to obtain the detection result corresponding to the target lymph node to be detected output by the image detection model. Specifically, the closer the detection result is to 0, the lower the possibility of lymph node metastasis in the target lymph node to be detected, and the closer the detection result is to 1, the higher the possibility of lymph node metastasis in the target lymph node to be detected.
[0282] In the embodiment provided by the present disclosure, training sample pairs and guidance information for sample pairs are constructed based on a data set of 1,205 patients with esophageal cancer collected. Based on this, a larger data set for the classification of LN metastasis of esophageal cancer can be constructed, and the label of the LN station is determined by the LN anatomical results in the data set. If all anatomical lymph nodes of the lymph node in the report are shown to be benign, the lymph node is benign, and if there are any metastatic lymph nodes in the report, it is marked as metastasis. The median size of the CT image is 512×512×91mm, and the median resolution is 0.795×0.795×5.0mm. All CT images are resampled to a consistent spatial resolution of 0.787×0.787×5.0mm. 3D training patches are generated by cropping 96×96×32 ROIs of the CT image and LN mask, centered on each LN station, and further using each LN-station binary map to "mask" the CT image by setting the voxels outside the LN-station to a constant value of -1024. Configuring local-global transformer blocks with L = 2, 4, and 3, respectively, and setting the threshold τ in the mask-guided margin loss to 0.8, results in better validation performance. This image detection model can be developed using PyTorch v1.12.1. For training, the AdamW optimizer with a learning rate of 1.6e-3 and a weight decay of 5e-4 can be used. The image detection model is trained for 250 epochs before convergence.
[0283] See Table 1 below, which shows a quantitative comparison of five-fold cross-validation of segmented data at the patient level for different image detection models and input settings. LMViT is the model structure of the image detection model provided in the embodiments of the present disclosure, and LMViT+Lmask is the image detection model trained according to the training method of the image detection model provided in the embodiments of the present disclosure. AUROC is the area under the characteristic curve, R@S80 is the recall rate with a specificity rate of 80%, S@R80 is the specificity with a recall rate of 80%, and MaskLN is whether the LN instance is used as a mask for implicit guidance.
[0284] Table 1
[0285] It can be found that when the CT image is used alone as input (without the LN instance mask), the LMViT provided by the embodiment of the present disclosure improves the AUROC to 0.8214, which is the highest among all the evaluation models, proving the effectiveness of the proposed local-global feature fusion architecture. When the LN instance mask is further used as an auxiliary input, all models show performance gains, and the AUROC exceeds 0.83. By applying the solution of the embodiment of the present disclosure, by carrying multiple target images corresponding to the target partition to be detected in the image processing task and inputting the multiple target images into the image processing model, the detection results corresponding to the target partition to be detected output by the image processing model can be obtained, thereby realizing automatic detection of whether there are abnormal objects to be detected in the target partition to be detected, and the image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0286] Corresponding to the above method embodiment, the present disclosure also provides an embodiment of a computer-aided diagnosis system for cancer cell metastasis to lymph nodes, the system comprising:
[0287] A first receiving module 1502 is configured to receive a CT image processing task, wherein the CT image processing task carries multiple CT images corresponding to a target mediastinal lymph node to be detected, and the CT image processing task is configured to detect whether cancer cells in the target mediastinal lymph node to be detected have metastasized to lymph nodes;
[0288] Input module 1504 is used to input the multiple CT images into a computer-aided diagnosis model to obtain a computer-aided diagnosis result corresponding to the target mediastinal lymph node to be detected, wherein the computer-aided diagnosis model determines the computer-aided diagnosis result based on the local image features and global image features of each CT image.
[0289] Corresponding to the above method embodiment, the present disclosure also provides an image processing device embodiment. FIG16 shows a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure. As shown in FIG16 , the device includes:
[0290] The receiving module 1602 is configured to receive an image processing task, wherein the image processing task carries multiple target images corresponding to the target to-be-detected partition, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target to-be-detected partition;
[0291] The detection module 1604 is configured to input multiple target images into the image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image.
[0292] Optionally, the image processing model includes a feature extraction layer, a feature fusion layer and an output layer, wherein the feature fusion layer is configured to fuse local image features and global image features of each target image;
[0293] Accordingly, the detection module 1604 is further configured to:
[0294] Input multiple target images into the feature extraction layer to obtain image feature information corresponding to each target image;
[0295] Input the feature information of each image into the feature fusion layer to obtain the image fusion information corresponding to each target image;
[0296] The image fusion information is input into the output layer to obtain the detection results corresponding to the target partition to be detected.
[0297] Optionally, the feature extraction layer includes n fully convolutional blocks connected in sequence, where n is a positive integer greater than or equal to 2;
[0298] Accordingly, the detection module 1604 is further configured to:
[0299] determining a target image to be processed among a plurality of target images;
[0300] Input the target image to be processed into the first full convolution block to obtain the first image feature information;
[0301] Input the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information, where 2≤i≤n;
[0302] Determine whether i is equal to n;
[0303] If not, then i is incremented by 1, and the operation of inputting the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information is continued;
[0304] If so, the i-th image feature information is determined as the image feature information corresponding to the target image to be processed.
[0305] Optionally, the feature fusion layer includes m local-global fusion blocks connected sequentially, where m is a positive integer greater than or equal to 2;
[0306] Accordingly, the detection module 1604 is further configured to:
[0307] Determining target image feature information from each image feature information;
[0308] Inputting the target image feature information into the first local-global fusion block to obtain the first image fusion information;
[0309] Input the j-1th image fusion information into the jth local-global fusion block to obtain the jth image fusion information, where 2≤j≤m;
[0310] Determine whether j is equal to m;
[0311] If not, then j is incremented by 1, and the operation of inputting the j-1th image fusion information into the jth local-global fusion block to obtain the jth image fusion information is continued;
[0312] If so, the j-th image fusion information is determined as the image fusion information corresponding to the target image feature information.
[0313] Optionally, the local-global fusion block includes at least one local-global feature fusion layer, which includes a local feature extraction sublayer, a global feature extraction sublayer, and a fusion layer. Accordingly, the detection module 1604 includes the t-th local-global feature fusion layer, which is configured as follows:
[0314] Acquire feature information of the image to be processed, wherein the feature information of the image to be processed includes feature information of the target image or fusion information of the t-1th image;
[0315] Input the feature information of the image to be processed into the local feature extraction sublayer to obtain the local feature information corresponding to the feature information of the image to be processed;
[0316] Input the feature information of the image to be processed into the global feature extraction sublayer to obtain the global feature information corresponding to the feature information of the image to be processed;
[0317] The local feature information and the global feature information are input into the fusion layer to obtain the image fusion information corresponding to the t-th local-global feature fusion layer.
[0318] Optionally, the local feature extraction sublayer includes a convolution processing unit; accordingly, the t-th local-global feature fusion layer is further configured as follows:
[0319] Input the feature information of the image to be processed into the local feature extraction sublayer to obtain the local feature information corresponding to the feature information of the image to be processed, including:
[0320] Inputting the feature information of the image to be processed into the convolution processing unit, and extracting the initial local feature information corresponding to the feature information of the image to be processed;
[0321] Perform feature mapping on the initial local feature information to obtain local feature information corresponding to the feature information of the image to be processed.
[0322] Optionally, the global feature extraction sublayer includes a codec image processing unit;
[0323] Accordingly, the tth local-global feature fusion layer is further configured as follows:
[0324] Perform feature mapping on the feature information of the image to be processed to obtain the feature information of the target image to be processed corresponding to the feature information of the image to be processed;
[0325] The target image to be processed feature information is input into the encoding and decoding image processing unit to obtain global feature information corresponding to the target image to be processed feature information.
[0326] Optionally, the tth local-global feature fusion layer is further configured as follows:
[0327] Splice local feature information and global feature information to obtain initial image fusion information;
[0328] Perform feature mapping on the initial image fusion information to obtain the image fusion information corresponding to the t-th local-global feature fusion layer.
[0329] Optionally, the image processing apparatus further includes a training module configured to:
[0330] Obtaining a training sample pair and guidance information for the training sample pair, wherein the training sample pair includes a training sample and a sample label, the training sample includes a plurality of sample images corresponding to the target partition to be detected, and the sample label is set to identify a sample result of the target partition to be detected;
[0331] Input the training sample pairs and guidance information into the initial image processing model to obtain the sample prediction results output by the initial image processing model and the target attention information of the target partition to be detected;
[0332] Calculate the first loss value of the initial image processing model based on the sample prediction result and the sample result;
[0333] Calculate the second loss value of the initial image processing model based on the target attention information and the guidance information;
[0334] The model parameters of the initial image processing model are adjusted according to the first loss value and the second loss value until a preset stop condition is reached to obtain an image processing model.
[0335] Optionally, the training module is further configured to:
[0336] Determining the position information of the object to be detected in the target to be detected area of each sample image;
[0337] The position information of the object to be detected is used as the guidance information corresponding to each sample image.
[0338] Optionally, the image processing model includes a feature extraction layer, a feature fusion layer and an output layer;
[0339] Accordingly, the training module is further configured as follows:
[0340] Splicing training samples and guidance information to obtain target splicing information;
[0341] Input the target splicing information into the feature extraction layer to obtain the feature extraction information output by the feature extraction layer;
[0342] Inputting the feature extraction information into the feature fusion layer to obtain feature fusion information output by the feature fusion layer and at least one attention feature map;
[0343] Input the feature fusion information into the output layer to obtain the sample prediction results corresponding to the target partition to be detected;
[0344] Aggregate at least one attention feature map to obtain target attention information corresponding to the target partition to be detected.
[0345] An image processing device provided by an embodiment of the present disclosure carries multiple target images corresponding to the target partition to be detected in the image processing task and inputs the multiple target images into the image processing model. It can obtain the detection results corresponding to the target partition to be detected output by the image processing model, thereby realizing automatic detection of whether there are abnormal objects to be detected in the target partition to be detected. The image processing model determines the detection results based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the objects to be detected.
[0346] The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solution of the image processing device and the technical solution of the above-mentioned image processing method are based on the same concept. For details not described in detail in the technical solution of the image processing device, please refer to the description of the technical solution of the above-mentioned image processing method.
[0347] Figure 17 shows a block diagram of a computing device according to one embodiment of the present disclosure. Components of the computing device 1700 include, but are not limited to, a memory 1710 and a processor 1720. The processor 1720 is connected to the memory 1710 via a bus 1730, and a database 1750 is used to store data.
[0348] The computing device 1700 also includes an access device 1740 that enables the computing device 1700 to communicate via one or more networks 1760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0349] In one embodiment of the present disclosure, the aforementioned components of computing device 1700 and other components not shown in FIG17 may also be connected to each other, for example, via a bus. It should be understood that the computing device block diagram shown in FIG17 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.
[0350] Computing device 1700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1700 may also be a mobile or stationary server.
[0351] The processor 1720 is configured to execute the following computer executable instructions, which implement the steps of the above method when executed by the processor.
[0352] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above method.
[0353] An embodiment of the present disclosure further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above method when executed by a processor.
[0354] The above is a schematic solution of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above method.
[0355] An embodiment of the present disclosure further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above method.
[0356] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above method belong to the same concept, and any details not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the above method.
[0357] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0358] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.
[0359] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present disclosure are not limited by the order of the actions described, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the disclosure are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure.
[0360] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0361] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).
[0362] The preferred embodiments of the present disclosure disclosed above are only used to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made based on the content of the embodiments of the present disclosure. The present disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can better understand and utilize the present disclosure. The present disclosure is limited by the claims and their full scope and equivalents. Industrial Applicability
[0363] The disclosed embodiment provides an image processing method, which receives an image processing task, wherein the image processing task carries multiple target images corresponding to a target partition to be detected, and the image processing task is set to detect whether there is an abnormal object to be detected in the target partition to be detected; the multiple target images are input into an image processing model to obtain a detection result corresponding to the target partition to be detected, wherein the image processing model determines the detection result based on the local image features and global image features of each target image. In this way, by carrying the multiple target images corresponding to the target partition to be detected in the image processing task and inputting the multiple target images into the image processing model, the detection result corresponding to the target partition to be detected output by the image processing model can be obtained, thereby enabling automatic detection of whether there is an abnormal object to be detected in the target partition to be detected, and the image processing model determines the detection result based on the local image features and global image features of each target image, which can improve the accuracy and efficiency of the detection results for the object to be detected.
Claims
1. An image processing method, include: Receiving an image processing task, wherein the image processing task carries a plurality of target images corresponding to the target to-be-detected partition, and the image processing task is configured to detect whether there is an abnormal to-be-detected object in the target to-be-detected partition; The multiple target images are input into an image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image.
2. The method according to claim 1, wherein the image processing model comprises a feature extraction layer, a feature fusion layer and an output layer, in, The feature fusion layer is configured to fuse local image features and global image features of each target image; The step of inputting the plurality of target images into an image processing model to obtain detection results corresponding to the target partitions to be detected includes: Inputting the multiple target images into the feature extraction layer to obtain image feature information corresponding to each target image; Inputting each image feature information into the feature fusion layer to obtain image fusion information corresponding to each target image; The image fusion information is input into the output layer to obtain the detection result corresponding to the target partition to be detected.
3. The method according to claim 2, wherein the feature extraction layer comprises n sequentially connected full convolution blocks, in, n is a positive integer greater than or equal to 2; The step of inputting the plurality of target images into the feature extraction layer to obtain image feature information corresponding to each target image includes: Determining a target image to be processed among the multiple target images; Inputting the target image to be processed into a first full convolution block to obtain first image feature information; Input the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information, where 2≤i≤n; Determine whether i is equal to n; If not, i is incremented by 1, and the operation of inputting the i-1th image feature information into the i-th full convolution block to obtain the i-th image feature information is continued; If so, the i-th image feature information is determined as the image feature information corresponding to the target image to be processed.
4. The method according to claim 2, wherein the feature fusion layer comprises m sequentially connected local-global fusion blocks, in, m is a positive integer greater than or equal to 2; The step of inputting the feature information of each image into the feature fusion layer to obtain the image fusion information corresponding to each target image includes: Determine target image feature information from each image feature information; Inputting the target image feature information into a first local-global fusion block to obtain first image fusion information; The j-1th image fusion information is input into the jth local-global fusion block to obtain the jth image fusion information, where: 2≤j≤m; Determine whether j is equal to m; If not, j is incremented by 1, and the operation of inputting the j-1th image fusion information into the jth local-global fusion block to obtain the jth image fusion information is continued; If so, the j-th image fusion information is determined as the image fusion information corresponding to the target image feature information.
5. According to the method of claim 4, the local-global fusion block comprises at least one local-global feature fusion layer, and the local-global feature fusion layer comprises a local feature extraction sublayer, a global feature extraction sublayer, and a fusion layer; For the tth local-global feature fusion layer, the method include: Acquire feature information of the image to be processed, wherein the feature information of the image to be processed includes feature information of the target image or fusion information of the t-1th image; Inputting the image feature information to be processed into the local feature extraction sublayer to obtain local feature information corresponding to the image feature information to be processed; Inputting the feature information of the image to be processed into the global feature extraction sublayer to obtain global feature information corresponding to the feature information of the image to be processed; The local feature information and the global feature information are input into the fusion layer to obtain image fusion information corresponding to the t-th local-global feature fusion layer.
6. The method according to claim 5, wherein the local feature extraction sublayer comprises a convolution processing unit; The feature information of the image to be processed is input into the local feature extraction sublayer to obtain local feature information corresponding to the feature information of the image to be processed, include: Inputting the feature information of the image to be processed into the convolution processing unit, and extracting initial local feature information corresponding to the feature information of the image to be processed; Feature mapping is performed on the initial local feature information to obtain local feature information corresponding to the feature information of the image to be processed.
7. The method according to claim 5, wherein the global feature extraction sublayer comprises a codec image processing unit; The feature information of the image to be processed is input into the global feature extraction sublayer to obtain the global feature information corresponding to the feature information of the image to be processed, include: Performing feature mapping on the feature information of the image to be processed to obtain feature information of the target image to be processed corresponding to the feature information of the image to be processed; The target image feature information to be processed is input into the encoding and decoding image processing unit to obtain global feature information corresponding to the target image feature information to be processed.
8. According to the method of claim 1, the image processing model is trained by the following steps: Obtaining training sample pairs and guidance information of the training sample pairs, in, The training sample pair includes a training sample and a sample label, the training sample includes a plurality of sample images corresponding to the target partition to be detected, and the sample label is set to identify the sample result of the target partition to be detected; The training sample pair and the guide information are input into the initial image processing model to obtain the initial image Processing the sample prediction results output by the model and the target attention information of the target partition to be detected; Calculating a first loss value of the initial image processing model according to the sample prediction result and the sample result; Calculating a second loss value of the initial image processing model according to the target attention information and the guidance information; The model parameters of the initial image processing model are adjusted according to the first loss value and the second loss value until a preset stop condition is reached to obtain an image processing model.
9. The method according to claim 8, wherein the image processing model comprises a feature extraction layer, a feature fusion layer and an output layer; Inputting the training sample pair and the guide information into an initial image processing model to obtain a sample prediction result output by the initial image processing model and target attention information of the target partition to be detected, including: Splicing the training sample and the guide information to obtain target splicing information; Inputting the target splicing information into the feature extraction layer to obtain feature extraction information output by the feature extraction layer; Inputting the feature extraction information into the feature fusion layer to obtain feature fusion information output by the feature fusion layer and at least one attention feature map; Inputting the feature fusion information into the output layer to obtain the sample prediction result corresponding to the target partition to be detected; Aggregate the at least one attention feature map to obtain target attention information corresponding to the target partition to be detected.
10. A CT image processing method, include: Receiving a CT image processing task, wherein the CT image processing task carries a plurality of CT images corresponding to the target partition to be detected, and the CT image processing task is configured to detect whether there are abnormal cells to be detected in the target partition to be detected; The multiple CT images are input into a CT image processing model to obtain a detection result corresponding to the target partition to be detected, wherein the CT image processing model determines the detection result based on local image features and global image features of each CT image.
11. A training method for an image processing model, applied to a cloud-side device, include: Acquire a training sample pair and guidance information of the training sample pair, wherein the training sample pair includes a training sample and a sample label, the training sample includes a plurality of sample images corresponding to the target partition to be detected, and the sample label is set to identify the sample result of the target partition to be detected; Inputting the training sample pair and the guidance information into an initial image processing model to obtain a sample prediction result output by the initial image processing model and target attention information of the target partition to be detected; Calculating a first loss value of the initial image processing model according to the sample prediction result and the sample result; According to the target attention information and the guidance information, the first 2. Loss value; Adjusting the model parameters of the initial image processing model according to the first loss value and the second loss value until a preset stop condition is reached, thereby obtaining the model parameters of the image processing model; The model parameters of the image processing model are sent to the end-side device.
12. An image processing method, include: Receive an image processing request sent by a user, wherein the image processing request includes an image processing task, the image processing task carries a plurality of target images corresponding to the target partition to be detected, and the image processing task is configured to detect whether there is an abnormal object to be detected in the target partition to be detected; Inputting the multiple target images into an image processing model to obtain detection results corresponding to the target partitions to be detected, wherein the image processing model determines the detection results based on local image features and global image features of each target image; The detection result corresponding to the target partition to be detected is sent to the user.
13. A computing device comprising: Memory and processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.
15. A computer-aided diagnosis method for cancer cell metastasis to lymph nodes. include: receiving a CT image processing task, wherein the CT image processing task carries a plurality of CT images corresponding to a target mediastinal lymph node to be detected, and the CT image processing task is configured to detect whether cancer cells in the target mediastinal lymph node to be detected have metastasized to lymph nodes; The multiple CT images are input into a computer-aided diagnosis model to obtain computer-aided diagnosis results corresponding to the target mediastinal lymph node to be detected, wherein the computer-aided diagnosis model determines the computer-aided diagnosis results based on local image features and global image features of each CT image.
16. A computer-aided diagnosis system for cancer cell metastasis to lymph nodes. include: A first receiving module is used to receive a CT image processing task, wherein the CT image processing task carries a plurality of CT images corresponding to a target mediastinal lymph node to be detected, and the CT image processing task is configured to detect whether cancer cells in the target mediastinal lymph node to be detected have metastasized to lymph nodes; An input module is used to input the multiple CT images into a computer-aided diagnosis model to obtain a computer-aided diagnosis result corresponding to the target mediastinal lymph node to be detected, wherein the computer-aided diagnosis model determines the computer-aided diagnosis result based on local image features and global image features of each CT image.
Citation Information
Patent Citations
Multi-modal image analysis method and system for cancer diagnosis
CN112489788A
Method and device for predicting abnormal cell metastasis degree and storage medium
CN113705595A
High-resolution remote sensing image target detection method and device based on attention mechanism
CN116128718A
Image processing method and device
CN116797554A
System and method for computer aided diagnosis of mammograms using multi-view and multi-scale information fusion
US20210375458A1
Cited By
Water body detection method and device
CN120032284A
Image processing method and image processing model training method
CN120782704A
Image processing model training method and image processing method
CN120853835A
