Image Processing Method, Model Training Method and Device, and Electronic Device

Abstract

The present application discloses an image processing method, a model training method, an apparatus, and an electronic device. The method includes: partitioning a mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region; according to the at least two mediastinal partition images, through at least two branch networks in a neural network model, performing lesion detection on different anatomical structures to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is at least used to detect mediastinal lesions in the anatomical structure of one mediastinal partition image. One branch network is at least used to detect lesions in the anatomical structure of one mediastinal partition image, that is to say, different branch networks are used for lesion detection of different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an image processing method, a model training method, an apparatus and an electronic device. Background Art

[0002] In recent years, it has become a trend to apply deep learning technology in the medical field to improve disease diagnosis and assist treatment. A large number of studies have also been carried out on the detection, recognition, quantitative evaluation, and diagnostic analysis of thoracic and pulmonary diseases in medical images using deep learning technology. However, the current research mainly focuses on the deep learning detection algorithms developed for lung field lesions, while ignoring the lesions in the mediastinal region. At the same time, due to the significantly different intensity distributions between the lung field and the mediastinum. For example, the lung field is mainly composed of air, while the mediastinum is completely composed of body tissues, and the anatomical structures in the lung field are different from those in the mediastinum. Therefore, the deep learning detection algorithms developed for lung field lesions cannot well detect mediastinal lesions, resulting in a significant reduction in the detection accuracy of mediastinal lesions. Summary of the Invention

[0003] In view of this, embodiments of the present application are committed to providing an image processing method, a model training method, an apparatus and an electronic device, which can accurately detect mediastinal lesions.

[0004] According to a first aspect of an embodiment of the present application, there is provided an image processing method, including: partitioning the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region; according to the at least two mediastinal partition images, through at least two branch networks in a neural network model, performing lesion detection on different anatomical structures to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is at least used to detect mediastinal lesions in the anatomical structure of one mediastinal partition image.

[0005] In one embodiment, each branch network includes a self-attention network. According to the at least two mediastinal partition images, through at least two branch networks in the neural network model, performing lesion detection on different anatomical structures to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, including: according to at least two sub-partition images corresponding to each mediastinal partition image, through the self-attention network, obtaining a feature map corresponding to each mediastinal partition image; according to the feature map corresponding to each mediastinal partition image, obtaining a total feature map; performing a function operation or a convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the pulmonary medical image.

[0006] In one embodiment, before obtaining the feature map corresponding to each mediastinal partition image through the self-attention network according to at least two sub-partition images corresponding to each mediastinal partition image, the method further includes: expanding one mediastinal partition image in two adjacent mediastinal partition images in the direction of the other mediastinal partition image to obtain an expanded mediastinal partition image, wherein the expanded mediastinal partition image and the other mediastinal partition image have an overlapping part; partitioning the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image.

[0007] In one embodiment, obtaining the feature map corresponding to each mediastinal partition image according to at least two sub-partition images corresponding to each mediastinal partition image through the self-attention network includes: obtaining a first feature map according to at least two sub-partition images corresponding to one mediastinal partition image in two adjacent mediastinal partition images through the self-attention network; obtaining a second feature map according to the first feature map and at least two sub-partition images corresponding to the other mediastinal partition image in two adjacent mediastinal partition images through the self-attention network.

[0008] In one embodiment, when two adjacent mediastinal partition images have an overlapping part, the overlapping part of the first feature map corresponds to a first overlapping feature map, and the overlapping part of the second feature map corresponds to a second overlapping feature map. Among them, obtaining the total feature map according to the feature map corresponding to each mediastinal partition image includes: performing weighted summation on the first overlapping feature map and the second overlapping feature map to obtain a total overlapping feature map; combining the total overlapping feature map, the feature map in the first feature map except the first overlapping feature map, and the feature map in the second feature map except the second overlapping feature map to obtain the total feature map.

[0009] In one embodiment, partitioning the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images includes: using a mediastinal window to perform a windowing operation on the region of interest image corresponding to the pulmonary medical image to obtain the mediastinal region; using a bone window to perform a windowing operation on the region of interest image corresponding to the pulmonary medical image to obtain a bone region, wherein the bone region includes the thoracic vertebra and the spinal column; using the straight line between the thoracic vertebra and the spinal column as the partitioning reference to partition the mediastinal region to obtain at least two mediastinal partition images.

[0010] In one embodiment, taking the straight line between the thoracic vertebra and the spine as the partitioning reference, the mediastinal region is partitioned to obtain at least two sub-partition images, including: the upper mediastinal partition image is the region image corresponding to the straight line on the side close to the thoracic vertebra with a first preset length; the lower mediastinal partition image is the region image corresponding to the straight line on the side close to the spine with a second preset length; the middle mediastinal partition image is the region image between the upper mediastinal partition image and the lower mediastinal partition image. Among them, the upper mediastinal partition image and the lower mediastinal partition image use the same branch network for lesion detection, and the middle mediastinal partition image uses another branch network for lesion detection.

[0011] In one embodiment, the method further includes: performing a thresholding operation on the pulmonary medical image to obtain a lung field mask; expanding the lung field mask outward to include the thoracic vertebra to obtain a region of interest mask; using the region of interest mask to perform a cropping operation on the pulmonary medical image to obtain a region of interest image.

[0012] According to the second aspect of the embodiments of the present application, a method for training a neural network model is provided, including: partitioning the mediastinal region of a pulmonary medical sample image to obtain at least two mediastinal partition sample images, where the pulmonary medical sample image is labeled with a lesion label; according to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, through at least two branch networks in the neural network model, obtaining a lesion prediction result of the mediastinal region of the pulmonary medical sample image, where one branch network is at least used to predict a mediastinal lesion in the anatomical structure of one mediastinal partition sample image; calculating a loss function value according to the lesion prediction result and the lesion label; updating the parameters of the neural network model according to the loss function value.

[0013] According to the third aspect of the embodiments of the present application, an image processing device is provided, including: a first acquisition module configured to partition the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region; a second acquisition module configured to perform lesion detection on different anatomical structures according to at least two mediastinal partition images through at least two branch networks in the neural network model to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is at least used to detect a mediastinal lesion in the anatomical structure of one mediastinal partition image.

[0014] According to a third aspect of the embodiments of the present application, there is provided a training device for a neural network model, including: a fourth acquisition module configured to partition the mediastinal region of a pulmonary medical sample image to obtain at least two mediastinal partition sample images, wherein the pulmonary medical sample image is labeled with a lesion label; a training module configured to obtain a lesion prediction result of the mediastinal region of the pulmonary medical sample image through at least two branch networks in the neural network model according to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, wherein one branch network is at least used to predict a mediastinal lesion in the anatomical structure of one mediastinal partition sample image; calculate a loss function value according to the lesion prediction result and the lesion label; and update the parameters of the neural network model according to the loss function value.

[0015] According to a fifth aspect of the embodiments of the present application, there is provided an electronic device, which includes: a processor; and a memory in which computer program instructions are stored, and when the computer program instructions are run by the processor, the processor is caused to execute the method mentioned in the above first aspect or second aspect.

[0016] According to a sixth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are run by the processor, the processor is caused to execute the method mentioned in the above first aspect or second aspect.

[0017] An image processing method provided by an embodiment of the present application first partitions the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, and then, according to the at least two mediastinal partition images, performs lesion detection on different anatomical structures of the mediastinal region through at least two branch networks in the neural network model to obtain a lesion detection result of the mediastinal region of the pulmonary medical image. One branch network is at least used to detect a lesion in the anatomical structure of one mediastinal partition image, that is to say, different branch networks are used for lesion detection of different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present application will become more obvious. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation to the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 It is a schematic diagram of the system architecture of the application scenario of the image processing method provided by an embodiment of the present application.

[0020] Figure 2 It is a schematic flowchart of an image processing method provided by an embodiment of the present application.

[0021] Figure 3 It is a schematic flowchart of an image processing method provided by another embodiment of the present application.

[0022] Figure 4 It is a schematic diagram of a mediastinal partition image provided by an embodiment of the present application.

[0023] Figure 5 It is a flowchart of a data processing process of an image processing method provided by an embodiment of the present application.

[0024] Figure 6 It is a flowchart of a process for obtaining a region of interest image corresponding to a pulmonary medical image provided by an embodiment of the present application.

[0025] Figure 7 It is a schematic flowchart of a training method for a neural network model provided by an embodiment of the present application.

[0026] Figure 8 It is a block diagram of an image processing apparatus provided by an embodiment of the present application.

[0027] Figure 9 It is a block diagram of a training apparatus for a neural network model provided by an embodiment of the present application.

[0028] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0030] The mediastinal region is a common site for various lesions. Mediastinal lesions mainly include hyperplasia, cysts, tumors, and lymph nodes metastasized from the lungs, etc. The detection of mediastinal lesions plays an important role in the early screening and diagnosis of related diseases. Currently, the deep learning detection algorithms developed for lung field lesions cannot detect mediastinal lesions well, and there are two reasons as follows. On the one hand, the lung field and the mediastinum have significantly different intensity distributions. The lung field is mainly composed of air, while the mediastinum is completely composed of body tissues. On the other hand, the anatomical structures in the lung field are different from those in the mediastinum. In the lung field, there are mainly bronchi and branch arteriovenous vessels, showing a sparse tree-like distribution. The mediastinum can be divided into three regions. The superior mediastinal region is located between the body of the sternum and the pericardium and is very narrow. The inferior mediastinal region is located between the pericardium and the spine and is used to accommodate the bifurcation of the trachea, the left and right main bronchi, the esophagus, blood vessels, lymph nodes, etc. The middle mediastinal region is located between the superior and inferior mediastinal regions and is used to accommodate the heart, the large blood vessels entering and leaving the heart, the phrenic nerve, and lymph nodes, etc. Since the anatomical structures contained in different mediastinal regions are different, the types and appearances of lesions in different mediastinal regions are also different. If the same neural network is used for lesion detection in different mediastinal regions, the accuracy of mediastinal lesion detection will be greatly reduced.

[0031] To solve the above problems, the embodiments of the present application provide an image processing method. First, by partitioning the mediastinal region of a pulmonary medical image, at least two mediastinal partition images can be obtained. Then, according to the at least two mediastinal partition images, through at least two branch networks in the neural network model, lesion detection is performed on different anatomical structures in the mediastinal region, and the lesion detection result of the mediastinal region of the pulmonary medical image is obtained. One branch network is at least used to detect lesions in the anatomical structure of one mediastinal partition image. That is to say, different branch networks are used for lesion detection of different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0032] Since the embodiments of the present application involve the application of neural networks, for the sake of easy understanding, the following first briefly introduces the relevant terms and related concepts such as neural networks that may be involved in the embodiments of the present application.

[0033] A neural network is an operation model composed of a large number of interconnected nodes (or neurons). Each node corresponds to a policy function, and the connection between every two nodes represents a weighted value for the signal passing through this connection, which is called a weight. A neural network generally includes multiple neural network layers that are cascaded with each other. The output of the i-th neural network layer is connected to the input of the i + 1-th neural network layer, and the output of the i + 1-th neural network layer is connected to the input of the i + 2-th neural network layer, and so on. After the training samples are input into the cascaded neural network layers, an output result is output through each neural network layer, and this output result serves as the input of the next neural network layer. Thus, the output is obtained through the calculation of multiple neural network layers. The predicted result of the output of the output layer is compared with the true target value, and then the weight matrix and policy function of each layer are adjusted according to the difference between the predicted result and the target value. The neural network continuously undergoes the above adjustment process using the training samples, so that the parameters such as the weights of the neural network are adjusted until the predicted result output by the neural network is consistent with the true target result. This process is called the training process of the neural network. After the neural network is trained, a neural network model can be obtained.

[0034] During the process of training a neural network, since it is desired that the output of the neural network is as close as possible to the value that is truly desired to be predicted, the predicted value of the current network can be compared with the truly desired target value, and then the weight vector of each layer of the neural network can be updated according to the difference between the two (of course, there is usually an initialization process before the first update, that is, parameters are preconfigured for each layer in the neural network). For example, if the predicted value of the network is high, the weight vector is adjusted to make it predict lower, and continuous adjustment is made until the neural network can predict the truly desired target value or a value that is very close to the truly desired target value. Therefore, a loss function or an objective function can be used to compare the difference between the predicted value and the target value, and they are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then, the training of the neural network becomes a process of minimizing this loss as much as possible.

[0035] The following combines Figure 1 to introduce in detail the system architecture of the application scenario of the image processing method mentioned in the embodiments of the present application. As Figure 1 shown, the application scenario provided by the embodiments of the present application involves a CT scanner 110, a server 120, and a computer device 130.

[0036] The CT scanner 110 is used to perform X-ray scanning on human tissues to obtain CT images of human tissues. In one embodiment, by scanning the lungs with the CT scanner 110, lung medical images can be obtained.

[0037] The computer device 120 can be a general-purpose computer or a computer device composed of dedicated integrated circuits, etc., and the embodiments of the present application do not limit this. For example, the computer device 120 can be a mobile terminal device such as a tablet computer, or it can also be a personal computer (PC), such as a laptop computer and a desktop computer, etc. The computer device 120 is connected to the CT scanner 110 through a communication network. Optionally, the communication network is a wired network or a wireless network. In some alternative embodiments, the computer device 120 receives the lung medical images sent by the CT scanner 110, the computer device 120 partitions the mediastinal region of the lung medical images to obtain at least two mediastinal partition images, and the computer device 120 performs lesion detection on different anatomical structures in the mediastinal region through at least two branch networks in the neural network model deployed thereon, so as to obtain the lesion detection result of the mediastinal region of the lung medical images.

[0038] The server 130 is a server, or composed of several servers, or a virtualization platform, or a cloud computing service center. The server 130 is connected to the computer device 120 through a communication network. Optionally, the communication network is a wired network or a wireless network. In some alternative embodiments, the server 130 receives the lung medical sample images collected by the computer device 110, and trains at least two branch networks with the lung medical sample images to obtain a neural network model for detecting lesions in the mediastinal region of lung medical images. The computer device 120 can send the lung medical images obtained from the CT scanner 110 to the server 130, the server 130 partitions the mediastinal region of the lung medical images to obtain at least two mediastinal partition images, and the server 130 performs lesion detection on different anatomical structures in the mediastinal region through at least two branch networks in the neural network model deployed thereon, so as to obtain the lesion detection result of the mediastinal region of the lung medical images, and send the lesion detection result to the computer device 120 for medical staff to view.

[0039] Embodiments of the present application provide an image processing method. First, by partitioning the mediastinal region of a pulmonary medical image, at least two mediastinal partition images can be obtained. Then, based on the at least two mediastinal partition images, through at least two branch networks in a neural network model, lesion detection is performed on different anatomical structures in the mediastinal region to obtain the lesion detection result of the mediastinal region of the pulmonary medical image. One branch network is at least used to detect lesions in the anatomical structure of one mediastinal partition image. That is to say, different branch networks are used for lesion detection of different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0040] The following will Figures 2 to 7 introduce in detail the image processing method mentioned in the embodiments of the present application.

[0041] Figure 2 is a schematic flowchart of the image processing method provided by an embodiment of the present application. Figure 2 The described method is executed by Figure 1 the server 130 mentioned in Figure 2 or other types of electronic devices with data processing functions. As

[0042] shown, the method includes the following steps.

[0043] Step S210: Partition the mediastinal region of the pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures in the mediastinal region.

[0044] The pulmonary medical image can be a medical image such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Computed Radiography (CR), or Digital radiography (DR). Embodiments of the present application do not make specific limitations on this. The mediastinal region refers to the region near the left and right mediastinal pleura, where there are the heart and large blood vessels entering and leaving the heart, the esophagus, the trachea, the thymus, nerves, and lymphatic tissues, etc.

[0045] For example, according to the left - right side method, the mediastinal region of the pulmonary medical image is divided into a left mediastinal partition image and a right mediastinal partition image; for another example, according to the upper - lower side method, the mediastinal region of the pulmonary medical image is divided into an upper mediastinal partition image, a middle mediastinal partition image, and a lower mediastinal partition image; for yet another example, according to the front - back side method, the mediastinal region of the pulmonary medical image is divided into a front mediastinal partition image and a back mediastinal partition image. The embodiments of the present application do not specifically limit the partitioning method of the mediastinal region, and the embodiments of the present application also do not specifically limit the number of at least two mediastinal partition images.

[0046] In one example, at least two mediastinal partition images are obtained through the following partitioning method: using a mediastinal window, perform a windowing operation on the region - of - interest image corresponding to the pulmonary medical image to obtain the mediastinal region; using a bone window, perform a windowing operation on the region - of - interest image corresponding to the pulmonary medical image to obtain a bone region, where the bone region includes the thoracic vertebra and the spine; taking the straight line between the thoracic vertebra and the spine as the partitioning reference, partition the mediastinal region to obtain at least two mediastinal partition images.

[0047] The window levels and window widths of the mediastinal window and the bone window are different, but the embodiments of the present application do not specifically limit the specific values of the window levels and window widths of the two. Using the specific window level and window width of the mediastinal window, performing a windowing operation on the region - of - interest image corresponding to the pulmonary medical image can obtain the mediastinal region corresponding to the pulmonary medical image. By using the specific window level and window width of the bone window, performing a windowing operation on the region - of - interest image corresponding to the pulmonary medical image can obtain the bone region.

[0048] It should be noted that after performing the windowing operation on the region - of - interest image, a thresholding operation can also be performed on the windowed region - of - interest image to determine that the foreground region is the mediastinal region or the bone region.

[0049] Determine the straight line between the thoracic vertebra and the spine, and take this straight line as the partitioning reference to partition the mediastinal region to obtain at least two mediastinal partition images. For example, divide the mediastinal region on the left side of the straight line into a left mediastinal partition image, and divide the mediastinal region on the right side of the straight line into a right mediastinal partition image; for another example, divide the straight line into at least two straight line segments, and the mediastinal region corresponding to each straight line segment is a mediastinal partition image.

[0050] The mediastinal partition image can only include the anatomical structures of the mediastinal region, that is, the mediastinal partition image is merely the partitioning result of the mediastinal region; the mediastinal partition image can also include not only the anatomical structures of the mediastinal region, but also the anatomical structures of the pulmonary region, that is, the mediastinal partition image extends to the pulmonary region on the basis of the partitioning result of the mediastinal region.

[0051] Preferably, the mediastinal partition image includes not only the anatomical structures of the mediastinal region but also the anatomical structures of the lung region. Since the lesions occurring in the anatomical structures of the mediastinal region may extend into the lung region, thus, it is possible to determine mediastinal lesions from the overall appearance, which is beneficial to the detection of mediastinal lesions.

[0052] Step S220: According to at least two mediastinal partition images, through at least two branch networks in the neural network model, perform lesion detection on different anatomical structures to obtain the lesion detection results of the mediastinal region of the pulmonary medical image.

[0053] Since the complexity of the anatomical structures in each mediastinal partition image is different, therefore, the specific network structure of the branch network for lesion detection can be determined according to the complexity of the anatomical structures. For the mediastinal partition image where the anatomical structure has a relatively high complexity, the network structure of the branch network can be more complex. For example, the network structure includes a decoder-encoder, etc.; for the mediastinal partition image where the anatomical structure has a relatively low complexity, the network structure of the branch network can be relatively simple. For example, the network structure can be composed only of basic layers such as several convolutional layers. The embodiments of the present application do not limit the specific network structure of the branch network, and those skilled in the art can make different selections according to actual needs.

[0054] The number of at least two branch networks can also be determined according to the complexity of the anatomical structures in each mediastinal partition image. For example, for two mediastinal partition images where the anatomical structures have a relatively low complexity, a single branch network can be used jointly to detect mediastinal lesions. That is to say, a single branch network is used to detect at least the mediastinal lesions in the anatomical structures of one mediastinal partition image. For the mediastinal partition image where the anatomical structure has a relatively high complexity, a single branch network is used separately to detect mediastinal lesions. This can reduce the complexity of the neural network model, thereby improving the efficiency of mediastinal lesion detection. The embodiments of the present application do not specifically limit the number of at least two branch networks, and those skilled in the art can make different selections according to actual needs.

[0055] By enabling two mediastinal partition images where the anatomical structures have a relatively low complexity to jointly use a single branch network, the weights of these two mediastinal partition images in this branch network can be shared, which helps the training of the neural network model and improves the training efficiency of the neural network model.

[0056] The lesion detection result can be the marking of mediastinal lesions and their probability values as real mediastinal lesions on the pulmonary medical image in the form of detection boxes or points, or it can also be a heat map. The heat map is an image representation for visualizing network attention, used to highlight the area where the mediastinal lesion is located on the pulmonary medical image in a special way, and different colors are used to highlight different levels of disease severity, which is more helpful for medical staff to diagnose mediastinal lesions. The embodiments of the present application do not specifically limit the form of the lesion detection result, and those skilled in the art can make different selections according to actual needs.

[0057] The heat map can be understood as a Gaussian heat map constructed with the center of the mediastinal lesion as the core. By controlling the threshold, points at different pixel positions of the heat map are highlighted with different colors, so that the area most likely to belong to the mediastinal lesion in the heat map can be determined.

[0058] An image processing method provided by an embodiment of the present application first partitions the mediastinal region of the pulmonary medical image to obtain at least two mediastinal partition images, and then, based on the at least two mediastinal partition images, uses at least two branch networks in the neural network model to detect lesions in different anatomical structures of the mediastinal region, so as to obtain the lesion detection result of the mediastinal region of the pulmonary medical image. One branch network is at least used to detect lesions in the anatomical structure of one mediastinal partition image, that is, different branch networks are used to detect lesions in different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0059] In an embodiment of the present application, each branch network at least includes a self-attention network. The self-attention network calculates based on the self-attention mechanism for the mediastinal partition image, enabling each sub-partition image in the mediastinal partition image to establish connections with other sub-partition images, thereby enhancing the feature representation. As Figure 3 shown, step S220 includes the following contents.

[0060] Step S310, based on at least two sub-partition images corresponding to each mediastinal partition image, use the self-attention network to obtain the feature map corresponding to each mediastinal partition image.

[0061] The self-attention network can be called a transformer network, or Transformer Network for short. Through the self-attention mechanism, each sub-partition image in the mediastinal partition image can establish connections with other sub-partition images. That is, the feature map corresponding to each mediastinal partition image is a feature-enhanced feature map.

[0062] Mediastinal lesions may exist at the intersection of two adjacent mediastinal partition images, that is, part of the mediastinal lesion is in one mediastinal partition image of the two adjacent mediastinal partition images, and the other part of the mediastinal lesion is in the other mediastinal partition image of the two adjacent mediastinal partition images. If mediastinal lesion detection is directly performed on each mediastinal partition image, the mediastinal lesion at the intersection cannot be detected or the mediastinal detection result is inaccurate. Therefore, at least two sub-partition images corresponding to each mediastinal partition image can be obtained in the following manner: expand one mediastinal partition image of the two adjacent mediastinal partition images in the direction of the other mediastinal partition image to obtain an expanded mediastinal partition image; partition the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image. Since the expanded mediastinal partition image and the other mediastinal partition image have an overlapping part, the expanded mediastinal partition image can include all mediastinal lesions.

[0063] The re-partitioning of the mediastinal partition image is performed according to position. For example, with a window of a preset size, the mediastinal partition image is intercepted to intercept the mediastinal partition image into four sub-partition images, which are located at the upper left position, upper right position, lower left position, and lower right position of the mediastinal partition image respectively.

[0064] In one example, according to at least two sub-partition images corresponding to one mediastinal partition image of two adjacent mediastinal partition images, a first feature map is obtained through the corresponding self-attention network; according to at least two sub-partition images corresponding to the other mediastinal partition image of the two adjacent mediastinal partition images, a second feature map is obtained through the corresponding self-attention network.

[0065] In another example, according to at least two sub-partition images corresponding to the expanded mediastinal partition image of two adjacent mediastinal partition images, a first feature map is obtained through the corresponding self-attention network; according to at least two sub-partition images corresponding to the other mediastinal partition image of the two adjacent mediastinal partition images, a second feature map is obtained through the corresponding self-attention network.

[0066] It should be noted that the two adjacent mediastinal partition images can use the same self-attention network or different self-attention networks, and the embodiments of the present application do not make specific limitations on this.

[0067] Through the self-attention mechanism of the self-attention network, not only can connections be established between each sub-partition image in the mediastinal partition image and other sub-partition images, but also connections can be established between the sub-partition images in two adjacent mediastinal partition images. Therefore, according to at least two sub-partition images corresponding to another mediastinal partition image in two adjacent mediastinal partition images, through the corresponding self-attention network, a second feature map is obtained, including: according to the first feature map and at least two sub-partition images corresponding to another mediastinal partition image in two adjacent mediastinal partition images, through the corresponding self-attention network, a second feature map is obtained. That is to say, through the self-attention mechanism of the self-attention network, connections can be established between the first feature map and at least two sub-partition images corresponding to another mediastinal partition image in two adjacent mediastinal partition images.

[0068] Step S320, obtain a total feature map according to the feature map corresponding to each mediastinal partition image.

[0069] In one example, for two adjacent mediastinal partition images that are extended, the two adjacent mediastinal partition images have an overlapping part. The overlapping part of the first feature map corresponds to a first overlapping feature map, and the overlapping part of the second feature map corresponds to a second overlapping feature map. The first overlapping feature map and the second overlapping feature map are weighted and summed to obtain a total overlapping feature map; the total overlapping feature map, the feature map in the first feature map except the first overlapping feature map, and the feature map in the second feature map except the second overlapping feature map are combined to obtain a total feature map.

[0070] In another example, for two adjacent mediastinal partition images that are not extended, the feature map corresponding to one mediastinal partition image in the two adjacent mediastinal partition images is directly combined with the feature map corresponding to the other mediastinal partition image in the two adjacent mediastinal partition images to obtain a total feature map.

[0071] Step S330, perform a function operation or a convolution operation on the total feature map to obtain a lesion detection result for the mediastinal region of the pulmonary medical image.

[0072] In one example, when the lesion detection structure is to mark the mediastinal lesion and its probability value as a true mediastinal lesion in the form of a detection box or points on the pulmonary medical image, a function operation is performed on the total feature map using an activation function to obtain a lesion detection result for the mediastinal region of the pulmonary medical image.

[0073] In another example, when the lesion detection structure is a heat map, at least one convolutional layer is used to perform a convolution operation on the total feature map once to obtain a lesion detection result for the mediastinal region of the pulmonary medical image.

[0074] In another embodiment of the present application, considering that mediastinal lesions may change the organizational structure, it may not be accurate enough to partition according to the organizational structure in the mediastinal region such as the heart or aorta. Therefore, according to the prior anatomical knowledge of the mediastinal region, different anatomical structures in the mediastinal region of the pulmonary medical image are partitioned. As Figure 4 shown, it shows a schematic diagram of partitioning the mediastinal region with the straight line between the thoracic vertebra and the spine as the partitioning reference. The straight line between the thoracic vertebra and the spine is denoted as L. At least two mediastinal partition images include the upper mediastinal partition image M1, the middle mediastinal partition image M2, and the lower mediastinal partition image M3. The regional image corresponding to the straight line L on the side close to the thoracic vertebra with the first preset length L1 is the upper mediastinal partition image M1; the regional image corresponding to the straight line L on the side close to the spine with the second preset length L3 is the lower mediastinal partition image M3; the regional image between the upper mediastinal partition image M1 and the lower mediastinal partition image M3 is the middle mediastinal partition image M2.

[0075] The embodiment of the present application does not specifically limit the value of the first preset length. For example, it can be 1 / 5 of the length of the straight line L. The embodiment of the present application also does not specifically limit the value of the second preset length. For example, it can be the length from the bottom end of the spine to the top end of the spine.

[0076] It should be noted that Figure 4 the shown mediastinal partition image includes not only the anatomical structure of the mediastinal region, but also the anatomical structure of the pulmonary region, that is, the mediastinal partition image extends to the pulmonary region on the basis of the partitioning result of the mediastinal region.

[0077] Among the three mediastinal partition images, the middle mediastinal partition image M2 has the largest area and the most complex anatomical structure; the upper mediastinal partition image M1 and the lower mediastinal partition image M3 have smaller areas and fewer anatomical structures. Therefore, in order to enhance the detection performance, improve the calculation efficiency, and reduce the calculation burden, different branch networks are adopted for different mediastinal partition images. For example, the upper mediastinal partition image and the lower mediastinal partition image adopt the same branch network for lesion detection. This branch network can be only a self-attention network. Since the areas of the upper mediastinal partition image and the lower mediastinal partition image are small, using the self-attention network for calculation will not cause an excessive calculation burden. The middle mediastinal partition image adopts another branch network for lesion detection. This branch network can be composed of a self-attention network and an encoder-decoder. Since the area of the middle mediastinal partition image is relatively large and the included anatomical structure is more complex, the encoder-decoder and the self-attention network are used together for feature extraction. In addition, the features of adjacent two sub-partition images in any mediastinal partition image are calculated in association, and the features of adjacent two mediastinal partition images are calculated in association.

[0078] That is to say, when at least two mediastinal partition images include an upper mediastinal partition image, a middle mediastinal partition image, and a lower mediastinal partition image, and the branch networks corresponding to the upper mediastinal partition image and the lower mediastinal partition image are self-attention networks, and the branch network corresponding to the middle mediastinal partition image is a self-attention network and an encoder-decoder, Figure 2 The step S220 shown in Figure 2 includes: expanding the upper mediastinal partition image and the lower mediastinal partition image in the direction of the middle mediastinal partition image respectively to obtain an expanded upper mediastinal partition image and an expanded lower mediastinal partition image; partitioning the expanded upper mediastinal partition image and the expanded lower mediastinal partition image respectively to obtain at least two sub-partition images corresponding to the expanded upper mediastinal partition image and at least two sub-partition images corresponding to the expanded lower mediastinal partition image; inputting the at least two sub-partition images corresponding to the expanded upper mediastinal partition image and the at least two sub-partition images corresponding to the expanded lower mediastinal partition image into the corresponding self-attention networks respectively for feature extraction to obtain a first feature map corresponding to the upper mediastinal partition image and a first feature map corresponding to the lower mediastinal partition image; inputting the middle mediastinal feature image into the encoder for encoding to obtain a decoded feature map; partitioning the decoded feature map to obtain at least two sub-partition images corresponding to the middle mediastinal feature image; inputting the first feature map corresponding to the upper mediastinal partition image, the first feature map corresponding to the lower mediastinal partition image, and the at least two sub-partition images corresponding to the middle mediastinal feature image into the corresponding self-attention networks to obtain a feature-enhanced feature map; inputting the feature-enhanced feature map into the decoder for decoding to obtain a second feature map; merging the first feature map corresponding to the upper mediastinal partition image, the first feature map corresponding to the lower mediastinal partition image, and the second feature map to obtain a total feature map; performing a function operation or a convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the pulmonary medical image. As Figure 5 shown in Figure 5 , it shows the data processing process of the image processing method as described above.

[0079] When merging the first feature map corresponding to the upper mediastinal partition image, the first feature map corresponding to the lower mediastinal partition image, and the second feature map, the weighted sum of the first overlapping feature map corresponding to the upper mediastinal partition image and the second overlapping feature map corresponding to the second feature map can be performed, and the weighted sum of the first overlapping feature map corresponding to the lower mediastinal partition image and the second overlapping feature map corresponding to the second feature map can be performed to obtain a total overlapping feature map; the total overlapping feature map, the feature map of the first feature map corresponding to the upper mediastinal partition image except the first overlapping feature map, the feature map of the first feature map corresponding to the lower mediastinal partition image except the first overlapping feature map, and the feature map of the second feature map except the second overlapping feature map are merged to obtain a total feature map.

[0080] For example, the encoder can be composed of a convolutional neural network, which includes two downsamplings. Therefore, the size of the encoded feature map on each axis is 1 / 4 of the size of the lung medical image. The decoder is also composed of a convolutional neural network, which includes two upsamplings and skip connections from the encoder to the decoder. The skip connections can combine the position information into the features to avoid position deviation in subsequent lesion localization. The embodiments of the present application do not specifically limit the specific composition of the encoder and the decoder.

[0081] As Figure 6 shown, it shows the process of obtaining the image of the region of interest corresponding to the lung medical image. Specifically, a thresholding operation is performed on the lung medical image to obtain a lung field mask; the lung field mask is extended outward to include the thoracic vertebra to obtain a region of interest mask; the lung medical image is cropped using the region of interest mask to obtain the region of interest image.

[0082] When extending the lung field mask outward to include the thoracic vertebra, the minimum bounding rectangle of the lung field mask can be taken, and the minimum bounding rectangle is extended outward to include the thoracic vertebra to obtain the region of interest mask.

[0083] First, considering that the lung field region is mainly composed of air and has a low intensity, a thresholding operation can be performed on the lung medical image to obtain the lung field mask. Second, in order to obtain the straight line between the thoracic vertebra and the spine for partitioning, the lung field mask is extended outward to include the thoracic vertebra. Finally, the lung medical image is cropped using the region of interest mask to obtain the region of interest image, and the region of interest image includes the mediastinal region and removes most of the irrelevant regions.

[0084] In summary, an image processing method provided by the embodiments of the present application divides the mediastinal region of the lung medical image into three parts based on anatomical prior knowledge, extracts features separately in each part, and performs feature interaction enhancement between neighborhoods in each part, thereby improving the computational efficiency and reducing the computational burden while maintaining accurate detection performance. In addition, it makes up for the deficiency that the current computer-aided algorithms lack the detection of mediastinal lesions in medical images, and further improves the application of AI in medical image analysis.

[0085] Next, in conjunction with Figure 7 the training method of the neural network model mentioned in the embodiments of the present application will be introduced in detail. Figure 7 The method described is executed by Figure 1 the server 130 mentioned in or other types of electronic devices with data processing functions. As Figure 7 shown, the training method of the neural network model provided by the embodiments of the present application includes the following steps.

[0086] It should be noted that some of the content mentioned in the following embodiments related to the training method of the neural network model is the same as that mentioned in the embodiments related to the image processing method. The following focuses on the differences between the two, and the same content will not be repeated. For details, please refer to the embodiments related to the image processing method.

[0087] Step S710: Partition the mediastinal region of the pulmonary medical sample image to obtain at least two mediastinal partition sample images.

[0088] The pulmonary medical sample image is labeled with lesion labels, but the specific form of the lesion labels is not specifically limited in the embodiments of the present application. The lesion labels can be mediastinal lesions marked in the form of detection boxes or points on the pulmonary medical sample image, or Gaussian heat maps constructed with the center of each mediastinal lesion on the pulmonary medical sample image as the core. The values of pixel points closer to the center are larger than those of pixel points farther from the center.

[0089] Step S720: According to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, obtain the lesion prediction results of the mediastinal region of the pulmonary medical sample image through at least two branch networks in the neural network model.

[0090] The specific form of the lesion prediction results is also determined according to the lesion labels. When the lesion labels are mediastinal lesions marked in the form of detection boxes or points on the pulmonary medical sample image, the lesion prediction results refer to the mediastinal lesions marked in the form of detection boxes or points on the pulmonary medical sample image and the probability values that they are real mediastinal lesions; when the lesion labels are Gaussian heat maps constructed with the center of each mediastinal lesion on the pulmonary medical sample image as the core, the lesion prediction results refer to the heat maps and the values of each pixel point in the heat maps.

[0091] Step S730: Calculate the loss function value according to the lesion prediction results and the lesion labels.

[0092] Determine the difference between the lesion prediction results and the lesion labels, and input the difference into the loss function to calculate the loss function value. In one example, when the lesion prediction result is a heat map, calculate the regression loss pixel by pixel to obtain the loss function value.

[0093] Step S740: Update the parameters of the neural network model according to the loss function value.

[0094] Perform gradient backpropagation on the loss function value to update the parameters in the neural network model, such as weights, biases, etc. The embodiments of the present application do not make specific limitations on this.

[0095] A training method for a neural network model provided by an embodiment of the present application first partitions the mediastinal region of a pulmonary medical sample image to obtain at least two mediastinal partition sample images, and then, according to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, obtains a lesion prediction result of the mediastinal region of the pulmonary medical sample image through at least two branch networks in the neural network model. Finally, according to the lesion prediction result and the lesion label, a loss function value is calculated, and according to the loss function value, the parameters of the neural network model are updated. One branch network in the trained neural network model is at least used to detect lesions in the anatomical structure of one mediastinal partition image, that is, different branch networks are used for lesion detection of different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0096] As described above in conjunction with Figures 2 to 7 , the method embodiments of the present application have been described in detail. Next, in conjunction with Figure 8 and Figure 9 , the apparatus embodiments of the present application will be described in detail. It should be understood that the descriptions of the method embodiments and the apparatus embodiments correspond to each other. Therefore, for parts not described in detail, reference may be made to the foregoing method embodiments.

[0097] Figure 8 FIG. is a schematic structural diagram of an image processing apparatus 800 provided by an embodiment of the present application. As shown in Figure 8 , the apparatus 800 may include: a first acquisition module 810 and a second acquisition module 820. These modules will be introduced in detail below. Figure 8 The apparatus 800 may include: a first acquisition module 810 and a second acquisition module 820. The following provides a detailed introduction to these modules.

[0098] The first acquisition module 810 is configured to partition the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region.

[0099] The second acquisition module 820 is configured to perform lesion detection on different anatomical structures according to at least two mediastinal partition images through at least two branch networks in the neural network model to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is used to detect lesions in the anatomical structure of one mediastinal partition image.

[0100] An image processing apparatus provided by an embodiment of the present application first partitions the mediastinal region of a pulmonary medical image to obtain at least two mediastinal partition images, and then, based on the at least two mediastinal partition images, uses at least two branch networks in a neural network model to detect lesions in different anatomical structures of the mediastinal region, thereby obtaining a lesion detection result for the mediastinal region of the pulmonary medical image. At least one branch network is used to detect lesions in the anatomical structure of at least one mediastinal partition image. That is to say, different branch networks are used to detect lesions in different mediastinal partition images to achieve optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0101] In an embodiment of the present application, each branch network includes a self-attention network. The second acquisition module 820 is further configured to, based on at least two sub-partition images corresponding to each mediastinal partition image, obtain a feature map corresponding to each mediastinal partition image through the self-attention network; obtain a total feature map based on the feature map corresponding to each mediastinal partition image; perform a function operation or a convolution operation on the total feature map to obtain a lesion detection result for the mediastinal region of the pulmonary medical image.

[0102] In an embodiment of the present application, the apparatus 800 further includes: an expansion module 830 configured to expand one of two adjacent mediastinal partition images in the direction of the other mediastinal partition image to obtain an expanded mediastinal partition image, where the expanded mediastinal partition image has an overlapping part with the other mediastinal partition image; a partitioning module 840 configured to partition the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image.

[0103] In an embodiment of the present application, when the second acquisition module 820 obtains a feature map corresponding to each mediastinal partition image through the self-attention network based on at least two sub-partition images corresponding to each mediastinal partition image, it is further configured to obtain a first feature map through the self-attention network based on at least two sub-partition images corresponding to one of two adjacent mediastinal partition images; and obtain a second feature map through the self-attention network based on the first feature map and at least two sub-partition images corresponding to the other mediastinal partition image among the two adjacent mediastinal partition images.

[0104] In an embodiment of the present application, when there is an overlapping part between two adjacent mediastinal partition images, the overlapping part of the first feature map corresponds to the first overlapping feature map, and the overlapping part of the second feature map corresponds to the second overlapping feature map. When the second acquisition module 820 obtains the total feature map according to the feature maps corresponding to each mediastinal partition image, it is further configured to perform weighted summation on the first overlapping feature map and the second overlapping feature map to obtain the total overlapping feature map; and combine the total overlapping feature map, the feature maps in the first feature map other than the first overlapping feature map, and the feature maps in the second feature map other than the second overlapping feature map to obtain the total feature map.

[0105] In an embodiment of the present application, the first acquisition module 810 is further configured to use a mediastinal window to perform a windowing operation on the image of the region of interest corresponding to the pulmonary medical image to obtain a mediastinal region; use a bone window to perform a windowing operation on the image of the region of interest corresponding to the pulmonary medical image to obtain a bone region, where the bone region includes the thoracic vertebra and the spinal column; and use the straight line between the thoracic vertebra and the spinal column as a partitioning reference to partition the mediastinal region to obtain at least two mediastinal partition images.

[0106] In an embodiment of the present application, when the first acquisition module 810 uses the straight line between the thoracic vertebra and the spinal column as a partitioning reference to partition the mediastinal region to obtain at least two sub-partition images, it is further configured to use the region image corresponding to the straight line on the side close to the thoracic vertebra with a first preset length as the upper mediastinal partition image; use the region image corresponding to the straight line on the side close to the spinal column with a second preset length as the lower mediastinal partition image; and use the region image between the upper mediastinal partition image and the lower mediastinal partition image as the middle mediastinal partition image.

[0107] In an embodiment of the present application, the upper mediastinal partition image and the lower mediastinal partition image use the same branch network for lesion detection, and the middle mediastinal partition image uses another branch network for lesion detection.

[0108] In an embodiment of the present application, the device 800 further includes: a third acquisition module 850, configured to perform a thresholding operation on the pulmonary medical image to obtain a lung field mask; expand the lung field mask outward to include the thoracic vertebra to obtain a region of interest mask; and use the region of interest mask to perform a cropping operation on the pulmonary medical image to obtain an image of the region of interest.

[0109] Figure 9 is a schematic structural diagram of a training device 900 of a neural network model provided by an embodiment of the present application. As Figure 9 shown, Figure 9 the training device 900 may include: a fourth acquisition module 910 and a training module 920. These modules will be introduced in detail below.

[0110] The fourth acquisition module 910 is configured to partition the mediastinal region of the lung medical sample image to obtain at least two mediastinal partition sample images, wherein the lung medical sample image is labeled with a lesion label.

[0111] The training module 920 is configured to obtain a lesion prediction result of the mediastinal region of the lung medical sample image through at least two branch networks in the neural network model according to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, wherein one branch network is at least used to predict the mediastinal lesion in the anatomical structure of one mediastinal partition sample image; calculate a loss function value according to the lesion prediction result and the lesion label; and update the parameters of the neural network model according to the loss function value.

[0112] An apparatus for training a neural network model provided by an embodiment of the present application first partitions the mediastinal region of the lung medical sample image to obtain at least two mediastinal partition sample images, then obtains a lesion prediction result of the mediastinal region of the lung medical sample image through at least two branch networks in the neural network model according to at least two mediastinal partition sample images corresponding to each mediastinal partition sample image, and finally calculates a loss function value according to the lesion prediction result and the lesion label, and updates the parameters of the neural network model according to the loss function value. One branch network in the trained neural network model is at least used to detect the lesion in the anatomical structure of one mediastinal partition image, that is, different branch networks are used for lesion detection for different mediastinal partition images to achieve the best lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

[0113] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Figure 10 The illustrated electronic device 1000 may include a memory 1010 and a processor 1020. The memory 1010 can be used to store executable code. The processor 1020 can be used to execute the executable code stored in the memory 1010 to implement the steps in the various methods described above. In some embodiments, the electronic device 1000 may further include a network interface 1030, and the data exchange between the processor 1000 and external devices can be realized through the network interface 1030.

[0114] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as Digital Video Disc (DVD)), or semiconductor media (such as Solid State Disk (SSD)), etc.

[0115] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments of the present application can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware

[0116] or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the

[0117] scope of this application.

[0118] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the

[0119] division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another

[0120] For the points, the couplings or direct couplings or communication connections shown or discussed among each other can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0121] The units described as separate components can be or may not be physically separated. As units shown

[0122] the components can be or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, can also be physically present separately for each unit, or two or more units can be integrated in one unit.

[0124] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art in the technical field disclosed by the present application can easily think of changes or substitutions within the technical scope disclosed by the present application, and all of them

[0125] should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the said claims.

Claims

1. An image processing method, characterized in that, Including: Partition the mediastinal region of the pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region; According to the at least two mediastinal partition images, through at least two branch networks in the neural network model, perform lesion detection on the different anatomical structures to obtain the lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is at least used to detect the mediastinal lesions in the anatomical structure of one mediastinal partition image, and the branch network includes a self-attention network; the step of according to the at least two mediastinal partition images, through at least two branch networks in the neural network model, perform lesion detection on the different anatomical structures to obtain the lesion detection result of the mediastinal region of the pulmonary medical image includes: According to at least two sub-partition images corresponding to each mediastinal partition image, obtain the feature map corresponding to each mediastinal partition image through the self-attention network; Obtain the total feature map according to the feature map corresponding to each mediastinal partition image; Perform a function operation on the total feature map to obtain the lesion detection result of the mediastinal region of the pulmonary medical image; Before obtaining the feature map corresponding to each mediastinal partition image according to at least two sub-partition images corresponding to each mediastinal partition image through the self-attention network, it further includes: Expand one mediastinal partition image in the adjacent two mediastinal partition images in the direction of the other mediastinal partition image to obtain an expanded mediastinal partition image, where the expanded mediastinal partition image and the other mediastinal partition image have an overlapping part; Partition the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image.

2. The method according to claim 1, wherein The step of obtaining the feature map corresponding to each mediastinal partition image according to at least two sub-partition images corresponding to each mediastinal partition image through the self-attention network includes: According to at least two sub-partition images corresponding to one mediastinal partition image in the adjacent two mediastinal partition images, obtain the first feature map through the self-attention network; According to the first feature map and at least two sub-partition images corresponding to the other mediastinal partition image in the adjacent two mediastinal partition images, obtain the second feature map through the self-attention network.

3. The method according to claim 2, wherein When the adjacent two mediastinal partition images have an overlapping part, the overlapping part of the first feature map corresponds to the first overlapping feature map, and the overlapping part of the second feature map corresponds to the second overlapping feature map, where the step of obtaining the total feature map according to the feature map corresponding to each mediastinal partition image includes: Perform weighted summation on the first overlapping feature map and the second overlapping feature map to obtain the total overlapping feature map; Merge the total overlapping feature map, the feature map in the first feature map except the first overlapping feature map, and the feature map in the second feature map except the second overlapping feature map to obtain the total feature map.

4. The method according to any one of claims 1 to 3, characterized in that Partitioning the mediastinal region of the pulmonary medical image to obtain at least two mediastinal partition images, including: Using a mediastinal window to perform a windowing operation on the region of interest image corresponding to the pulmonary medical image to obtain the mediastinal region; Using a bone window to perform a windowing operation on the region of interest image corresponding to the pulmonary medical image to obtain a bone region, where the bone region includes the thoracic vertebra and the spinal column; Taking the straight line between the thoracic vertebra and the spinal column as the partitioning reference to partition the mediastinal region to obtain the at least two mediastinal partition images.

5. The method according to claim 4, characterized in that, The step of taking the straight line between the thoracic vertebra and the spinal column as the partitioning reference to partition the mediastinal region to obtain the at least two sub-partition images, including: Taking the region image corresponding to the straight line on the side close to the thoracic vertebra with a first preset length as the upper mediastinal partition image; Taking the region image corresponding to the straight line on the side close to the spinal column with a second preset length as the lower mediastinal partition image; Taking the region image between the upper mediastinal partition image and the lower mediastinal partition image as the middle mediastinal partition image, wherein, the upper mediastinal partition image and the lower mediastinal partition image use the same branch network for lesion detection, and the middle mediastinal partition image uses another branch network for lesion detection.

6. The method according to claim 4, characterized in that It further includes: Performing a thresholding operation on the pulmonary medical image to obtain a lung field mask; Expanding the lung field mask outward to include the thoracic vertebra to obtain a region of interest mask; Using the region of interest mask to perform a cropping operation on the pulmonary medical image to obtain the region of interest image.

7. An image processing apparatus, characterized in that, Including: A first acquisition module configured to partition the mediastinal region of the pulmonary medical image to obtain at least two mediastinal partition images, where different mediastinal partition images include different anatomical structures of the mediastinal region; A second acquisition module configured to perform lesion detection on the different anatomical structures according to the at least two mediastinal partition images through at least two branch networks in a neural network model to obtain a lesion detection result of the mediastinal region of the pulmonary medical image, where one branch network is at least used to detect mediastinal lesions in the anatomical structure of one mediastinal partition image, and the branch network includes a self-attention network; the step of performing lesion detection on the different anatomical structures according to the at least two mediastinal partition images through at least two branch networks in a neural network model to obtain a lesion detection result of the mediastinal region of the pulmonary medical image includes: According to at least two sub-partition images corresponding to each mediastinal partition image, obtaining a feature map corresponding to each mediastinal partition image through the self-attention network; Obtaining a total feature map according to the feature map corresponding to each mediastinal partition image; Performing a function operation on the total feature map to obtain a lesion detection result of the mediastinal region of the pulmonary medical image; Before obtaining a feature map corresponding to each mediastinal partition image according to at least two sub-partition images corresponding to each mediastinal partition image through the self-attention network, it further includes: Expand one of the adjacent mediastinal partition images towards the direction where the other mediastinal partition image is located to obtain an expanded mediastinal partition image, wherein the expanded mediastinal partition image and the other mediastinal partition image have an overlapping part; Partition the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image.

8. An electronic device, comprising: A processor; And A memory in which computer program instructions are stored, and when the computer program instructions are run by the processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images