Image processing method and device

Abstract

The invention discloses an image processing method and device, and the method comprises the steps: enabling a first lung image to comprise a lung, a bronchus part having a first medical position relation with the lung, and an artery part having a second medical position relation with the lung, and enabling the first lung image to be inputted into a first bronchus segmentation model for segmentation, obtaining a bronchus segmentation result; segmenting the lung based on a plurality of lung segmentation models constructed based on the first medical position relationship and the bronchial segmentation result to obtain a lung segmentation result, the plurality of lung segmentation models including segmentation models constructed for different lung segments; and obtaining an artery segmentation result based on the lung segmentation result, the second medical position relation and an artery segmentation model. According to the technical scheme, the accuracy of artery segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of computer image processing technology, and particularly to an image processing method and device. Background Art

[0002] Currently, in various thoracic surgeries, especially lung surgeries, it is usually necessary to perform fine segmentation on bronchi, pulmonary arteries, and lung tissues. Such segmentation not only helps to clarify the position and function of each anatomical structure, but also provides a more accurate operation perspective for surgeons, thus ensuring the safety and effectiveness of the surgery. During the surgical planning process, especially the segmentation of the pulmonary artery, has become a key link. However, due to the complex structure of the pulmonary artery and significant individual differences, especially in terms of branching and orientation, in some cases, the accuracy of artery segmentation is relatively low.

[0003] In view of this, how to improve the accuracy of artery segmentation has become a technical problem to be urgently solved. Summary of the Invention

[0004] In view of this, the embodiments of this application provide an image processing method and device, which can improve the accuracy of artery segmentation.

[0005] In a first aspect, the embodiments of this application provide an image processing method. The first lung image includes the lungs, a bronchial part having a first medical positional relationship with the lungs, and an arterial part having a second medical positional relationship with the lungs. The method includes: inputting the first lung image into a first bronchus segmentation model for segmentation to obtain a bronchus segmentation result; segmenting the lungs using multiple lung segmentation models constructed based on the first medical positional relationship and the bronchus segmentation result to obtain a lung segmentation result, where the multiple lung segmentation models include segmentation models constructed for different lung segments; and obtaining an artery segmentation result based on the lung segmentation result, the second medical positional relationship, and an artery segmentation model.

[0006] In a second aspect, the embodiments of this application provide an image processing device, including: a bronchus segmentation module, configured to input the first lung image into a first bronchus segmentation model for segmentation to obtain a bronchus segmentation result, where the first lung image includes the lungs, a bronchial part having a first medical positional relationship with the lungs, and an arterial part having a second medical positional relationship with the lungs; a lung segmentation module, configured to segment the lungs using multiple lung segmentation models constructed based on the first medical positional relationship and the bronchus segmentation result to obtain a lung segmentation result, where the multiple lung segmentation models include segmentation models constructed for different lung segments; and an artery segmentation module, configured to obtain an artery segmentation result based on the lung segmentation result, the second medical positional relationship, and an artery segmentation model.

[0007] The embodiments of the present application provide an image processing method and apparatus. By inputting the first lung image into the first bronchus segmentation model for segmentation to obtain the bronchus segmentation result, then segmenting the lungs using multiple lung segmentation models constructed based on the first medical positional relationship between the lungs and the bronchus part and the bronchus segmentation result to obtain the lung segmentation result, and finally obtaining the artery segmentation result based on the lung segmentation result, the second medical positional relationship between the lungs and the artery part, and the artery segmentation model. The embodiments of the present application first obtain the bronchus segmentation result in combination with the medical background, then obtain the lung segmentation result according to the bronchus segmentation result, and further obtain the artery segmentation result according to the lung segmentation result, improving the accuracy and precision of artery segmentation. Description of the Drawings

[0008] The drawings are used to provide a further understanding of the present disclosure and form a part of the specification. They are used together with the embodiments of the present disclosure to explain the present disclosure and do not constitute a limitation to the present disclosure. By describing the detailed exemplary embodiments with reference to the drawings, the above and other features and advantages will become more apparent to those skilled in the art. In the drawings:

[0009] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application.

[0010] Figure 2 is a flowchart of an image processing method provided by an exemplary embodiment of the present application.

[0011] Figure 3 is a schematic diagram of a bronchus segmentation result provided by an exemplary embodiment of the present application.

[0012] Figure 4 is a schematic diagram of a lung segmentation result provided by an exemplary embodiment of the present application.

[0013] Figure 5 is a schematic diagram of an artery segmentation result provided by an exemplary embodiment of the present application.

[0014] Figure 6 is a flowchart of an image processing method provided by another exemplary embodiment of the present application.

[0015] Figure 7 is a flowchart of an image processing method provided by yet another exemplary embodiment of the present application.

[0016] Figure 8 is a flowchart of an image processing method provided by still another exemplary embodiment of the present application.

[0017] Figure 9 is a schematic diagram of the mapping relationship between the bronchus and lung segments provided by an exemplary embodiment of the present application.

[0018] Figure 10 It is a schematic structural diagram of an image processing apparatus provided by an exemplary embodiment of the present application.

[0019] Figure 11 It is a block diagram of an electronic device for image processing provided by an exemplary embodiment of the present application. Detailed implementation manners

[0020] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0021] Currently, thoracic surgery (especially lung surgery) usually requires a detailed segmentation of the bronchi, arteries (i.e., pulmonary arteries), and lungs in order to better plan surgical operations. Among them, the lungs and bronchi are usually divided into 18 segments. This segmentation process not only involves dividing the lungs into different lung segments, but also requires corresponding division of the bronchi and arteries to ensure that the diseased area can be accurately removed during the surgical process and normal tissues can be retained to the greatest extent.

[0022] However, when implementing the surgery, there are still certain challenges in the accuracy of artery segmentation. Although modern medical imaging technologies and surgical planning tools have been widely used, due to the complex anatomical structure and large variations of arteries, it is difficult to accurately segment the arteries. This lack of accuracy may affect the fineness and safety of the surgery. Especially when dealing with diseases involving multiple lung segments, if the artery segmentation is not accurate enough, it may lead to unnecessary resection of some healthy tissues, or the diseased area cannot be completely removed, thereby affecting the postoperative recovery of the patient.

[0023] In view of the above problems, the embodiments of the present application provide an image processing method. Next, various non-limiting embodiments of the present application will be specifically introduced with reference to the accompanying drawings.

[0024] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a CT scanner 130, a server 120, and a computer device 110. The computer device 110 can obtain medical images (such as lung images) from the CT scanner 130 used for X-ray scanning of human tissues. At the same time, the computer device 110 can also be connected to the server 120 through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0025] The computer device 110 can be a general-purpose computer or a computer device composed of dedicated integrated circuits, etc., and the embodiments of the present application do not make specific limitations thereto. For example, the computer device 110 can be a mobile terminal device such as a tablet computer, or can also be a personal computer (PC), such as a laptop computer and a desktop computer, etc. Those skilled in the art can know that the number of the above computer devices 110 can be one or more, and their types can be the same or different. For example, the above computer device 110 can be one, or the above computer device 110 can be dozens or hundreds, or more. The embodiments of the present application do not limit the number and device type of the computer device 110.

[0026] The server 120 can be a single server, or composed of several servers, or a virtualization platform, or a cloud computing service center.

[0027] Figure 2 It is a schematic flowchart of an image processing method provided by an exemplary embodiment of the present application. Figure 2 The method is executed by a computing device, such as a server. As Figure 2 shown, the image processing method includes the following contents.

[0028] In one embodiment, the first lung image includes a lung, a bronchial part having a first medical positional relationship with the lung, and an arterial part having a second medical positional relationship with the lung. The lung can include multiple lung segments, each lung segment is supplied with air by an independent tertiary bronchus, and each lung segment is supplied with blood by an independent tertiary arterial branch, that is, each lung segment has an independent bronchial branch, and each lung segment has an independent arterial branch to supply blood. The first medical positional relationship can be understood as the positional relationship between the bronchial segment and the lung segment, that is, the corresponding relationship between the bronchial segment and the lung segment. The second medical positional relationship can be understood as the positional relationship between the arterial segment and the lung segment, that is, the corresponding relationship between the arterial segment and the lung segment.

[0029] S210: Input the first lung image into the first bronchial segmentation model for segmentation to obtain a bronchial segmentation result.

[0030] Specifically, the first lung image can be a lung CT image including bronchial lobation results, lung lobation results, and arterial lobation results. It should be noted that since the segmentation operation (for example, bronchial segmentation, lung segmentation, etc.) is performed inside the corresponding lobe, the bronchus, lung, and artery are first lobed.

[0031] In one embodiment, the bronchus includes multiple bronchial segments, and the bronchial segmentation result may include the segmentation results of multiple bronchial segments. That is to say, the bronchial segmentation result can be understood as being composed of the segmentation results respectively corresponding to multiple bronchial segments. For example, referring to Figure 3 the bronchial segmentation result shown, where the bronchial segments of different colors represent multiple different types of bronchial segments (for example, the first bronchial segment or the second bronchial segment).

[0032] Input the first lung image into the first bronchial segmentation model to segment each of the multiple bronchial segments one by one, obtain the segmentation result of each bronchial segment, and then construct the bronchial segmentation result according to the segmentation results respectively corresponding to the multiple bronchial segments. The first bronchial segmentation model is used to segment the bronchial part in the first lung image.

[0033] S220: Segment the lungs using multiple lung segmentation models constructed based on the first medical positional relationship and the bronchial segmentation result to obtain the lung segmentation result.

[0034] In one embodiment, the multiple lung segmentation models include segmentation models constructed for different lung segments.

[0035] Specifically, according to the first medical positional relationship, determine the mapping relationship between the lung segments and the bronchial segmentation result. Different lung segments in different positions can be considered as different types of lung segments, and then set the corresponding lung segmentation models for different types of lung segments. And use the constructed multiple lung segmentation models for different types to segment the first lung image to obtain the lung segmentation result. It should be noted that the first lung image at this time is a lung image including the bronchial segmentation result.

[0036] For example, referring to Figure 4 the lung segmentation result shown, where the lung segments of different colors represent different types of lung segments.

[0037] S230: Based on the lung segmentation result, the second medical positional relationship, and the artery segmentation model, obtain the artery segmentation result.

[0038] In one embodiment, the artery segmentation result may be composed of the segmentation results of multiple artery segments.

[0039] Specifically, input the first lung image containing the lung segmentation result into the artery segmentation model to obtain the artery segmentation result. It should be understood that the artery segmentation result obtained at this time may include the segmentation results of multiple artery segments, and the segmentation results of the multiple artery segments are not classified. Then, according to the positional relationship between the lung segmentation result mapped by the second medical positional relationship and the artery segmentation result, determine the categories of the segmentation results of the multiple artery segments to obtain the segmentation results of the multiple artery segments, that is, the artery segmentation result.

[0040] For example, the parameter Figure 5 shows the arterial segmentation result, where the arterial segments of different colors represent different types of arterial segments.

[0041] It can be seen that in the embodiment of the present application, the first lung image is input into the first bronchus segmentation model for segmentation to obtain the bronchus segmentation result, and then the lungs are segmented by multiple lung segmentation models constructed based on the first medical positional relationship between the lungs and the bronchus part and the bronchus segmentation result to obtain the lung segmentation result. Finally, based on the lung segmentation result, the second medical positional relationship between the lungs and the artery part, and the artery segmentation model, the arterial segmentation result is obtained. This enables the embodiment of the present application to first obtain the bronchus segmentation result in combination with the medical background, then obtain the lung segmentation result according to the bronchus segmentation result, and further obtain the arterial segmentation result according to the lung segmentation result, improving the accuracy and precision of arterial segmentation.

[0042] In an embodiment of the present application, the bronchus segmentation result includes the segmentation results of multiple bronchus segments. Among them, inputting the first lung image into the first bronchus segmentation model for segmentation to obtain the bronchus segmentation result includes: inputting the first lung image into the first bronchus segmentation model to segment each of the multiple bronchus segments one by one to obtain the segmentation result of each bronchus segment; constructing the segmentation results of multiple bronchus segments based on the segmentation results of each bronchus segment.

[0043] Specifically, the first bronchus segmentation model may include a sparse convolution model and a bronchus classification prediction model. The sparse convolution (spconv) model can be understood as a feature extraction backbone network for extracting the feature data of the bronchus part, and the bronchus classification prediction model can be understood as a classification prediction head for obtaining the segmentation result of each bronchus segment.

[0044] In an embodiment, the feature data of the bronchus part in the first lung image is extracted by the sparse convolution model, and then the feature data and the encoded data of each bronchus segment are input into the bronchus classification prediction model, thereby obtaining the segmentation results of multiple bronchus segments. The encoded data of the bronchus segment can be understood as the identification data of the bronchus segment, and each bronchus segment has its unique encoded data.

[0045] It should be noted that for the specific description of the embodiment of the present application, please refer to Figure 6 the description of the embodiment. To avoid repetition, it will not be elaborated here.

[0046] It can be seen that in the embodiment of the present application, by segmenting the bronchus segment by segment and constructing the bronchus segmentation result based on the segmentation result of each bronchus segment, the segmentation accuracy of each bronchus segment and the accuracy of the bronchus segmentation result are improved.

[0047] Figure 6 It is a schematic flowchart of an image processing method provided by another exemplary embodiment of the present application. Figure 6 The embodiment is Figure 2 A further description of step S210 in the embodiment. The image processing method includes the following contents.

[0048] In one embodiment, the plurality of bronchial segments include a first bronchial segment. The first bronchial segment can be any one of the plurality of bronchial segments, and the embodiments of the present application do not specifically limit the first bronchial segment.

[0049] S610: Extract the feature data of the bronchial part in the first lung image through a sparse convolution model.

[0050] Specifically, the sparse convolution model is used to extract the features of the bronchial part. Sparse convolution is a convolution method effective in processing sparse data (such as point cloud data). It saves computing resources compared to traditional convolution and is applicable to scenarios that need to process sparse and irregular data. That is to say, the first bronchial segmentation model first extracts the high-dimensional features of the bronchial part through the sparse convolution model, that is, the feature data of the bronchial part.

[0051] S620: Input the feature data and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result of the first bronchial segment.

[0052] Specifically, the encoded data can be understood as the unique identification data of the first bronchial segment, and the encoding method of the encoded data can adopt one-hot encoding, etc. For example, the encoded data of the first bronchial segment can be the one-hot encoding corresponding to the first bronchial segment. It should be noted that one-hot encoding is to convert the category of each segment into a vector representation, and only the bit corresponding to the category in this vector is 1, and the other bits are 0.

[0053] The bronchial classification prediction model can adopt the Mask2Former structure, which is a structure suitable for instance (i.e., each bronchial segment) segmentation. The task of instance segmentation is to distinguish different instances (such as the first bronchial segment) in the first lung image, and each instance may be a part of an object (i.e., the bronchus). That is to say, for each bronchial segment, the bronchial classification prediction model can output which part of the bronchus it belongs to (i.e., which bronchial segment) and the type of the bronchial segment.

[0054] In one embodiment, input the feature data of the bronchial part and the one-hot encoding of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result of the first bronchial segment, that is, the instance where the first bronchial segment is located.

[0055] It can be seen from this that in the embodiment of the present application, the feature extraction part obtains the feature data of the bronchial part, and then predicts each part of the bronchus segment by segment, improving the prediction accuracy of each bronchial segment. At the same time, the sparse convolution model is used to extract the feature data of the bronchial part, saving computing resources and improving the adaptability of the embodiment of the present application.

[0056] In an embodiment of the present application, the multiple bronchial segments include at least one second bronchial segment that has been segmented. Among them, inputting the feature data and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result of the first bronchial segment includes: inputting the segmentation results of at least one second bronchial segment, the feature data, and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result.

[0057] Specifically, the second bronchial segment can be the bronchial segment for which the segmentation prediction has been completed and output by the bronchial classification prediction model. The number of second bronchial segments can be one or more, and the embodiment of the present application does not make specific limitations on this.

[0058] In an embodiment, when performing segmentation prediction on the first bronchial segment, not only the first bronchial segment that needs to be segmented and predicted currently should be considered, but also the information of all the previously segmented second bronchial segments should be considered. That is to say, when performing segmentation prediction on the first bronchial segment, the bronchial classification prediction model will jointly use the segmentation results of all known second bronchial segments, the feature data of the bronchial part, and the encoded data of the first bronchial segment as inputs to facilitate the output of the first segmentation result of the first bronchial segment.

[0059] It should be noted that the segment-by-segment prediction method adopted in the embodiment of the present application can be regarded as an autoregressive process, and the prediction of each bronchial segment depends on the previously predicted segmentation results.

[0060] It can be seen from this that in the embodiment of the present application, the feature extraction part obtains the feature data of the bronchial part, and then performs segmentation prediction on each segment of the bronchus segment by segment, and each prediction combines the previously predicted segmentation results, ensuring that the segmentation results of each bronchial segment are more accurate and consistent.

[0061] Figure 7 It is a schematic flowchart of an image processing method provided by another exemplary embodiment of the present application. Figure 7 The embodiment is Figure 6 The steps further included after step S620 in the embodiment. This image processing method further includes the following content.

[0062] S710: Encode the first segmentation result to obtain the encoded first segmentation result.

[0063] Specifically, the first bronchial segmentation model may further include a translation model, which is used to generate a corresponding encoded description according to the input first segmentation result. That is to say, the translation model can convert the segmentation result of each bronchial segment into a form of encoded description.

[0064] In one embodiment, the first segmentation prediction result of the first bronchial segment is input into the translation model, and then the encoded first segmentation result is output.

[0065] The translation model can adopt Image Translation (IT) technology to achieve two-way translation of "image to description" and "description to image". For example, it may be to translate the segmentation result in the form of an image into a description in a certain encoded format, or generate an image of the segmentation result according to the encoded description. The translation model required in this way can be trained using a Conditional Generative Adversarial Network (CGAN). It should be noted that CGAN is a generative model, which can generate an image related to the input condition and has strong image generation ability.

[0066] It should be noted that the training of the translation model can be two-way. It can be trained by converting the image into an encoded description, or by converting the encoded description into an image. By adopting the above training method, the translation model has flexible conversion ability.

[0067] S720: Input the encoded first segmentation result into the language judgment model to obtain the predicted probability value for the first bronchial segment.

[0068] Specifically, the first bronchial segmentation model may further include a language judgment model. The language judgment model may be a Generative Pretrained Transformer (GPT), and the embodiments of the present application do not make specific limitations on this. The large language judgment model can use the trained GPT as pre-training to judge whether the first segmentation result predicted by the bronchial classification prediction model is correct. By using these pre-trained GPTs, the language judgment model can utilize its powerful language understanding ability to judge the rationality of the predicted first segmentation result.

[0069] In one embodiment, the result judged by the language judgment model can be reflected in the form of a probability value, that is, the output result of the language judgment model can be the predicted probability value (that is, the probability value of correct prediction).

[0070] It should be noted that when training the language judgment model, the encoded descriptions generated by the translation model can be used as input to further train the language judgment model. In this way, the language judgment model can not only understand ordinary language information, but also specifically judge the correctness of the first segmentation result.

[0071] It should be understood that the processes of translation and judgment of the segmentation results are carried out paragraph by paragraph. Therefore, for each bronchial segment, the language judgment model will judge whether the predicted segmentation result is correct. Specifically, it is to calculate the probability of the current segment (for example, the first bronchial segment) being predicted correctly.

[0072] S730: When the predicted probability value is less than the preset probability threshold, re-segment the first bronchial segment to obtain the second segmentation result of the first bronchial segment.

[0073] Specifically, the predicted probability value output by the language judgment model is judged. When the predicted probability value is greater than the preset probability threshold, it indicates that the first segmentation result predicted for the first bronchial segment is correct, and at this time, the segmentation prediction of the first bronchial segment is completed. Or, when the predicted probability value is less than the preset probability threshold, it indicates that the first segmentation result predicted for the first bronchial segment may be incorrect. At this time, the first bronchus will be re-segmented and predicted, so as to correct or optimize the incorrect first segmentation result and obtain the second segmentation result of the first bronchial segment. The preset probability threshold can be 0.5, 0.55 or 0.6, etc., and the embodiments of the present application do not make specific limitations on this.

[0074] It should be noted that Figure 7 The process of the illustrated embodiment can be understood as first generating an encoded description (for example, image to description) of each bronchial segment through the translation model. Then, the language judgment model based on the trained GPT is used to judge whether the predicted segmentation result of each bronchial segment is correct. If the predicted probability value of a certain bronchial segment (for example, the first bronchial segment) is lower than the preset probability threshold, error correction is performed.

[0075] It can be seen from this that the embodiments of the present application, by combining the image translation technology and the judgment ability of the large language judgment model, can perform accuracy judgment and error correction in the segmentation prediction of each bronchial segment, improving the accuracy of the segmentation result of the bronchial segment prediction.

[0076] In an embodiment of the present application, re-segmenting the first bronchial segment to obtain the second segmentation result of the first bronchial segment includes: obtaining the feature data of the first bronchial segment; inputting the feature data of the first bronchial segment, the first segmentation result and the encoded data of the first bronchial segment into the second bronchial segmentation model to obtain the second segmentation result, where the second segmentation result is different from the first segmentation result.

[0077] Specifically, the server may further include a second bronchial segmentation model, where the structure of the second bronchial segmentation model is similar to that of the first bronchial segmentation model. And the second bronchial segmentation model may include a sparse convolution model and a bronchial classification prediction model, where the sparse convolution model may be shared with the first bronchial segmentation model. That is to say, the above two models use the same feature extraction part when processing the input data, so as to ensure that both models extract information from the same features.

[0078] It should be noted that the second bronchial segmentation model can be understood as an error correction model, which corrects the first segmentation result with segmentation errors.

[0079] In one embodiment, the input of the second bronchial segmentation model not only includes the feature data of the first bronchial segment and the encoded data of the first bronchial segment, but also includes the incorrect first segmentation result. That is to say, the second bronchial segmentation model not only corrects the prediction of the first bronchial segment, but also needs to consider whether there are errors in the previous prediction (i.e., the first segmentation result). The output of the second bronchial segmentation model is the correction of the first segmentation result, that is, the second segmentation result. Its purpose is to adjust and correct the incorrect part according to the first segmentation result.

[0080] In one embodiment, when training the second bronchial segmentation model, a loss function (loss_diff) can be added to ensure that the output of the second bronchial segmentation model is inconsistent with the first segmentation result. That is to say, the training objective of the second bronchial segmentation model is not only to output the correct second segmentation result, but also to require different adjustments to the prediction result during each correction. This can prevent the second bronchial segmentation model from making repetitive errors and ensure the second bronchial segmentation model plays its role.

[0081] The loss function loss_diff emphasizes that the second bronchial segmentation model needs to change the predicted segmentation result, ensuring that the segmentation result predicted by the second bronchial segmentation model is different from the previous one, rather than simply repeating or copying the incorrect prediction of the first segmentation result. The loss function can also adopt Mean Squared Error (MSE), cross-entropy loss, or Mean Absolute Error (MAE), etc. The embodiments of the present application do not specifically limit the loss function.

[0082] The second bronchial segmentation model only makes one prediction. Whether the predicted second segmentation result is correct or not, the second bronchial segmentation model will stop running. This is because most of the bronchial segmentation problems occur at only a few positions. Therefore, the corrected second segmentation result can be obtained by re-predicting once, without the need to repeatedly make multiple predictions.

[0083] It can be seen from this that when the predicted probability value obtained in the embodiment of the present application is lower than the preset probability threshold, the second bronchial segmentation model will intervene and attempt to correct the errors in the first segmentation result, which helps to improve the accuracy and precision of the predicted segmentation result. And the second bronchial segmentation model only runs once and will stop regardless of whether the result is correct or wrong, aiming to correct the problem by re-predicting a small number of incorrect segmentation results, improve the accuracy of the final predicted segmentation result, and avoid unnecessary repeated calculations.

[0084] Figure 8 It is a schematic flowchart of an image processing method provided by another exemplary embodiment of the present application. Figure 8 The embodiment is Figure 2 A further description of step S220 in the embodiment. The image processing method includes the following contents.

[0085] In one embodiment, referring to Figure 4 , the lungs include multiple lung segments.

[0086] It should be noted that the task of lung segmentation can be carried out for lung lobes, that is, segment prediction of lung segments within the lung lobes.

[0087] S810: Determine the mapping relationship between the lung segments and the bronchial segmentation result according to the first medical positional relationship, and set multiple lung segmentation models for multiple lung segments.

[0088] Preferably, the number of lung segmentation models can be set to 5.

[0089] Specifically, according to the mapping relationship between the lung segments and the bronchial segmentation result reflected by the first medical positional relationship, the lung segments in different positions are defined as different types of lung segments, and then multiple lung segmentation models are set for the different types of lung segments. For example, referring to Figure 9 , the bronchial segmentation result can be used to indicate the distribution of bronchi in different positions within the lung lobe, and these bronchial segmentation results within the lung lobe can indicate different regions of the lung lobe.

[0090] The lung segmentation model can be a multi-class linear regression classifier. For example, 5 lung segmentation models can be understood as 5 independent classifiers, and each classifier can handle classification problems of multiple classes (i.e., "multi-class").

[0091] S820: Segment the first lung image by using multiple lung segmentation models to obtain a lung segmentation result.

[0092] Specifically, each lung segmentation model (or each classifier) inputs the bronchial segmentation result within the current lung lobe (e.g., the upper lobe of the right lung), that is, the input of each lung segmentation model is the segmentation information of a bronchus within the lung lobe. The bronchial segmentation result within the current lung lobe can be used to indicate the distribution of the bronchi at different positions within the lung lobe, and these bronchial segmentation results can represent different regions of the lung lobe.

[0093] Furthermore, the lung segmentation model is used to classify all points (e.g., pixel points) within the lung lobe. Each lung segmentation model learns and judges the input bronchial segmentation result within the lung lobe, and based on this bronchial segmentation result, classifies all points in the entire lung lobe (i.e., all positions within the lung lobe). Finally, the lung lobe is segmented into different regions or categories to obtain the lung segmentation result corresponding to the lung lobe. Finally, for each lung lobe, the lung segmentation result corresponding to each lung lobe is obtained, and the lung segmentation result is constructed based on the lung segmentation result corresponding to each lung lobe.

[0094] It can be seen from this that by adopting multiple lung segmentation models in the embodiments of the present application, it is possible to optimize according to the characteristics of different regions respectively, which helps to better cope with the diversity of the lung lobe structure, improves the accuracy of lung segmentation, and optimizes the processing efficiency of lung segmentation.

[0095] In an embodiment of the present application, based on the lung segmentation result, the second medical position relationship, and the artery segmentation model, an artery segmentation result is obtained, including: inputting the first lung image containing the lung segmentation result into the artery segmentation model to obtain an artery segmentation result; determining the mapping relationship between the lung segmentation result and the artery segmentation result according to the second medical position relationship; and determining the category of the artery segmentation result according to the mapping relationship to obtain the artery segmentation result.

[0096] Specifically, the artery segmentation result may include the segmentation results of multiple artery segments. Inputting the first lung image containing the lung segmentation result into the artery segmentation model, the segmentation results of multiple artery segments are obtained, where the artery segmentation model may be an instance segmentation model, such as a model with a Mask2Former structure. The embodiments of the present application do not make specific limitations on this. It should be noted that the artery segmentation result (i.e., the segmentation results of multiple artery segments) has not been classified at this time.

[0097] Furthermore, according to the second medical position relationship recorded in the medical background, the mapping relationship between the lung segmentation result and the artery segmentation result is determined, that is, the position of each artery segment is analyzed. For example, it may be to check the distribution position of each artery segment and see what proportion of each lung segment the artery segment occupies. The artery segment will ultimately be classified into the category with the largest proportion of the lung segment where the artery segment is located. For example, if a certain artery segment is mostly located in lung segment A (accounting for 90%), and only a small part is located in lung segment B (accounting for 10%), then this artery segment will be classified as lung segment A.

[0098] It can be seen from this that in the embodiment of the present application, the arteries are reasonably divided according to the lung segmentation result. By judging the proportion of each arterial segment in each lung segment, the arterial segment is divided into the lung segment with the largest proportion, so as to achieve accurate arterial segmentation, ensure that the arterial segmentation result can reflect the anatomical structure of the lung lobe, and ensure consistency with the lung segmentation result.

[0099] In an embodiment of the present application, before inputting the first lung image into the first bronchial segmentation model for segmentation to obtain the bronchial segmentation result, it further includes: segmenting the second lung image through a bronchial lobar model, a lung lobar model, and an arterial lobar model to obtain the first lung image, where the first lung image includes the bronchial lobar result, the lung lobar result, and the arterial lobar result.

[0100] It should be noted that bronchial lobation refers to the further branching of the bronchus into smaller bronchi and supplying different lung lobes. This branching structure helps to distribute air throughout the lungs. Lung lobation is a way of dividing the organizational structure of the lung, which means that the lung is divided into several parts. Among them, the human lung is usually divided into two parts, the left lung has two lobes (i.e., the upper lobe and the lower lobe), and the right lung has three lobes (i.e., the upper lobe, the middle lobe, and the lower lobe). Arterial lobation refers to the branching structure of the artery when supplying an organ or tissue. Arterial lobation can be understood as that during the branching process of the artery, in order to ensure that blood can be supplied to different parts or regions, the artery may be divided into multiple branches and distributed to the corresponding regions like tree branches.

[0101] Specifically, the second lung image is a medical image, and this medical image can be a Computed Tomography (CT) image, a Digital Radiography (DR) image, a Magnetic Resonance Imaging (MRI) image, etc. For example, the second lung image can be a lung CT image, and the second lung image can include the lung, the bronchial part, and the arterial part.

[0102] The bronchial lobar model can be a U-net model or other deep learning models, and this bronchial lobar model is used to perform lobation on the bronchi in the second lung image. The lung lobar model can be a U-net model or other deep learning models, and this lung lobar model is used to perform lobation on the lungs in the second lung image. The arterial lobar model can be a U-net model or other deep learning models, and this arterial lobar model is used to perform lobation on the arteries in the second lung image.

[0103] In one embodiment, the server may include a lobar model, which includes a bronchial lobar model, a pulmonary lobar model, and an arterial lobar model. After the second pulmonary image is input into the lobar model, the bronchial lobar model, the pulmonary lobar model, and the arterial lobar model in the lobar model may perform lobation on the bronchi, lungs, and arteries in the second pulmonary image simultaneously or in a preset order to obtain a first pulmonary image including bronchial lobation results, pulmonary lobation results, and arterial lobation results.

[0104] It can be seen that since the segmentation tasks (i.e., bronchial segmentation, arterial segmentation, and pulmonary segmentation) are all performed in the corresponding lobes, the lobation operation is first performed in the embodiments of the present application, laying a foundation for the subsequent segmentation operation.

[0105] Figure 10 It is a schematic structural diagram of an image processing device provided by an exemplary embodiment of the present application. As Figure 10 shown, the image processing device 1000 includes: a lobation module 1010, a bronchial segmentation module 1020, a pulmonary segmentation module 1030, and an arterial segmentation module 1040.

[0106] The bronchial segmentation module 1020 is configured to input the first pulmonary image into a first bronchial segmentation model for segmentation to obtain bronchial segmentation results, where the first pulmonary image includes a lung, a bronchial part having a first medical positional relationship with the lung, and an arterial part having a second medical positional relationship with the lung; the pulmonary segmentation module 1030 is configured to segment the lung using a plurality of pulmonary segmentation models constructed based on the first medical positional relationship and the bronchial segmentation results to obtain pulmonary segmentation results, where the plurality of pulmonary segmentation models include segmentation models constructed for different lung segments; the arterial segmentation module 1040 is configured to obtain arterial segmentation results based on the pulmonary segmentation results, the second medical positional relationship, and an arterial segmentation model.

[0107] The embodiments of the present application provide an image processing device. By inputting the first pulmonary image into the first bronchial segmentation model for segmentation to obtain bronchial segmentation results, then segmenting the lung using a plurality of pulmonary segmentation models constructed based on the first medical positional relationship between the lung and the bronchial part and the bronchial segmentation results to obtain pulmonary segmentation results, and finally obtaining arterial segmentation results based on the pulmonary segmentation results, the second medical positional relationship between the lung and the arterial part, and the arterial segmentation model, the embodiments of the present application first obtain bronchial segmentation results in combination with the medical background, then obtain pulmonary segmentation results based on the bronchial segmentation results, and further obtain arterial segmentation results based on the pulmonary segmentation results, improving the accuracy and precision of arterial segmentation.

[0108] According to an embodiment of the present application, the bronchial segmentation result includes the segmentation results of multiple bronchial segments. The bronchial segmentation module 1020 is configured to input the first lung image into the first bronchial segmentation model to segment multiple bronchial segments one by one, obtain the segmentation result of each bronchial segment; and construct the segmentation results of multiple bronchial segments based on the segmentation results of each bronchial segment.

[0109] According to an embodiment of the present application, the multiple bronchial segments include a first bronchial segment. The bronchial segmentation module 1020 is configured to extract the feature data of the bronchial part in the first lung image through a sparse convolution model; input the feature data and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result of the first bronchial segment.

[0110] According to an embodiment of the present application, the multiple bronchial segments include at least one second bronchial segment whose segmentation has been completed. The bronchial segmentation module 1020 is configured to input the segmentation results of the at least one second bronchial segment, the feature data, and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result.

[0111] According to an embodiment of the present application, the bronchial segmentation module 1020 is configured to encode the first segmentation result to obtain the encoded first segmentation result; input the encoded first segmentation result into the language judgment model to obtain the predicted probability value for the first bronchial segment; in the case where the predicted probability value is less than the preset probability threshold, re-segment the first bronchial segment to obtain the second segmentation result of the first bronchial segment.

[0112] According to an embodiment of the present application, the bronchial segmentation module 1020 is configured to obtain the feature data of the first bronchial segment; input the feature data of the first bronchial segment, the first segmentation result, and the encoded data of the first bronchial segment into the second bronchial segmentation model to obtain the second segmentation result, where the second segmentation result is different from the first segmentation result.

[0113] According to an embodiment of the present application, the lung includes multiple lung segments. The lung segmentation module 1030 is configured to determine the mapping relationship between the lung segments and the bronchial segmentation results according to the first medical positional relationship, set different lung segmentation models for different lung segments, where the lung segments and the lung segmentation models are in one-to-one correspondence; use the multiple lung segmentation models to segment the first lung image to obtain the lung segmentation result.

[0114] According to an embodiment of the present application, the artery segmentation module 1040 is configured to input the first lung image including the lung segmentation result into the artery segmentation model to obtain the artery segmentation result; determine the mapping relationship between the lung segmentation result and the artery segmentation result according to the second medical positional relationship; determine the category of the artery segmentation result according to the mapping relationship to obtain the artery segmentation result.

[0115] According to an embodiment of the present application, the lobulation module 1010 is configured to segment a second lung image through a bronchial lobulation model, a lung lobulation model, and an arterial lobulation model to obtain a first lung image, where the first lung image includes a bronchial lobulation result, a lung lobulation result, and an arterial lobulation result.

[0116] It should be understood that the specific working processes and functions of the lobulation module 1010, the bronchial segmentation module 1020, the lung segmentation module 1030, and the arterial segmentation module 1040 in the above embodiments can be referred to the descriptions in the image processing method provided in the above Figures 1 to 9 embodiment. To avoid repetition, they will not be elaborated here.

[0117] Figure 11 is a block diagram of an electronic device for image processing provided by an exemplary embodiment of the present application.

[0118] Referring to Figure 11 , the electronic device 1100 includes a processing component 1110, which further includes one or more processors, and memory resources represented by a memory 1120 for storing instructions executable by the processing component 1110, such as application programs. The application programs stored in the memory 1120 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1110 is configured to execute instructions to perform the above image processing method.

[0119] The electronic device 1100 may further include a power component configured to perform power management of the electronic device 1100, a wired or wireless network interface configured to connect the electronic device 1100 to a network, and an input / output (I / O) interface. The electronic device 1100 may be operated based on an operating system stored in the memory 1120, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0120] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the above-mentioned electronic device 1100, enables the above-mentioned electronic device 1100 to execute an image processing method, including: The first lung image includes a lung, a bronchial part having a first medical positional relationship with the lung, and an arterial part having a second medical positional relationship with the lung, where the method includes: inputting the first lung image into a first bronchial segmentation model for segmentation to obtain a bronchial segmentation result; segmenting the lung using a plurality of lung segmentation models constructed based on the first medical positional relationship and the bronchial segmentation result to obtain a lung segmentation result, where the plurality of lung segmentation models include segmentation models constructed for different lung segments; obtaining an arterial segmentation result based on the lung segmentation result, the second medical positional relationship, and an arterial segmentation model.

[0121] The embodiments of the present application also provide a computer program product, including a computer program, the computer program is used to execute the steps of the image processing method described in the above method embodiments, for details, reference can be made to the above method embodiments, and details will not be repeated here. This computer program product can be specifically implemented in a manner of hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0122] All of the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, and details will not be repeated here one by one.

[0123] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0124] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and details will not be repeated here.

[0125] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, the functional units in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0128] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, and other media that can store program verification codes.

[0129] It should be noted that in the description of this application, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise stated, the meaning of "multiple" is two or more.

[0130] The above are only the preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent replacements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The first lung image includes a lung, a bronchial part having a first medical positional relationship with the lung, and an arterial part having a second medical positional relationship with the lung. Wherein, the method includes: Inputting the first lung image into a first bronchial segmentation model for segmentation to obtain a bronchial segmentation result; Segmenting the lung by using a plurality of lung segmentation models constructed based on the first medical positional relationship and the bronchial segmentation result to obtain a lung segmentation result, wherein the plurality of lung segmentation models include segmentation models constructed for different lung segments; Obtaining an arterial segmentation result based on the lung segmentation result, the second medical positional relationship, and an arterial segmentation model.

2. The image processing method according to claim 1, wherein The bronchial segmentation result includes segmentation results of a plurality of bronchial segments. Wherein, the step of inputting the first lung image into the first bronchial segmentation model for segmentation to obtain a bronchial segmentation result includes: Inputting the first lung image into the first bronchial segmentation model to segment each of the plurality of bronchial segments one by one to obtain a segmentation result of each bronchial segment; Constructing the segmentation result of the plurality of bronchial segments based on the segmentation result of each bronchial segment.

3. The image processing method according to claim 2, wherein The plurality of bronchial segments include a first bronchial segment. Wherein, the step of inputting the first lung image into the first bronchial segmentation model to segment each of the plurality of bronchial segments one by one to obtain a segmentation result of each bronchial segment includes: Extracting feature data of the bronchial part in the first lung image through a sparse convolution model; Inputting the feature data and the encoded data of the first bronchial segment into a bronchial classification prediction model to obtain a first segmentation result of the first bronchial segment.

4. The image processing method according to claim 3, wherein The plurality of bronchial segments include at least one second bronchial segment that has been segmented. Wherein, the step of inputting the feature data and the encoded data of the first bronchial segment into a bronchial classification prediction model to obtain a first segmentation result of the first bronchial segment includes: Inputting the segmentation result of the at least one second bronchial segment, the feature data, and the encoded data of the first bronchial segment into the bronchial classification prediction model to obtain the first segmentation result.

5. The image processing method according to claim 3, wherein After the step of inputting the feature data and the encoded data of the first bronchial segment into a bronchial classification prediction model to obtain a first segmentation result of the first bronchial segment, it further includes: Encoding the first segmentation result to obtain an encoded first segmentation result; Inputting the encoded first segmentation result into a language judgment model to obtain a predicted probability value for the first bronchial segment; In the case where the predicted probability value is less than a preset probability threshold, re-segmenting the first bronchial segment to obtain a second segmentation result of the first bronchial segment.

6. The image processing method according to claim 5, wherein The step of re-segmenting the first bronchial segment to obtain a second segmentation result of the first bronchial segment includes: Obtaining the feature data of the first bronchial segment; Inputting the feature data of the first bronchial segment, the first segmentation result, and the encoded data of the first bronchial segment into a second bronchial segmentation model to obtain the second segmentation result. Wherein, the second segmentation result is different from the first segmentation result.

7. The image processing method according to claim 1, characterized in that, The lungs include multiple lung segments, wherein, segmenting the lungs using the multiple lung segmentation models constructed based on the first medical positional relationship and the bronchial segmentation result to obtain a lung segmentation result, including: determining the mapping relationship between the lung segments and the bronchial segmentation result according to the first medical positional relationship, and setting multiple lung segmentation models for the multiple lung segments; segmenting the first lung image using the multiple lung segmentation models to obtain the lung segmentation result.

8. The image processing method according to claim 1, wherein Obtaining an artery segmentation result based on the lung segmentation result, the second medical positional relationship, and the artery segmentation model, including: inputting the first lung image including the lung segmentation result into the artery segmentation model to obtain an artery segmentation result; determining the mapping relationship between the lung segmentation result and the artery segmentation result according to the second medical positional relationship; determining the category of the artery segmentation result according to the mapping relationship to obtain the artery segmentation result.

9. The image processing method according to any one of claims 1 to 8, characterized in that, Before segmenting the first lung image by inputting it into the first bronchial segmentation model to obtain a bronchial segmentation result, it further includes: segmenting a second lung image through a bronchial lobation model, a lung lobation model, and an artery lobation model to obtain the first lung image, wherein the first lung image includes a bronchial lobation result, a lung lobation result, and an artery lobation result.

10. An image processing apparatus, characterized in that, including: a bronchial segmentation module, configured to segment a first lung image by inputting it into a first bronchial segmentation model to obtain a bronchial segmentation result, wherein the first lung image includes lungs, a bronchial part having a first medical positional relationship with the lungs, and an artery part having a second medical positional relationship with the lungs; a lung segmentation module, configured to segment the lungs using multiple lung segmentation models constructed based on the first medical positional relationship and the bronchial segmentation result to obtain a lung segmentation result, wherein the multiple lung segmentation models include segmentation models constructed for different lung segments; an artery segmentation module, configured to obtain an artery segmentation result based on the lung segmentation result, the second medical positional relationship, and the artery segmentation model.

Images