Lung segment bifurcation recognition method and device, electronic device, and computer-readable storage medium

Abstract

A lung segment bifurcation recognition method, device, electronic device and computer readable storage medium, wherein the method comprises: acquiring an image data set, the image data set containing a plurality of image data with labels, the labels being used to mark the identification information of the bifurcation contained in the image data, the identification information of each lung segment corresponding to the bifurcation and the position information; using the image data set as training data, and inputting the training data into a pre-constructed bifurcation recognition network for training to obtain a trained bifurcation recognition network; inputting a target image collected by a bronchoscope into the trained bifurcation recognition network for instance segmentation to obtain the identification information of a target bifurcation contained in the target image output by the trained bifurcation recognition network, the identification information of each lung segment belonging to the target bifurcation and the position information. The method, device, electronic device and computer readable storage medium in the present application can realize fast and accurate recognition of lung segment bifurcation.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for identifying lung segment bifurcation. Background Technology

[0002] With the advancement of computer vision technology, medical endoscopes have developed rapidly in recent decades. Bronchial endoscopes are widely used, but due to the numerous branches and tree-like structure of the bronchi, bronchial endoscopes often fail to distinguish bronchial bifurcations in clinical applications, making it difficult to accurately locate the specific part of the bronchus in the current endoscopic image. This can affect patient examinations or surgeries. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and computer-readable storage medium for identifying lung segment bifurcation, which can realize bifurcation structure location identification based on neural networks, thereby improving the accuracy and efficiency of identification.

[0004] This application provides a method for identifying lung segment bifurcation, including:

[0005] An image dataset is acquired, comprising several labeled images. The labels are used to mark the identification information of bifurcations, the identification information of each lung segment corresponding to the bifurcation, and the location information of the images. The image dataset is used as training data and input into a pre-constructed bifurcation recognition network for training, resulting in a trained bifurcation recognition network. The target image acquired by bronchoscopy is input into the trained bifurcation recognition network for instance segmentation, resulting in the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network.

[0006] This application also provides a lung segment bifurcation identification device, including:

[0007] The acquisition module is used to acquire an image dataset containing several labeled image data. The labels are used to mark the identification information of the bifurcation, the identification information of each lung segment corresponding to the bifurcation, and the location information of the image data. The training module is used to use the image dataset as training data and input the training data into a pre-constructed bifurcation recognition network for training to obtain a trained bifurcation recognition network. The processing module is used to input the target image acquired by the bronchoscopy into the trained bifurcation recognition network for instance segmentation to obtain the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network.

[0008] One aspect of this application also provides an electronic device, including:

[0009] Memory and processor;

[0010] The memory stores executable computer programs;

[0011] The processor coupled to the memory invokes the executable computer program stored in the memory to perform the steps of the lung segment bifurcation identification method described above.

[0012] In one aspect, this application also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, implements the lung segment bifurcation identification method provided in the above embodiments.

[0013] As can be seen from the above embodiments of this application, image data is acquired. The image dataset contains several labeled image data. The labels are used to mark the identification information of the bifurcation contained in the image data, the identification information of each lung segment corresponding to the bifurcation, and the location information. The image dataset is used as training data, and the training data is input into a pre-constructed bifurcation recognition network for training to obtain the trained bifurcation recognition network. The target image acquired by the bronchoscopy is input into the trained bifurcation recognition network for instance segmentation to obtain the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network. This can achieve accurate and fast identification of the specific structure and location of the bifurcation and lung segments in the target image. Compared with the prior art, it improves the accuracy and speed of identifying bifurcation and lung segments in bronchoscopic images. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram illustrating the implementation process of a lung segment bifurcation identification method provided in an embodiment of this application;

[0016] Figure 2 A schematic diagram of a lung bronchus with bifurcation information labeled;

[0017] Figure 3 A schematic diagram illustrating the implementation process of a lung segment bifurcation identification method provided in another embodiment of this application;

[0018] Figure 4A This is a schematic diagram of the recognition result of lung segment bifurcation identification using a non-instance segmentation algorithm for the purposes of this application;

[0019] Figure 4B for Figure 4A A schematic diagram showing the identification results of the corresponding lung segment bifurcation identified by the instance segmentation algorithm of this application;

[0020] Figure 5A This is a schematic diagram of another recognition result for lung segment bifurcation identified using a non-instance segmentation algorithm in this application;

[0021] Figure 5B for Figure 5A A schematic diagram showing the identification results of the corresponding lung segment bifurcation identified by the instance segmentation algorithm of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a lung segment bifurcation identification device provided in another embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] See Figure 1 This application provides a flowchart of the implementation of a lung segment bifurcation identification method according to an embodiment. This method can be applied to electronic devices, such as desktop computers, servers, and other non-mobile electronic devices that can process data, or mobile electronic devices such as smartphones, tablets, laptops, and robots that can process data.

[0026] like Figure 1 As shown in the figure, this application provides a method for identifying lung segment bifurcation, the method comprising:

[0027] Step S101: Obtain an image dataset, which contains several labeled image data;

[0028] This label is used to mark the identification information of the bifurcation in the image data, the identification information of each lung segment corresponding to the bifurcation, and the location information.

[0029] Specifically, for information on the identification of bronchial segments and bifurcation points in the lungs, please refer to [link to relevant documentation]. Figure 2 According to the lung structure, it is divided into left and right sides, with a total of 18 lung segments in the left and right bronchi. The portion between each tracheal branch with a terminal port and the terminal port of that tracheal branch is defined as a lung segment. Figure 2 The right bronchus is divided into 10 segments, including B1 to B10: apical, posterior, anterior, lateral, medial, dorsal, medial basal, anterior basal, lateral basal, and posterior basal. The left bronchus is divided into 8 segments, including B1-2 to B10: apical-posterior, posterior, anterior, lateral, medial, medial anterior basal, lateral basal, and posterior basal. The bifurcation points of the bronchi throughout the lungs are considered the bifurcation points. Figure 2 In the middle, the bronchi have 17 bifurcations, including 0 to 16.

[0030] The identification information is Figure 2 The numbers B1-B10 are used to identify segments, and numbers 0-16 are used to identify forks. The coordinate information is in planar coordinates, expressed in the form of (x, y).

[0031] Step S102: Use the image dataset as training data and input the training data into the pre-constructed bifurcation recognition network for training to obtain the trained bifurcation recognition network.

[0032] The bifurcation identification network can specifically be a convolutional neural network capable of instance segmentation, such as the first-generation location-based segmentation target network SOLOv1 (SOLO, Segmenting Objects by Locations) and the second-generation location-based segmentation target network SOLOv2.

[0033] The image dataset, which includes the identification information of each lung segment, the identification information of the bifurcation, and the coordinates of the lung segment and the bifurcation, is used as training data to train the bifurcation recognition network, resulting in the trained bifurcation recognition network.

[0034] Step S103: Input the target image acquired by the bronchoscopy into the trained bifurcation recognition network for instance segmentation, and obtain the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network.

[0035] Accurately and quickly identifying the structural location of the bronchi in the target image can reduce the registration range with the virtual bronchial structure map. Registration only needs to be performed with the same location in the virtual bronchial structure map, which improves the registration speed. After approval, the 6D degrees of freedom are updated for navigation, assisting the endoscope to enter the lungs.

[0036] In this embodiment, image data is acquired. This image dataset contains several labeled image data. The labels are used to mark the identification information of the bifurcation contained in the image data, the identification information of each lung segment corresponding to the bifurcation, and the location information. This image dataset is used as training data and input into a pre-constructed bifurcation recognition network for training. The trained bifurcation recognition network is obtained. The target image acquired by the bronchoscopy is input into the trained bifurcation recognition network for instance segmentation. The target bifurcation identification information, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image are obtained from the output of the trained bifurcation recognition network. This can achieve accurate and fast identification of the specific structure and location of the bifurcation and lung segments in the target image. Compared with the prior art, it improves the accuracy and speed of identifying bifurcation and lung segments in bronchoscopic images.

[0037] like Figure 3 As shown, in another embodiment, the lung segment bifurcation identification method mainly includes:

[0038] Step S201: Train the keyframe detection network and identify keyframes using the keyframe detection network;

[0039] Several bronchial images labeled with their actual types are used as input training samples to train the constructed keyframe detection network. The keyframe detection network calculates and outputs the predicted type of the bronchial image. Based on the difference between the predicted type and the actual type, the parameters of the keyframe detection network are adjusted to obtain the trained keyframe detection network.

[0040] The categories include normal image type and abnormal image type. The normal image type indicates that the image is clear and has no glare or bubbles, while the abnormal image type indicates that the image has glare, bubbles, or blur that affects normal display.

[0041] Bronchial video images from several users were collected and input into a pre-trained keyframe detection network to identify the type of each bronchial video image. Bronchial video images of the normal image type were identified as keyframes, and the image dataset was composed of these keyframes.

[0042] This keyframe detection network can also directly output keyframes and non-keyframes.

[0043] Furthermore, the labels in step S101 are obtained by performing an instance segmentation annotation operation on the image data.

[0044] Specifically, typical instance segmentation requires the use of bounding boxes from object detection as an aid. However, due to the irregularity of lung segments, these bounding boxes are prone to containing noise, for example... Figure 4A and Figure 5A ,in, Figure 4A It's the 12th bifurcation, containing four lung segments: B7-B10. Figure 5A This is the 16th bifurcation, containing three lung segments: B1, B2, and B3. Different rectangular bounding boxes can easily cause overlap of lung segments, for example... Figure 5A B1 and B2 in the code introduce noise, increasing the training difficulty of the bifurcation recognition network, and the rectangular bounding boxes cannot accurately contain lung segments, for example... Figure 4A Examples include B7 in the example. Furthermore, instance segmentation annotations can yield noise-free instance segmentation bounding boxes. Figure 4B It corresponds Figure 4A The instance segmentation annotation box obtained by instance segmentation. Figure 5B It corresponds Figure 5A The instance segmentation bounding boxes obtained by instance segmentation do not contain noise and can accurately include and match lung segments.

[0045] Step S102 uses the image dataset as training data to train the pre-constructed bifurcation recognition network, specifically as follows:

[0046] Sample features of each image data in the image dataset are extracted separately. The extracted sample features of any image data are input into the bifurcation recognition network to obtain the prediction result output by the bifurcation recognition network. The prediction result includes the predicted label information of the bifurcation contained in any image data, the predicted label information of the lung segment belonging to the bifurcation, and the predicted location. The bifurcation recognition network is adjusted according to the difference information between the label of any image data and the obtained prediction result to obtain the trained bifurcation recognition network.

[0047] The bifurcation identification network includes a feature extraction network module, a lung segment identification network module, and a lung segment instance segmentation network module. The trained bifurcation identification network includes these three modules after training.

[0048] Step S103 involves inputting the target image acquired by the bronchoscopy into the trained bifurcation recognition network for instance segmentation, obtaining the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network, specifically:

[0049] The feature extraction network module can be ResNet (Deep residual network), MobileNet, or ShuffleNet, used to extract the texture and shape features of each lung segment in the image data.

[0050] The feature extraction network module inputs the extracted feature information to the lung segment recognition network module. This lung segment recognition network module identifies the identifiers of all lung segments in the target image based on the texture and shape features of each lung segment input by the feature extraction network module. These identifiers, such as B1-B10, are then used by the lung segment instance segmentation network module to generate instance segmentation bounding boxes corresponding to all lung segments in the target image based on the identified lung segment identifiers. (See [link to instance segmentation bounding boxes]). Figure 4B and Figure 5B As shown.

[0051] The lung segment instance segmentation network module is used to segment lung segments in real time to identify the lung segments in an image and output image data with labeled lung segment identification information and lung segment coordinate information.

[0052] The principle behind SOLOv2 network processing for instance segmentation of image data is as follows:

[0053] The acquired lung segment image is divided into an S×S grid. The grid containing the center of the lung segment corresponds to a binary mask. The lung segment image corresponds to S*S masks.

[0054] SOLOv2 decomposes the mask header into kernel branches and feature branches, corresponding to kernel learning and feature learning, respectively. The kernel and feature branches perform convolution to extract features and generate a mask. The maximum number of masks is achieved when every grid cell contains a target. This convolutional method initially filters out some grid cells without targets, reducing subsequent computation. A mask refers to using a selected image, graphic, or object to occlude (fully or partially) the image being processed, controlling the area or process of image processing. The non-maximum suppression matrix (NMS) calculates the intersection-union ratio (IoU) between predicted masks. The IoU can be used to calculate the mask score of each mask and delete masks with scores below a preset threshold. Multiple masks can be calculated in parallel, improving the speed of the maximum suppression algorithm and selecting the optimal prediction instance from multiple prediction instances. The intersection-union ratio (IoU) describes the degree of overlap between two bounding boxes and can be represented in matrix form. A low mask score for a particular mask indicates a high overlap between that mask and other masks.

[0055] The trained bifurcation recognition network also includes a bifurcation recognition module, which maintains a mapping relationship between the identification information of any bifurcation and the identification information of the lung segment belonging to that bifurcation. The bifurcation recognition module is used to determine the identification information of the bifurcations contained in the target image based on the identification information of all lung segments in the target image and the mapping relationship.

[0056] The target image with this bifurcation structure acquired in real time is an image that needs to be identified and labeled with the identification information of lung segments and bifurcation points, such as a real-time image of the lung bronchi of a person currently being examined or operated on via endoscopy.

[0057] Furthermore, the lung segment bifurcation identification method in this application embodiment also includes:

[0058] During bronchoscopy, the bronchoscope typically needs to be positioned directly over a specific lung segment containing a bifurcation to allow for successful entry. Therefore, after acquiring the target image, the bronchoscope must be adjusted to face the desired lung segment. Based on a pre-planned path to the area of ​​medical interest and the target image, the marker information for the specific lung segment the bronchoscope needs to enter can be determined. Using this marker information, the coordinates of the center point of the designated lung segment in the target image are calculated, and the offset between the bronchoscope's center point coordinates and the designated lung segment's center point coordinates is determined. The bronchoscope can then be moved or its angle adjusted based on this offset. This area of ​​medical interest is the location of the lesion.

[0059] The lung segment bifurcation identification method provided in this embodiment acquires image data. This image dataset contains several labeled image data points. The labels are used to mark the identification information of the bifurcation points contained in the image data, the identification information of each lung segment corresponding to the bifurcation, and the location information. This image dataset is used as training data, and the training data is input into a pre-constructed bifurcation identification network for training. The trained bifurcation identification network is then obtained. The target image acquired by bronchoscopy is input into the trained bifurcation identification network for instance segmentation. The resulting data, output by the trained bifurcation identification network, identifies the target bifurcation points contained in the target image and the segments belonging to those bifurcation points. By identifying and mapping the various lung segments, the system can accurately and quickly determine the specific structure and location of bifurcations and lung segments in the target image. Compared to prior art, this improves the accuracy and speed of identifying bifurcations and lung segments in bronchoscopic images. Furthermore, based on the pre-defined planned path to the area of ​​medical interest and the target image, the system determines the identification information of the specific lung segment that the bronchoscope needs to enter. Based on the identification information of the specified lung segment, the system calculates the coordinates of the center point of the specified lung segment in the target image and determines the offset between the coordinates of the center point of the bronchoscope and the center point of the specified lung segment. This offset is used to guide the movement of the bronchoscope, achieving assisted navigation and improving the accuracy and efficiency of navigation.

[0060] See Figure 6 , Figure 6 This is a schematic diagram of a lung segment bifurcation recognition device according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. This device can be an electronic device with data processing capabilities, or it can be configured as a virtual module within an electronic device with data processing capabilities. Figure 6 As shown, the device includes:

[0061] The acquisition module 401 is used to acquire an image dataset, which contains several labeled image data. The labels are used to mark the identification information of the bifurcation, the identification information of each lung segment corresponding to the bifurcation, and the location information contained in the image data.

[0062] The training module 402 is used to use the image dataset as training data and input the training data into the pre-built bifurcation recognition network for training, so as to obtain the trained bifurcation recognition network.

[0063] The processing module 403 is used to input the target image acquired by the bronchoscopy into the trained bifurcation recognition network for instance segmentation, and obtain the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network.

[0064] Furthermore, the acquisition module 401 is also used to acquire bronchial video images of several users and input them into a pre-trained keyframe detection network to identify the type of each frame of bronchial video image, including normal image type and abnormal image type; the bronchial video images of the corresponding normal image type are identified as keyframes, and the keyframes constitute an image dataset.

[0065] The training module 402 is also used to train the constructed keyframe detection network by taking several bronchial images labeled with their actual types as input training samples; calculating the predicted type of the output bronchial image through the keyframe detection network; and adjusting the parameters of the keyframe detection network based on the difference information between the predicted type and the actual type to obtain the trained keyframe detection network.

[0066] The training module 402 is also used to extract sample features of each image data in the image dataset; by inputting the extracted sample features of any image data into the bifurcation recognition network, the prediction result output by the bifurcation recognition network is obtained. The prediction result includes the prediction label information of the bifurcation contained in the image data, the prediction label information of the lung segment belonging to the bifurcation, and the prediction location; the bifurcation recognition network is adjusted according to the difference information between the label of the image data and the obtained prediction result to obtain the trained bifurcation recognition network.

[0067] The trained bifurcation recognition network includes a feature extraction network module, a lung segment recognition network module, and a lung segment instance segmentation network module.

[0068] The feature extraction network module is used to extract the texture and shape features of each lung segment in the target image;

[0069] The lung segment recognition network module is used to identify the identification information of all lung segments in the target image based on the texture and shape features of each lung segment;

[0070] The lung segment instance segmentation network module is used to generate instance segmentation bounding boxes corresponding to all lung segments in the target image based on the identification information of all lung segments in the target image.

[0071] Furthermore, the trained bifurcation recognition network also includes a bifurcation recognition module, which maintains a mapping relationship between the identification information of any bifurcation and the identification information of the lung segment belonging to that bifurcation. The bifurcation recognition module is used to determine the identification information of the bifurcations contained in the target image based on the identification information of all lung segments in the target image and the mapping relationship.

[0072] The processing module 403 is also used to determine the identification information of the designated lung segment that the bronchoscope needs to enter based on the preset planned path to the medical focus and the target image; calculate the center point coordinates of the designated lung segment in the target image based on the identification information of the designated lung segment; and determine the offset between the center point coordinates of the bronchoscope and the center point coordinates of the designated lung segment, the offset being used to guide the movement of the bronchoscope.

[0073] The specific process by which each module implements its respective function can be found in the relevant content of the above embodiments, and will not be repeated here.

[0074] The lung segment bifurcation recognition device provided in this embodiment acquires image data. This image dataset contains several labeled image data points. The labels are used to mark the identification information of the bifurcation points, the identification information of each lung segment corresponding to the bifurcation, and the location information. This image dataset is used as training data and input into a pre-constructed bifurcation recognition network for training. The trained bifurcation recognition network is then obtained. The target image acquired by a bronchoscope is input into the trained bifurcation recognition network for instance segmentation. The target image output by the trained bifurcation recognition network contains the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information. This allows for accurate and rapid identification of the specific structure and location of bifurcations and lung segments in the target image. Compared with prior art, this improves the accuracy and speed of identifying bifurcations and lung segments in bronchoscopic images. Furthermore, it enables assisted navigation, improving the accuracy and efficiency of navigation.

[0075] See Figure 7 The present application provides a schematic diagram of the hardware structure of an electronic device according to an embodiment. Figure 7 As shown, the electronic device includes a memory 601 and a processor 602.

[0076] The memory 601 stores an executable computer program 603. The processor 602, coupled to the memory 601, calls the executable computer program 603 stored in the memory to execute the lung segment bifurcation identification method provided in the above embodiment.

[0077] For example, the computer program 603 can be divided into one or more modules / units, which are stored in the memory 601 and executed by the processor 602 to complete the present invention. The one or more modules / units may include various modules in the lung segment bifurcation recognition device in the above embodiments, such as: acquisition module 401, annotation module 402, training module 403, and processing module 404. These are used to implement the above-described lung segment bifurcation recognition method.

[0078] Furthermore, the device also includes:

[0079] At least one input device and at least one output device.

[0080] The processor 602, memory 601, input devices, and output devices mentioned above can be connected via a bus.

[0081] The input device can be a camera, touch panel, physical buttons, or mouse, etc. The output device can be a display screen.

[0082] Furthermore, the device may include more components than illustrated, or combine certain components, or different components, such as network access devices, sensors, etc.

[0083] The processor 602 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0084] The memory 601 can be, for example, a hard disk drive, non-volatile memory (such as flash memory or other electronically programmable erasure-restricted memory used to form a solid-state drive), volatile memory (such as static or dynamic random access memory), etc., and this application embodiment is not limited thereto. Specifically, the memory 601 can be an internal storage unit of the electronic device, such as the hard disk or RAM of the electronic device. The memory 601 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. Further, the memory 601 can include both internal storage units and external storage devices of the electronic device. The memory 601 is used to store computer programs and other programs and data required by the terminal. The memory 601 can also be used to temporarily store data that has been output or will be output.

[0085] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device of the above embodiments, and the computer-readable storage medium may be the aforementioned... Figure 7 The memory 601 in the illustrated embodiment stores a computer program on the computer-readable storage medium. When executed by a processor, the program implements the lung segment bifurcation identification method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0087] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0089] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0090] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0094] The above is a description of the lung segment bifurcation identification method, apparatus, electronic device, and computer-readable storage medium provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for identifying lung segment bifurcation, characterized in that, include: Obtain an image dataset containing several labeled image data, wherein the labels are used to mark the identification information of the bifurcation, the identification information of each lung segment corresponding to the bifurcation, and the location information of the image data; The image dataset is used as training data, and the training data is input into a pre-constructed bifurcation recognition network for training to obtain the trained bifurcation recognition network. The target image acquired by the bronchoscopy is input into the trained bifurcation recognition network for instance segmentation, and the target bifurcation identification information, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image are obtained by the trained bifurcation recognition network. The step of inputting the target image acquired by bronchoscopy into a trained bifurcation recognition network for instance segmentation, and obtaining the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network, includes: Based on the pre-defined planned path to the area of ​​medical interest and the target image, the identification information of the designated lung segment that the bronchoscope needs to enter is determined; Based on the identification information of the specified lung segment, calculate the coordinates of the center point of the specified lung segment in the target image; The offset between the center point coordinates of the bronchoscope and the center point coordinates of the designated lung segment is determined, and the offset is used to guide the movement of the bronchoscope. The step of using the image dataset as training data to input into a pre-constructed bifurcation recognition network for training, and obtaining the trained bifurcation recognition network includes: Extract sample features from each image data in the image dataset; By inputting the sample features of any extracted image data into the bifurcation recognition network, the prediction result output by the bifurcation recognition network is obtained. The prediction result includes the predicted identification information of the bifurcation contained in any image data, the predicted identification information of the lung segment belonging to the bifurcation, and the predicted location. The bifurcation recognition network is adjusted based on the difference between the label of any image data and the obtained prediction result to obtain the trained bifurcation recognition network.

2. The method as described in claim 1, characterized in that, The acquired image dataset includes: The system collects bronchial video images from several users and inputs them into a pre-trained keyframe detection network to identify the type of each frame of the bronchial video image. The type includes normal image type and abnormal image type. The corresponding bronchial video images belonging to the normal image type are identified as keyframes, and the image dataset is composed of each keyframe.

3. The method as described in claim 2, characterized in that, Prior to obtaining the image dataset, the following steps are included: Several bronchial images labeled with their actual types were used as input training samples to train the constructed keyframe detection network. The keyframe detection network calculates and outputs the predicted type of the bronchial image, and adjusts the parameters of the keyframe detection network based on the difference between the predicted type and the actual type, thus obtaining the trained keyframe detection network.

4. The method as described in claim 1, characterized in that, The method further includes: The labels are obtained by performing instance segmentation and annotation operations on the image data.

5. The method as described in claim 1, characterized in that, The trained bifurcation recognition network includes a feature extraction network module, a lung segment recognition network module, and a lung segment instance segmentation network module; the feature extraction network module is used to extract the texture and shape features of each lung segment in the target image; The lung segment recognition network module is used to identify the identification information of all lung segments in the target image based on the texture and shape features of each lung segment; The lung segment instance segmentation network module is used to generate instance segmentation bounding boxes corresponding to all lung segments in the target image based on the identification information of all lung segments in the target image.

6. The method as described in claim 5, characterized in that, The trained bifurcation recognition network further includes a bifurcation recognition module. The bifurcation recognition module maintains a mapping relationship between the identification information of any bifurcation and the identification information of the lung segment belonging to any bifurcation. The bifurcation recognition module is used to determine the identification information of the bifurcations contained in the target image based on the identification information of all lung segments in the target image and the mapping relationship.

7. A lung segment bifurcation identification device, characterized in that, include: The acquisition module is used to acquire an image dataset, which contains several labeled image data. The labels are used to mark the identification information of the bifurcation, the identification information of each lung segment corresponding to the bifurcation, and the location information contained in the image data. The training module is used to use the image dataset as training data and input the training data into a pre-constructed bifurcation recognition network for training, so as to obtain the trained bifurcation recognition network. The processing module is used to input the target image acquired by the bronchoscopy into the trained bifurcation recognition network for instance segmentation, and obtain the identification information of the target bifurcation, the identification information of each lung segment belonging to the target bifurcation, and the location information of the target image output by the trained bifurcation recognition network. The training module is further used to extract sample features from each image data in the image dataset; by inputting the extracted sample features of any image data into the bifurcation recognition network, the prediction result output by the bifurcation recognition network is obtained. The prediction result includes the predicted identification information of the bifurcation contained in any image data, the predicted identification information of the lung segment belonging to the bifurcation, and the predicted location; the bifurcation recognition network is adjusted according to the difference information between the label of any image data and the obtained prediction result to obtain the trained bifurcation recognition network. The processing module is further configured to determine the identification information of the designated lung segment that the bronchoscope needs to enter based on the preset planned path to the medical focus and the target image; calculate the center point coordinates of the designated lung segment in the target image based on the identification information of the designated lung segment; and determine the offset between the center point coordinates of the bronchoscope and the center point coordinates of the designated lung segment, wherein the offset is used to guide the movement of the bronchoscope.

8. The lung segment bifurcation identification device as described in claim 7, characterized in that, The trained bifurcation recognition network includes a feature extraction network module, a lung segment recognition network module, and a lung segment instance segmentation network module. The feature extraction network module is used to extract the texture and shape features of each lung segment in the target image. The lung segment recognition network module is used to identify the identification information of all lung segments in the target image based on the texture and shape features of each lung segment. The lung segment instance segmentation network module is used to generate instance segmentation bounding boxes corresponding to all lung segments in the target image based on the identification information of all lung segments in the target image. The trained bifurcation recognition network further includes a bifurcation recognition module. The bifurcation recognition module maintains a mapping relationship between the identification information of any bifurcation and the identification information of the lung segment belonging to any bifurcation. The bifurcation recognition module is used to determine the identification information of the bifurcations contained in the target image based on the identification information of all lung segments in the target image and the mapping relationship.

9. An electronic device, characterized in that, include: Memory and processor; The memory stores executable computer programs; The processor coupled to the memory invokes the executable computer program stored in the memory to perform the lung segment bifurcation identification method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the lung segment bifurcation identification method as described in any one of claims 1 to 6.

Images