A segmentation method, device, electronic equipment and storage medium

Abstract

The application discloses a segmentation method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring image information of a to-be-segmented image setting surface, the setting surface comprising a main reference surface and an auxiliary reference surface; inputting the image information into a segmentation model to acquire a segmentation result of the to-be-segmented image output by the segmentation model; wherein the structural complexity of a main feature extraction module in the segmentation model is higher than that of an auxiliary feature extraction module in the segmentation model, the main feature extraction module is used for extracting main image features of main image information, the auxiliary feature extraction module is used for extracting auxiliary image features in auxiliary image information, the main image information is image information corresponding to the main reference surface, and the auxiliary image information is image information corresponding to the auxiliary reference surface. By using the method, image segmentation can be performed based on the image information of the to-be-segmented image setting surface, different modules are used to process different setting surfaces during segmentation, and the segmentation precision is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and particularly relate to a segmentation method and device, electronic equipment and storage medium. BACKGROUND

[0002] Image segmentation is a technology and process of dividing an image into several specific regions with unique properties and proposing a target of interest. It is a key step from image processing to image analysis.

[0003] Existing image segmentation methods mainly include manual segmentation, traditional semi-automatic segmentation method, and fully automatic segmentation based on deep learning. Among them, the image segmentation method based on deep learning shows great potential.

[0004] However, for small target and complex structure image segmentation tasks, the image segmentation method based on deep learning has the technical problem of low segmentation accuracy. SUMMARY

[0005] Embodiments of the present application provide a segmentation method, device, electronic equipment and storage medium, which effectively improve the accuracy of image segmentation.

[0006] In a first aspect, embodiments of the present application provide a segmentation method, comprising:

[0007] Obtaining image information of a to-be-segmented image setting surface, the setting surface comprising a main reference surface and an auxiliary reference surface;

[0008] Inputting the image information into a segmentation model to obtain a segmentation result of the to-be-segmented image output by the segmentation model;

[0009] Wherein, the structure complexity of a main feature extraction module in the segmentation model is higher than that of an auxiliary feature extraction module in the segmentation model, the main feature extraction module is used to extract main image features of main image information, and the auxiliary feature extraction module is used to extract auxiliary image features in auxiliary image information, the main image information is image information corresponding to the main reference surface, and the auxiliary image information is image information corresponding to the auxiliary reference surface.

[0010] Optionally, the segmentation model comprises a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module and a decoding module; and correspondingly, inputting the image information into the segmentation model to obtain the segmentation result of the to-be-segmented image output by the segmentation model comprises:

[0011] Inputting the main image information into the main feature extraction module to obtain main image features;

[0012] Inputting the auxiliary image information into the auxiliary feature extraction module to obtain auxiliary image features;

[0013] input the auxiliary image features into the feature rotation module for rotation transformation;

[0014] merge the main image features and the rotated auxiliary image features to obtain fused image features;

[0015] input the fused image features into the decoding module to obtain a segmentation result of the image to be segmented.

[0016] Optionally, the number of the auxiliary feature extraction modules is equal to the number of the auxiliary reference surfaces; the network structures of the auxiliary feature extraction modules are the same or different; when the number of the auxiliary reference surfaces is at least two, the auxiliary image features corresponding to each of the auxiliary reference surfaces are input into the corresponding auxiliary feature extraction module.

[0017] Optionally, the main feature extraction module comprises a plurality of main feature extraction networks, each main feature extraction network comprising an encoding submodule and a center residual submodule, and each main feature extraction network is connected through the corresponding encoding submodule; correspondingly, the main image information is input into the main feature extraction module to obtain main image features, comprising:

[0018] inputting the main target information into each layer of the main feature extraction network to obtain corresponding image features;

[0019] when the main feature extraction network is a first layer main feature extraction network, the main target information is the main image information; when the main feature extraction network is a non-first layer main feature extraction network, the main target information is the output of the encoding submodule of the last layer of the main feature extraction network, and the main image features comprise the image features output by each layer of the main feature extraction network.

[0020] Optionally, the auxiliary feature extraction module comprises a plurality of auxiliary feature extraction networks, each layer of the auxiliary feature extraction network being connected in sequence; correspondingly, the auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features, comprising:

[0021] inputting the auxiliary target information into each layer of the auxiliary feature extraction network to obtain corresponding image features;

[0022] when the auxiliary feature extraction network is a first layer auxiliary feature extraction network, the auxiliary target information is the auxiliary image information; when the auxiliary feature extraction network is a non-first layer auxiliary feature extraction network, the auxiliary target information is the output of the last layer of the auxiliary feature extraction network, the auxiliary image features comprise the image features output by each layer of the auxiliary feature extraction network, and the number of the auxiliary feature extraction networks is less than the number of the main feature extraction networks included in the main feature extraction module.

[0023] Optionally, the decoding module comprises a plurality of decoding networks, the number of decoding networks comprised by the decoding module is equal to the number of main feature extraction networks comprised by the main feature extraction module; the outputs of the set sub-modules in the target decoding network except the first layer decoding network are respectively up-sampled to the inputs of the last layer decoding network; correspondingly, the fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented, comprising:

[0024] The fused image features are input into each decoding network to obtain a segmentation result, the segmentation result being the sum of the outputs of the first layer decoding network and the set layer decoding network in each decoding network;

[0025] The last layer decoding network in the decoding module uses an expansion convolution with different expansion convolution rates to process the input fused image features, the last layer decoding network in the decoding module processes the input through a residual sub-module or the last layer decoding network in the decoding module processes the input through a residual sub-module and a convolution sub-module, the set sub-module in the last layer decoding network being an intermediate convolution sub-module, and the set sub-module in the decoding network except the last layer decoding network in the target decoding network being a residual sub-module.

[0026] Optionally, the loss function of the segmentation model is determined based on the outputs of the plurality of decoding networks of the decoding module and the corresponding label values, the plurality of decoding networks for determining the loss function comprising the first layer decoding network, and the loss function of the first layer decoding network being determined based on the weighted foreground overlap coefficient and the weighted background overlap coefficient.

[0027] In a second aspect, an embodiment of the present application further provides a segmentation device, comprising:

[0028] A first acquisition module is configured to acquire image information of a set surface of an image to be segmented, the set surface comprising a main reference surface and an auxiliary reference surface.

[0029] A second acquisition module is configured to input the image information into a segmentation model and acquire a segmentation result of the image to be segmented output by the segmentation model.

[0030] In the segmentation model, the structural complexity of the main feature extraction module is higher than that of the auxiliary feature extraction module, the main feature extraction module is configured to extract main image features of main image information, and the auxiliary feature extraction module is configured to extract auxiliary image features of auxiliary image information, the main image information being image information corresponding to the main reference surface, and the auxiliary image information being image information corresponding to the auxiliary reference surface.

[0031] In a third aspect, an embodiment of the present application further provides an electronic device, comprising:

[0032] one or more processors;

[0033] a storage device for storing one or more programs;

[0034] The one or more programs are executed by the one or more processors, so that the one or more processors implement the segmentation method provided by the embodiments of the present application.

[0035] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the segmentation method provided by the embodiments of the present application.

[0036] The embodiments of the present application provide a segmentation method, device, electronic equipment and storage medium. First, image information of a to-be-segmented image setting surface is acquired, and the setting surface includes a main reference surface and an auxiliary reference surface. Then, the image information is input into a segmentation model, and a segmentation result of the to-be-segmented image output by the segmentation model is acquired. The structural complexity of a main feature extraction module in the segmentation model is higher than the structural complexity of an auxiliary feature extraction module in the segmentation model. The main feature extraction module is used to extract main image features of main image information. The auxiliary feature extraction module is used to extract auxiliary image features in auxiliary image information. The main image information is image information corresponding to the main reference surface, and the auxiliary image information is image information corresponding to the auxiliary reference surface. By using the above technical solution, image segmentation can be performed based on the image information of the to-be-segmented image setting surface. Different modules are used to process different setting surfaces during segmentation, and the segmentation precision is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A flowchart of a segmentation method provided for the first embodiment of the present application is shown in the figure.

[0038] Figure 2 A sequence of original CT images and a lung nodule region extraction image provided for the example embodiment of the present application are shown in the figure.

[0039] Figure 3 A flowchart of a segmentation method provided for the second embodiment of the present application is shown in the figure.

[0040] Figure 4 An image information of a setting surface provided for the example embodiment of the present application is shown in the figure.

[0041] Figure 5 A network architecture of a segmentation model provided for the example embodiment of the present application is shown in the figure.

[0042] Figure 6 A network architecture of a main feature extraction module provided for the example embodiment of the present application is shown in the figure.

[0043] Figure 7 A structure diagram of an encoding submodule provided for an example embodiment of the present application is shown in FIG. 1;

[0044] Figure 8 A network structure diagram of an auxiliary feature extraction module provided for an example embodiment of the present application is shown in FIG. 2;

[0045] Figure 9 A structure diagram of a decoding module provided for an example embodiment of the present application is shown in FIG. 3;

[0046] Figure 10 A structure diagram of an intermediate convolutional submodule provided for an example embodiment of the present application is shown in FIG. 4;

[0047] Figure 11 A structure diagram of a residual submodule provided for an example embodiment of the present application is shown in FIG. 5;

[0048] Figure 12 A structure diagram of a segmentation device provided for Embodiment Three of the present application is shown in FIG. 6;

[0049] Figure 13 A structure diagram of an electronic device provided for Embodiment Four of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, rather than all the structures.

[0051] Before the example embodiments are discussed in more detail, it should be mentioned that some of the example embodiments are described as processes or methods depicted as flow diagrams. Although the flow diagrams describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently or at the same time. In addition, the order of the operations can be rearranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figures. The processes can correspond to methods, functions, routines, subroutines, subprograms, etc. In addition, the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] The term "comprising" and its variations used in the present application are open, i.e. "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment".

[0053] It should be noted that the "first", "second" and the like mentioned in the present application are only used to distinguish the corresponding content, and are not used to limit the order or mutual dependence.

[0054] It should be noted that the modification of "one" and "multiple" mentioned in the present application is illustrative but not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0055] Embodiment one

[0056] Figure 1 A flowchart of a segmentation method provided for the first embodiment of the present application. The method can be applied to the segmentation of an image to be segmented. The method can be executed by a segmentation device, which can be realized by software and / or hardware, and is generally integrated on an electronic device, including but not limited to computers and mobile phones and the like.

[0057] The present application is suitable for image segmentation, including but not limited to image segmentation tasks of small targets and complex structures (such as brain tumors / liver tumors) of any medical images (such as MRI), and has wide application prospects.

[0058] Taking the segmentation of pulmonary nodules as an example, lung cancer is one of the malignant tumor diseases with the fastest growth of incidence and mortality rate in the world, which seriously threatens human health. Low-dose CT (Low-Dose Computed Tomography, LDCT) screening has been proved to be a practical technology, which can improve the detection and classification accuracy of early diagnosis of lung nodules, and help to reduce mortality. Lung nodule analysis is one of the effective cancer prevention interventions. By processing and analyzing chest CT images, suspicious nodules in the lungs are detected, and the size (diameter, volume) and type of the nodules are further determined, so that doctors can evaluate the patient's condition and determine the best treatment plan for the patient according to these information.

[0059] On the basis of lung nodule detection, lung nodule segmentation is an extremely important link: the calculation of lung nodule volume, the measurement of diameter, the three-dimensional modeling of lung tissue organs and lesions, surgical simulation and surgical planning all depend on the accurate segmentation of lung nodules, thereby providing an objective basis for the diagnosis and treatment of diseases.

[0060] The segmentation methods of pulmonary nodules mainly include manual segmentation, traditional semi-automatic segmentation method, and fully automatic segmentation based on deep learning. The manual segmentation method is time-consuming and labor-intensive, and is severely dependent on the personal experience of the operator, with poor segmentation consistency and repeatability. The traditional semi-automatic image segmentation method mainly includes threshold-based, region-based, edge-based, graph-based and energy functional-based methods:

[0061] (1)Threshold-based method mainly considers the gray level features of the image, and is suitable for images in which the target gray value is uniformly distributed outside the background gray value;

[0062] (2)Region-based method divides the image into different regions according to the similarity criterion, and only considers the spatial neighborhood information;

[0063] (3)Edge-based method mainly uses the discontinuity (i.e. pixel mutation) of local features of the image to obtain the boundary of the image target;

[0064] (4)Graph theory and energy functional-based method can handle more complex images, but the algorithm has a huge amount of calculation and requires human prior information.

[0065] In summary, the traditional segmentation method is difficult to achieve full-automatic and accurate segmentation due to the low utilization rate of information and features, complex calculation, large amount of calculation, low precision, the need for human prior and interaction, etc.

[0066] In recent years, image segmentation methods based on deep learning have shown great potential. Convolutional Neural Networks (CNNs) can automatically extract image features through convolutional layers, pooling layers, activation functions, etc. Network layers can obtain different spatial scales, textures, global and local image information, and achieve end-to-end image segmentation under the supervision of image labels.

[0067] For lung CT images, the main difficulties of lung nodule segmentation are:

[0068] (1) In the lung CT sequence, only a few slices exist, causing a serious imbalance between positive and negative samples;

[0069] (2) The lung nodule area is small, which belongs to the small target segmentation task, and the segmentation difficulty is high;

[0070] (3) There are many types of lung nodules, various shapes, complex textures, no fixed shape and structure, and high segmentation difficulty.

[0071] To solve the above technical problems, as shown in Figure 1 The segmentation method provided by Embodiment One of the present application comprises the following steps:

[0072] S110, obtaining image information of a to-be-segmented image setting surface, the setting surface comprising a main reference surface and an auxiliary reference surface.

[0073] The application scenario of the present application can be a scenario of segmenting a target object. The image to be segmented can be understood as an image to be segmented. The target object to be segmented can be included in the image to be segmented. The target object is not limited here and includes but is not limited to small targets and complex structures. The set surface can be a cross section of the target object. The set surface is not limited here and can include a main reference surface and an auxiliary reference surface when segmenting the image to be segmented. The main reference surface and the auxiliary reference surface are not limited by the present application and can be determined based on the target object to be segmented or the actual segmentation scenario. The main reference surface can be considered as the main analysis reference surface. The auxiliary reference surface can be considered as the auxiliary analysis reference surface. The present application uses deep learning technology to propose a segmentation method when segmenting the image to be segmented, referring to the method and habit of doctors when reading films. The main reference surface and the auxiliary reference surface of the image to be segmented are efficiently utilized to provide segmentation accuracy. The main reference surface and the auxiliary reference surface are not limited here and include but are not limited to one or more of the transverse, sagittal and coronal surfaces of the image to be segmented. The auxiliary reference surface can be a surface of the image to be segmented that is not selected as the main reference surface. For example, the main reference surface is the transverse surface and the auxiliary reference surface is the sagittal and coronal surfaces.

[0074] It can be understood that the image to be segmented can be an image obtained by processing an original image. The processing method is not limited here. The original image can be considered as an image including a target object. The electronic device can first extract a local image including a target object from the original image, and then pre-process the local image to obtain the image to be segmented. The pre-processing method is not limited here. The pre-processing method in the training stage and the application stage of the segmentation model can be different.

[0075] The present application does not limit how to extract the local image, as long as the local image includes the target object. It should be noted that when extracting the local image, if the coordinates corresponding to the local image exceed the coordinates of the edge of the original image, the exceeding part can be supplemented with 0 pixels.

[0076] The pre-processing method of the present application can include normalization operation. The present application can perform nonlinear processing before normalization operation, such as limiting the pixel value of the target image within a set range (i.e. realizing data truncation), and setting the pixel value exceeding the range to the boundary value of the set range with the smallest difference. The target image can be a local image or an image formed after pre-processing of the local image except normalization operation.

[0077] The original image is the original CT image sequence, also known as CT sequence data, and the local image is a 3D (Three-dimensional, 3D) image region containing a complete lung nodule region. The process of obtaining the image to be segmented is described as follows:

[0078] Figure 2 A kind of original CT image sequence and lung nodule region extraction image schematic diagram provided for the example embodiment of the present application, CT sequence data (i.e. original CT image sequence / original image) obtained in clinic, as shown in Figure 2 When performing lung nodule segmentation task, the 3D (Three-dimensional, 3D) image region containing complete lung nodule region, i.e. local image, needs to be extracted with lung nodule center as reference. Lung nodule detection and positioning method can adopt artificial reading film detection and positioning or mature detection algorithm. The lung nodule 3D region data extraction method in the present application is as follows:

[0079] According to the pixel position (x, y, z) of the lung nodule center, a cubic image region with length, width and height of rmm (for example, r = 45mm, which can be an empirical value) is extracted, i.e. local image. Wherein, the lung nodule center is located at the center of the cube. The boundary pixels of the cubic image region are calculated as follows:

[0080] (1) According to the lung nodule center and the length, width and height of the original CT image sequence, the vertex coordinates of the extracted cubic image region are calculated:

[0081] x min = x-r / spacing*0.5;

[0082] x max = x+r / spacing*0.5;

[0083] Wherein, x min , x max represent the minimum and maximum values of the x-axis direction of the extracted cubic image region in the pixel coordinate system, and spacing represents the voxel spacing in the x-axis direction of the image, with unit of mm. Similarly, replace x with y and z in the formula, y min , y max and z min , z max in the y-axis direction and z-axis direction can be obtained.

[0084] According to the acquired vertex coordinates, a cuboid image region containing the lung nodule can be extracted from the original CT image sequence. If the vertex coordinates exceed the size of the original image during the extraction process (whether the size exceeds the size of the original image is determined based on the length, width and height of the original CT image sequence), the exceeding part can be supplemented with 0 pixel value. The actual physical length of the extracted cuboid image region is r x r x r mm, the pixel length is l = r / spacing, and the image size is l x l x l, as shown in Figure 2

[0085] The preprocessing of the local image refers to normalization operation, flipping, expansion operation and adding Gaussian noise of the cuboid image region. It can be understood that the normalization operation can be the last operation of the image preprocessing.

[0086] Specifically, the normalization operation refers to normalizing the image pixel value range to (-1, 1), that is:

[0087] I = 2 x (I - I min ) / (I max -I min )-1;

[0088] In the formula, I represents the pixel value matrix of the sequence image currently subjected to the normalization operation, I min , and I max represent the minimum value and the maximum value of the current image pixel value matrix, respectively. The minimum and maximum values of the pixel value matrix can be set to fixed values by the application:

[0089] I min = -1024;

[0090] I max = 400;

[0091] Before normalization, I needs to be subjected to nonlinear transformation, that is, the pixel value less than I min becomes -1024, and the pixel value greater than I max becomes 400.

[0092] Flipping refers to mirror image operation of the image, and randomly flipping the image along any single axis or multiple axes of the x-axis, y-axis and z-axis to realize data augmentation of the training data (note that no flipping operation is performed in the algorithm inference application stage, that is, the segmentation model application stage). The training data can be considered as the training sample in the segmentation model training stage. Gaussian noise refers to adding Gaussian noise with the same size (l x l x l) as the input image to the image. No Gaussian noise is added in the segmentation model application stage. Adding Gaussian noise in the training stage can realize the requirement of data diversification and improve the accuracy of the segmentation model.

[0093] ​In the training stage of the segmentation model, the training samples can be derived from a public sample dataset (for example, the public dataset: Luna16, LNDb2019, or a self-made dataset, etc.). After obtaining the public sample dataset, the application can first extract local images, and then pre-process the pre-processed data and the corresponding label values to input into the to-be-trained model, so as to obtain the trained segmentation model.

[0094] The image pre-processing means in the training stage and the application stage of the segmentation model are different. For example, the pre-processing means in the training stage include, but are not limited to, normalization operation, flipping, expansion operation, and adding Gaussian noise. In the application stage of the segmentation model, the pre-processing means include, but are not limited to, normalization operation.

[0095] It can be understood that the to-be-segmented image can be a multi-dimensional picture, and the image information of the set surface can be image information obtained by processing the to-be-segmented image, such as rotating the to-be-segmented image. The processing means is not limited here.

[0096] S120, inputting the image information into the segmentation model to obtain a segmentation result of the to-be-segmented image output by the segmentation model.

[0097] The structure complexity of the main feature extraction module in the segmentation model is higher than that of the auxiliary feature extraction module in the segmentation model. The main feature extraction module is used to extract main image features of main image information. The auxiliary feature extraction module is used to extract auxiliary image features in auxiliary image information. The main image information is image information corresponding to the main reference surface. The auxiliary image information is image information corresponding to the auxiliary reference surface.

[0098] The segmentation result can be a result obtained by segmenting a target object in the to-be-segmented image. The segmentation structure can be an image including only the target object.

[0099] In this embodiment, the feature extraction module included in the segmentation model can be divided into a main feature extraction module and an auxiliary feature extraction module. The number of the main feature extraction module and the auxiliary feature extraction module is not limited, as long as the number of the feature extraction module is equal to the number of the image information of the set surface. The network structure of each main feature extraction module can be the same or different, and the network structure of each auxiliary feature extraction module can be the same or different.

[0100] The main reference surface can be input to a main feature extraction module, which can be a module for extracting image features in image information of the main reference surface. The auxiliary reference surface can be input to an auxiliary feature extraction module, which can be a module for extracting image features in image information of the auxiliary reference surface. The present application inputs the main reference surface and the auxiliary reference surface to the main feature extraction module and the auxiliary feature extraction module respectively to realize full-automatic segmentation of the image to be segmented based on a multi-branch network.

[0101] The network structure of the main feature extraction module is more complex than that of the auxiliary feature extraction module, so as to extract more and finer image features in the image information of the main reference surface. The structural complexity can be regarded as the complexity of the network structure. The structural complexity can be reflected in the number of layers of the network structure, which is not limited herein. The present embodiment does not limit the network structure of the segmentation model.

[0102] The segmentation method provided by the first embodiment of the present application first acquires image information of a set surface of an image to be segmented, wherein the set surface includes a main reference surface and an auxiliary reference surface; then inputs the image information into a segmentation model to acquire a segmentation result of the image to be segmented output by the segmentation model; wherein the structural complexity of a main feature extraction module in the segmentation model is higher than that of an auxiliary feature extraction module in the segmentation model, the main feature extraction module is used to extract main image features of main image information, and the auxiliary feature extraction module is used to extract auxiliary image features in auxiliary image information, the main image information is image information corresponding to the main reference surface, and the auxiliary image information is image information corresponding to the auxiliary reference surface. By using the above technical solution, image segmentation can be performed based on the image information of the set surface of the image to be segmented, different modules are used to process different set surfaces during segmentation, and the segmentation accuracy is improved.

[0103] Embodiment two

[0104] Figure 3 The flowchart of the segmentation method provided by the second embodiment of the present application is based on the above embodiments. In the present embodiment, the segmentation model specifically includes a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module and a decoding module; and the image information input into the segmentation model to acquire the segmentation result of the image to be segmented output by the segmentation model specifically includes:

[0105] The main image information is input into the main feature extraction module to obtain main image features;

[0106] The auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features;

[0107] The auxiliary image features are input into the feature rotation module for rotation transformation;

[0108] The main image features and the rotated auxiliary image features are merged and then input into the feature fusion module to obtain the fused image features;

[0109] The fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented.

[0110] For details not covered in this embodiment, please refer to Embodiment 1.

[0111] like Figure 3 As shown, the segmentation method provided in Embodiment 2 of the present invention includes the following steps:

[0112] S210. Obtain image information of the set surface of the image to be segmented.

[0113] S220. Input the main image information into the main feature extraction module to obtain the main image features.

[0114] The segmentation model includes a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module, and a decoding module.

[0115] The main feature extraction module analyzes the main image information. The auxiliary feature extraction module analyzes the auxiliary image information. The feature rotation module can be considered as a module that performs feature rotation transformations, used for pixel alignment, such as aligning pixels in each defined surface. The feature fusion module can be considered as a module that performs feature fusion. The decoding module can be considered as a module that complementarily fuses the information from the defined surfaces to extract the target object.

[0116] When inputting image information into the segmentation model, the main image information is fed into the main feature extraction module to extract the main image features. The specific network structure of the main feature extraction module is not specified here.

[0117] S230. Input the auxiliary image information into the auxiliary feature extraction module to obtain auxiliary image features.

[0118] When inputting image information into the segmentation model, auxiliary image information is fed into the auxiliary feature extraction module to extract auxiliary image features. The specific network structure of the auxiliary feature extraction module is not specified here.

[0119] Taking a lung nodule as the target, Figure 4An illustrative diagram of setting image information of a surface is provided for an example embodiment of the present application. The present application draws on the method and habit of doctors when reading a film, that is, a transverse surface is mainly used as a reading view (i.e., a main reference surface), and a coronal surface and a sagittal surface are used as auxiliary reading views (i.e., auxiliary reference surfaces), and the three views are combined, so that information can be complementary, and the target object can be observed more carefully and comprehensively, such as Figure 4 As shown in the figure, the image performance of the same nodule under different views is shown.

[0120] Similar to doctors reading a film, when learning image features, a neural network can also learn images of different views to obtain image features under different perspectives, and then fuse the image features, that is, complement the information, so as to achieve a higher-precision lung nodule segmentation result.

[0121] S240, input the auxiliary image features into the feature rotation module for rotation transformation.

[0122] The auxiliary image features are input into the feature rotation module for feature rotation to realize the correspondence of the pixel points. The structure of the feature rotation module is not limited here.

[0123] S250, merge the main image features and the rotated auxiliary image features, and input the merged features into the feature fusion module to obtain fused image features.

[0124] When the fused image features are obtained, the main image features and the auxiliary image features can be merged first. Then the merged features are input into the feature fusion module to obtain the fused image features.

[0125] S260, input the fused image features into the decoding module to obtain a segmentation result of the image to be segmented.

[0126] The structure of the decoding module is not limited here. The decoding module can include a residual sub-module, an intermediate convolution sub-module, and a convolution sub-module. The intermediate convolution sub-module can be used to realize the extraction and fusion of information. The residual sub-module can use the residual idea to retain more detailed information in the segmentation result. The convolution sub-module can realize channel transformation. The decoding module effectively utilizes image features of each layer, and improves the segmentation precision.

[0127] Figure 5 A network architecture diagram of a segmentation model is provided for an example embodiment of the present application. The network framework of the segmentation model is shown in Figure 5

[0128] ​The segmentation model proposed in this invention is an auto-encoder (AE) structure. It uses the extracted 3D region containing lung nodules (a cube image or a pre-processed cube image, 1×1×1) as image input. Three views (i.e., image information of the defined planes) are input to three encoders, namely the feature extraction module (Transverse Plane Encoder, Coronal Plane Encoder, and Sagittal Plane Encoder) for feature extraction. The transverse plane is the primary reference view, also known as the primary reference plane. Therefore, this invention designs a more complex Transverse Plane Encoder structure to extract more and finer features. The coronal and sagittal planes are auxiliary reference views, also known as auxiliary reference planes. Due to limitations in computer memory and considerations for model size, this invention can design the Coronal Plane Encoder and Sagittal Plane Encoder with the same structure, which is more lightweight and has fewer network layers compared to the Transverse Plane Encoder.

[0129] After obtaining the image features extracted from the three views, the image features need to be fused. First, the auxiliary image features, i.e., the feature matrices, of the coronal and sagittal planes are rotated ( Figure 5 The feature rotation module preceding the feature fusion module (not shown) aligns each pixel in the matrix with the pixels of the cross-section. Next, a concatenation operation is performed on features corresponding to the same resolution. After obtaining the merged features, the information extracted from each view needs to be further fused using the Feature Fusion Module. The structure of the Feature Fusion Module can be Conv1×1×1+GN+ReLU operations. Thus, five multi-scale image features are obtained, which are the fused image features.

[0130] It is important to note that the feature extraction module in this example includes a transverse plane encoder, a coronal plane encoder, and a sagittal plane encoder. The transverse plane encoder can be considered the main feature extraction module in this example, while the coronal plane encoder and sagittal plane encoder can be considered auxiliary feature extraction modules. The decoding module in this example can be... Figure 5 Shared decoding in.

[0131] It should be noted that in this example embodiment, the set plane includes the cross-section, the coronal plane, and the sagittal plane.

[0132] The embodiment two of the present application provides a segmentation method, which specifically realizes the structure of the segmentation model and the specific operation of obtaining the segmentation result based on image information. By using the method, each scale image feature can be effectively analyzed, and the segmentation precision of the target object is improved.

[0133] On the basis of the above-mentioned embodiments, variant embodiments of the above-mentioned embodiments are proposed, and it should be noted that, in order to make the description brief, only the differences from the above-mentioned embodiments are described in the variant embodiments.

[0134] In one embodiment, the number of the auxiliary feature extraction modules is equal to the number of the auxiliary reference surfaces; the network structures of the auxiliary feature extraction modules are the same or different; when the number of the auxiliary reference surfaces is at least two, the auxiliary image features corresponding to each of the auxiliary reference surfaces are input into the corresponding auxiliary feature extraction module.

[0135] In the present embodiment, one auxiliary image information corresponds to one auxiliary feature extraction module, and the network structures of each auxiliary feature extraction module can be the same or different.

[0136] In one embodiment, the main feature extraction module comprises: a plurality of main feature extraction networks, each main feature extraction network comprising an encoding submodule and a center residual submodule, and each main feature extraction network is connected through the corresponding encoding submodule; accordingly, the main image information is input into the main feature extraction module to obtain the main image feature, comprising:

[0137] The main target information is input into each layer of the main feature extraction network to obtain the corresponding image feature.

[0138] When the main feature extraction network is the first layer main feature extraction network, the main target information is the main image information, when the main feature extraction network is a non-first layer main feature extraction network, the main target information is the output of the encoding submodule included in the last layer main feature extraction network, and the main image feature comprises the image feature output by each layer of the main feature extraction network.

[0139] The image feature output by each layer of the main feature extraction network is the sum of the output of the corresponding encoding submodule and the output of the corresponding center residual submodule. The input of the residual submodule of each main feature extraction network is the output of the corresponding encoding submodule.

[0140] The encoding submodule can comprise a convolution layer and a pooling layer, or only comprise a convolution layer, and the structure of the encoding submodule is not specifically limited here. The encoding submodule can be used for feature extraction.

[0141] Figure 6This is a schematic diagram of the network architecture of a main feature extraction module provided in an example embodiment of the present invention. In this example, the main feature extraction module is a Transverse Plane Encoder, and the network structure diagram of the main feature extraction module is shown below. Figure 6 As shown in the diagram. Here, Conv represents 3D convolution, GN represents Group Normalization (GN), and ReLU represents the activation function. Each encoding module, i.e., the encoding sub-module, has a consistent structure, all being a ConvEnc-Module structure. It should be noted that max pooling is not required in ConvEnc-Module1. Through the main feature extraction module, five image features at different scales can be obtained, namely Feature x1 to Feature x5. Conv1×1×1+GN+ReLU can be considered as the central residual sub-module. Figure 6 It includes five main feature extraction networks, each of which can be considered as a main feature extraction network.

[0142] Figure 7 This is a schematic diagram of the structure of an encoding submodule provided in an example embodiment of the present invention. See also: Figure 7 The encoding submodule consists of three convolutional layers and one pooling layer (i.e., max pooling 2×2×2). The first layer of the encoding submodule in the main feature extraction module may include three convolutional layers but not a pooling layer. The first layer of the encoding submodule in the auxiliary feature extraction module may include three convolutional layers and one pooling layer.

[0143] The number of convolutional layers included in the encoding submodule is not limited, nor is the size of the convolutional kernel in each convolutional layer. For example, Conv1×1×1+GN is the first convolutional layer, and Conv3×3×3+GN+ReLU is the second convolutional layer. The kernel size of the second convolutional layer can be larger than the kernel size of the first convolutional layer. The three convolutional layers in the encoding submodule can include two second convolutional layers and one first convolutional layer.

[0144] In one embodiment, the auxiliary feature extraction module includes: multiple auxiliary feature extraction networks, each layer of which is connected sequentially; correspondingly, the auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features, including:

[0145] The auxiliary target information is input into each layer of the auxiliary feature extraction network to obtain the corresponding image features;

[0146] When the auxiliary feature extraction network is a first layer auxiliary feature extraction network, the auxiliary target information is the auxiliary image information, and when the auxiliary feature extraction network is a non-first layer auxiliary feature extraction network, the auxiliary target information is the output of a previous layer auxiliary feature extraction network. The auxiliary image features include image features output by each layer auxiliary feature extraction network, and the number of auxiliary feature extraction networks is less than the number of main feature extraction networks included in the main feature extraction module.

[0147] The embodiment is not limited to the structure of each layer auxiliary feature extraction network. For example, the first layer auxiliary feature extraction network can include two convolution operations with different convolution kernel sizes. The auxiliary feature extraction network other than the first layer auxiliary feature extraction network can be implemented by an encoding submodule.

[0148] Figure 8 A network structure diagram of an auxiliary feature extraction module is provided for the example embodiment of the present application. In the example, the auxiliary feature extraction module is a Coronal Plane Encoder and a Sagittal Plane Encoder. The network structure of the auxiliary feature extraction module is shown in FIG. 6. Figure 8 Conv5x5x52 indicates a convolution operation with a convolution kernel of 5x5x5 and a convolution step of 2; and ConvEnc-Module1 and ConvEnc-Module2 have the same structure as ConvEnc-Module in the Transverse Plane Encoder. Through the auxiliary feature extraction module, three different scale image features in the coronal plane and the sagittal plane can be obtained, i.e., Feature x2’~Feature x4’.

[0149] Figure 8 The auxiliary feature extraction module includes three layer auxiliary feature extraction networks, i.e., a layer auxiliary feature extraction network formed by Conv5x5x52+GN+Relu and Conv3x3x32+GN+Relu (the image feature corresponding to the layer is Feature x2’), a layer auxiliary feature extraction network formed by ConvEnc-Module1 (the image feature corresponding to the layer is Feature x3’), and a layer auxiliary feature extraction network formed by ConvEnc-Module2 (the image feature corresponding to the layer is Feature x4’).

[0150] It should be noted that the size of the convolution kernel of the convolution operation in the main feature extraction module and the auxiliary feature extraction module is not limited, the first layer auxiliary feature extraction network in the auxiliary feature extraction module includes two convolution operations with different sizes of convolution kernels. The size of the convolution kernel of the convolution operation in the center residual submodule in the main feature extraction module can be smaller than the size of the convolution kernel of the convolution operation in the auxiliary feature extraction module. The size of the convolution kernel of the convolution operation in the center residual submodule in the main feature extraction module can be equal to the size of the convolution kernel of the first convolution layer, or smaller than the size of the convolution kernel of the second convolution layer.

[0151] In one embodiment, the decoding module includes a plurality of decoding networks, the number of decoding networks included in the decoding module is equal to the number of main feature extraction networks included in the main feature extraction module; the outputs of the set submodules in the target decoding network except the first layer decoding network are respectively up-sampled to the inputs of the last layer decoding network; correspondingly, the fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented, including:

[0152] The fused image features are input into each decoding network to obtain a segmentation result, which is the sum of the outputs of the first layer decoding network and the set layer decoding network in each decoding network;

[0153] The last layer decoding network in the decoding module uses dilated convolution with different dilated convolution rates to process the input fused image features. Except for the last layer decoding network in the decoding module, the input is processed by a residual submodule or the input is processed by a residual submodule and a convolution submodule in the decoding module. The set submodule in the last layer decoding network is an intermediate convolution submodule, and the set submodule in the decoding network except the last layer decoding network in the target decoding network is a residual submodule.

[0154] The number of fused image features is equal to the number of decoding networks, and the fused image features are respectively input into the corresponding decoding networks to obtain a segmentation result after processing by the decoding module.

[0155] The structure of the highest layer decoding network, i.e. the last layer decoding network, is not limited as long as it uses dilated convolution with different dilated convolution rates. The highest layer decoding network can obtain features with rich detail information and multiple scales.

[0156] The network structure of the residual submodule is not limited as long as it can use the residual idea to retain more rich detail information in the segmentation result, such as the residual submodule can be realized by convolution operations with different sizes of convolution kernels, activation functions and normalization operations.

[0157] The decoding network other than the highest layer decoding network can first merge the input features, and then input to the residual sub-module and the convolution sub-module in turn; or input the merged features to the residual sub-module.

[0158] The embodiment does not limit the set layer decoding network to which layer decoding network, such as the decoding network of the first set layer in the decoding module, and the embodiment obtains the segmentation result in combination with the output of the set sub-module to facilitate obtaining more semantic information.

[0159] The means of adding the outputs of the first layer decoding network and the set layer decoding network is not limited, such as up-sampling the output of the lowest layer decoding network in the first layer decoding network and the set layer decoding network to the previous layer, and then adding the output of the previous layer convolution sub-module to serve as the output of the decoding network, and the output of the first layer decoding network can be the segmentation result, which is the addition of the output of the first layer decoding network convolution sub-module and the up-sampled output.

[0160] Figure 9 A structural diagram of a decoding module provided by an example embodiment of the present application is shown in FIG. 1. The decoding part adopts a structure of shared weights, which can more effectively complementally fuse and utilize the information of each view, and can reduce the demand of the model on the video memory and reduce the size of the model. Figure 9 As shown in the figure, the structure of the decoding module is mainly composed of a ConvMid-Module intermediate convolution sub-module (the last layer decoding network), a Residual Module residual sub-module, and a Conv-Module convolution sub-module. The layer where Feature x1 is located is the first layer decoding network. The first to third layer decoding networks each include an intermediate convolution sub-module and a residual sub-module. The fourth layer decoding network includes a residual sub-module. The fifth layer decoding network includes an intermediate convolution sub-module.

[0161] It should be noted that, Figure 9 Fusion Feature x2 can be considered as a fused image feature, which is obtained by fusing the main image feature Feature x2 and the auxiliary image feature Feature x2' after processing. Fusion Feature x3 and Fusion Feature x4 are the same, and are not described here.

[0162] Figure 10 A structural diagram of an intermediate convolution sub-module provided by an example embodiment of the present application is shown in FIG. 2. The present application performs separate module operations on high layer features Feature x5. The module structure of the intermediate convolution sub-module is as shown in the figure. Figure 10As shown in the ConvMid-Module, in order to obtain detailed and multi-scale features, the application adds dilated convolutions with different dilated convolution rates in the intermediate convolution sub-module of the decoding module, and different levels of features are extracted and fused through concatenate operation. In the last intermediate convolution sub-module, the application uses a non-local module to capture long-distance information (the non-local module is a publicly disclosed method, which will not be described in detail here).

[0163] Figure 11 A structural diagram of a residual sub-module provided for an example embodiment of the application. The decoding module also includes four residual sub-modules Residual Module with the same structure, and the detailed network structure is as follows Figure 11 As shown in the Residual-Module, the residual idea is mainly used to retain more detailed information in the segmentation result.

[0164] The Conv-Module in the decoding module is a convolution sub-module, which is used to output a 2-channel feature map from multi-channel features, and can be a conv1x1x1 convolution. By summing different levels of features, the feature information of each layer can be effectively utilized to improve the segmentation accuracy.

[0165] In one embodiment, the loss function of the segmentation model is determined based on the outputs of multiple decoding networks of the decoding module and corresponding label values. The multiple decoding networks for determining the loss function include a first layer decoding network, and the loss function of the first layer decoding network is determined based on the weighted foreground overlap coefficient and the weighted background overlap coefficient.

[0166] The multiple decoding networks can be set layer decoding networks. When determining the loss function, the outputs of the multiple decoding networks are combined to obtain more semantic information. The loss function of the first layer decoding network can be determined based on the foreground overlap coefficient, the background overlap coefficient and the weight, such as the loss function of the first layer decoding network being determined based on the weighted sum of 1 minus the foreground overlap coefficient and 1 minus the background overlap coefficient. The foreground overlap coefficient can be the overlap coefficient of the target object, and the background overlap coefficient can be considered as the overlap coefficient of the background. The weights of the foreground overlap coefficient and the background overlap coefficient are not limited here.

[0167] The technical means for determining the loss function adopted by different decoding networks can be different, which is not limited here.

[0168] Specifically, when determining the loss function based on the outputs of multiple decoding networks and their corresponding label values, different calculation methods can be used to determine the respective loss functions of different decoding networks and their corresponding label values. Finally, the sum of the loss functions corresponding to each decoding network is used to determine the loss function of the segmentation model. For example, this invention uses the Sigmoid activation function in the last layer of the segmentation model. The optimization objective of the segmentation model is the sum of the weighted Dice loss and the binary cross-entropy loss between the segmentation result predicted by the network and the gold standard label (Ground Truth, GT). Considering that features at different levels provide different information, more detailed information is needed for y1 with the highest resolution, so Dice loss is used; for y2 and y3 (the outputs of the set layer decoding networks), more semantic information is provided, and the loss is calculated using binary cross-entropy, i.e.:

[0169] Loss=Dice_loss(y1,gt)+BCE(y2,gt)+BCE(y3,gt)

[0170]

[0171]

[0172] Dice_loss=λ(1-Dice forground )+(1-λ)(1-Dice background )

[0173] In the formula, y1, y2, and y3 are respectively Figure 9 The decoding output in the dataset refers to the output of a portion of the decoding networks; gt represents the gold standard image label, i.e., the label value; Dice is the overlap index, mainly used to calculate the similarity between two sets; S gt S represents the set of pixels of the Ground Truth. p This represents the set of pixels in the segmentation result to be evaluated; in BCE, y represents the value of Ground Truth, and p i This represents the output value of the logic layer. In Dice_loss, the Dice coefficients for the foreground and background are calculated separately according to the formula, i.e., Dice... forground and Dice background To alleviate the severe imbalance in the foreground and background pixel ratio during lung nodule segmentation, different weights can be assigned to different Dice losses, where λ is the weight value for the foreground (e.g., λ = 0.9), thus obtaining the corresponding Dice_loss. Here, the target object can be considered the foreground, and other objects can be considered the background.

[0174] In the above formula, Loss can be considered as the loss function of the segmentation model, and the outputs of the plurality of decoding networks include y1, y2 and y3. gt is the corresponding label value. Dice_loss(y1, gt) is the loss function of the first layer segmentation network determined based on the foreground overlap coefficient, the background overlap coefficient and the weight.

[0175] According to the network structure and optimization target provided in the application, a corresponding training model can be constructed, and a corresponding segmentation model can be obtained by training through a corresponding lung nodule segmentation data set (for example, a public data set: Luna16, LNDb2019, or a self-made data set).

[0176] The embodiment of the application provides the following specific implementation modes on the basis of the technical solutions of the above embodiments.

[0177] The present example takes the lung nodule included in the image to be segmented as an example to illustrate the method for automatically segmenting the lung nodule in the image to be segmented, which includes the following three steps:

[0178] Step 1: lung nodule 3D region data (i.e., local image) extraction and image preprocessing;

[0179] Step 2: design and construction of a lung nodule segmentation neural network model (i.e., a segmentation model) of a Multi-View multi-branch method, and model training;

[0180] Step 3: model application, forward inference of the model, and thus obtaining a lung nodule segmentation result (i.e., a segmentation result in the application).

[0181] Taking the segmentation of the lung nodule as an example, the model application, forward inference of the model on new data (i.e., an original image), and the segmentation result are obtained. After obtaining new clinical data (i.e., an original image) and the detection result of the lung nodule (i.e., the center position coordinates of the lung nodule), the lung nodule 3D region data is extracted according to step 1, and a normalization operation is performed (note that the flipping, adding Gaussian noise operation and expansion operation are not performed at this time). The normalized data is input into the trained network model (i.e., the segmentation model), and the forward inference of the network is performed, and thus the final segmentation result is obtained.

[0182] The segmentation model of the application has a method precision higher than that of all online methods in the LNDb Grand Challenge (https: / / lndb.grand-challenge.org / Home / ) online evaluation, and the details are as follows:

[0183] The public data sets Luna16 and LNDb2019 are used as the data set for model training. Since each lung nodule in the public data set is labeled by multiple doctors, in order to reduce the difference between the labels of different doctors, the common labeled pixels between two doctors are taken as the ground truth of lung nodule labeling, about 2000 lung nodules and corresponding mask labels are obtained, and are used as the training set of the model. After the model training is completed, the independent test set provided by LNDb2019 is forward inferred to obtain the lung nodule segmentation result, and online evaluation is performed. Table 1 is an effect comparison table of the segmentation model provided by the application and other models, and the results of the method of the application and the results of a method of Challenge are as shown in the following table. It can be seen that the method of the application is better than the corresponding method in six evaluation indexes.

[0184] Table 1 is an effect comparison table of the segmentation model provided by the application and other models

[0185]

[0186] In summary, the application is aimed at the problems of imbalance between positive and negative samples, high segmentation difficulty and the like in lung nodule segmentation. By referring to the reading process and method of doctors, a multi-branch lung nodule segmentation deep learning model (MVE-SDNet) based on Multi-View, i.e., a segmentation model, is designed, which can more effectively utilize the image information under different perspectives (transverse, sagittal and coronal) of three-dimensional images, extract more rich image features, form information complementation, and effectively improve the accuracy of lung nodule segmentation. The segmentation result can provide more reliable and objective basis.

[0187] Compared with the existing methods, the difference of the method is at least:

[0188] (1) The application is a fully automatic lung nodule segmentation method, which does not need human intervention;

[0189] (2) Compared with the manual segmentation of the reading personnel, the method has repeatability and consistency of segmentation result, is more reliable, faster and lower in cost;

[0190] (3) The application refers to the reading process and method of doctors, and the designed MVE-SDNet can more effectively utilize the image information under different perspectives (transverse, sagittal and coronal) of three-dimensional images, extract more rich image features, form information complementation, and effectively improve the accuracy of lung nodule segmentation.

[0191] (4) The application has generalizability and can be very conveniently applied to other tissue, organ and data modal task (such as MRI image brain tumor, CT image liver tumor segmentation task and the like).

[0192] The application proposes to learn from the process and ideas of doctors reading films, and designs a multi-branch lung nodule segmentation network based on Multi-View. The specific structure and implementation of each module of the network are replaceable, and can be adjusted appropriately according to requirements. The data preprocessing methods, such as normalization and data augmentation strategies, are replaceable.

[0193] Embodiment three

[0194] Figure 12 A structural schematic diagram of a segmentation device provided for the third embodiment of the application. The device can be applicable to the case of segmenting an image to be segmented, wherein the device can be realized by software and / or hardware, and is generally integrated on an electronic device.

[0195] As shown in Figure 12 , the device comprises:

[0196] A first acquisition module 31 is configured to acquire image information of a set surface of an image to be segmented, wherein the set surface comprises a main reference surface and an auxiliary reference surface.

[0197] A second acquisition module 32 is configured to input the image information into a segmentation model, and acquire a segmentation result of the image to be segmented output by the segmentation model.

[0198] In the segmentation model, the structural complexity of a main feature extraction module is higher than that of an auxiliary feature extraction module. The main feature extraction module is configured to extract main image features of main image information, and the auxiliary feature extraction module is configured to extract auxiliary image features of auxiliary image information. The main image information corresponds to the main reference surface, and the auxiliary image information corresponds to the auxiliary reference surface.

[0199] In this embodiment, the device first acquires image information of a set surface of an image to be segmented through the first acquisition module 31, wherein the set surface comprises a main reference surface and an auxiliary reference surface. Then, the second acquisition module 32 inputs the image information into a segmentation model, and acquires a segmentation result of the image to be segmented output by the segmentation model.

[0200] The embodiment provides a segmentation device, which can perform image segmentation based on image information of a set surface of an image to be segmented. Different modules are used to process different set surfaces during segmentation, thereby improving the segmentation accuracy.

[0201] In one embodiment, the segmentation model comprises a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module, and a decoding module. Correspondingly, the second acquisition module 32 is specifically configured to:

[0202] input the main image information into the main feature extraction module to obtain main image features;

[0203] inputting the auxiliary image information into an auxiliary feature extraction module to obtain auxiliary image features;

[0204] inputting the auxiliary image features into the feature rotation module for rotation transformation;

[0205] merging the main image features and the rotated auxiliary image features to obtain fused image features;

[0206] inputting the fused image features into the decoding module to obtain a segmentation result of the image to be segmented.

[0207] In one embodiment, the number of auxiliary feature extraction modules is equal to the number of auxiliary reference surfaces; the network structures of the auxiliary feature extraction modules are the same or different; when the number of auxiliary reference surfaces is at least two, the auxiliary image features corresponding to each auxiliary reference surface are input into the corresponding auxiliary feature extraction module.

[0208] In one embodiment, the main feature extraction module includes a plurality of main feature extraction networks, each main feature extraction network including an encoding submodule and a center residual submodule, and each main feature extraction network is connected through the corresponding encoding submodule; correspondingly, the second acquisition module 32 inputs the main image information into the main feature extraction module to obtain main image features, including:

[0209] inputting main target information into each main feature extraction network to obtain corresponding image features;

[0210] When the main feature extraction network is a first main feature extraction network, the main target information is the main image information; when the main feature extraction network is a non-first main feature extraction network, the main target information is the output of the encoding submodule included in the last main feature extraction network; and the main image features include the image features output by each main feature extraction network.

[0211] In one embodiment, the auxiliary feature extraction module includes a plurality of auxiliary feature extraction networks, and each auxiliary feature extraction network is connected in sequence; correspondingly, the second acquisition module 32 inputs the auxiliary image information into the auxiliary feature extraction module to obtain auxiliary image features, including:

[0212] inputting auxiliary target information into each auxiliary feature extraction network to obtain corresponding image features;

[0213] When the auxiliary feature extraction network is a first layer auxiliary feature extraction network, the auxiliary target information is the auxiliary image information, and when the auxiliary feature extraction network is a non-first layer auxiliary feature extraction network, the auxiliary target information is the output of the last layer auxiliary feature extraction network. The auxiliary image features include image features output by each layer of the auxiliary feature extraction network, and the number of the auxiliary feature extraction networks is less than the number of the main feature extraction networks included in the main feature extraction module.

[0214] In one embodiment, the decoding module includes a plurality of decoding networks, and the number of the decoding networks included in the decoding module is equal to the number of the main feature extraction networks included in the main feature extraction module. The outputs of the target decoding networks except the first layer decoding network are respectively up-sampled to the inputs of the last layer decoding network. Correspondingly, the fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented, including:

[0215] The fused image features are input into each decoding network to obtain a segmentation result, and the segmentation result is the sum of the outputs of the first layer decoding network and the set layer decoding network in each decoding network.

[0216] The last layer decoding network in the decoding module processes the input fused image features by using an expansion convolution with different expansion convolution rates. The decoding network in the decoding module except the last layer decoding network processes the input by using a residual sub-module, or the decoding network in the decoding module except the last layer decoding network processes the input by using a residual sub-module and a convolution sub-module. The set sub-module in the last layer decoding network is an intermediate convolution sub-module, and the set sub-module in the decoding network except the last layer decoding network in the target decoding network is a residual sub-module.

[0217] In one embodiment, the loss function of the segmentation model is determined based on the outputs of the plurality of decoding networks of the decoding module and the corresponding label values. The plurality of decoding networks for determining the loss function include the first layer decoding network, and the loss function of the first layer decoding network is determined based on the weighted foreground overlap coefficient and the weighted background overlap coefficient.

[0218] The segmentation device described above can perform the segmentation method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of performing the method.

[0219] Embodiment Four

[0220] Figure 13 A structural schematic diagram of an electronic device provided by Embodiment Four of the present application. As shown in the figure, Figure 13 The electronic device provided by Embodiment Four of the present application includes one or more processors 41 and storage devices 42. The processor 41 in the electronic device can be one or more,Figure 13 The electronic device can include one or more processors 41; a memory 42 storing one or more programs; and the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement the segmentation method according to any one of the embodiments of the present application.

[0221] The electronic device can further include an input device 43 and an output device 44.

[0222] The processor 41, the memory 42, the input device 43 and the output device 44 in the electronic device can be connected by a bus or other means, Figure 13 For example, by bus connection.

[0223] The memory 42 in the electronic device can be used to store one or more programs as a computer readable storage medium, and the program can be a software program, a computer executable program and a module, such as the program instructions / modules corresponding to the segmentation method provided in the first or second embodiment of the present application (for example, the first acquisition module 31 and the second acquisition module 32 shown in the segmentation device). Figure 12 The processor 41 executes the software program, instruction and module stored in the memory 42, so as to execute various function applications and data processing of the electronic device, that is, to implement the segmentation method in the above method embodiments.

[0224] The memory 42 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 42 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the memory 42 can further include a memory remotely arranged with respect to the processor 41, which can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0225] The input device 43 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. The output device 44 can include a display device such as a display screen.

[0226] And when the above electronic device includes one or more programs executed by the one or more processors 41, the program performs the following operations:

[0227] Obtain image information of an image to be segmented, and the image to be segmented includes a main reference surface and an auxiliary reference surface;

[0228] inputting the image information into a segmentation model, and obtaining a segmentation result of the image to be segmented output by the segmentation model;

[0229] The structure complexity of the main feature extraction module in the segmentation model is higher than that of the auxiliary feature extraction module in the segmentation model.

[0230] Embodiment Five

[0231] Embodiment Five of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to perform a segmentation method, the method comprising: obtaining image information of a set surface of an image to be segmented, the set surface comprising a main reference surface and an auxiliary reference surface;

[0232] inputting the image information into a segmentation model, and obtaining a segmentation result of the image to be segmented output by the segmentation model;

[0233] The structure complexity of the main feature extraction module in the segmentation model is higher than that of the auxiliary feature extraction module in the segmentation model.

[0234] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0235] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer readable program code is contained. Such propagated data signal can take a variety of forms, including, but not limited to, electro-magnetic, optical or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a storage medium and that can communicate, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device.

[0236] The program code contained on the computer readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wire line, optical fiber cable, radio frequency (RF), and the like, or any suitable combination thereof.

[0237] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0238] It is to be understood that the above description is directed to the preferred embodiments and that those skilled in the art will be able to devise various modifications which, although not specifically described herein, embody the principles of the application and are included within the spirit and scope of the application. Accordingly, while the preferred embodiments have been described above, those skilled in the art will understand that they are not to be limited to the preferred embodiments, but are to include all such embodiments falling within the scope of the application as defined by the appended claims.

Claims

1. A segmentation method, characterized in that, include: Obtain image information of the defined plane of the image to be segmented, wherein the defined plane includes a main reference plane and an auxiliary reference plane; The image information is input into the segmentation model to obtain the segmentation result of the image to be segmented output by the segmentation model; In this segmentation model, the structural complexity of the main feature extraction module is higher than that of the auxiliary feature extraction module. The main feature extraction module is used to extract the main image features from the main image information, and the auxiliary feature extraction module is used to extract the auxiliary image features from the auxiliary image information. The main image information is the image information corresponding to the main reference plane, and the auxiliary image information is the image information corresponding to the auxiliary reference plane. The segmentation model includes a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module, and a decoding module. Correspondingly, the image information is input into the segmentation model to obtain the segmentation result of the image to be segmented output by the segmentation model, including: The main image information is input into the main feature extraction module to obtain the main image features; The auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features; The auxiliary image features are input into the feature rotation module for rotation transformation; The main image features and the rotated auxiliary image features are merged and then input into the feature fusion module to obtain the fused image features; The fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented; The decoding module includes multiple decoding networks, the number of which is equal to the number of main feature extraction networks in the main feature extraction module. The outputs of sub-modules in the target decoding network (excluding the first-layer decoding network) are upsampled to the inputs of the previous-layer decoding network. Correspondingly, the fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented, including: The fused image features are input into each of the decoding networks to obtain a segmentation result, which is the sum of the outputs of the first layer decoding network and the set layer decoding network in each of the decoding networks. The decoding network in the last layer of the decoding module uses dilated convolution with different dilation rates to process the fused image features of the input. The decoding network in the decoding module, excluding the last layer, processes the input through a residual submodule or through a combination of residual and convolutional submodules. The submodule in the last layer of the decoding network is set as an intermediate convolutional submodule. The submodules in the target decoding network, excluding the last layer of the decoding network, are set as residual submodules.

2. The method according to claim 1, characterized in that, The number of auxiliary feature extraction modules is equal to the number of auxiliary reference surfaces; the network structures of each auxiliary feature extraction module may be the same or different; when the number of auxiliary reference surfaces is at least two, the auxiliary image information corresponding to each auxiliary reference surface is input into the corresponding auxiliary feature extraction module.

3. The method according to claim 1, characterized in that, The main feature extraction module includes: multiple main feature extraction networks, each main feature extraction network including an encoding submodule and a central residual submodule, and each main feature extraction network is connected through its corresponding encoding submodule; correspondingly, the main image information is input into the main feature extraction module to obtain the main image features, including: The main target information is input into each layer of the main feature extraction network to obtain the corresponding image features; When the main feature extraction network is a first-layer main feature extraction network, the main target information is the main image information. When the main feature extraction network is not a first-layer main feature extraction network, the main target information is the output of the coding sub-module included in the previous layer main feature extraction network. The main image features include the image features output by each layer main feature extraction network.

4. The method according to claim 1, characterized in that, The auxiliary feature extraction module includes: multiple auxiliary feature extraction networks, with each layer of the auxiliary feature extraction network connected sequentially; correspondingly, the auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features, including: The auxiliary target information is input into each layer of the auxiliary feature extraction network to obtain the corresponding image features; When the auxiliary feature extraction network is the first layer of auxiliary feature extraction network, the auxiliary target information is the auxiliary image information. When the auxiliary feature extraction network is not the first layer of auxiliary feature extraction network, the auxiliary target information is the output of the previous layer of auxiliary feature extraction network. The auxiliary image features include the image features output by each layer of auxiliary feature extraction network. The number of auxiliary feature extraction networks is less than the number of main feature extraction networks included in the main feature extraction module.

5. The method according to claim 1, characterized in that, The loss function of the segmentation model is determined based on the outputs of multiple decoding networks of the decoding module and the corresponding label values. Among the multiple decoding networks that determine the loss function, there is a first-layer decoding network. The loss function of the first-layer decoding network is determined based on the weighted foreground overlap coefficient and the weighted background overlap coefficient.

6. A dividing device, characterized in that, include: The first acquisition module is used to acquire image information of the set surface of the image to be segmented, wherein the set surface includes a main reference surface and an auxiliary reference surface; The second acquisition module is used to input the image information into the segmentation model and acquire the segmentation result of the image to be segmented output by the segmentation model; In this segmentation model, the structural complexity of the main feature extraction module is higher than that of the auxiliary feature extraction module. The main feature extraction module is used to extract the main image features from the main image information, and the auxiliary feature extraction module is used to extract the auxiliary image features from the auxiliary image information. The main image information is the image information corresponding to the main reference surface, and the auxiliary image information is the image information corresponding to the auxiliary reference surface. The segmentation model includes a main feature extraction module, an auxiliary feature extraction module, a feature rotation module, a feature fusion module, and a decoding module. Correspondingly, the second acquisition module is specifically used for: The main image information is input into the main feature extraction module to obtain the main image features; The auxiliary image information is input into the auxiliary feature extraction module to obtain auxiliary image features; The auxiliary image features are input into the feature rotation module for rotation transformation; The main image features and the rotated auxiliary image features are merged and then input into the feature fusion module to obtain the fused image features; The fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented; The decoding module includes multiple decoding networks, the number of which is equal to the number of main feature extraction networks in the main feature extraction module. The outputs of sub-modules in the target decoding network (excluding the first-layer decoding network) are upsampled to the inputs of the previous-layer decoding network. Correspondingly, the fused image features are input into the decoding module to obtain the segmentation result of the image to be segmented, including: The fused image features are input into each of the decoding networks to obtain a segmentation result, which is the sum of the outputs of the first layer decoding network and the set layer decoding network in each of the decoding networks. The decoding network in the last layer of the decoding module uses dilated convolution with different dilation rates to process the fused image features of the input. The decoding network in the decoding module, excluding the last layer, processes the input through a residual submodule or through a combination of residual and convolutional submodules. The submodule in the last layer of the decoding network is set as an intermediate convolutional submodule. The submodules in the target decoding network, excluding the last layer of the decoding network, are set as residual submodules.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the segmentation method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the segmentation method as described in any one of claims 1-5.

Images