Image segmentation method and device, electronic equipment, medium and product

Abstract

The invention discloses an image segmentation method and device, electronic equipment, a medium and a product. The method comprises the following steps: acquiring a plurality of images to be segmented; wherein the to-be-segmented image comprises a to-be-segmented object; processing the plurality of to-be-segmented images according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each to-be-segmented image; wherein the target image segmentation model comprises a feature extraction sub-model, a feature lower sampling sub-model, a feature upper sampling sub-model and a residual error convolution sub-model; and determining object segmentation data corresponding to the to-be-segmented object based on the object segmentation results corresponding to the plurality of to-be-segmented images. According to the technical scheme, the global features and the local detail features of the image are extracted based on the neural network model, and then the global features and the local detail features are combined to accurately segment the to-be-segmented object contained in the image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to an image segmentation method, device, electronic equipment, medium and product. Background Art

[0002] Pancreas segmentation helps quantify pancreatic volume and tissue characteristics for studying the pathological development of pancreatic fibrosis and chronic pancreatitis. Pancreas segmentation helps analyze different regions of the pancreas, helps study the function of pancreatic islet cells, and supports the study of endocrine diseases such as diabetes.

[0003] In the related art, pancreas segmentation of medical images is usually performed by medical staff identifying and segmenting the pancreas in the image based on their own experience.

[0004] However, the disadvantages of this segmentation method are: low segmentation accuracy, time-consuming and labor-intensive, and low segmentation efficiency. Summary of the invention

[0005] The present invention provides an image segmentation method, device, electronic device, medium and product to achieve the effect of extracting global features and local detail features of an image based on a neural network model, and then combining the global features and local detail features to accurately segment the objects to be segmented contained in the image.

[0006] According to one aspect of the present invention, there is provided an image segmentation method, the method comprising:

[0007] Acquire a plurality of images to be segmented; wherein the images to be segmented include the object to be segmented; the plurality of images to be segmented are two-dimensional images obtained by slicing the three-dimensional scanning data corresponding to the target part including the object to be segmented;

[0008] Processing a plurality of images to be segmented according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each of the images to be segmented; wherein the target image segmentation model includes a feature extraction submodel, a feature downsampling submodel, a feature upsampling submodel and a residual convolution submodel; the feature downsampling submodel includes a plurality of feature downsampling modules, each of which includes at least three first convolution units for extracting local features and at least one second convolution unit for extracting global features; the feature upsampling submodel includes a plurality of feature upsampling modules, each of which includes at least three third convolution units for extracting local features and at least one fourth convolution unit for extracting global features;

[0009] The object segmentation data corresponding to the object to be segmented is determined based on the object segmentation results corresponding to the plurality of images to be segmented.

[0010] According to another aspect of the present invention, there is provided an image segmentation device, the device comprising:

[0011] An image acquisition module, used to acquire a plurality of images to be segmented; wherein the images to be segmented include objects to be segmented; the plurality of images to be segmented are two-dimensional images obtained by slicing three-dimensional scanning data corresponding to target parts including the objects to be segmented;

[0012] An object segmentation module, used to process the plurality of images to be segmented according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each of the images to be segmented; wherein the target image segmentation model includes a feature extraction sub-model, a feature downsampling sub-model, a feature upsampling sub-model and a residual convolution sub-model; the feature downsampling sub-model includes a plurality of feature downsampling modules, each of which includes at least three first convolution units for extracting local features and one second convolution unit for extracting global features; the feature upsampling sub-model includes a plurality of feature upsampling modules, each of which includes at least three third convolution units for extracting local features and one fourth convolution unit for extracting global features;

[0013] The segmentation data determination module is used to determine the object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image segmentation method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image segmentation method described in any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, a computer program product is provided. The computer program product comprises a computer program. When the computer program is executed by a processor, the image segmentation method according to any embodiment of the present invention is implemented.

[0020] The technical solution of the embodiment of the present invention obtains multiple images to be segmented; wherein the images to be segmented contain objects to be segmented, and further, processes the multiple images to be segmented according to a pre-trained target image segmentation model to obtain object segmentation results corresponding to each image to be segmented, and further, determines object segmentation data corresponding to the objects to be segmented based on the object segmentation results corresponding to the multiple images to be segmented, thereby solving the problems of low object segmentation accuracy, laboriousness and low segmentation efficiency existing in the related art, and realizes the extraction of global features and local detail features of the image based on the neural network model, and then, combines the global features and local detail features to accurately segment the objects to be segmented contained in the image, thereby improving the segmentation accuracy, and by setting at least one convolutional network for extracting global features in the neural network model, it is achieved that while ensuring the model segmentation accuracy, the amount of calculation and model parameters are effectively controlled to improve the segmentation efficiency.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flowchart of an image segmentation method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flowchart of an image segmentation method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of a model structure of a feature downsampling module provided in Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of a model structure of a feature upsampling module provided in Embodiment 2 of the present invention;

[0027] Figure 5 is a schematic diagram of a model structure of a target image segmentation model provided in Embodiment 2 of the present invention;

[0028] Figure 6 is a flowchart of an image segmentation method provided according to Embodiment 3 of the present invention;

[0029] Figure 7 is a structural schematic diagram of an image segmentation device provided according to a fourth embodiment of the present invention;

[0030] Figure 8 It is a schematic diagram of the structure of an electronic device for implementing the image segmentation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any corresponding variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 1 is a flowchart of an image segmentation method provided by the first embodiment of the present invention. This embodiment is applicable to the case of segmenting objects contained in an image. The method can be executed by an image segmentation device. The image segmentation device can be implemented in the form of hardware and / or software. The image segmentation device can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:

[0035] S110, acquiring a plurality of images to be segmented, wherein the images to be segmented contain objects to be segmented.

[0036] Among them, the image to be segmented may be an image to be segmented. In this embodiment, the multiple images to be segmented are two-dimensional images obtained by slicing the three-dimensional scanning data corresponding to the target part containing the object to be segmented. The object to be segmented may be an object to be segmented. Alternatively, the object to be segmented may also be an object that needs to be separately marked out from the image. The object to be segmented may be any object, optionally, the pancreas, heart or other objects. The target part is the part containing the object to be segmented. Optionally, when the object to be segmented is the pancreas, the target part is the abdomen. The three-dimensional scanning data may be relevant data information obtained after scanning the target part based on a scanning device. Optionally, the three-dimensional scanning data may include but is not limited to CT images. In this embodiment, the three-dimensional scanning data corresponding to the target part containing the object to be segmented may be obtained. Further, the acquired three-dimensional scanning data may be sliced. Then, the image to be segmented may be determined based on the multiple slice images obtained after slicing.

[0037] Optionally, on the basis of the above-mentioned technical solutions, it also includes: obtaining three-dimensional scanning data corresponding to the target part of the object to be segmented; slicing the three-dimensional scanning data in a preset direction according to a preset thickness to obtain multiple images to be processed; and processing the multiple images to be processed according to a preset image preprocessing method to obtain multiple images to be segmented.

[0038] Among them, the preset direction can be any direction of the three directions included in the three-dimensional scanning data. Optionally, the preset direction is the Z-axis direction. The preset thickness can be any slice thickness, optionally, 0.5 mm, 1 mm or 2 mm, etc. The image to be processed can be a slice image obtained by slicing the three-dimensional scanning data. The image to be processed can be a two-dimensional image. The image preprocessing method includes at least image normalization and data enhancement. The data enhancement method can include multiple methods, optionally including at least one of rotation, translation, elastic transformation and noise addition.

[0039] It should be noted that the three-dimensional scanning data may be acquired in real time from a medical imaging device, may be acquired from an imaging database, or may be a scanning data transmission received from an external device, and this embodiment does not specifically limit this.

[0040] As an optional implementation of this embodiment, the target part can be scanned based on a scanning device to obtain three-dimensional scanning data corresponding to the target part, and the three-dimensional scanning data obtained by scanning can be acquired. Further, the acquired three-dimensional scanning data can be sliced ​​in the Z-axis direction according to a preset thickness, and the sliced ​​image obtained by slicing is used as an image to be processed to obtain multiple images to be processed. Further, for each image to be processed, the image to be processed can be normalized and data enhanced, and the image obtained after processing is used as an image to be segmented to obtain multiple images to be segmented.

[0041] S120 , processing a plurality of images to be segmented according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each image to be segmented.

[0042] The target image segmentation model may be a neural network model that uses an image as an input object to identify and segment objects contained in the image. The target image segmentation model may include at least one sub-model, and the at least one sub-model may include a feature extraction sub-model, a feature downsampling sub-model, a feature upsampling sub-model, and a residual convolution sub-model.

[0043] The feature extraction submodel may be a neural network model for extracting features from an image. The feature extraction submodel may be a neural network model of any model structure, and may optionally be a convolutional network.

[0044] Among them, the feature downsampling submodel can be a neural network model for downsampling features to obtain shallow feature information of the image. The feature downsampling submodel can include multiple feature downsampling modules, and each feature downsampling module is connected in sequence. That is to say, in the case of downsampling the image features based on the feature downsampling submodel, the image features can be downsampled in sequence based on multiple feature downsampling modules in the feature downsampling submodel. Each feature downsampling module has the same model structure. Each feature downsampling module can include at least three first convolution units for extracting local features and at least one second convolution unit for extracting global features. The first convolution unit can be a neural network model for extracting local features of image features. The first convolution unit can be any convolutional neural network capable of extracting local image features, and optionally, a convolutional neural network including a first convolution kernel. The second convolution unit can be a neural network model for extracting global features of image features. The second convolution unit can be any convolutional neural network capable of extracting global image features, and optionally, a convolutional neural network including a second convolution kernel. In this embodiment, the kernel size of the first convolution kernel may be smaller than the kernel size of the second convolution kernel, and the kernel size of the second convolution kernel satisfies a preset size threshold. The preset size threshold may be any kernel size value, optionally, 15×15, 16×16, or 17×17, etc. Exemplarily, the kernel size of the first convolution kernel may be 3×3; the kernel size of the second convolution kernel may be 17×17.

[0045] Among them, the feature upsampling submodel can be a neural network model for upsampling features to obtain deep feature information of the image. The feature upsampling submodel can include multiple feature upsampling modules, and each feature upsampling module is connected in sequence. That is to say, in the case of upsampling image features based on the feature upsampling submodel, the image features can be upsampled in sequence based on multiple feature upsampling modules in the feature upsampling submodel. Each feature upsampling module has the same model structure. Each feature upsampling module can include at least three third convolution units for extracting local features and at least one fourth convolution unit for extracting global features. The third convolution unit can be a neural network model for extracting local features of image features. The third convolution unit can be any convolutional neural network capable of extracting local image features, and optionally, it is a convolutional neural network containing a first convolution kernel. The fourth convolution unit can be a neural network model for extracting global features of image features. The fourth convolution unit can be any convolutional neural network capable of extracting global image features, and optionally, it is a convolutional neural network containing a second convolution kernel.

[0046] It should be noted that, in general, increasing the size of the convolution kernel will bring a lot of computational overhead. In this case, in order to effectively control the amount of computation and reduce the computational overhead, global feature extraction can also be performed based on a convolution network containing a large convolution kernel. In this embodiment, at least one second convolution unit and at least one fourth convolution unit for extracting global features can be set in the target image segmentation model, and the second convolution unit is set in the middle of at least three first convolution units, and the fourth convolution unit is set in the middle of at least three third convolution units. Exemplarily, it is assumed that the feature downsampling module includes three first convolution units, namely, the first convolution unit A, the first convolution unit B, the second convolution unit C, and a second convolution unit. Furthermore, the model structure of the feature downsampling module can be that the first convolution unit A is connected to the first convolution unit B, the first convolution unit B is connected to the second convolution unit, and the second convolution unit is connected to the first convolution unit C.

[0047] Among them, the residual convolution sub-model can be a neural network model for performing residual processing on image features. It can be understood that in a deep neural network model, as the number of network layers increases, the gradient will gradually decay or grow during the back propagation process, causing the gradient to disappear or explode. In order to alleviate the situation of gradient disappearance and gradient explosion and improve model performance, a residual network can be used to perform residual processing on features. Optionally, the residual convolution sub-model is a residual convolution network containing a convolution layer and a residual layer.

[0048] In practical applications, when segmenting the pancreas in an image, the pancreatic region in the CT image can be identified and segmented by automated pancreatic segmentation. However, the morphology of the pancreas in CT images is relatively complex and often presents an irregular shape. Compared with other objects in CT images, the outline of the pancreas is usually unclear and easily confused with surrounding tissues (such as the duodenum, stomach, spleen, etc.). The contrast between the pancreas and surrounding tissues in CT images is usually low, the boundaries are blurred, and it is difficult to accurately segment. In addition, the pancreas is usually surrounded by fat tissue, which further increases the difficulty of segmentation.

[0049] In view of the above situation, in this embodiment, a target image segmentation model that extracts global features and local features of an image and combines global features and local features to perform object segmentation on the image can be used to process the image to be segmented containing the object to be segmented (such as the pancreas). Then, an object segmentation result corresponding to the image to be segmented can be output. Among them, the object segmentation result can be the object region annotation result of the object to be segmented in the image to be segmented. That is, the image region corresponding to the object to be segmented is distinguished and displayed in the image to be segmented. Exemplarily, in the case where the object to be segmented contained in the image to be segmented is the pancreas, the object segmentation result can be an image in which the pancreatic region is annotated or distinguished in the image to be segmented.

[0050] In this embodiment, when multiple images to be segmented containing objects to be segmented are obtained, the multiple images to be segmented can be input into a pre-trained target image segmentation model. Further, the multiple images to be segmented can be processed based on the feature extraction sub-model, feature downsampling sub-model, feature upsampling sub-model and residual convolution sub-model in the target image segmentation model, and the object segmentation result corresponding to each image to be segmented can be output.

[0051] S130 . Determine object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented.

[0052] The object segmentation data may be three-dimensional scan data of the object to be segmented. Alternatively, the object segmentation data may also be understood as data that marks or displays the area corresponding to the image to be segmented in the three-dimensional scan data corresponding to the target part. Optionally, the object segmentation data may be an image that marks the object to be segmented in the CT image of the target part.

[0053] In this embodiment, when object segmentation results corresponding to multiple images to be segmented are obtained, the multiple images to be segmented are obtained by slicing the three-dimensional scanning data. In order to be able to observe the object to be segmented more three-dimensionally, three-dimensional scanning data corresponding to the object to be segmented can be constructed based on the object segmentation results corresponding to the multiple images to be segmented, and the constructed three-dimensional scanning data can be used as the object segmentation data corresponding to the object to be segmented.

[0054] Optionally, object segmentation data corresponding to the object to be segmented is determined based on the object segmentation results corresponding to multiple images to be segmented, including: superimposing the object segmentation results corresponding to the multiple images to be segmented in a preset direction according to a preset image arrangement order, and using the image obtained after superposition as the object segmentation data corresponding to the object to be segmented.

[0055] The preset image arrangement order may be the order of positions of slice images in a preset direction determined when segmenting the three-dimensional scanning data.

[0056] As an optional implementation of this embodiment, when object segmentation results corresponding to multiple images to be segmented are obtained, the results to be segmented corresponding to the multiple images to be segmented can be arranged in a preset direction according to a preset image arrangement order, and the multiple images to be segmented that have been arranged can be superimposed. Further, the image obtained after superposition can be used as object segmentation data corresponding to the object to be segmented.

[0057] The technical solution of the embodiment of the present invention obtains multiple images to be segmented; wherein the images to be segmented contain objects to be segmented, and further, processes the multiple images to be segmented according to a pre-trained target image segmentation model to obtain object segmentation results corresponding to each image to be segmented, and further, determines object segmentation data corresponding to the objects to be segmented based on the object segmentation results corresponding to the multiple images to be segmented, thereby solving the problems of low object segmentation accuracy, laboriousness and low segmentation efficiency existing in the related art, and realizes the extraction of global features and local detail features of the image based on the neural network model, and then, combines the global features and local detail features to accurately segment the objects to be segmented contained in the image, thereby improving the segmentation accuracy, and by setting at least one convolutional network for extracting global features in the neural network model, it is achieved that while ensuring the model segmentation accuracy, the amount of calculation and model parameters are effectively controlled to improve the segmentation efficiency.

[0058] Embodiment 2

[0059] Figure 2 This is a flow chart of an image segmentation method provided in the second embodiment of the present invention. On the basis of the above embodiment, multiple images to be segmented are input into the target image segmentation model, and then, based on the feature extraction sub-model, feature downsampling sub-model, feature upsampling sub-model and residual convolution sub-model in the target image segmentation model, the multiple images to be segmented are processed, and the object segmentation results corresponding to each image to be segmented are output. The specific implementation method can be referred to the technical solution of this embodiment. Among them, the technical terms that are the same or similar to the above embodiment are not repeated here.

[0060] like Figure 2 As shown, the method includes:

[0061] S210, obtaining a plurality of images to be segmented; wherein the images to be segmented include objects to be segmented.

[0062] S220, inputting a plurality of images to be segmented into a pre-trained target image segmentation model.

[0063] In this embodiment, when multiple images to be segmented containing objects to be segmented are obtained, the multiple images to be segmented can be input into a pre-trained target image segmentation model. Furthermore, the multiple images to be segmented can be processed based on the target image segmentation model.

[0064] S230 , extracting features of a plurality of images to be segmented based on a feature extraction sub-model to obtain a first feature to be processed.

[0065] Among them, the first feature to be processed can be a low-dimensional feature vector of multiple images to be segmented, and the information contained in the low-dimensional feature vector can be used for subsequent tasks. It can be understood that the original image data usually has a high dimension, and directly processing it may be very time-consuming and computationally intensive. Therefore, feature extraction can be performed on the image to extract feature information in the image while reducing the dimension, and then, subsequent tasks can be continued based on the extracted image features.

[0066] In this embodiment, when multiple images to be segmented are input into the target image segmentation model, first, feature extraction can be performed on the multiple images to be segmented based on the feature extraction submodel in the target image segmentation model. Then, the extracted multiple image features can be used as the first features to be processed.

[0067] S240. Downsample the first feature to be processed in sequence based on the multiple feature downsampling modules included in the feature downsampling sub-model to obtain a second feature to be processed.

[0068] Among them, the second feature to be processed can be shallow feature information obtained after downsampling the image features in sequence based on multiple feature downsampling modules.

[0069] In this embodiment, when the first feature to be processed is obtained, the first feature to be processed can be input into the feature downsampling sub-model. Further, the first feature to be processed can be processed in sequence based on the multiple feature downsampling modules included in the feature downsampling sub-model to obtain the second feature to be processed.

[0070] In this embodiment, each feature downsampling module has the same model structure, and therefore, each feature downsampling module downsamples its corresponding model input features in the same way. To facilitate explanation of the process of determining the second feature to be processed, the feature downsampling process can be explained by taking any one feature downsampling module downsampling its corresponding model input features as an example.

[0071] Optionally, based on multiple feature downsampling modules included in the feature downsampling sub-model, the first feature to be processed is downsampled in sequence to obtain the second feature to be processed, including: inputting the first feature to be processed into the feature downsampling sub-model; for each feature downsampling module, using the model output feature of the previous feature downsampling module of the current feature downsampling module as the model input feature of the current feature downsampling module, and inputting the model input feature into the current feature downsampling module; processing the model input feature based on at least two first convolution units to obtain the first feature; evenly dividing the first feature to obtain at least four features to be applied; inputting any feature to be applied as the target application feature The first convolution unit is used for feature extraction to obtain a first feature to be spliced, and the first feature to be spliced ​​is spliced ​​with other features to be applied except the target application feature in at least four features to be applied to obtain a second feature; the second feature is input into the first convolution unit for feature extraction to obtain a third feature, and residual processing is performed on the third feature and the model output feature of the first first convolution unit in at least two first convolution units to obtain a model output feature corresponding to the current feature downsampling module; when the current feature downsampling module is the last feature downsampling module in the feature downsampling sub-model, the model output feature of the current feature downsampling module is used as the second feature to be processed.

[0072] Among them, the current feature downsampling module is any one of the multiple feature downsampling modules. The model output feature is the feature output by the corresponding feature downsampling module. The model input feature is the feature input to the corresponding feature downsampling module. It should be noted that, when the current feature downsampling module is the first feature downsampling module in the feature downsampling sub-model, its corresponding model input feature can be the first feature to be processed input to the feature downsampling sub-model.

[0073] As an optional implementation of this embodiment, when the first feature to be processed is obtained, the first feature to be processed can be input into the feature downsampling submodel. Further, for each feature downsampling module in the feature downsampling submodel, when the model output feature of the previous feature upsampling module of the current feature downsampling module is obtained, the model output feature can be used as the model input feature of the current feature downsampling module, and the model input feature can be input into the current feature downsampling module. Further, the model input feature can be processed in sequence based on at least two first convolution units in the current feature downsampling module, and the first feature can be obtained. Further, the first feature can be averaged to divide the first feature into at least four features, and the feature obtained after the average division is used as the feature to be applied, and at least four features to be applied are obtained. Further, any one of the at least four features to be applied can be input into the second convolution unit as the target application feature, so as to perform global feature extraction on the target application feature based on the second convolution unit, and the extracted feature is used as the first feature to be spliced. Afterwards, the first feature to be spliced ​​can be spliced ​​with other features to be applied except the target application feature in at least four features to be applied, and the feature obtained after splicing is used as the second feature. Further, the second feature can be input into the first convolution unit to extract the second feature based on the first convolution unit, and the extracted feature is used as the third feature. Further, the model output feature of the first first convolution unit among the at least two first convolution units can be obtained, and the model output feature and the third feature can be subjected to residual processing, and the feature obtained after the residual processing can be used as the model output feature corresponding to the current feature downsampling module. Further, in the case where the current feature downsampling module is the last feature downsampling module in the feature downsampling sub-model, the model output feature corresponding to the current feature downsampling module can be used as the second feature to be processed.

[0074] It should be noted that the second feature is the feature obtained by splicing the first feature to be spliced ​​with other features to be applied except the target application feature in at least four features to be applied. Furthermore, the second feature combines the local features extracted based on the first convolution unit and the global features extracted based on the second convolution unit.

[0075] For example, Figure 3 Schematic diagram of a model structure of a feature downsampling module provided by an embodiment of the present invention. Figure 3As shown, the feature downsampling module includes three convolution networks with convolution kernels of 3×3 (first convolution unit) and one convolution network with convolution kernels of 17×17 (second convolution unit). When the model input feature corresponding to the feature downsampling module is obtained, the model input feature can be input into the feature downsampling module, and the model input feature can be processed in sequence based on two convolution networks with convolution kernels of 3×3, and the first feature can be obtained. Further, the first feature can be divided into four equal parts to divide the first feature into four features to obtain four features to be applied. Further, any one of the four features to be applied can be input into a convolution network with a convolution kernel of 17×17 to obtain the first feature to be spliced. The first feature to be spliced ​​is spliced ​​with the remaining three features to be applied in the four features to be applied, and the feature obtained after splicing is used as the second feature. Further, the second feature can be input into a convolution network with a convolution kernel of 3×3 to obtain the third feature. Furthermore, the third feature and the model output feature corresponding to the first convolution network with a convolution kernel of 3×3 can be added, and the feature obtained after the addition is used as the model output feature corresponding to the feature downsampling module.

[0076] S250. Based on the multiple feature upsampling modules included in the feature upsampling sub-model, upsample the second feature to be processed and the model output features of each feature downsampling module in the feature downsampling sub-model in turn, and obtain the third feature to be processed.

[0077] Among them, the third feature to be processed can be deep feature information obtained after upsampling the image features in sequence based on multiple feature upsampling modules.

[0078] In this embodiment, when the second feature to be processed is obtained, the second feature to be processed can be input into the feature upsampling sub-model. Furthermore, in order to fuse the deep feature information and the shallow feature information to obtain more refined image features, and perform object segmentation based on the image features to improve segmentation accuracy, the model output features of each feature downsampling module in the feature downsampling sub-model can be input into the corresponding feature upsampling module. Furthermore, the second feature to be processed and the model output features of each feature downsampling module can be upsampled in turn based on multiple feature upsampling modules to obtain the third feature to be processed.

[0079] Optionally, based on multiple feature upsampling modules included in the feature upsampling sub-model, the second feature to be processed and the model output features of each feature downsampling module in the feature downsampling sub-model are upsampled in sequence to obtain a third feature to be processed, including: taking the second feature to be processed as the model output feature of the last feature downsampling module, and taking the model output feature and the model output feature of the feature downsampling module before the last feature downsampling module as the model input feature of the first feature upsampling module; for other feature upsampling modules in the feature upsampling sub-model except the first feature upsampling module, concatenating the model output features of the feature upsampling module before the current feature upsampling module and the model output features of the feature downsampling module connected to the current feature upsampling module as the model input features of the current feature upsampling module, and processing the model input features based on the current feature upsampling module to obtain the model output features of the current feature upsampling module; until the current feature upsampling module is the last feature upsampling module, and taking the model output features of the last feature upsampling module as the third feature to be processed.

[0080] Among them, the feature downsampling module connected to the current feature upsampling module can be a feature downsampling module that has a connection relationship with the current feature upsampling module. It should be noted that in order to be able to splice two features, the feature downsampling module connected to the current feature upsampling module can be a feature downsampling module whose corresponding model output feature has the same feature size as the model input feature of the current upsampling module.

[0081] As an optional implementation of this embodiment, the second feature to be processed can be used as the model output feature of the last feature downsampling module, and the model output feature and the model output feature of the last feature downsampling module of the last feature downsampling module are spliced, and the feature obtained after splicing is used as the model input feature of the first feature upsampling module. Further, for other feature upsampling modules except the first feature upsampling module in the feature upsampling sub-model, the model output feature of the last feature upsampling module of the current feature upsampling module and the model output feature of the feature downsampling module connected to the current feature upsampling module are spliced, and the feature obtained after splicing is used as the model input feature of the current feature upsampling module. Then, the model input feature can be processed based on the current feature upsampling module to obtain the model output feature corresponding to the current feature upsampling module. Further, in the case where the current feature upsampling module is the last feature upsampling module in the feature upsampling sub-model, the model output feature of the last feature upsampling module can be used as the third feature to be processed.

[0082] In this embodiment, for each feature upsampling module in the feature upsampling sub-model, when the model input feature corresponding to the current feature upsampling module is determined, the model input feature can be input to the current feature upsampling module. Furthermore, the model input feature can be processed based on the third convolution unit and the fourth convolution unit in the current feature upsampling module to obtain the model output feature corresponding to the current feature upsampling module.

[0083] Optionally, the model input features are processed based on the current feature upsampling module to obtain model output features corresponding to the current feature upsampling module, including: inputting the model input features into the current feature upsampling module; processing the model input features in sequence based on at least two third convolution units to obtain a fourth feature; equally dividing the fourth feature to obtain at least four features to be used; inputting any feature to be used as a target use feature into the fourth convolution unit for feature extraction to obtain a second feature to be spliced, and splicing the second feature to be spliced ​​with other features to be used in at least four features to be used except the target use feature to obtain a fifth feature; inputting the fifth feature into the third convolution unit for feature extraction to obtain a sixth feature, and performing residual processing on the sixth feature and the model output feature of the first third convolution unit of at least two third convolution units to obtain a model output feature corresponding to the current feature upsampling module.

[0084] As an optional implementation of the first embodiment of the present invention, for each feature upsampling module in the feature upsampling submodel, when the model input feature corresponding to the current feature upsampling module is determined, the model input feature can be input into the current feature upsampling module. Further, the model input feature can be processed in sequence based on at least two third convolution units in the current feature upsampling module to obtain a fourth feature. Further, the fourth feature can be evenly divided to divide the fourth feature into at least four features, and the feature obtained after the even division is used as the feature to be used to obtain at least four features to be used. Further, any one of the at least four features to be used can be input as the target use feature to the fourth convolution unit, so as to perform global feature extraction on the target use feature based on the fourth convolution unit, and the extracted feature is used as the second feature to be spliced. Afterwards, the second feature to be spliced ​​can be spliced ​​with other features to be used in the at least four features to be used except the target use feature, and the feature obtained after splicing is used as the fifth feature. Furthermore, the fifth feature can be input into the third convolution unit to extract the fifth feature based on the third convolution unit, and the extracted feature can be used as the sixth feature. Furthermore, the model output feature of the first third convolution unit among the at least two third convolution units can be obtained, and the model output feature and the sixth feature can be subjected to residual processing, and the feature obtained after the residual processing can be used as the model output feature corresponding to the current feature upsampling module.

[0085] It should be noted that the fifth feature is the feature obtained by splicing the second feature to be spliced ​​with other features to be used except the target feature in at least four features to be used. Furthermore, the fifth feature combines the local features extracted based on the third convolution unit and the global features extracted based on the fourth convolution unit.

[0086] For example, Figure 4 Schematic diagram of a model structure of a feature upsampling module provided in an embodiment of the present invention. Figure 4As shown, the feature upsampling module includes three convolution networks with convolution kernels of 3×3 (third convolution unit) and one convolution network with a convolution kernel of 17×17 (fourth convolution unit). When the model input feature corresponding to the feature upsampling module is obtained, the model input feature can be input into the feature upsampling module, and the model input feature can be processed in sequence based on two convolution networks with convolution kernels of 3×3, and the fourth feature can be obtained. Further, the fourth feature can be divided into four equal parts to divide the fourth feature into four features to obtain four features to be used. Further, any one of the four features to be used can be input into a convolution network with a convolution kernel of 17×17 for feature extraction to obtain a second feature to be spliced. The second feature to be spliced ​​is spliced ​​with the remaining three features to be used in the four features to be used, and the feature obtained after splicing is used as the fifth feature. Further, the fifth feature can be input into a convolution network with a convolution kernel of 3×3 to obtain a sixth feature. Furthermore, residual processing can be performed on the sixth feature and the model output feature corresponding to the first convolution network with a convolution kernel of 3×3, and the feature obtained after the residual processing is used as the model output feature corresponding to the feature upsampling module.

[0087] S260. Perform residual processing on the third feature to be processed and the first feature to be processed based on the residual convolution sub-model to obtain an object segmentation result corresponding to each image to be segmented.

[0088] In this embodiment, when the third feature to be processed output by the feature upsampling sub-model is obtained, the third feature to be processed and the first feature to be processed output by the feature extraction sub-model can be spliced, and the spliced ​​feature can be input into the residual convolution sub-model. Then, the spliced ​​feature can be processed based on the residual convolution sub-model to obtain an object segmentation result corresponding to each image to be segmented.

[0089] For example, Figure 5 Schematic diagram of a model structure of a target image segmentation model provided by an embodiment of the present invention. Figure 5As shown, the target image segmentation model includes a first residual convolution network (feature extraction submodel), N residual convolution downsampling networks (feature downsampling modules), N-1 residual convolution upsampling networks (feature upsampling modules) and a second residual convolution network (residual convolution submodel). In the case of obtaining multiple images to be segmented, the multiple images to be segmented are input into the target image segmentation model; the features of the multiple images to be segmented are extracted based on the first residual convolution network to obtain the first feature to be processed; the first feature to be processed is sequentially downsampled based on the N residual convolution downsampling networks to obtain the second feature to be processed; the second feature to be processed and the model output features of each residual convolution downsampling network are sequentially upsampled based on the N-1 residual convolution upsampling networks to obtain the third feature to be processed; the third feature to be processed and the first feature to be processed are residually processed based on the second residual convolution network to obtain the object segmentation result corresponding to each image to be segmented.

[0090] S270 . Determine object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented.

[0091] The technical solution of the embodiment of the present invention is as follows: by inputting multiple images to be segmented into a pre-trained target image segmentation model; further, extracting features of the multiple images to be segmented based on a feature extraction submodel to obtain a first feature to be processed; further, downsampling the first feature to be processed in turn based on multiple feature downsampling modules included in the feature downsampling submodel to obtain a second feature to be processed; further, upsampling the second feature to be processed and the model output features of each feature downsampling module in the feature downsampling submodel in turn based on multiple feature upsampling modules included in the feature upsampling submodel to obtain a third feature to be processed; further, residual processing is performed on the third feature to be processed and the first feature to be processed based on a residual convolution submodel to obtain an object segmentation result corresponding to each image to be segmented, thereby achieving the extraction of global features and local detail features of the image based on a neural network model, and then, combining the global features and the local detail features to accurately segment the object to be segmented contained in the image, thereby improving the segmentation accuracy, and by setting at least one convolution network for extracting global features in the neural network model, while ensuring the model segmentation accuracy, effectively controlling the amount of calculation and model parameters to improve the segmentation efficiency.

[0092] Embodiment 3

[0093] Figure 6is a flow chart of an image segmentation method provided by the third embodiment of the present invention. On the basis of the above embodiment, a target image segmentation model can also be pre-trained to perform object segmentation processing on multiple images to be segmented containing objects to be segmented based on the target image segmentation model, thereby determining the object segmentation result corresponding to the images to be segmented. Among them, the technical terms that are the same as or corresponding to the above embodiment are not repeated here.

[0094] like Figure 6 As shown, the method includes:

[0095] S310: Obtain training sample data for training a target image segmentation model.

[0096] The training sample data includes a plurality of sample segmentation images and a theoretical object labeling result corresponding to each sample segmentation image; the sample segmentation image contains an object to be segmented.

[0097] The plurality of sample segmentation images are two-dimensional images obtained by slicing three-dimensional scanning data corresponding to the target part of the object to be segmented. The sample segmentation image may be an image reconstructed by an image reconstruction model, or an image pre-stored in a storage space. The theoretical object annotation result may be a true annotation result of the object area obtained after object segmentation of the sample segmentation image.

[0098] In this embodiment, before training the model to be trained, multiple training sample data may be obtained first to train the model based on the training sample data. In order to improve the performance of the model, as much and rich training sample data as possible may be obtained, and multiple sample segmented images containing the objects to be segmented may be obtained. Further, the objects to be segmented in each sample segmented image are annotated to obtain theoretical object annotation results corresponding to each image to be segmented. Thus, rich training sample data is constructed based on the above method.

[0099] S320: Training the training sample data based on the model to be trained to obtain a loss value.

[0100] Among them, the model to be trained can be a neural network model whose model parameters are default values ​​or initial values. In this embodiment, the model to be trained includes a feature extraction submodel, a feature downsampling submodel, a feature upsampling submodel and a residual convolution submodel. The feature downsampling submodel includes a plurality of feature downsampling modules, each of which includes at least three first convolution units for extracting local features and at least one second convolution unit for extracting global features; the feature upsampling submodel includes a plurality of feature upsampling modules, each of which includes at least a plurality of third convolution units for extracting local features and at least one fourth convolution unit for extracting global features. The loss value can be a numerical value that characterizes the degree of difference between the predicted output and the actual output.

[0101] In this embodiment, when the training sample data for training the target image segmentation model is obtained, multiple sample segmentation models in the training sample data can be input into the model to be trained, so as to process multiple sample segmentation images based on the model to be trained, and output the model predicted segmentation results corresponding to the sample segmentation images. Furthermore, the model predicted segmentation results can be compared with the theoretical object labeling results corresponding to each sample segmentation image in the training sample data to determine the loss value.

[0102] S330, based on the loss value, the model parameters in the model to be trained are modified, and the convergence of the loss function in the model to be trained is taken as the training goal to obtain at least one model to be verified.

[0103] The loss function may be a function determined based on the loss value and used to characterize the degree of difference between the predicted output and the actual output. The model to be verified may be a model obtained by correcting the model parameters in the training model according to the loss value. The number of models to be corrected may be consistent with the number of training iterations (or the number of model parameter corrections).

[0104] In this embodiment, when the loss value is obtained, the model parameters in the model to be trained can be corrected based on the loss value, the corrected model can be used as the first model to be processed, and the model to be processed can be backed up and saved. Further, the process of processing the training sample data and determining the loss value can be repeated. Furthermore, the model parameters in the model to be processed can be corrected based on the obtained loss value, the corrected model can be used as the second model to be processed, and the model to be processed can be backed up and saved. Further, when the convergence of the loss function is detected, such as the training error of the loss function is less than the preset error, or the error change trend tends to be stable, the iterative training can be stopped at this time, and the model obtained at this time can be used as the model to be verified. Furthermore, the saved backup model can be retrieved, and the retrieved backup model and the model to be verified obtained at this time can be used as the model to be verified, so as to obtain at least one model to be verified.

[0105] S340, verifying at least one model to be verified based on the test sample, and using the model to be verified whose verification result meets the preset conditions as the target image segmentation model.

[0106] The test sample may be a part of the training sample data, or may be training sample data re-determined based on the training sample data determination process described in step S310. The preset condition may include at least one of the accuracy reaching a preset accuracy threshold, the recall reaching a preset recall threshold, the mean square error reaching a first error threshold, and the root mean square error reaching a second error threshold.

[0107] In this embodiment, in order to evaluate the model performance of at least one model to be verified, a test sample can be obtained, and for at least one model to be verified, the model to be verified can be tested and verified based on the test sample, and a verification result corresponding to the model to be verified can be obtained. Further, it can be determined whether the verification result of at least one model to be verified meets the preset conditions. Then, the model to be verified whose verification result meets the preset conditions can be used as the target image segmentation model.

[0108] S350, obtaining a plurality of images to be segmented; wherein the images to be segmented include objects to be segmented.

[0109] S360: Process multiple images to be segmented according to the pre-trained target image segmentation model to obtain an object segmentation result corresponding to each image to be segmented.

[0110] S370 . Determine object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented.

[0111] The technical solution of the embodiment of the present invention obtains training sample data involved in training the target image segmentation model; further, the training sample data is trained based on the model to be trained to obtain a loss value; further, the model parameters in the model to be trained are corrected based on the loss value, and the convergence of the loss function in the model to be trained is used as the training goal to obtain at least one model to be verified; further, at least one model to be verified is verified based on the test sample, and the model to be verified whose verification result meets the preset conditions is used as the target image segmentation model, thereby achieving the effect of training a target image segmentation model capable of accurately performing object segmentation based on the constructed training sample data, effectively improving the accuracy and robustness of the model, and effectively improving the model performance.

[0112] Embodiment 4

[0113] Figure 7 Schematic diagram of the structure of an image segmentation device provided by Embodiment 4 of the present invention. Figure 7 As shown, the device includes: an image acquisition module 410, an object segmentation module 420 and a segmentation data determination module 430.

[0114] Among them, the image acquisition module 410 is used to acquire multiple images to be segmented; wherein the images to be segmented include objects to be segmented; the multiple images to be segmented are two-dimensional images obtained by slicing the three-dimensional scanning data corresponding to the target part including the object to be segmented; the object segmentation module 420 is used to process the multiple images to be segmented according to the target image segmentation model obtained in advance, and obtain the object segmentation result corresponding to each image to be segmented; wherein the target image segmentation model includes a feature extraction submodel, a feature downsampling submodel, a feature upsampling submodel and a residual convolution submodel; the feature downsampling submodel includes multiple feature downsampling modules, each of which includes at least three first convolution units for extracting local features and a second convolution unit for extracting global features; the feature upsampling submodel includes multiple feature upsampling modules, each of which includes multiple third convolution units for extracting local features and a fourth convolution unit for extracting global features; the segmentation data determination module 430 is used to determine the object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the multiple images to be segmented.

[0115] The technical solution of the embodiment of the present invention obtains multiple images to be segmented; wherein the images to be segmented contain objects to be segmented, and further, processes the multiple images to be segmented according to a pre-trained target image segmentation model to obtain object segmentation results corresponding to each image to be segmented, and further, determines object segmentation data corresponding to the objects to be segmented based on the object segmentation results corresponding to the multiple images to be segmented, thereby solving the problems of low object segmentation accuracy, laboriousness and low segmentation efficiency existing in the related art, and realizes the extraction of global features and local detail features of the image based on the neural network model, and then, combines the global features and local detail features to accurately segment the objects to be segmented contained in the image, thereby improving the segmentation accuracy, and by setting at least one convolutional network for extracting global features in the neural network model, it is achieved that while ensuring the model segmentation accuracy, the amount of calculation and model parameters are effectively controlled to improve the segmentation efficiency.

[0116] Optionally, the device further includes: a scanning data acquisition module, a data slicing module, and a module for determining images to be segmented. The scanning data acquisition module is used to acquire three-dimensional scanning data corresponding to the target part of the object to be segmented; the data slicing module is used to slice the three-dimensional scanning data in a preset direction according to a preset thickness to obtain multiple images to be processed; the module for determining images to be segmented is used to process the multiple images to be processed according to a preset image preprocessing method to obtain multiple images to be segmented; the image preprocessing method at least includes image normalization and data enhancement.

[0117] Optionally, the object segmentation module includes: an image input unit, a feature extraction unit, a feature downsampling unit, a feature upsampling unit and a segmentation result output unit.

[0118] An image input unit, used for inputting a plurality of images to be segmented into a pre-trained target image segmentation model;

[0119] A feature extraction unit, used for extracting features of the plurality of images to be segmented based on a feature extraction sub-model to obtain a first feature to be processed;

[0120] A feature downsampling unit, configured to sequentially downsample the first feature to be processed based on a plurality of feature downsampling modules included in the feature downsampling sub-model, and obtain a second feature to be processed;

[0121] A feature upsampling unit, configured to sequentially upsample the second feature to be processed and the model output feature of each feature downsampling module in the feature downsampling submodel based on a plurality of feature upsampling modules included in the feature upsampling submodel, and obtain a third feature to be processed;

[0122] A segmentation result output unit is used to perform residual processing on the third feature to be processed and the first feature to be processed based on the residual convolution sub-model to obtain an object segmentation result corresponding to each image to be segmented.

[0123] Optionally, the feature downsampling unit includes: a feature input subunit, a model input feature determination subunit, a first feature determination subunit, a feature averaging subunit, a feature splicing subunit, a model output feature determination subunit and a second feature to be processed determination subunit.

[0124] A feature input subunit, used for inputting the first feature to be processed into the feature downsampling submodel;

[0125] A model input feature determination subunit is used to, for each of the feature downsampling modules, use the model output feature of the previous feature downsampling module of the current feature downsampling module as the model input feature of the current feature downsampling module, and input the model input feature into the current feature downsampling module;

[0126] A first feature determination subunit, configured to process the model input feature based on at least two first convolution units in the current feature downsampling module to obtain a first feature;

[0127] A feature averaging subunit, used for averaging the first feature to obtain at least four features to be applied;

[0128] A feature splicing subunit, used for inputting any of the features to be applied as a target application feature into the second convolution unit for feature extraction to obtain a first feature to be spliced, and splicing the first feature to be spliced ​​with other features to be applied among at least four features to be applied except the target application feature to obtain a second feature;

[0129] A model output feature determination subunit, used for inputting the second feature into the first convolution unit for feature extraction to obtain a third feature, and performing residual processing on the third feature and the model output feature of the first of the at least two first convolution units to obtain a model output feature corresponding to the current feature downsampling module;

[0130] The second feature to be processed determining subunit is used to use the model output feature of the current feature downsampling module as the second feature to be processed when the current feature downsampling module is the last feature downsampling module in the feature downsampling submodel.

[0131] Optionally, the feature upsampling unit includes: a model input feature determination subunit, a feature processing subunit and a third to-be-processed feature determination subunit.

[0132] A model input feature determination subunit is used to use the second feature to be processed as the model output feature of the last feature downsampling module, and use the model output feature and the model output feature of the previous feature downsampling module of the last feature downsampling module as the model input feature of the first feature upsampling module;

[0133] A feature processing subunit is used for splicing the model output features of the feature upsampling module before the current feature upsampling module and the model output features of the feature downsampling module connected to the current feature upsampling module for the feature upsampling modules other than the first feature upsampling module in the feature upsampling submodel, so as to serve as the model input features of the current feature upsampling module, and processing the model input features based on the current feature upsampling module to obtain the model output features of the current feature upsampling module;

[0134] The third feature to be processed determining subunit is used to determine whether the current feature upsampling module is the last feature upsampling module and use the model output feature of the last feature upsampling module as the third feature to be processed.

[0135] Optionally, a feature processing subunit is specifically used to input the model input feature into the current feature upsampling module; based on at least two third convolution units in the current feature upsampling module, the model input feature is processed in sequence to obtain a fourth feature; the fourth feature is evenly divided to obtain at least four features to be used; any of the features to be used is input as a target use feature into the fourth convolution unit for feature extraction to obtain a second feature to be spliced, and the second feature to be spliced ​​is spliced ​​with other features to be used in at least four features to be used except the target use feature to obtain a fifth feature; the fifth feature is input into the third convolution unit for feature extraction to obtain a sixth feature, and residual processing is performed on the sixth feature and the model output feature of the first of the at least two third convolution units to obtain a model output feature corresponding to the current feature upsampling module.

[0136] Optionally, the segmentation data determination module is specifically used to superimpose object segmentation results corresponding to multiple images to be segmented in a preset direction according to a preset image arrangement order, and use the image obtained after superposition as object segmentation data corresponding to the object to be segmented.

[0137] Optionally, the device also includes: a sample data acquisition module, a loss value determination module, a model parameter correction module and a model verification module.

[0138] A sample data acquisition module, used to acquire training sample data for training the target image segmentation model; wherein the training sample data includes a plurality of sample segmentation images and a theoretical object labeling result corresponding to each of the sample segmentation images; the sample segmentation images include objects to be segmented;

[0139] A loss value determination module, used for training the training sample data based on the model to be trained to obtain a loss value;

[0140] A model parameter correction module, used to correct the model parameters in the model to be trained based on the loss value, take the convergence of the loss function in the model to be trained as the training goal, and obtain at least one model to be verified;

[0141] The model verification module is used to verify at least one of the models to be verified based on the test samples, and use the model to be verified whose verification result meets the preset conditions as the target image segmentation model.

[0142] Optionally, the first convolution unit and the third convolution unit include a first convolution kernel; the second convolution unit and the fourth convolution unit include a second convolution kernel; the core size of the first convolution kernel is smaller than the core size of the second convolution kernel, and the core size of the second convolution kernel meets a preset size threshold.

[0143] The image segmentation device provided in the embodiment of the present invention can execute the image segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0144] Embodiment 5

[0145] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0146] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0147] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image segmentation method.

[0149] In some embodiments, the image segmentation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image segmentation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image segmentation method in any other appropriate manner (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Acquire a plurality of images to be segmented; wherein the images to be segmented include the object to be segmented; the plurality of images to be segmented are two-dimensional images obtained by slicing the three-dimensional scanning data corresponding to the target part including the object to be segmented; Processing a plurality of images to be segmented according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each of the images to be segmented; wherein the target image segmentation model includes a feature extraction submodel, a feature downsampling submodel, a feature upsampling submodel and a residual convolution submodel; the feature downsampling submodel includes a plurality of feature downsampling modules, each of which includes at least three first convolution units for extracting local features and at least one second convolution unit for extracting global features; the feature upsampling submodel includes a plurality of feature upsampling modules, each of which includes at least three third convolution units for extracting local features and at least one fourth convolution unit for extracting global features; The object segmentation data corresponding to the object to be segmented is determined based on the object segmentation results corresponding to the plurality of images to be segmented.

2. The image segmentation method according to claim 1, characterized in that: Also includes: Acquire three-dimensional scanning data corresponding to the target part of the object to be segmented; Slicing the three-dimensional scanning data in a preset direction according to a preset thickness to obtain a plurality of images to be processed; The plurality of images to be processed are processed respectively according to a preset image preprocessing method to obtain a plurality of images to be segmented; wherein the image preprocessing method at least includes image normalization and data enhancement.

3. The image segmentation method according to claim 1, characterized in that: The step of processing the plurality of images to be segmented according to the pre-trained target image segmentation model to obtain an object segmentation result corresponding to each of the images to be segmented includes: Inputting the plurality of images to be segmented into a pre-trained target image segmentation model; Extracting features of the plurality of images to be segmented based on the feature extraction sub-model to obtain first features to be processed; Downsampling the first to-be-processed feature in sequence based on a plurality of feature downsampling modules included in the feature downsampling sub-model, and obtaining a second to-be-processed feature; Based on the multiple feature upsampling modules included in the feature upsampling sub-model, the second feature to be processed and the model output features of each feature downsampling module in the feature downsampling sub-model are sequentially upsampled to obtain a third feature to be processed; Based on the residual convolution sub-model, residual processing is performed on the third feature to be processed and the first feature to be processed to obtain an object segmentation result corresponding to each of the images to be segmented.

4. The image segmentation method according to claim 3, characterized in that: The plurality of feature downsampling modules included in the feature downsampling sub-model sequentially downsample the first feature to be processed to obtain a second feature to be processed, including: Inputting the first feature to be processed into the feature downsampling sub-model; For each of the feature downsampling modules, the model output features of the feature downsampling module before the current feature downsampling module are used as the model input features of the current feature downsampling module, and the model input features are input into the current feature downsampling module; Processing the model input feature based on at least two first convolution units in the current feature downsampling module to obtain a first feature; The first feature is evenly divided to obtain at least four features to be applied; Input any of the features to be applied as a target application feature into a second convolution unit for feature extraction to obtain a first feature to be spliced, and splice the first feature to be spliced ​​with other features to be applied except the target application feature in at least four of the features to be applied to obtain a second feature; Inputting the second feature into the first convolution unit for feature extraction to obtain a third feature, and performing residual processing on the third feature and the model output feature of the first of the at least two first convolution units to obtain a model output feature corresponding to the current feature downsampling module; In the case where the current feature downsampling module is the last feature downsampling module in the feature downsampling sub-model, the model output feature of the current feature downsampling module is used as the second feature to be processed.

5. The image segmentation method according to claim 3, characterized in that: The plurality of feature upsampling modules included in the feature upsampling sub-model sequentially upsample the second feature to be processed and the model output features of each of the feature downsampling modules in the feature downsampling sub-model, and obtain a third feature to be processed, including: Using the second feature to be processed as the model output feature of the last feature downsampling module, and using the model output feature and the model output feature of the previous feature downsampling module before the last feature downsampling module as the model input feature of the first feature upsampling module; For the feature upsampling modules other than the first feature upsampling module in the feature upsampling sub-model, the model output features of the feature upsampling module before the current feature upsampling module and the model output features of the feature downsampling module connected to the current feature upsampling module are concatenated to serve as the model input features of the current feature upsampling module, and the model input features are processed based on the current feature upsampling module to obtain the model output features of the current feature upsampling module; Until the current feature upsampling module is the last feature upsampling module, and the model output feature of the last feature upsampling module is used as the third feature to be processed.

6. The image segmentation method according to claim 5, characterized in that: The processing of the model input feature based on the current feature upsampling module to obtain the model output feature of the current feature upsampling module includes: Inputting the model input features into the current feature upsampling module; Processing the model input features in sequence based on at least two third convolution units in the current feature upsampling module to obtain a fourth feature; The fourth feature is evenly divided to obtain at least four features to be used; Input any of the features to be used as a target use feature into a fourth convolution unit for feature extraction to obtain a second feature to be spliced, and splice the second feature to be spliced ​​with other features to be used except the target use feature in at least four of the features to be used to obtain a fifth feature; The fifth feature is input into the third convolution unit for feature extraction to obtain the sixth feature, and residual processing is performed on the sixth feature and the model output feature of the first third convolution unit among the at least two third convolution units to obtain the model output feature corresponding to the current feature upsampling module.

7. The image segmentation method according to claim 1, characterized in that: The determining the object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented includes: The object segmentation results corresponding to the plurality of images to be segmented are superimposed in a preset direction according to a preset image arrangement order, and the image obtained after the superposition is used as the object segmentation data corresponding to the object to be segmented.

8. The image segmentation method according to claim 1, characterized in that: Also includes: Acquire training sample data for training the target image segmentation model; wherein the training sample data includes a plurality of sample segmentation images corresponding to the object to be segmented and an object annotation result corresponding to each of the sample segmentation images; Training the training sample data based on the model to be trained to obtain a loss value; Modifying the model parameters in the model to be trained based on the loss value, taking the convergence of the loss function in the model to be trained as a training goal, and obtaining at least one model to be verified; At least one of the models to be verified is verified based on the test sample, and the model to be verified whose verification result meets the preset conditions is used as the target image segmentation model.

9. The image segmentation method according to any one of claims 1 to 8, characterized in that: The first convolution unit and the third convolution unit include a first convolution kernel; the second convolution unit and the fourth convolution unit include a second convolution kernel; the core size of the first convolution kernel is smaller than the core size of the second convolution kernel, and the core size of the second convolution kernel meets a preset size threshold.

10. An image segmentation device, characterized in that: include: An image acquisition module, used to acquire a plurality of images to be segmented; wherein the images to be segmented include objects to be segmented; the plurality of images to be segmented are two-dimensional images obtained by slicing three-dimensional scanning data corresponding to target parts including the objects to be segmented; An object segmentation module, used to process the plurality of images to be segmented according to a pre-trained target image segmentation model to obtain an object segmentation result corresponding to each of the images to be segmented; wherein the target image segmentation model includes a feature extraction sub-model, a feature downsampling sub-model, a feature upsampling sub-model and a residual convolution sub-model; the feature downsampling sub-model includes a plurality of feature downsampling modules, each of which includes at least three first convolution units for extracting local features and one second convolution unit for extracting global features; the feature upsampling sub-model includes a plurality of feature upsampling modules, each of which includes at least three third convolution units for extracting local features and one fourth convolution unit for extracting global features; The segmentation data determination module is used to determine the object segmentation data corresponding to the object to be segmented based on the object segmentation results corresponding to the plurality of images to be segmented.

Images