Training method of automatic image segmentation model, automatic image segmentation method and system

Abstract

The present disclosure relates to the fields of artificial intelligence and image processing, and provides a training method for an automatic image segmentation model, an automatic image segmentation method and a system, the training method comprising: obtaining a training sample set and sample annotation information; inputting the training sample set into a first branch network and a second branch network of the automatic image segmentation model, respectively, to obtain a first feature and a second feature, respectively; using a shallow fusion module to fuse the first feature and the second feature to obtain a fusion feature; using a feature aggregation module to obtain a predicted segmentation result of a lesion in each training sample based on the fusion feature; and training the automatic image segmentation model by comparing the predicted segmentation result and the sample annotation information. The present disclosure can solve the problem that the prediction result of the automatic segmentation of the model is not accurate enough, and can take into account the global distribution and local details in the image to identify the lesions in the image, avoid ignoring details at different scales, and improve the accuracy of the segmentation result.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and image processing, and in particular to a training method for an automatic image segmentation model, an automatic image segmentation method and a system. Background Art

[0002] With the advancement of artificial intelligence and image processing technology, artificial intelligence-based imaging image analysis technology has also developed rapidly. This technology can be used to identify and segment lesions in medical images such as computed tomography (CT) images, thereby helping medical personnel to observe lesions more clearly.

[0003] However, since the morphology of the lesions themselves is often diverse, the irregular morphology of the lesions brings difficulties to the automatic segmentation task of artificial intelligence, making it difficult for the model to accurately segment the lesions. Taking the segmentation of lung nodule lesions as an example, in related technologies, the morphological characteristics of the nodules themselves and the surrounding habitat are often ignored. Due to the large changes in the size of the nodules, the blurred and irregular contours of the nodules, the key feature information of the lesions cannot be effectively extracted and fully integrated, making the prediction results of the automatic segmentation of the network model less than ideal. Summary of the invention

[0004] The present disclosure provides a training method for an automatic image segmentation model, an automatic image segmentation method and a system, so as to at least solve the problem in the related art that the automatic segmentation prediction results of the model are not accurate enough due to the large changes in the morphological scale of the lesions and the irregular edges. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a training method for an automatic image segmentation model is provided, the training method comprising: obtaining a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample comprises image data of physiological tissue containing a lesion, and each sample annotation information represents the lesion position in the corresponding training sample; inputting the training sample set into a first branch network and a second branch network of the automatic image segmentation model respectively, and obtaining a first feature extracted from the first branch network and a second feature extracted from the second branch network respectively, wherein the first branch network can extract local features in the training sample, and the second branch network can extract global features in the training sample; utilizing a shallow fusion module of the automatic image segmentation model to fuse the first feature and the second feature to obtain a fused feature; utilizing a feature aggregation module of the automatic image segmentation model to obtain a predicted segmentation result for the lesion in each training sample based on the fused feature; and training the automatic image segmentation model by comparing the predicted segmentation result with the sample annotation information.

[0006] Optionally, the first feature and the second feature are obtained in the following manner: feature extraction at multiple feature scales is performed on each training sample in the training sample set in the first branch network and the second branch network, respectively, to obtain the first feature and the second feature, wherein the number of channels of the extracted features in the feature extraction at different feature scales is different, the first feature includes multiple first-scale features corresponding one-to-one to the multiple feature scales, the second feature includes multiple second-scale features corresponding one-to-one to the multiple feature scales, the first scale feature corresponds one-to-one to the second scale feature, and the fused feature includes multiple scale fusion features corresponding one-to-one to the multiple feature scales.

[0007] Optionally, the shallow fusion module includes a branch feature extraction module, wherein, for any feature scale, the shallow fusion module obtains the scale fusion feature in the following manner: respectively inputting a first scale feature and a second scale feature corresponding to the current feature scale into the branch feature extraction module to respectively obtain a first extracted feature corresponding to the first scale feature and a second extracted feature corresponding to the second scale feature, wherein the branch feature extraction module is used to perform channel compression of multiple compression scales on the input scale feature, and merge the multiple compressed features obtained by compression, wherein the number of compressed channels in the channel compression at different compression scales is different; by introducing a multi-attention mechanism, the weights of different channel features in the first extracted feature and the second extracted feature are respectively adjusted to obtain a first branch feature corresponding to the first extracted feature and a second branch feature corresponding to the second extracted feature; the first branch feature and the second branch feature are aggregated to obtain the scale fusion feature of the current feature scale.

[0008] Optionally, the branch feature extraction module obtains the first extraction feature or the second extraction feature in the following manner: according to the channel of the current scale feature, the current scale feature is split into multiple split features, wherein the current scale feature is the first scale feature or the second scale feature; performing depth convolution and point-by-point convolution on the multiple split features respectively to obtain multiple convolution features corresponding to the multiple split features one by one; performing channel compression on the multiple convolution features according to each compression scale in the multiple compression scales respectively to obtain the multiple compressed features corresponding to the multiple compression scales one by one; merging the multiple compressed features to obtain the extraction feature corresponding to the current scale feature.

[0009] Optionally, the automatic image segmentation model includes an upsampling superposition module, and the training method further includes: using the upsampling superposition module, according to the minimum number of channels of each scale fusion feature in the fusion feature, upsampling the scale fusion feature with more channels than the minimum number of channels to obtain a sampling fusion feature; using the upsampling superposition module, superimposing the scale fusion feature with the minimum number of channels to the sampling fusion feature to obtain the superimposed sampling fusion feature; using the feature aggregation module, based on the superimposed sampling fusion feature, obtaining the predicted segmentation result, wherein, after each upsampling, the scale fusion feature with the same number of channels as the scale fusion feature obtained by the current sampling is superimposed on the scale fusion feature obtained by the current sampling.

[0010] Optionally, the feature aggregation module obtains the predicted segmentation result in the following manner: performing global maximum pooling and global average pooling on the fused features, respectively, to obtain first pooling features corresponding to the global maximum pooling and second pooling features corresponding to the global average pooling, respectively; performing multiple convolutions on the first pooling features and the second pooling features, respectively, to obtain first pooling convolution features corresponding to the first pooling features and second pooling convolution features corresponding to the second pooling features; superimposing the first pooling convolution features and the second pooling convolution features to obtain superimposed features; fusing the superimposed features with the fused features to obtain the predicted segmentation result.

[0011] According to a second aspect of the present disclosure, there is provided an automatic image segmentation method, the automatic image segmentation method comprising: acquiring a target image, wherein the target image is image data of a physiological tissue containing a lesion; inputting the target image into an automatic image segmentation model, and using the automatic image segmentation model to segment the lesion in the target image to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method for the automatic image segmentation model described in the present disclosure.

[0012] According to a third aspect of the present disclosure, a training system for an automatic image segmentation model is provided, the training system comprising: a sample acquisition unit, configured to acquire a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample comprises image data of a physiological tissue containing a lesion, and each sample annotation information represents a lesion position in the corresponding training sample; an extraction unit, configured to input the training sample set into a first branch network and a second branch network of the automatic image segmentation model, respectively, to obtain a first feature extracted from the first branch network and a second feature extracted from the second branch network, respectively, wherein the first branch network can extract local features in the training sample, and the second branch network can extract global features in the training sample; a fusion unit, configured to fuse the first feature and the second feature using a shallow fusion module of the automatic image segmentation model to obtain a fused feature; a prediction unit, configured to obtain a predicted segmentation result of the lesion in each training sample based on the fused feature using a feature aggregation module of the automatic image segmentation model; and a training unit, configured to train the automatic image segmentation model by comparing the predicted segmentation result with the sample annotation information.

[0013] According to a fourth aspect of the present disclosure, there is provided an automatic image segmentation system, comprising: an image acquisition unit, configured to acquire a target image, wherein the target image is image data of a physiological tissue containing a lesion; and an image segmentation unit, configured to input the target image into an automatic image segmentation model, and use the automatic image segmentation model to segment the lesion in the target image to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method of the automatic image segmentation model described in the present disclosure.

[0014] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein when the instructions executable by the processor are executed by the processor, the processor is prompted to execute the training method of the automatic image segmentation model or the automatic image segmentation method according to the present disclosure.

[0015] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the automatic image segmentation model or the automatic image segmentation method according to the present disclosure.

[0016] According to a seventh aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the computer executable instructions are executed by at least one processor, the training method of the automatic image segmentation model or the automatic image segmentation method according to the present disclosure is implemented.

[0017] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0018] According to the present disclosure, a dual-branch network can be used to extract different first features and second features from the image data through a first branch network for extracting local features and a second branch network for extracting global features. The first features and the second features are then fused, and a predicted segmentation result is obtained based on the fused features. In this way, the image can be identified by extracting features from different angles such as local and global, and more and richer information can be extracted from the image. The lesions in the image can be identified by taking into account both the global distribution and local details in the image, avoiding ignoring details at different scales, and improving the accuracy of the segmentation results.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 It is a schematic flowchart of a training method of an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0022] Figure 2 It is a schematic block diagram of an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0023] Figure 3 It is a schematic block diagram of a Transformer model in an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0024] Figure 4 It is a schematic flowchart of the steps of obtaining scale fusion features in the training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure.

[0025] Figure 5 It is a schematic block diagram of a shallow fusion module in an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0026] Figure 6It is a schematic flowchart of the steps of obtaining extracted features in the training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure.

[0027] Figure 7 It is a schematic flowchart of the steps of obtaining predicted segmentation results in the training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure.

[0028] Figure 8 It is a schematic block diagram of a weight aggregation module in an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0029] Fig. 9 It is a schematic flowchart of an application example of a training method for an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0030] Fig.10 is a schematic flowchart of an automatic image segmentation method according to an exemplary embodiment of the present disclosure.

[0031] Fig.11 It is a schematic block diagram of a training system for an automatic image segmentation model according to an exemplary embodiment of the present disclosure.

[0032] Fig.12 is a schematic block diagram of an automatic image segmentation system according to an exemplary embodiment of the present disclosure.

[0033] Fig.13 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] In view of the problem in the related art that the automatic segmentation prediction results of the model are not accurate enough due to the large changes in the scale of lesion morphology and irregular edges, the exemplary embodiments of the present disclosure propose a training method for an automatic image segmentation model, an automatic image segmentation method, a training system for an automatic image segmentation model, an automatic image segmentation system, an electronic device, a computer-readable storage medium and a computer program product, which can solve or at least alleviate the above problems.

[0038] In a first aspect of an exemplary embodiment of the present disclosure, a method for training an automatic image segmentation model is provided.

[0039] The training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure can be applied to scenarios where users interact with software. For example, the software can be loaded on the user terminal, and the user can input model training instructions on the user terminal. The user terminal can obtain the final automatic image segmentation model by executing the training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure.

[0040] Specifically, the user terminal can obtain a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample includes image data of physiological tissue containing lesions, and each sample annotation information represents the lesion location in the corresponding training sample.

[0041] The user terminal can also input the training sample set into the first branch network and the second branch network of the image automatic segmentation model respectively, and obtain the first feature extracted from the first branch network and the second feature extracted from the second branch network respectively, wherein the first branch network is used to extract local features in the training samples, and the second branch network is used to extract global features in the training samples.

[0042] The user terminal can also use the shallow fusion module of the automatic image segmentation model to fuse the first feature and the second feature to obtain a fused feature. It can also use the feature aggregation module of the automatic image segmentation model to obtain a predicted segmentation result for the lesion in each training sample based on the fused feature. The automatic image segmentation model can also be trained by comparing the predicted segmentation result and the sample annotation information.

[0043] The above-mentioned user terminal can be, for example, a tablet computer, a laptop computer, a digital assistant, a wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios. For example, the user can request a training model from the server through the network on the user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.). The server can complete the request by executing the training method of the automatic image segmentation model according to the exemplary embodiment of the present disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0044] According to the training method of the automatic image segmentation model of the exemplary embodiment of the present disclosure, a network segmentation model with dual-branch feature extraction and feature fusion can be used to automatically segment lesions from images, and images can be identified by extracting more and richer information from different feature extraction angles such as local and global. Lesions in images can be identified by taking into account both the global distribution and local details in the images, avoiding ignoring details at different scales, and improving the accuracy of the segmentation results.

[0045] The following will refer to Figures 1 to 9 An example of a training method for an automatic image segmentation model according to an embodiment of the present disclosure is described. The automatic image segmentation model may include a first branch network, a second branch network, a shallow fusion module, and a feature aggregation module. The first branch network and the second branch network may be used to extract features from input training samples, the shallow fusion module may be used to fuse the features output by the first branch network and the second branch network, and the feature aggregation module may be used to obtain a predicted segmentation result for the lesion in each training sample based on the fused features.

[0046] like Figure 1 As shown, the training method of the automatic image segmentation model may include the following steps:

[0047] In step S110, a training sample set and sample annotation information corresponding to each training sample in the training sample set may be obtained.

[0048] Here, the training samples may include image data of physiological tissues containing lesions, and each sample annotation information may represent the location of the lesion in the corresponding training sample.

[0049] As an example, the image data may be obtained by capturing physiological tissue with lesions, the physiological tissue may be any part of the human body or animal body, and the lesion may refer to the abnormal part of the physiological tissue, for example, the physiological tissue may be the lungs, and the lesion may be a lung nodule. In the embodiments of the present disclosure, the image data may refer to data that reproduces a substance in a visually perceptible manner, which may also be referred to as "image data" in this article. The method of capturing the image data may include, but is not limited to, computed tomography (CT), magnetic resonance imaging, X-rays, cameras, microscopic imaging, etc. However, the embodiments of the present disclosure do not place particular restrictions on the method of acquiring image data, the location of the physiological tissue, the type of lesion, etc.

[0050] In step S110, the training sample set may be obtained by: acquiring an original image data set, wherein the original image data set includes original image data of physiological tissues containing lesions; and preprocessing the original image data set to obtain the above training sample set.

[0051] As an example, preprocessing the original image data set may include resampling and / or slicing the original image data set. Here, the resampling operation may include unifying the overall size of all the original image data, and the slicing operation may include slicing the three-dimensional original image data into two-dimensional data to facilitate reading and recognition of the network model.

[0052] In addition, the preprocessing of the original image dataset may also include data enhancement of the original image dataset or the resampled and / or sliced ​​original image dataset, where data enhancement refers to generating image data based on the data in the current original image dataset, and adding the generated image data to the original image dataset to enhance the amount of data in the dataset. As an example, the image data may be generated by randomly flipping the original image data in the original image dataset horizontally and / or vertically, and the generated image data may be added back to the original image dataset.

[0053] In this step, the sample annotation information may be, for example, but not limited to, a segmentation mask of the lesion position in the corresponding image data. As an example, the lesion position in the original image data may be manually located and annotated to obtain a segmentation mask of the lesion as the corresponding sample annotation information, for example, a professional in the field of imaging medicine may perform manual annotation to obtain a segmentation mask to achieve fully supervised learning.

[0054] After obtaining the training sample data set, the training sample data set can be divided into a training set, a validation set, and a test set for respectively using for model training, validation, and testing.

[0055] In step S120, the training sample set may be input into the first branch network and the second branch network of the automatic image segmentation model respectively, to obtain the first feature extracted from the first branch network and the second feature extracted from the second branch network respectively.

[0056] Here, the first branch network can extract local features in the training samples, and the second branch network can extract global features in the training samples.

[0057] As an example, Figure 2 As shown, the first branch network may include a convolutional neural network (CNN), and the second branch network may include a Transformer network, wherein the CNN network can capture the feature details of the local area of ​​interest from a local modeling, while the Transformer network can obtain the global context information of the extracted features from a global modeling.

[0058] By jointly extracting the features of image data through a dual-branch network, image features can be captured from different dimensions. Compared with the solution of using a single network to extract features, more image information can be obtained, and global and local feature complementarity can be achieved, which is conducive to more accurate identification and segmentation of lesions.

[0059] Although CNN network and Transformer network are used as examples for illustration here, the embodiments of the present disclosure are not limited thereto and other networks may also be used for implementation, as long as the features extracted by the two networks focus on different aspects and can capture global features and local features respectively.

[0060] In addition, as an example, in step S120, the first branch network and the second branch network can be used to extract features of different scales on the image data respectively to obtain features of different scales, and the amount of information contained in these features and the dimensions of attention to the information are different.

[0061] For example, the shallow fusion module can obtain the first feature and the second feature in the following manner: perform feature extraction on multiple feature scales on each training sample in the training sample set in the first branch network and the second branch network respectively to obtain the first feature and the second feature.

[0062] Here, in the feature extraction at different feature scales, the number of channels of the extracted features is different, the first feature includes multiple first-scale features corresponding one-to-one to multiple feature scales, the second feature includes multiple second-scale features corresponding one-to-one to multiple feature scales, the first scale feature corresponds one-to-one to the second scale feature, and the fused feature includes multiple scale fused features corresponding one-to-one to multiple feature scales.

[0063] Specifically, feature extraction is performed on the training samples at each feature scale in multiple feature scales to obtain scale features corresponding to each feature scale. The first scale feature output by the first branch network can correspond to the second scale feature output by the second branch network one by one. The number of channels of the corresponding pair of scale features is the same, and they can be fused to obtain the scale fusion feature at the feature scale. As an example, each channel can correspond to a feature matrix. In addition, the number of feature scales and the number of feature channels corresponding to each feature scale described in this article can be set according to actual needs.

[0064] by Figure 2 For example, four feature scales can be preset, and the image data can be input into the first branch network (e.g. Figure 2 In a CNN network), the first branch network can output the first scale features corresponding to each feature scale, for example Figure 2 The number of channels of these first scale features is different and can be reduced or increased in sequence, for example, Figure 2 The number of channels of each first scale feature c1 increases gradually from left to right.

[0065] Correspondingly, the image data is also input into the second branch network to Figure 2 For example, the second branch network may include a Transformer network, which may perform a downsampling operation on the image data and output an initial extracted feature obtained by the downsampling operation. The initial extracted feature may be upsampled according to four preset feature scales to obtain a second scale feature t1 corresponding to each feature scale, thereby corresponding to the number of channels of the first scale feature of the first branch network.

[0066] The structure of the Transformer network can be, for example, Figure 3 As shown, the image data can be first divided into blocks to obtain block images; the block images can be linearly transformed and flattened through the key component K, the query component Q and the value component V respectively; the output of the key component K and the output of the query component Q can be element-by-element multiplied to obtain a first operation result; the first operation result and the output of the value component V can be element-by-element multiplied to obtain a second operation result; the second operation result can be deformed and reconstructed to obtain the above-mentioned initial extracted features.

[0067] Although it is described here that the second scale feature is obtained by downsampling based on the output of the Transformer network, the embodiments of the present disclosure are not limited thereto. For example, other networks may be used to directly output the second scale feature. In addition, the internal structure of the Transformer network is not limited to Figure 3 The structure (TR) shown can be adjusted according to actual needs or the existing Transformer network structure can be adopted.

[0068] In the above process, the fewer the number of channels of a feature, the shallower the information contained in the feature, and the shallow information helps to focus on local features, such as the texture and edge features of the lesion; the more the number of channels of a feature, the deeper the information contained in the feature, and the visualization of deep information is relatively abstract, which is difficult for humans to understand, and helps to focus on global contextual associations. Therefore, in the embodiments of the present disclosure, by extracting features with different numbers of channels at different feature scales, different levels of information can be obtained, so that lesions in the image can be identified more comprehensively and fully from different levels and different semantic depths.

[0069] In step S130, the shallow fusion module of the automatic image segmentation model may be used to fuse the first feature and the second feature to obtain a fused feature.

[0070] Specifically, the automatic image segmentation model may further include a shallow fusion module. In this step, the shallow fusion module may fuse the outputs of two parallel branch networks. The fusion method may adopt an existing feature fusion method, such as feature superposition.

[0071] Preferably, according to an embodiment of the present disclosure, an adaptive fusion method is provided, which can obtain information on more scales by performing channel splitting, channel compression and channel merging on features and introducing an attention mechanism, so as to further increase the model's perception level of image data, which is conducive to improving the segmentation effect of the model.

[0072] Specifically, refer to Figure 2 and Figure 5 , a shallow fusion module (SF) can be used to fuse the first feature and the second feature, wherein the shallow fusion module can include a branch feature extraction module (PBC).

[0073] like Figure 4 As shown, in an example where the first feature and the second feature distribution include multiple scale features, for any feature scale, the shallow fusion module can obtain the scale fusion feature in the following manner:

[0074] In step S410, the first scale feature and the second scale feature corresponding to the current feature scale may be respectively input into the branch feature extraction module to obtain a first extracted feature corresponding to the first scale feature and a second extracted feature corresponding to the second scale feature.

[0075] Here, the branch feature extraction module can be used to perform channel compression at multiple compression scales on the input scale feature, and merge the multiple compressed features obtained by compression, wherein the number of channels compressed in the channel compression at different compression scales is different.

[0076] Specifically, the branch feature extraction module can perform feature extraction on different channels for each of the corresponding first scale features and second scale features; then the features extracted from each channel can be compressed. During the compression process, multiple compression scales can be selected to obtain multiple compressed features; finally, these compressed features can be merged to obtain extracted features corresponding to the input scale features.

[0077] As an example, refer to Figure 5 , the branch feature extraction module can have a parallel branch convolution structure, for example. Figure 6 As shown, the branch feature extraction module can obtain the first extraction feature or the second extraction feature in the following manner:

[0078] In step S610, the current scale feature may be channel-split according to the channels of the current scale feature to obtain a plurality of split features.

[0079] Here, the current scale feature may be a first scale feature or a second scale feature. As described above, each scale feature may have multiple feature channels. In this step, each feature channel may be split to obtain a split feature corresponding to each channel.

[0080] In step S620, depthwise convolution and pointwise convolution may be performed on the plurality of split features respectively to obtain a plurality of convolution features corresponding one-to-one to the plurality of split features.

[0081] In this step, depthwise convolution and pointwise convolution can be used in sequence to compress each split feature at different multi-scale compression channels (e.g. Figure 5 Specifically, we can first perform a deep convolution operation on the classification feature to obtain a deep convolution kernel, and then perform a point-by-point convolution on the deep convolution kernel to obtain a point-by-point convolution kernel as the convolution feature.

[0082] In step S630, channel compression may be performed on the multiple convolutional features according to each compression scale of the multiple compression scales, to obtain multiple compressed features corresponding to the multiple compression scales.

[0083] As an example, Figure 5 As shown in FIG. 3 , three compression scales can be preset, namely 1 / 6, 1 / 3 and 1 / 2. The convolutional features can be compressed according to the compression ratio of each compression scale to obtain the corresponding compressed features. Figure 5 Taking the compression ratios of 1 / 6, 1 / 3 and 1 / 2 as an example, three compressed features can be obtained by compressing the convolution features according to compression ratios of 1 / 6, 1 / 3 and 1 / 2 respectively. For example, when the convolution feature has 6 channels, channel compression can be performed through the convolution operation. Taking the compression ratios of 1 / 6, 1 / 3 and 1 / 2 as examples, three compressed features with 1 channel, 2 channels and 3 channels respectively can be obtained.

[0084] In step S640, multiple compressed features may be merged to obtain an extracted feature corresponding to the current scale feature.

[0085] In this step, all compressed features may be merged together and aligned with the original input features, for example, they may be superimposed to obtain extracted features corresponding to the input scale features, such as the first extracted features or the second extracted features.

[0086] In the above way, extracting features for channels at multiple compression scales enables the network model to capture the detailed information of small-scale features, while obtaining the contextual information of large-scale features, thereby providing the model with rich feature representation, enhancing its expressiveness and improving segmentation performance.

[0087] Return to reference Figure 4 In step S420, a multi-attention mechanism can be introduced to adjust the weights of different channel features in the first extracted feature and the second extracted feature, respectively, to obtain a first branch feature corresponding to the first extracted feature and a second branch feature corresponding to the second extracted feature.

[0088] As an example, the shallow fusion module may also include a first attention module and a second attention module, wherein the first attention module may be used to adjust the weights of different channel features in the first extracted feature to obtain a first branch feature; and the second attention module may be used to adjust the weights of different channel features in the second extracted feature to obtain a second branch feature.

[0089] Preferably, Figure 5 For example, the first attention module can adopt the efficient channel attention mechanism (EfficientChannel Attention, ECA), and the second attention module can adopt the coordinate attention mechanism (CoordinateAttention, CA).

[0090] Here, efficient channel attention can automatically adjust channel weights according to input features and adaptively adjust the importance of channel features, so that the network can better learn useful features and suppress irrelevant features. Coordinate attention can effectively capture the spatial structure in the feature map by explicitly modeling the information of the spatial dimension. At the same time, considering the coordinate information of each position in the feature map, it can adaptively adjust the weight of the feature channel, emphasize important areas and suppress irrelevant areas, thereby enhancing feature representation and improving the perception of the target location.

[0091] The above-mentioned ECA and CA are only preferred examples, and the embodiments of the present disclosure are not limited thereto, and other existing attention mechanisms may also be adopted.

[0092] In step S430, the first branch feature and the second branch feature may be aggregated to obtain a scale fusion feature of the current feature scale.

[0093] In this step, the first branch feature and the second branch feature may be aggregated, for example, spliced, to obtain a scale fusion feature.

[0094] As an example, Figure 5 As shown, the shallow fusion module can also include a weight aggregation module (WC). The weight aggregation module can be used to fuse the feature information of the first branch feature and the second branch feature. The module can use the Softmax function to normalize the learnable parameters, and adaptively assign weights w1 and w2 by performing element-by-element product operations on the first branch feature c3 and the second branch feature t3, respectively, and finally concatenate and fuse these features to generate a feature fusion block sfb, i.e., a scale fusion feature. Here, the learnable parameters refer to the parameters that can be iteratively updated in the back propagation process after the loss function is optimized.

[0095] In addition, optionally, in the above process, residual connection and maximum pooling (MaxPool) operations may be performed on the extracted features output by the branch feature extraction module.

[0096] Specifically, the extracted features output by the branch feature extraction module and the mid-scale features input to the branch feature extraction module can be residually connected to obtain residual connection features. Here, residual connection is used to prevent information loss during the extraction process. This operation takes into account that some original features will be lost after the convolution operation is performed to extract features. The loss of feature information can be compensated by superimposing the features extracted by the branch feature extraction module with the original input features.

[0097] You can also perform a maximum pooling operation on the scale feature, and concatenate the residual connection feature with the sampling feature after the maximum pooling operation to obtain a concatenated feature, for example Figure 5c2 and t2 shown. In this way, redundant feature information can be removed while retaining important feature information, which is beneficial to the generalization of the network model and reduces overfitting.

[0098] In addition, the extracted features output by the branch feature extraction module can also be convolved, for example, Figure 5 In , the concatenated features c2 and t2 can be convolved (Conv), and the convolved features are input into the subsequent attention module.

[0099] Although the above reference Figure 5 An example description of the fusion processing of the shallow fusion module is given, but the embodiments of the present disclosure are not limited to this. Some of the processing processes in the above processing can be omitted or replaced. For example, the residual connection, maximum pooling operation and / or convolution operation process can be omitted, and the convolution process before multi-attention feature extraction can also be omitted.

[0100] Return to reference Figure 1 In step S140, the feature aggregation module of the automatic image segmentation model can be used to obtain the predicted segmentation result of the lesion in each training sample based on the fusion features.

[0101] Specifically, the automatic image segmentation model may further include a feature aggregation module. In this step, as an example, the feature aggregation module may directly aggregate the fused features to obtain the final predicted segmentation result.

[0102] As another example, before obtaining the final predicted segmentation result based on the fused features, a dense connection operation based on element-by-element superposition may be performed to perform long-distance learning of the model.

[0103] Specifically, the automatic image segmentation model may further include an upsampling superposition module. The training method according to the embodiment of the present disclosure may further include: using the upsampling superposition module, according to the minimum number of channels of each scale fusion feature in the fusion feature, upsampling the scale fusion feature with more channels than the minimum number of channels to obtain the sampling fusion feature; using the upsampling superposition module, superimposing the scale fusion feature with the minimum number of channels onto the sampling fusion feature to obtain the superimposed sampling fusion feature; using the feature aggregation module, based on the superimposed sampling fusion feature, obtaining the predicted segmentation result.

[0104] Here, after each upsampling, the scale fusion feature having the same number of channels as the scale fusion feature obtained by the current sampling is superimposed on the scale fusion feature obtained by the current sampling.

[0105] The processing of the upsampling stacking module can be as follows Figure 2Specifically, in the example where the fusion feature includes multiple scale fusion features, since the number of channels corresponding to each feature scale is different, each scale fusion feature has a different number of channels. The scale fusion features with more channels can be upsampled step by step from more channels to fewer channels according to the number of channels of each feature scale until the upsampled number is the minimum number of channels of all scale fusion features.

[0106] by Figure 2 Taking the scale fusion features S1, S2, S3 and S4 in as examples, the number of channels decreases from feature S1 to feature S4, that is, feature S1 can be the scale fusion feature with the largest number of channels, and feature S4 can be the scale fusion feature with the smallest number of channels. Taking feature S1 as an example, feature S1 can be upsampled from the largest number of channels to the smallest number of channels to obtain feature S1-1 with the same number of channels as feature S2. After upsampling, feature S2 with the same number of channels as feature S1-1 can be superimposed on the currently sampled feature S1-1 to obtain feature S1-1'. Then, feature S1-1' can continue to be upsampled to obtain feature S1-2 with the same number of channels as feature S3. After this upsampling, feature S3 with the same number of channels as feature S1-2 and feature S2-1 obtained by upsampling feature S2 can be superimposed on feature S1-2 currently sampled to obtain feature S1-2'. According to this method, feature S1-3 can be obtained, and features with the same number of channels can be superimposed on feature S1-3 to obtain superimposed sampled fusion feature S1-3' corresponding to feature S1. By analogy, upsampling of each scale fusion feature S1, S2, S3 and S4 can be completed. Here, since feature S4 is the feature with the smallest number of channels, it can be superimposed on other features without upsampling. It only needs to be superimposed on the last layer sampling result of features S1, S2 and S3.

[0107] Through the above method, in the densely connected structure based on element-by-element superposition, the feature fusion blocks in the same layer have the same feature size and can be superimposed and fused with each other. The upsampling operation can sample the lower-layer features to the same size of the upper-layer features. In this way, the use of element-by-element superposition can promote inter-layer information interaction of fused features and feature fusion within fused features, while reducing the amount of calculation and the amount of parameters, saving computing costs.

[0108] Return to step S140, if Figure 2As shown, the automatic image segmentation model may include a feature aggregation module (FA). Here, in one example, the feature aggregation module may respectively aggregate the above-mentioned superimposed sampling fusion features to obtain respectively predicted segmentation results; in another example, the feature aggregation model may also directly aggregate the fusion features of each scale to obtain respectively predicted segmentation results.

[0109] Specifically, if Figure 7 As shown in the figure, for any superimposed sampling fusion feature or scale fusion feature, the feature aggregation module can obtain the predicted segmentation result in the following ways:

[0110] In step S710, global maximum pooling and global average pooling may be performed on the fused features, respectively, to obtain first pooling features corresponding to the global maximum pooling and second pooling features corresponding to the global average pooling, respectively.

[0111] For example, Figure 8 As shown, the input features can be convolved or the input features can be convolved (for example Figure 8 After the 1×1×C group convolution Conv_1 shown in the figure, where C represents the number of channels), adaptive maximum pooling (Amap) and adaptive average pooling (Aavp) are performed to obtain the weights of the attention features, and the first pooling features and the second pooling features are obtained. Such pooling operations can compress the size of the feature map and retain the original number of channels, which is beneficial for the network model to extract important global information of the features and eliminate redundant information.

[0112] In step S720, the first pooling feature and the second pooling feature may be convolved multiple times respectively to obtain a first pooling convolution feature corresponding to the first pooling feature and a second pooling convolution feature corresponding to the second pooling feature.

[0113] by Figure 8 For example, the first pooling feature and the second pooling feature can be convolved multiple times, for example, 1×1×C / r convolution Conv_2 and 1×1×C convolution Conv_3 are performed to obtain the first pooling convolution feature and the second pooling convolution feature. Figure 8 In , Relu represents the activation function, C represents the number of channels, and r represents the number of groups of grouped convolution.

[0114] In step S730, the first pooled convolution feature and the second pooled convolution feature may be superimposed to obtain a superimposed feature.

[0115] by Figure 7For example, an aggregation module similar to the weight aggregation module WC in the shallow fusion module can be used to superimpose the first pooled convolutional features and the second pooled convolutional features. Compared with the weight aggregation module in the shallow fusion module, the difference between the aggregation module used here can only be that the splicing operation of the weight aggregation module in the shallow fusion module is replaced by the superposition operation. Figure 8 In , Sigmoid represents the activation function.

[0116] In step S740, the superimposed features and the fused features may be fused to obtain a predicted segmentation result.

[0117] In this step, the superimposed features can be fused with the fused features of the original input to obtain the predicted segmentation result, for example Figure 8 In addition, optionally, the features obtained by fusion of the superimposed features and the fusion features of the original input can be convolved, for example, a 1×1×1 convolution Conv_4 can be performed, and the convolved features are used as the predicted segmentation results.

[0118] Through the above process, the learnable parameters can be used to dynamically allocate weight information, and the importance of each channel can be further explored, which is conducive to the enhancement of effective feature information and the weakening of irrelevant feature information.

[0119] Return to reference Figure 1 In step S150, the automatic image segmentation model may be trained by comparing the predicted segmentation results and the sample annotation information.

[0120] Specifically, the prediction loss can be determined based on the predicted segmentation results and sample annotation information. By optimizing the prediction loss, the model parameters (such as the learnable parameters mentioned above) can be adjusted to continuously optimize the model's prediction results and achieve model training.

[0121] Here, the optimization goal for training the automatic image segmentation model can be:

[0122]

[0123] in, Represents the predicted segmentation result, represents the actual segmentation result, L Represents the loss function that needs to be optimized.

[0124] As an example, the loss function can be expressed by the following formula (1):

[0125] (1)

[0126] in, Indicates the batch processing size, Nrepresents the total number of training samples, and represents the pre-set weight hyperparameter, Indicates i The actual segmentation results of training samples are Indicates i However, the form of the loss function is not limited to the above example, and the prediction loss can also be determined by other existing loss calculation methods.

[0127] In addition, in an example where the fusion feature includes multiple scale fusion features, the predicted segmentation results can be obtained for each scale fusion feature or for multiple sampled scale fusion features obtained by upsampling these scale fusion features.

[0128] by Figure 2 For example, each feature aggregation module outputs a corresponding predicted segmentation result, and the loss can be calculated for each predicted segmentation result, and the loss for each predicted segmentation result can be superimposed as the final loss. After the model is trained, the predicted segmentation results output by different feature aggregation modules can be superimposed to obtain the final predicted segmentation result.

[0129] Fig. 9 FIG. 1 shows an application example process of a training method for an automatic image segmentation model according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, taking lung nodule CT image data as an example, in step S910, lung nodule CT image data can be obtained. In step S920, the image data can be annotated by the imaging doctor to obtain the annotation information. In step S930, the image data can be resampled and sliced ​​to obtain a training sample data set. In step S940, the training sample data set can be divided into a training set, a validation set, and a test set, which are used for model training, validation, and testing, respectively. In step S950, the training sample can be subjected to dual-branch feature extraction through the first branch network and the second branch network to obtain a first extracted feature and a second extracted feature. In step S960, the first extracted feature and the second extracted feature can be feature fused. In step S970, the image automatic segmentation model can be trained, and in step S980, the trained model is used to obtain the final segmentation result.

[0130] The training method according to the exemplary embodiment of the present disclosure can solve the problem that the prediction results of the automatic segmentation of the network model are not ideal due to the large changes in the size of nodules and the fuzzy and irregular contours of the nodules, which makes it impossible to effectively extract and fully integrate the key feature information, and can improve the accuracy of automatic segmentation of lesions.

[0131] In addition, according to an exemplary embodiment of the present disclosure, an automatic image segmentation model based on dual-branch feature extraction and feature fusion is proposed. By capturing the global and local information of the image data, it can accurately locate the lesion area and extract key feature information. It also allocates feature weights through adaptive feature fusion and global dual-pooling attention, which is beneficial to improving the segmentation performance accuracy of the lesion.

[0132] In addition, according to an exemplary embodiment of the present disclosure, a shallow fusion module is designed in the model. In the shallow fusion module, feature information is extracted by using multi-scale convolution, residual connection and multi-attention, weights are adaptively allocated using learnable parameters, and finally spliced ​​and fused to generate a fusion block with highly fused semantic features, which is conducive to the network model learning deep feature information and fitting functions and improving generalization capabilities.

[0133] In addition, according to an exemplary embodiment of the present disclosure, by using a densely connected structure based on element-by-element superposition in the model, information interaction between semantic fusion block layers and feature fusion within blocks can be promoted, feature enhancement is performed at the top level of the model through global dual-pooling attention in the feature aggregation module, and a segmentation map is generated by superimposing the feature map of the original input, which is conducive to obtaining better image segmentation results of the lesions.

[0134] In a second aspect of an exemplary embodiment of the present disclosure, an automatic image segmentation method is provided. The automatic image segmentation method can be applied to a scenario where a user interacts with software, for example, the software can be loaded on a user terminal, the user can input an automatic image segmentation instruction on the user terminal, and the user terminal can obtain a final segmented lesion by executing the automatic image segmentation method according to an exemplary embodiment of the present disclosure.

[0135] like Fig.10 As shown, the automatic image segmentation method may include: step S1010, a target image may be acquired, wherein the target image is image data of a physiological tissue containing a lesion; step S1020, the target image may be input into an automatic image segmentation model, and the lesion in the target image may be segmented using the automatic image segmentation model to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method of the automatic image segmentation model described in the exemplary embodiment of the present disclosure.

[0136] The specific steps and details of the automatic image segmentation method according to the exemplary embodiment of the present disclosure can be implemented and understood similarly to the training method of the automatic image segmentation model described in the first aspect above, and accordingly, the automatic image segmentation method has the same or similar beneficial effects as the training method of the automatic image segmentation model described above, which will not be repeated here.

[0137] In a third aspect of the exemplary embodiments of the present disclosure, a training system for an automatic image segmentation model is provided, such as Fig.11 As shown, the training system includes a sample acquisition unit 1110 , an extraction unit 1120 , a fusion unit 1130 , a prediction unit 1140 and a training unit 1150 .

[0138] The sample acquisition unit 1110 is configured to acquire a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample includes image data of physiological tissue containing lesions, and each sample annotation information represents the lesion location in the corresponding training sample.

[0139] The extraction unit 1120 is configured to input the training sample set into the first branch network and the second branch network of the image automatic segmentation model, respectively, to obtain the first feature extracted from the first branch network and the second feature extracted from the second branch network, respectively, wherein the first branch network can extract local features in the training samples, and the second branch network can extract global features in the training samples.

[0140] The fusion unit 1130 is configured to use a shallow fusion module of the automatic image segmentation model to fuse the first feature and the second feature to obtain a fused feature.

[0141] The prediction unit 1140 is configured to obtain a predicted segmentation result of the lesion in each training sample based on the fusion features by using the feature aggregation module of the automatic image segmentation model.

[0142] The training unit 1150 is configured to train the automatic image segmentation model by comparing the predicted segmentation result and the sample annotation information.

[0143] As an example, the first feature and the second feature are obtained in the following manner: feature extraction at multiple feature scales is performed on each training sample in the training sample set in the first branch network and the second branch network, respectively, to obtain the first feature and the second feature, wherein the number of channels of the extracted features in the feature extraction at different feature scales is different, the first feature includes multiple first-scale features corresponding one-to-one to multiple feature scales, the second feature includes multiple second-scale features corresponding one-to-one to multiple feature scales, the first scale feature corresponds one-to-one to the second scale feature, and the fused feature includes multiple scale fused features corresponding one-to-one to multiple feature scales.

[0144] As an example, the shallow fusion module includes a branch feature extraction module, wherein, for any feature scale, the shallow fusion module obtains a scale fusion feature in the following manner: the first scale feature and the second scale feature corresponding to the current feature scale are respectively input into the branch feature extraction module to obtain a first extracted feature corresponding to the first scale feature and a second extracted feature corresponding to the second scale feature, respectively, wherein the branch feature extraction module is used to perform channel compression of multiple compression scales on the input scale feature, and merge the multiple compressed features obtained by compression, wherein the number of compressed channels in the channel compression at different compression scales is different; by introducing a multi-attention mechanism, the weights of different channel features in the first extracted feature and the second extracted feature are respectively adjusted to obtain a first branch feature corresponding to the first extracted feature and a second branch feature corresponding to the second extracted feature; the first branch feature and the second branch feature are aggregated to obtain a scale fusion feature of the current feature scale.

[0145] As an example, the branch feature extraction module obtains the first extraction feature or the second extraction feature in the following manner: according to the channel of the current scale feature, the current scale feature is split into multiple split features, where the current scale feature is the first scale feature or the second scale feature; deep convolution and point-by-point convolution are performed on the multiple split features respectively to obtain multiple convolution features corresponding to the multiple split features one by one; channel compression is performed on the multiple convolution features according to each compression scale in the multiple compression scales respectively to obtain multiple compression features corresponding to the multiple compression scales one by one; and the multiple compression features are merged to obtain the extraction feature corresponding to the current scale feature.

[0146] As an example, the automatic image segmentation model includes an upsampling superposition module, and the training method also includes: using the upsampling superposition module, according to the minimum number of channels of each scale fusion feature in the fusion feature, upsampling the scale fusion feature with more channels than the minimum number of channels to obtain the sampling fusion feature; using the upsampling superposition module, superimposing the scale fusion feature with the minimum number of channels on the sampling fusion feature to obtain the superimposed sampling fusion feature; using the feature aggregation module, based on the superimposed sampling fusion feature, obtaining a predicted segmentation result, wherein after each upsampling, the scale fusion feature with the same number of channels as the scale fusion feature obtained by the current sampling is superimposed on the scale fusion feature obtained by the current sampling.

[0147] As an example, the feature aggregation module obtains the predicted segmentation result in the following manner: performing global maximum pooling and global average pooling on the fused features, respectively, to obtain the first pooling feature corresponding to the global maximum pooling and the second pooling feature corresponding to the global average pooling, respectively; performing multiple convolutions on the first pooling feature and the second pooling feature, respectively, to obtain the first pooling convolution feature corresponding to the first pooling feature and the second pooling convolution feature corresponding to the second pooling feature; superimposing the first pooling convolution feature and the second pooling convolution feature to obtain the superimposed feature; fusing the superimposed feature with the fused feature to obtain the predicted segmentation result.

[0148] Regarding the device in the above-mentioned embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. Each unit in the training device of the automatic image segmentation model can execute the corresponding steps in the method according to the training method of the automatic image segmentation model in the method embodiment of the first aspect above, which will not be elaborated here.

[0149] In a fourth aspect of the exemplary embodiments of the present disclosure, there is provided an automatic image segmentation system, such as Fig.12 As shown, the automatic image segmentation system includes an image acquisition unit 1210 and an image segmentation unit 1220 .

[0150] The image acquisition unit 1210 is configured to acquire a target image, wherein the target image is image data of a physiological tissue containing a lesion; the image segmentation unit 1220 is configured to input the target image into an automatic image segmentation model, and use the automatic image segmentation model to segment the lesion in the target image to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method of the automatic image segmentation model of the exemplary embodiment of the present disclosure.

[0151] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0152] Fig.13 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Fig.13 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the automatic image segmentation model or the automatic image segmentation method as described in the above exemplary embodiments.

[0153] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0154] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0155] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0156] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0157] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0158] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the training method of the image automatic segmentation model or the image automatic segmentation method as described in the above exemplary embodiments. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a B D-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0159] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer executable instructions, and when the computer executable instructions are executed by at least one processor, the training method of the automatic image segmentation model or the automatic image segmentation method according to the exemplary embodiment of the present disclosure is implemented.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0161] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an automatic image segmentation model, characterized in that: The training method comprises: Acquire a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample includes image data of physiological tissue containing a lesion, and each sample annotation information represents a lesion position in the corresponding training sample; Inputting the training sample set into a first branch network and a second branch network of an automatic image segmentation model respectively, and obtaining a first feature extracted from the first branch network and a second feature extracted from the second branch network respectively, wherein the first branch network can extract local features in the training sample, and the second branch network can extract global features in the training sample; Using the shallow fusion module of the automatic image segmentation model, the first feature and the second feature are fused to obtain a fused feature; Using the feature aggregation module of the automatic image segmentation model, based on the fusion features, a predicted segmentation result of the lesion in each training sample is obtained; By comparing the predicted segmentation result with the sample annotation information, the automatic image segmentation model is trained. The first feature and the second feature are obtained by: performing feature extraction on multiple feature scales for each training sample in the training sample set in the first branch network and the second branch network respectively to obtain the first feature and the second feature, Among them, the number of channels of the features extracted in the feature extraction at different feature scales is different, the first feature includes a plurality of first scale features corresponding one-to-one to the plurality of feature scales, the second feature includes a plurality of second scale features corresponding one-to-one to the plurality of feature scales, the first scale feature corresponds one-to-one to the second scale feature, the fusion feature includes a plurality of scale fusion features corresponding one-to-one to the plurality of feature scales, The shallow fusion module includes a branch feature extraction module, wherein for any feature scale, the shallow fusion module obtains the scale fusion feature by the following method: Inputting a first scale feature and a second scale feature corresponding to the current feature scale into the branch feature extraction module respectively, so as to obtain a first extracted feature corresponding to the first scale feature and a second extracted feature corresponding to the second scale feature respectively, wherein the branch feature extraction module is used to perform channel compression of multiple compression scales on the input scale feature, and merge the multiple compressed features obtained by compression, wherein the number of channels compressed in the channel compression at different compression scales is different; By introducing a multi-attention mechanism, the weights of different channel features in the first extracted feature and the second extracted feature are adjusted respectively, so as to obtain a first branch feature corresponding to the first extracted feature and a second branch feature corresponding to the second extracted feature; Aggregate the first branch feature and the second branch feature to obtain a scale fusion feature of the current feature scale, The branch feature extraction module obtains the first extraction feature or the second extraction feature in the following manner: According to the channel of the current scale feature, split the current scale feature into multiple split features, wherein the current scale feature is the first scale feature or the second scale feature; Performing depthwise convolution and pointwise convolution on the multiple split features respectively to obtain multiple convolution features corresponding to the multiple split features one by one; Performing channel compression on the multiple convolution features according to each compression scale of the multiple compression scales, respectively, to obtain the multiple compressed features corresponding to the multiple compression scales one by one; The multiple compressed features are merged to obtain an extracted feature corresponding to the current scale feature, Wherein, the automatic image segmentation model includes an upsampling superposition module, and the training method further includes: Using the upsampling superposition module, according to the minimum number of channels of each scale fusion feature in the fusion feature, upsampling the scale fusion feature with a channel number greater than the minimum number of channels to obtain a sampled fusion feature; Using the upsampling superposition module, superimposing the scale fusion feature with the minimum number of channels onto the sampling fusion feature to obtain the superimposed sampling fusion feature; Using the feature aggregation module, based on the superimposed sampling fusion features, the predicted segmentation result is obtained, After each upsampling, the scale fusion feature having the same number of channels as the scale fusion feature obtained by the current sampling is superimposed on the scale fusion feature obtained by the current sampling.

2. The training method according to claim 1, characterized in that: The feature aggregation module obtains the predicted segmentation result in the following manner: Performing global maximum pooling and global average pooling on the fused features respectively, so as to obtain a first pooling feature corresponding to the global maximum pooling and a second pooling feature corresponding to the global average pooling respectively; Performing multiple convolutions on the first pooling feature and the second pooling feature respectively to obtain a first pooling convolution feature corresponding to the first pooling feature and a second pooling convolution feature corresponding to the second pooling feature; Superimposing the first pooled convolution feature and the second pooled convolution feature to obtain a superimposed feature; The superimposed feature is fused with the fused feature to obtain the predicted segmentation result.

3. An automatic image segmentation method, characterized in that: The automatic image segmentation method comprises: Acquiring a target image, wherein the target image is image data of a physiological tissue containing a lesion; The target image is input into an automatic image segmentation model, and the lesions in the target image are segmented using the automatic image segmentation model to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method of the automatic image segmentation model according to claim 1 or 2.

4. A training system for an automatic image segmentation model, characterized in that: The training system comprises: A sample acquisition unit is configured to acquire a training sample set and sample annotation information corresponding to each training sample in the training sample set, wherein the training sample includes image data of physiological tissue containing a lesion, and each sample annotation information represents a lesion position in the corresponding training sample; An extraction unit is configured to input the training sample set into a first branch network and a second branch network of an automatic image segmentation model, respectively, to obtain a first feature extracted from the first branch network and a second feature extracted from the second branch network, respectively, wherein the first branch network can extract local features in the training sample, and the second branch network can extract global features in the training sample; a fusion unit configured to fuse the first feature and the second feature using a shallow fusion module of the automatic image segmentation model to obtain a fused feature; A prediction unit is configured to obtain a predicted segmentation result of a lesion in each training sample based on the fusion feature by using a feature aggregation module of the automatic image segmentation model; A training unit is configured to train the automatic image segmentation model by comparing the predicted segmentation result with the sample annotation information, The extraction unit is configured to obtain the first feature and the second feature in the following manner: performing feature extraction on multiple feature scales on each training sample in the training sample set in the first branch network and the second branch network, respectively, to obtain the first feature and the second feature, wherein the number of channels of the features extracted in the feature extraction on different feature scales is different, the first feature includes multiple first scale features corresponding to the multiple feature scales one-to-one, the second feature includes multiple second scale features corresponding to the multiple feature scales one-to-one, the first scale feature corresponds to the second scale feature one-to-one, the fusion feature includes multiple scale fusion features corresponding to the multiple feature scales one-to-one, The shallow fusion module includes a branch feature extraction module, wherein for any feature scale, the shallow fusion module obtains the scale fusion feature in the following manner: the first scale feature and the second scale feature corresponding to the current feature scale are respectively input into the branch feature extraction module to respectively obtain a first extraction feature corresponding to the first scale feature and a second extraction feature corresponding to the second scale feature, wherein the branch feature extraction module is used to perform channel compression of multiple compression scales on the input scale feature, and merge the multiple compressed features obtained by compression, wherein the number of compressed channels in the channel compression at different compression scales is different; by introducing a multi-attention mechanism, the weights of different channel features in the first extraction feature and the second extraction feature are respectively adjusted to obtain a first branch feature corresponding to the first extraction feature and a second branch feature corresponding to the second extraction feature; the first branch feature and the second branch feature are aggregated to obtain a scale fusion feature of the current feature scale, The branch feature extraction module obtains the first extraction feature or the second extraction feature in the following manner: according to the channel of the current scale feature, the current scale feature is split into multiple split features, wherein the current scale feature is the first scale feature or the second scale feature; the multiple split features are respectively subjected to depth convolution and point-by-point convolution to obtain multiple convolution features corresponding to the multiple split features; according to each compression scale of the multiple compression scales, the multiple convolution features are respectively subjected to channel compression to obtain the multiple compression features corresponding to the multiple compression scales; the multiple compression features are merged to obtain the extraction feature corresponding to the current scale feature. Among them, the image automatic segmentation model includes an upsampling superposition module, and the fusion unit is further configured to: use the upsampling superposition module to upsample the scale fusion features with more channels than the minimum number of channels according to the minimum number of channels of each scale fusion feature in the fusion feature to obtain a sampling fusion feature; use the upsampling superposition module to superimpose the scale fusion features with the minimum number of channels on the sampling fusion features to obtain the superimposed sampling fusion features; use the feature aggregation module to obtain the predicted segmentation result based on the superimposed sampling fusion features, wherein after each upsampling, the scale fusion features with the same number of channels as the scale fusion features obtained by the current sampling are superimposed on the scale fusion features obtained by the current sampling.

5. An automatic image segmentation system, characterized in that: The automatic image segmentation system comprises: An image acquisition unit is configured to acquire a target image, wherein the target image is image data of a physiological tissue containing a lesion; An image segmentation unit is configured to input the target image into an automatic image segmentation model, and use the automatic image segmentation model to segment the lesions in the target image to obtain a segmentation result, wherein the automatic image segmentation model is trained according to the training method of the automatic image segmentation model according to claim 1 or 2.

6. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, Wherein, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the automatic image segmentation model according to claim 1 or 2 or the automatic image segmentation method according to claim 3.

7. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the automatic image segmentation model according to claim 1 or 2 or the automatic image segmentation method according to claim 3.

8. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the training method of the automatic image segmentation model according to claim 1 or 2 or the automatic image segmentation method according to claim 3 is implemented.

Images