Image segmentation method, image segmentation device, electronic device, and storage medium

Abstract

The present disclosure provides an image segmentation method, an image segmentation device, an electronic device and a storage medium, which can be applied to the fields of artificial intelligence and finance. The method comprises: acquiring an initial image to be segmented; performing feature mapping processing on the initial image by using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, the feature encoder comprising n sub-encoders connected in sequence, each sub-encoder outputting one first target feature map; processing the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map constitute a feature group; processing the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmented image, wherein the segmented image comprises at least one object to be segmented.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence and financial technology, and particularly relates to an image segmentation method, an image segmentation device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Image segmentation technology is widely used in many fields, such as bill image recognition, bank card recognition, remote sensing image segmentation, etc. This segmentation technology mainly relies on algorithms to realize the segmentation of specific regions. At present, the structure of this kind of image segmentation algorithm is generally composed of feature extraction and feature coding. The algorithm extracts features from the input image information in the encoder stage, and enlarges the segmented image to the original image size in the decoder part to realize segmentation.

[0003] However, most image segmentation algorithms fuse the feature maps in adjacent stages together in order to enhance the carrying capacity of pixel shallow feature map information, without considering the global context information carried by each feature map, thereby causing poor accuracy of image segmentation. SUMMARY

[0004] In view of the above problems, the present disclosure provides an image segmentation method, an image segmentation device, an electronic device, a computer readable storage medium and a program product.

[0005] According to a first aspect of the present disclosure, an image segmentation method is provided, comprising:

[0006] obtaining an initial image to be segmented, wherein the initial image comprises at least one object to be segmented;

[0007] performing feature mapping processing on the initial image by using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map;

[0008] processing the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group;

[0009] processing the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmented image, wherein the segmented image comprises at least one object to be segmented.

[0010] According to an embodiment of the present disclosure, the feature mapping processing of the initial image by the feature encoder in the image segmentation model to obtain the first target feature map carrying global context information includes:

[0011] For each sub-encoder, the image pixel features in the input image are extracted by using the first convolutional layer, wherein the input image includes the initial image or the first target feature map output by the sub-encoder connected to the input end of the sub-encoder;

[0012] The image pixel features are processed by using the dense neural network to output the third target feature map;

[0013] The third target feature map is input into the random inactivation layer to output the first target feature map corresponding to the sub-encoder.

[0014] According to an embodiment of the present disclosure, before the dense neural network processes the image pixel features, the method further includes:

[0015] The image pixel features are processed by using the batch normalization layer to obtain normalized image pixel features, so that the dense neural network processes the normalized image pixel features;

[0016] After the third target feature map is input into the random inactivation layer, the method further includes:

[0017] The first target feature map output by the random inactivation layer is input into the max-pooling layer to output the pooled first target feature map, so that the pooled first target feature map is input into the attention mechanism layer or the sub-encoder connected to the output end of the sub-encoder.

[0018] According to an embodiment of the present disclosure, the dense neural network includes a first dense processing block, at least one second dense processing block, and a convolutional pooling layer arranged between any two adjacent dense processing blocks;

[0019] The processing of the image pixel features by using the dense neural network to output the third target feature map includes:

[0020] The image pixel features are processed by using the first dense processing block to output the fourth target feature map;

[0021] The fourth target feature map is processed by using the convolutional pooling layer to output the fifth target feature map;

[0022] The fifth target feature map is processed by using the second dense processing block to output the third target feature map.

[0023] According to an embodiment of the present disclosure, the dense processing block includes i hole convolutional layers.

[0024] wherein any of the first dense processing block and the second dense processing block is configured to process an input feature map in the following manner:

[0025] in a case that i = 1, processing the input feature map by using the ith dilated convolution layer to obtain a first intermediate feature, wherein the input feature map comprises the image pixel feature or the fifth target feature map;

[0026] in a case that i ≠ 1, processing the fusion feature by using the ith dilated convolution layer to obtain a second intermediate feature, wherein the second intermediate feature comprises the fourth target feature map or the third target feature map, and the fusion feature is generated according to the image pixel feature and a plurality of intermediate features output by the previous i-1 dilated convolution layers.

[0027] According to an embodiment of the present disclosure, the processing of the first target feature map output by the nth sub-encoder by using the attention mechanism layer in the image segmentation model to obtain the second target feature map comprises:

[0028] processing the first target feature map output by the nth sub-encoder by using a plurality of pooling layers to obtain a plurality of third intermediate features;

[0029] for each of the third intermediate features, performing convolution processing on the third intermediate feature by using a first convolution layer to obtain a fourth intermediate feature;

[0030] performing up-sampling processing on a plurality of the fourth intermediate features by using an up-sampling layer to obtain a fifth intermediate feature;

[0031] generating a fusion intermediate feature according to the fifth intermediate feature and the first feature map output by the nth sub-encoder;

[0032] processing the fusion intermediate feature by using a compression excitation layer to obtain the second target feature map.

[0033] According to an embodiment of the present disclosure, the processing of the fusion intermediate feature by using the compression excitation layer to obtain the second target feature map comprises:

[0034] processing the fusion intermediate feature by using a second convolution layer to obtain a sixth intermediate feature;

[0035] performing global pooling processing on the sixth intermediate feature by using a squeeze function layer to obtain an initial global information vector;

[0036] performing excitation operation on the initial global information vector by using an excitation layer to obtain a target global information vector, wherein the target global information vector comprises a plurality of channel data and a weight corresponding to each of the channel data;

[0037] Based on the plurality of weights, the target global information vector and the sixth intermediate feature are scaled to obtain a seventh intermediate feature;

[0038] The seventh intermediate feature is upsampled to obtain the second target feature map.

[0039] According to an embodiment of the present disclosure, the plurality of convolutional neural networks are connected in sequence, and the number of convolutional neural networks is the same as the number of sub-encoders;

[0040] The processing of the first target feature map and the second target feature map in each feature group by the convolutional neural network in the image segmentation model obtains a segmentation image, including:

[0041] In the case of m = 1, the second target feature map in the feature group and the first target feature map output by the n-th sub-encoder are processed by the m-th convolutional neural network to obtain an intermediate output feature map;

[0042] In the case of m ≠ 1, the intermediate output feature map output by the (m-1)-th convolutional neural network and the first target feature map output by the (n-m+1)-th sub-encoder are processed by the m-th convolutional neural network to generate an intermediate output feature map, wherein when m is the maximum value, the intermediate output feature map represents the segmentation image.

[0043] According to an embodiment of the present disclosure, the image segmentation model is obtained by the following method:

[0044] An image training set is obtained, wherein the image training set includes a plurality of training images containing objects to be cut and at least one segmentation label corresponding to each training image;

[0045] The training images are processed by a feature encoder to obtain n first training feature maps carrying global context information, wherein the feature encoder includes n sub-encoders connected in sequence, and each sub-encoder outputs one first training feature map;

[0046] The first training feature map output by the n-th sub-encoder is processed by an attention mechanism layer to obtain a second training feature map, wherein one first training feature map and one second training feature map form a training feature group;

[0047] The first training feature map and the second training feature map in each training feature group are processed by a convolutional neural network to obtain a training segmentation image;

[0048] The training segmentation image and the segmentation label are input into a loss function to output a loss result;

[0049] According to the loss result, the network parameters of the initial segmentation model are iteratively adjusted to generate a trained image segmentation model, wherein the initial segmentation model comprises the feature encoder, the attention mechanism layer, and the convolutional neural network.

[0050] According to an embodiment of the present disclosure, the training of the image segmentation model further comprises:

[0051] According to the plurality of training segmentation images and the segmentation labels corresponding to each training segmentation image, a segmentation accuracy of the image segmentation model is generated.

[0052] In a case where the segmentation accuracy is less than a preset accuracy threshold, the image segmentation model is iteratively trained using the image training set to obtain a retrained image segmentation model.

[0053] According to an embodiment of the present disclosure, the training of the image segmentation model further comprises:

[0054] According to the plurality of training segmentation images and the segmentation labels corresponding to each training segmentation image, a classification accuracy and a recall rate of the image segmentation model are respectively calculated.

[0055] According to the classification accuracy and the recall rate, a balance score is generated.

[0056] In a case where the balance score is less than a preset score threshold, the image segmentation model is iteratively trained using the image training set to obtain a retrained image segmentation model.

[0057] A second aspect of the present disclosure provides an image segmentation device, comprising:

[0058] A first obtaining module is configured to obtain an initial image to be segmented, wherein the initial image comprises at least one object to be segmented.

[0059] A feature mapping module is configured to perform feature mapping processing on the initial image using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map.

[0060] An attention module is configured to process the first target feature map output by the nth sub-encoder using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group.

[0061] a convolution module configured to process the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmentation image, wherein the segmentation image includes at least one object to be segmented.

[0062] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

[0063] A fourth aspect of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method described above.

[0064] A fifth aspect of the present disclosure further provides a computer program product comprising a computer program that, when executed by a processor, implements the method described above.

[0065] According to embodiments of the present disclosure, by processing the initial image to be segmented by using the plurality of sub-encoders in the feature encoder, the n first target feature maps carrying global context information can be obtained, by processing the first target feature map output by the nth sub-encoder by using the attention mechanism layer, the second target feature map with higher attention can be obtained, and by processing the plurality of feature groups by using the convolutional neural network, the segmentation image with higher accuracy can be obtained, thereby avoiding the problem of low image segmentation accuracy caused by not considering the global context information carried by each feature map in the related art, and reducing the usage rate of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0066] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0067] Figure 1 An application scenario diagram of the image segmentation method according to embodiments of the present disclosure is schematically shown;

[0068] Figure 2 A flowchart of the image segmentation method according to embodiments of the present disclosure is schematically shown;

[0069] Figure 3 A structure diagram of the image segmentation model according to embodiments of the present disclosure is schematically shown;

[0070] Figure 4 A structure diagram of the sub-encoder according to embodiments of the present disclosure is schematically shown;

[0071] Figure 5A structural diagram of an attention mechanism layer according to an embodiment of the present disclosure is schematically shown.

[0072] Figure 6 A flowchart of a training method of an image segmentation model according to an embodiment of the present disclosure is schematically shown.

[0073] Figure 7 A structural block diagram of an image segmentation device according to an embodiment of the present disclosure is schematically shown.

[0074] Figure 8 A structural block diagram of a training device of an image segmentation model according to an embodiment of the present disclosure is schematically shown; and

[0075] Figure 9 A block diagram of an electronic device adapted to implement the training method of the image segmentation model or the image segmentation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0076] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it is to be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and techniques have been omitted in order to avoid obscuring the concepts of the present disclosure.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the present disclosure. As used herein, the term "includes" and "comprises" and the like means the inclusion or presence of the stated features, steps, operations, and / or components, but not the exclusion or absence of one or more other features, steps, operations, or components.

[0078] All terms used herein, including technical and scientific terms, have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or overly formal way.

[0079] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally construed that the meaning of the expression is the same as that of the expression "one or more of A, B, and C" (e.g., "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0080] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of data (such as, but not limited to, user personal information) comply with relevant laws and regulations, necessary security measures are taken, and the public order and good customs are not violated.

[0081] Embodiments of the present disclosure provide an image segmentation method, an image segmentation device, an electronic device and a storage medium. The method comprises: obtaining an initial image to be segmented, wherein the initial image comprises at least one object to be segmented; performing feature mapping processing on the initial image by using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map; processing the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group; processing the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmented image, wherein the segmented image comprises at least one object to be segmented.

[0082] Figure 1 An application scenario diagram of the image segmentation method according to an embodiment of the present disclosure is schematically shown.

[0083] As shown in Figure 1 According to the application scenario 100 of the embodiment, the first terminal device 101, the second terminal device 102, the third terminal device 103, the network 104 and the server 105 are included. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0084] A user can use at least one of the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0085] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0086] The server 105 can be a server providing various services, for example, a background management server (only for example) providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server can perform analysis and the like on received user requests and the like, and feed back the processing result (for example, a webpage, information, or data, or the like obtained or generated according to the user request) to the terminal device.

[0087] It should be noted that the image segmentation method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the image segmentation apparatus provided in the embodiments of the present disclosure can generally be arranged in the server 105. The image segmentation method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the image segmentation apparatus provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0088] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0089] Figure 2 An illustrative flowchart of an image segmentation method according to an embodiment of the present disclosure is shown. Figure 3 An illustrative structural diagram of an image segmentation model according to an embodiment of the present disclosure is shown.

[0090] As Figure 2 shown, the image segmentation method of this embodiment includes operation S210 to operation S240.

[0091] In operation S210, an initial image to be segmented is obtained, wherein the initial image includes at least one object to be segmented;

[0092] In operation S220, the initial image is subjected to feature mapping processing using a feature encoder in the image segmentation model, to obtain n first target feature maps carrying global context information, wherein the feature encoder includes n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map;

[0093] In operation S230, the first target feature map output by the n-th sub-encoder is processed by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group.

[0094] In operation S240, the first target feature map and the second target feature map in each feature group are processed by using a convolutional neural network in the image segmentation model to obtain a segmentation image, wherein the segmentation image includes at least one object to be segmented.

[0095] According to an embodiment of the present disclosure, the initial image to be segmented can be obtained by using a common image acquisition device.

[0096] According to an embodiment of the present disclosure, the attention mechanism layer can be a feature pyramid attention mechanism layer.

[0097] According to an embodiment of the present disclosure, referring to Figure 3 In image segmentation, the initial image to be segmented (i.e., Input in Figure 3 is processed by using a feature encoder composed of n sub-encoders connected in sequence for feature mapping, so that each sub-encoder outputs a first target feature map, and the first target feature map output by the last sub-encoder is processed by using an attention mechanism layer to realize feature weight addition to pay more attention to features in the target region, thereby obtaining a second target feature map with enhanced pixel receptive field.

[0098] According to an embodiment of the present disclosure, the first target feature map and the second target feature map in each feature group are processed by using multiple convolutional neural networks (CNNs), thereby obtaining at least one segmentation image (i.e., Output in Figure 3 ), and each segmentation image includes at least one object to be segmented.

[0099] According to an embodiment of the present disclosure, the initial image to be segmented is processed by using multiple sub-encoders in the feature encoder, thereby obtaining n first target feature maps carrying global context information, the first target feature map output by the n-th sub-encoder is processed by using an attention mechanism layer, thereby obtaining a second target feature map with higher attention, and multiple feature groups are processed by using convolutional neural networks, thereby obtaining a segmentation image with higher accuracy, thereby avoiding the problem of low image segmentation accuracy caused by not considering the global context information carried by each feature map in the related art, and reducing the usage rate of computing resources.

[0100] Figure 4 A structural diagram of a sub-encoder according to an embodiment of the present disclosure is schematically shown.

[0101] According to an embodiment of the present disclosure, the initial image is subjected to feature mapping processing by using a feature encoder in the image segmentation model to obtain n first target feature maps carrying global context information, including the following operations:

[0102] For each sub-encoder, the image pixel features in the input image are extracted by using the first convolutional layer, wherein the input image includes the initial image or the first target feature map output by the sub-encoder connected to the input end of the sub-encoder;

[0103] The image pixel features are processed by using the dense neural network to output a third target feature map;

[0104] The third target feature map is input into the random inactivation layer to output the first target feature map corresponding to the sub-encoder.

[0105] According to an embodiment of the present disclosure, referring to Figure 3 and Figure 4 , when the input image is the initial image, the first sub-encoder processes the initial image at this time, the first convolutional layer in the first sub-encoder extracts the image pixel features of the initial image, and then the image pixel features are input into the dense neural network DenseNet for processing to obtain a third target feature map. The third target feature map is input into the random inactivation layer DropOut to output the first target feature map corresponding to the sub-encoder.

[0106] According to an embodiment of the present disclosure, the role of the dense neural network DenseNet is to make the output third target feature map contain as much feature information as possible, so that the subsequent convolutional neural network can obtain a segmentation image with better segmentation effect when performing image segmentation.

[0107] According to an embodiment of the present disclosure, since the dense neural network DenseNet realizes feature reuse by superimposing the feature maps obtained by the current layer and all previous layers, the captured pixel features will be too complex, and the calculation result is prone to overfitting. Therefore, before subsequent processing (such as before inputting into the attention mechanism layer or before inputting into the next sub-encoder), the third target feature map output by the dense neural network is input into a random inactivation layer DropOut for regularization processing to reduce the complexity of the features in the first target feature map, thereby improving the accuracy of the first target feature map.

[0108] According to an embodiment of the present disclosure, referring to Figure 4 , before the dense neural network processes the image pixel features, the following operations are further included:

[0109] The image pixel features are processed by using the batch normalization layer to obtain normalized image pixel features, so that the dense neural network processes the normalized image pixel features;

[0110] According to an embodiment of the present disclosure, after inputting the third target feature map into the random inactivation layer, the following operation is further included:

[0111] The first target feature map output by the random inactivation layer is input into a max pooling layer, and a pooled first target feature map is output, so as to input the pooled first target feature map into an attention mechanism layer or a sub-encoder connected to an output end of the sub-encoder.

[0112] According to an embodiment of the present disclosure, in order to speed up the image segmentation, a batch normalization (BN) layer can be used to process the image pixel features, so that the normalized image pixel features are input into the dense neural network.

[0113] According to an embodiment of the present disclosure, the max pooling layer MaxPooling is used to extract main feature information in the first target feature map, and the obtained pooled first target feature map is passed to the next sub-encoder or the attention mechanism layer for further processing.

[0114] According to an embodiment of the present disclosure, the dense neural network includes a first dense processing block and at least one second dense processing block and a convolutional pooling layer arranged between any two adjacent dense processing blocks.

[0115] According to an embodiment of the present disclosure, referring to Figure 4 , the image pixel features are processed by the dense neural network to output a third feature map, including the following operations:

[0116] The image pixel features are processed by the first dense processing block to output a fourth target feature map;

[0117] The fourth target feature map is processed by the convolutional pooling layer to output a fifth target feature map;

[0118] The fifth target feature map is processed by the second dense processing block to output the third target feature map.

[0119] According to an embodiment of the present disclosure, the convolutional pooling layer includes at least one convolutional layer Conv, at least one batch normalization layer BN, and at least one pooling layer Pooling.

[0120] According to an embodiment of the present disclosure, in order to avoid that the convolutional neural network will seriously lose pixel information as the convolution extends, the dense neural network composed of two dense processing blocks is used to realize the retention and reuse of pixel feature information.

[0121] According to an embodiment of the present disclosure, the first dense processing block outputs a fourth target feature map by reusing pixel feature information in the process of processing image pixel features, the fourth target feature map is processed by a convolution pooling layer to output a fifth target feature map, and the second dense processing block reuses pixel feature information to process the fifth target feature map to obtain the third target feature map.

[0122] According to an embodiment of the present disclosure, the dense neural network composed of two dense processing blocks is used to realize the retention and reuse of pixel feature information, and the third target feature map generated after the reuse is helpful to improve the segmentation effect of the segmented image.

[0123] According to an embodiment of the present disclosure, referring to Figure 4 , the dense processing block includes i dilated convolution layers.

[0124] According to an embodiment of the present disclosure, any one of the first dense processing block and the second dense processing block processes the input feature map in the following manner:

[0125] In the case of i = 1, the input feature map is processed by the i th dilated convolution layer to obtain a first intermediate feature, wherein the input feature map includes image pixel features or the fifth target feature map;

[0126] In the case of i ≠ 1, the fusion feature is processed by the i th dilated convolution layer to obtain a second intermediate feature, wherein the second intermediate feature includes the fourth target feature map or the third target feature map, and the fusion feature is generated according to the image pixel features and the intermediate features output by the previous i-1 dilated convolution layers.

[0127] According to an embodiment of the present disclosure, since the processing procedures of the first dense processing block and the second dense processing block are the same, the processing procedure of the first dense processing block is exemplarily described.

[0128] According to an embodiment of the present disclosure, the first intermediate feature is output after the first dilated convolution layer processes the image pixel features, the first intermediate feature output by the first dilated convolution layer and the image pixel features are processed by the second dilated convolution layer at the same time, and the first intermediate feature corresponding to the second dilated convolution layer is output, the two first intermediate features output by the first dilated convolution layer and the second dilated convolution layer and the image pixel features are processed by the third dilated convolution layer at the same time, and the first intermediate feature corresponding to the third dilated convolution layer is output, and so on, the i-1 first intermediate features (the i-1 first intermediate features constitute the fusion feature) output by the previous i-1 dilated convolution layers and the image pixel features are processed by the i th dilated convolution layer at the same time, and the second intermediate feature corresponding to the i th dilated convolution layer is output.

[0129] According to embodiments of this disclosure, the main advantages of the above-described dense processing block structure are enhanced feature propagation, encouragement of feature reuse, and a significant reduction in the number of parameters. Simultaneously, while extracting features within the dense processing block, it can increase the receptive field of the pixel. Let l represent the current dilated convolutional layer receiving all the preceding feature maps, such as image pixel features and multiple intermediate features output from the first i-1 dilated convolutional layers, denoted as x0, x1, ..., x... l-1 Then the second intermediate feature x output by the current dilated convolutional layer l As shown in formula (2).

[0130] x l =H l ([x0,x1,…,x l-1 (1)

[0131] Among them, H l The dilated convolution operation is represented by formula (1), which represents the cascade of feature maps.

[0132] According to embodiments of this disclosure, the dilated convolutional layer includes at least one batch normalization layer (BN), a dilated convolutional neural network (AtrousCNN), and an activation function (ReLU).

[0133] It should be noted that this disclosure only sets two dense processing blocks. The reason is that setting too many dense processing blocks does not improve the segmentation effect of the image, but may lead to overfitting. Therefore, the number of dense processing blocks needs to be set according to the actual needs. However, when setting multiple dense processing blocks, it is preferable to connect the two dense processing blocks through a convolutional pooling layer.

[0134] Figure 5 A schematic diagram of the attention mechanism layer according to an embodiment of the present disclosure is shown.

[0135] According to embodiments of this disclosure, such as Figure 5 As shown, the first target feature map output by the nth sub-encoder is processed using the attention mechanism layer in the image segmentation model to obtain the second target feature map, including the following operations:

[0136] Multiple pooling layers are used to process the first target feature map output by the nth sub-encoder to obtain multiple third intermediate features;

[0137] For each third intermediate feature, the first convolutional layer is used to perform convolution processing on the third intermediate feature to obtain the fourth intermediate feature;

[0138] The fifth intermediate feature is obtained by upsampling multiple fourth intermediate features using an upsampling layer.

[0139] Based on the fifth intermediate feature and the first feature map output by the nth sub-encoder, a fused intermediate feature is generated;

[0140] The fusion intermediate feature is processed by using a compression excitation layer to obtain a second target feature map.

[0141] According to an embodiment of the present disclosure, the first target feature map output by the nth sub-encoder is simultaneously input into four pooling layers Pool using 1x1, 2x2, 3x3 and 6x6 different scale sizes respectively to perform a pooling operation in parallel to enhance the pixel receptive field, thereby obtaining a plurality of different third intermediate features. The third intermediate features are processed by using a first convolution layer Conv to obtain fourth intermediate features.

[0142] According to an embodiment of the present disclosure, the plurality of fourth intermediate features are processed by using an up-sampling layer Upsample to obtain fifth intermediate features, and the first feature map output by the nth sub-encoder is processed by using a convolution layer to obtain an output intermediate feature. The output intermediate feature and the fifth intermediate features are fused to obtain a fusion intermediate feature.

[0143] According to an embodiment of the present disclosure, it can be known from the above that the convolution operation of different convolution kernels is a kind of local receptive field convolution work of guiding pixels on a single channel by using a convolution kernel. By modeling the mutual dependence between feature map channels and the importance of different channels, the learning of convolution characteristics can be enhanced, and these information can be utilized through subsequent conversion. Therefore, the fusion intermediate feature is processed by using a compression excitation layer (Squeeze-Excitation, SE) to realize the sharing, conversion and interaction ability of information between channels, thereby obtaining a second target feature map.

[0144] According to an embodiment of the present disclosure, as shown in Figure 5 The fusion intermediate feature is processed by using a compression excitation layer to obtain a second target feature map, including the following operations:

[0145] The fusion intermediate feature is processed by using a second convolution layer to obtain a sixth intermediate feature;

[0146] The sixth intermediate feature is processed by using a squeeze function layer to obtain an initial global information vector;

[0147] The initial global information vector is processed by using an excitation layer to obtain a target global information vector, wherein the target global information vector includes a plurality of channel data and a weight corresponding to each channel data;

[0148] Based on the plurality of weights, the target global information vector and the sixth intermediate feature are processed by using a scaling operation to obtain a seventh intermediate feature;

[0149] The seventh intermediate feature is processed by using an up-sampling operation to obtain a second target feature map.

[0150] According to an embodiment of the present disclosure, the fusion intermediate feature can be expressed as Input SE , and the second convolution layer performs a convolution operation on the fusion intermediate feature, as shown in formula (2):

[0151] F c :Input SE →U,Input SE ∈R H×W×C ,U∈R H′×W′×C′ (2)

[0152] wherein F c represents a convolution operation, and V=[V1, V2, …, V i ] represents a set of filters in the second convolution layer. V i represents the i-th convolution kernel. The output of the convolution operation is expressed as the sixth intermediate feature U, as shown in formula (3) and formula (4):

[0153] U=[U1,U2,…,U j ] (3)

[0154]

[0155] wherein × represents a convolution operator, U j ∈R H ′×W′ . V s represents that each convolution kernel acts on the U j corresponding channel feature map.

[0156] According to an embodiment of the present disclosure, the fusion intermediate feature is processed by the second convolution layer to obtain the sixth intermediate feature U, wherein the length, width and height of the fusion intermediate feature are C×W×H, and the length, width and height of the sixth intermediate feature U are C′×W′×H′. The sixth intermediate feature U is taken as the input of the squeeze function layer to perform a global pooling operation to obtain an initial global information vector Z. The fusion intermediate feature includes a plurality of sub-features, and each sub-feature is processed by the global pooling operation according to formula (5) to obtain a sub-vector Z m corresponding to each sub-feature. Formula (5) is exemplarily described with respect to the sub-feature C′, and the global pooling operation of other sub-features is modified correspondingly with reference to formula (5).

[0157]

[0158] wherein F sq represents a squeeze function. The initial global information vector Z=[Z1,...,Z C′ ], Z∈R 1×1×C′According to an embodiment of the present disclosure, the initial global information vector is subjected to an excitation operation as shown in formula (6) by using the excitation layer, to obtain a target global information vector S.

[0159] S=F ex (Z,W)=σ(W2δ(W1Z)) (6)

[0160] wherein F ex represents an excitation function.

[0161] According to an embodiment of the present disclosure, the excitation layer can obtain a target global information vector S=[S1,...,S C′ ] of a 1×1×C′ tensor, so as to calculate the weight of c′ channel data, that is, the importance of different channel data is screened, wherein the excitation layer at least includes one global pooling layer and two full connection layers, wherein the activation functions of the global pooling layer and the full connection layer are σ and δ respectively, W1 and W2 are weights, σ is a sigmoid function, and δ is a ReLu function. At this time, different weights have been generated for each channel data, and the target global information vector S and the sixth intermediate feature U are proportionally scaled as shown.

[0162] Y=F scale (U j ,S k )=S k U k (7)

[0163] wherein j and k are one-to-one corresponding, that is, the 1×1 channel data with weight is multiplied by the corresponding H′×W′ channel data, and j, k∈[1, C′].

[0164] According to an embodiment of the present disclosure, the seventh intermediate feature Y with enhanced receptive field and carrying context information is gradually up-sampled to obtain a second target feature map.

[0165] According to an embodiment of the present disclosure, referring to Figure 3 , a plurality of convolutional neural networks are connected in sequence, and the number of convolutional neural networks is the same as the number of sub-encoders.

[0166] According to an embodiment of the present disclosure, the convolutional neural network in the image segmentation model is used to process the first target feature map and the second target feature map in each feature group to obtain a segmented image, including the following operations:

[0167] In the case of m=1, the second target feature map in the feature group and the first target feature map output by the n-th sub-encoder are processed by the m-th convolutional neural network to obtain an intermediate output feature map.

[0168] In the case of m≠1, the intermediate output feature map is generated by processing the intermediate output feature map output by the (m-1)th convolutional neural network and the first target feature map output by the (n-m+1)th sub-encoder, wherein the intermediate output feature map represents the segmentation image when m is the maximum value.

[0169] According to an embodiment of the present disclosure, an example is exemplarily illustrated with m=n=4, in the case of m=1, i.e., the first convolutional neural network processes the first target feature map output by the fourth sub-encoder and the second target feature map output by the attention mechanism layer, and the first convolutional neural network outputs the intermediate output feature map.

[0170] According to an embodiment of the present disclosure, in the case of m=2, i.e., the second convolutional neural network processes the first target feature map output by the third (n=4, m=2, i.e., n-m+1=3) sub-encoder and the intermediate output feature map output by the first (m=2, m-1=1) convolutional neural network, and the second convolutional neural network outputs the intermediate output feature map.

[0171] According to an embodiment of the present disclosure, in the case of m=3, i.e., the third convolutional neural network processes the first target feature map output by the second (n=4, m=3, i.e., n-m+1=2) sub-encoder and the intermediate output feature map output by the second (m=3, m-1=2) convolutional neural network, and the third convolutional neural network outputs the intermediate output feature map.

[0172] According to an embodiment of the present disclosure, in the case of m=4, i.e., the fourth convolutional neural network processes the first target feature map output by the first (n=4, m=4, i.e., n-m+1=1) sub-encoder and the intermediate output feature map output by the third (m=4, m-1=3) convolutional neural network, and the fourth convolutional neural network outputs the intermediate output feature map, which is the segmentation image.

[0173] Figure 6 A flowchart of a training method of an image segmentation model according to an embodiment of the present disclosure is schematically shown.

[0174] As shown in Figure 6 The image segmentation model of this embodiment is obtained by training through operation S610 to operation S660.

[0175] In operation S610, an image training set is obtained, wherein the image training set includes a plurality of training images containing an object to be cut and at least one segmentation label corresponding to each training image;

[0176] In operation S620, the training image is processed by a feature encoder to obtain n first feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first feature map.

[0177] In operation S630, the first feature map output by the n th sub-encoder is processed by an attention mechanism layer to obtain a second feature map, wherein one first feature map and one second feature map form a feature group.

[0178] In operation S640, the first feature map and the second feature map in each feature group are processed by a convolutional neural network to obtain a training segmentation image.

[0179] In operation S650, the training segmentation image and the segmentation label are input into a loss function to output a loss result.

[0180] In operation S660, the network parameters of the initial segmentation model are iteratively adjusted according to the loss result to generate a trained image segmentation model, wherein the initial segmentation model comprises the feature encoder, the attention mechanism layer, and the convolutional neural network.

[0181] According to an embodiment of the present disclosure, the training image includes a bill or a bank card, and the training image can be obtained by using any image acquisition device, such as a camera, a video camera, or a remote sensing satellite.

[0182] According to an embodiment of the present disclosure, the loss function can be a cross-entropy function.

[0183] According to an embodiment of the present disclosure, referring to Figure 3 , during training, the feature encoder composed of n sub-encoders connected in sequence is used to perform feature mapping processing on the training image (i.e. Figure 3 Input), so that each sub-encoder outputs one first feature map, and the attention mechanism layer is used to process the first feature map output by the last sub-encoder to realize the addition of feature weights so that the image segmentation model pays more attention to the features in the target region, thereby obtaining a second feature map with enhanced pixel receptive field.

[0184] According to an embodiment of the present disclosure, the first feature map and the second feature map in each feature group are processed by a convolutional neural network CNN to obtain a training segmentation image (i.e. Figure 3 Output), the training segmentation image and the segmentation label are input into a loss function to output a loss result, and the network parameters of the initial segmentation model are iteratively adjusted based on the loss result to generate a trained image segmentation model.

[0185] According to an embodiment of the present disclosure, by processing the training image by using the plurality of sub-encoders in the feature encoder, n first training feature maps carrying global context information can be obtained, by processing the first training feature map output by the nth sub-encoder by using the attention mechanism layer, a second training feature map with higher attention can be obtained, and by processing the plurality of training feature groups by using the convolutional neural network, a training segmentation image with higher accuracy can be obtained, thereby avoiding the problem of low image segmentation accuracy caused by not considering the global context information carried by each feature map in the related art, and reducing the usage rate of computing resources.

[0186] According to an embodiment of the present disclosure, the training method of the image segmentation model further includes the following operations:

[0187] According to the plurality of training segmentation images and the segmentation label corresponding to each training segmentation image, the segmentation accuracy of the image segmentation model is generated;

[0188] In the case where the segmentation accuracy is less than the preset accuracy threshold, the image segmentation model is iteratively trained by using the image training set to obtain a retrained image segmentation model.

[0189] According to an embodiment of the present disclosure, the preset accuracy threshold can be set according to actual conditions, for example, it can be 95%.

[0190] According to an embodiment of the present disclosure, in order to evaluate the performance of the proposed model, MeanIoU is used to evaluate the segmentation accuracy of semantic segmentation, as shown in formula (8):

[0191]

[0192] Wherein, k represents the total number of classes to be segmented, P ii represents the total number of pixels belonging to class i but predicted as class i, P ji represents the total number of pixels belonging to class i but predicted as class j.

[0193] According to an embodiment of the present disclosure, in the case where the segmentation accuracy MeanIoU is less than the preset accuracy threshold, the image segmentation model is iteratively trained by using the image training set to obtain a retrained image segmentation model.

[0194] According to an embodiment of the present disclosure, the accuracy of the image segmentation model is distinguished by the segmentation accuracy MeanIoU to improve the segmentation effect of the image segmentation model, so that the segmentation accuracy of the finally used image segmentation model is higher.

[0195] According to an embodiment of the present disclosure, the training method of the image segmentation model further includes the following operations:

[0196] According to the plurality of training segmentation images and the segmentation label corresponding to each training segmentation image, a classification accuracy and a recall rate of the image segmentation model are respectively calculated;

[0197] According to the classification accuracy and the recall rate, a balance score is generated;

[0198] In a case where the balance score is less than a preset score threshold, the image segmentation model is iteratively trained by using the image training set, and a retrained image segmentation model is obtained.

[0199] According to an embodiment of the present disclosure, the classification accuracy Accuracy is as shown in formula (9):

[0200]

[0201] Wherein, the meanings of TP, TN, FP and FN are as shown in Table 1.

[0202] Table 1

[0203]

[0204] Wherein, positive and negative can respectively refer to two classification results, for example, positive and negative in training classification can respectively refer to whether there is an object to be segmented in the training segmentation image, such as a bill or a bank card, and similarly, positive and negative in label classification can respectively refer to whether there is an object to be segmented based on the segmentation label. TP, FN, FP and TN are respectively represented by letters corresponding to numerical values.

[0205] According to an embodiment of the present disclosure, a comparative experiment is performed on an existing image segmentation method (such as U-Net and FCN16s) and the image segmentation model of the present disclosure by using a comparative data set including a plurality of verification images for the comparative experiment and a verification label corresponding to each verification image, and a comparative result as shown in Table 2 is obtained.

[0206] Table 2

[0207] Method Acc (%) MeanIoU (%) U-Net 95.60 86.31 FCN16s 91.70 72.73 The present disclosure 97.47 91.59

[0208] As can be seen from Table 2, the segmentation accuracy and the classification accuracy of the image segmentation model of the present disclosure are both better than those of the existing image segmentation method.

[0209] In an alternative embodiment, the classification accuracy Precision (P) is as shown in formula (9):

[0210]

[0211] According to an embodiment of the present disclosure, the recall rate Recall (R) is as shown in formula (10):

[0212]

[0213] According to an embodiment of the present disclosure, the balanced score F1-score is generated according to the classification accuracy and the recall rate i As shown in formula (11):

[0214]

[0215] According to an embodiment of the present disclosure, in the case that the balanced score is less than a preset score threshold, the image segmentation model is iteratively trained by using the image training set to obtain a retrained image segmentation model, wherein the preset score threshold can be specifically set according to actual requirements, for example, can be 95%.

[0216] According to an embodiment of the present disclosure, the existing image segmentation method and the image segmentation model of the present disclosure are compared by using the contrast data set again, and the comparison results shown in Table 3 are obtained.

[0217] Table 3

[0218]

[0219] As shown in Table 2, the classification accuracy P, the recall rate R and the balanced score F1-score of the image segmentation model of the present disclosure i are all better than those of the existing image segmentation method.

[0220] Figure 7 The structural block diagram of the image segmentation device according to an embodiment of the present disclosure is schematically shown.

[0221] As shown in Figure 7 , the image segmentation device 700 comprises a first acquisition module 710, a feature mapping module 720, an attention module 730 and a convolution module 740.

[0222] The first acquisition module 710 is configured to acquire an initial image to be segmented, wherein the initial image comprises at least one object to be segmented;

[0223] The feature mapping module 720 is configured to perform feature mapping processing on the initial image by using a feature encoder in the image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map;

[0224] The attention module 730 is configured to process the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group.

[0225] The convolution module 740 is configured to process the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model, to obtain a segmentation image, wherein the segmentation image at least includes a to-be-segmented object.

[0226] According to an embodiment of the present disclosure, by processing the initial image to be segmented by using the plurality of sub-encoders in the feature encoder, the n first target feature maps carrying global context information can be obtained, by processing the first target feature map output by the nth sub-encoder by using the attention mechanism layer, the second target feature map with higher attention can be obtained, and by processing the plurality of feature groups by using the convolutional neural network, the segmentation image with higher accuracy can be obtained, thereby avoiding the problem of low image segmentation accuracy caused by not considering the global context information carried by each feature map in the related art, and reducing the usage rate of computing resources.

[0227] According to an embodiment of the present disclosure, the feature mapping module 720 includes a convolution extraction sub-module, a dense sub-module, and a random inactivation sub-module.

[0228] The convolution extraction sub-module is configured to extract image pixel features in an input image by using a first convolutional layer for each sub-encoder, wherein the input image includes an initial image or a first target feature map output by a sub-encoder connected to the input end of the sub-encoder;

[0229] The dense sub-module is configured to process the image pixel features by using a dense neural network to output a third target feature map;

[0230] The random inactivation sub-module is configured to input the third target feature map into a random inactivation layer to output a first target feature map corresponding to the sub-encoder.

[0231] According to an embodiment of the present disclosure, the feature mapping module 720 further includes a normalization sub-module and a first pooling sub-module.

[0232] The normalization sub-module is configured to process the image pixel features by using a batch normalization layer to obtain normalized image pixel features, so that the dense neural network processes the normalized image pixel features.

[0233] The first pooling sub-module is configured to input the first target feature map output by the random inactivation layer into a max-pooling layer to output a pooled first target feature map, so that the pooled first target feature map is input into the attention mechanism layer or a sub-encoder connected to the output end of the sub-encoder.

[0234] According to an embodiment of the present disclosure, the dense neural network includes a first dense processing block and at least one second dense processing block and a convolutional pooling layer arranged between any two adjacent dense processing blocks.

[0235] According to an embodiment of the present disclosure, the dense submodule comprises a first dense unit, a convolution pooling unit, and a second dense unit.

[0236] The first dense unit is configured to process the image pixel feature by using a first dense processing block, and output a fourth target feature map.

[0237] The convolution pooling unit is configured to process the fourth target feature map by using a convolution pooling layer, and output a fifth target feature map.

[0238] The second dense unit is configured to process the fifth target feature map by using a second dense processing block, and output a third target feature map.

[0239] According to an embodiment of the present disclosure, the dense processing block comprises i dilated convolution layers.

[0240] According to an embodiment of the present disclosure, any one of the first dense processing block and the second dense processing block is configured to process an input feature map by using a first processing subunit and a second processing subunit.

[0241] The first processing subunit is configured to, in a case where i = 1, process the input feature map by using the i-th dilated convolution layer to obtain a first intermediate feature, wherein the input feature map comprises the image pixel feature or the fifth target feature map.

[0242] The second processing subunit is configured to, in a case where i ≠ 1, process a fusion feature by using the i-th dilated convolution layer to obtain a second intermediate feature, wherein the second intermediate feature comprises the fourth target feature map or the third target feature map, and the fusion feature is generated according to the image pixel feature and a plurality of intermediate features output by the previous i-1 dilated convolution layers.

[0243] According to an embodiment of the present disclosure, the attention module 730 comprises a second pooling submodule, a first convolution submodule, an up-sampling submodule, a fusion submodule, and an excitation submodule.

[0244] The second pooling submodule is configured to process the first target feature map output by the n-th sub-encoder by using a plurality of pooling layers respectively to obtain a plurality of third intermediate features.

[0245] The first convolution submodule is configured to, for each third intermediate feature, perform convolution processing on the third intermediate feature by using a first convolution layer to obtain a fourth intermediate feature.

[0246] The up-sampling submodule is configured to perform up-sampling processing on the plurality of fourth intermediate features by using an up-sampling layer to obtain a fifth intermediate feature.

[0247] The fusion submodule is configured to generate a fusion intermediate feature according to the fifth intermediate feature and the first feature map output by the n-th sub-encoder.

[0248] The excitation submodule is configured to process the fused intermediate feature by using a compressed excitation layer to obtain a second target feature map.

[0249] According to an embodiment of the present disclosure, the excitation submodule comprises a convolution unit, a global pooling unit, an excitation unit, a scaling unit, and an up-sampling unit.

[0250] The convolution unit is configured to process the fused intermediate feature by using a second convolution layer to obtain a sixth intermediate feature.

[0251] The global pooling unit is configured to perform global pooling processing on the sixth intermediate feature by using a squeeze function layer to obtain an initial global information vector.

[0252] The excitation unit is configured to perform excitation operation on the initial global information vector by using an excitation layer to obtain a target global information vector, wherein the target global information vector comprises a plurality of channel data and a weight corresponding to each channel data.

[0253] The scaling unit is configured to perform scaling processing on the target global information vector and the sixth intermediate feature based on the plurality of weights to obtain a seventh intermediate feature.

[0254] The up-sampling unit is configured to perform up-sampling processing on the seventh intermediate feature to obtain the second target feature map.

[0255] According to an embodiment of the present disclosure, the plurality of convolutional neural networks are sequentially connected, and the number of convolutional neural networks is the same as the number of sub-encoders.

[0256] According to an embodiment of the present disclosure, the convolution module 740 comprises a second convolutional submodule and a third convolutional submodule.

[0257] The second convolutional submodule is configured to, in the case of m = 1, process the second target feature map in the feature group and the first target feature map output by the n-th sub-encoder by using the m-th convolutional neural network to obtain an intermediate output feature map.

[0258] The third convolutional submodule is configured to, in the case of m ≠ 1, process the intermediate output feature map output by the m-1-th convolutional neural network and the first target feature map output by the n-m+1-th sub-encoder by using the m-th convolutional neural network to generate an intermediate output feature map, wherein when m is the maximum value, the intermediate output feature map represents a segmentation image.

[0259] Figure 8 The structure block diagram of the training device of the image segmentation model according to an embodiment of the present disclosure is schematically shown.

[0260] As Figure 8As shown, the training device 800 of the image segmentation model of the embodiment includes a second acquisition module 810, a feature encoding module 820, a second attention module 830, a second convolution module 840, a loss module 850, and an iteration module 860.

[0261] The second acquisition module 810 is configured to acquire an image training set, wherein the image training set includes a plurality of training images containing objects to be cut and at least one segmentation label corresponding to each training image.

[0262] The feature encoding module 820 is configured to perform feature mapping processing on the training images by using a feature encoder to obtain n first feature maps carrying global context information, wherein the feature encoder includes n sub-encoders connected in sequence, and each sub-encoder outputs one first feature map.

[0263] The second attention module 830 is configured to process the first feature map output by the nth sub-encoder by using an attention mechanism layer to obtain a second feature map, wherein one first feature map and one second feature map form a feature group.

[0264] The second convolution module 840 is configured to process the first feature map and the second feature map in each feature group by using a convolutional neural network to obtain a training segmentation image.

[0265] The loss module 850 is configured to input the training segmentation image and the segmentation label into a loss function to output a loss result.

[0266] The iteration module 860 is configured to iteratively adjust network parameters of an initial segmentation model according to the loss result to generate a trained image segmentation model, wherein the initial segmentation model includes the feature encoder, the attention mechanism layer, and the convolutional neural network.

[0267] According to embodiments of the present disclosure, by processing the training images by using the plurality of sub-encoders in the feature encoder, n first feature maps carrying global context information can be obtained, by processing the first feature map output by the nth sub-encoder by using the attention mechanism layer, a second feature map with higher attention can be obtained, and by processing the plurality of feature groups by using the convolutional neural network, a segmentation image with higher accuracy can be obtained, thereby avoiding the problem of low image segmentation accuracy caused by not considering the global context information carried by each feature map in the related art, and reducing the usage rate of computing resources.

[0268] According to embodiments of the present disclosure, the training device 800 of the image segmentation model further includes an accuracy module and a second iteration module.

[0269] The accuracy module is configured to generate segmentation accuracy of the image segmentation model according to the plurality of training segmentation images and the segmentation label corresponding to each training segmentation image.

[0270] a second iteration module configured to iteratively train the image segmentation model using the image training set when the segmentation accuracy is less than the preset accuracy threshold, to obtain a retrained image segmentation model.

[0271] According to an embodiment of the present disclosure, the training apparatus 800 of the image segmentation model further comprises a calculation module, a generation module, and a third iteration module.

[0272] The calculation module is configured to calculate a classification accuracy and a recall rate of the image segmentation model respectively according to the plurality of training segmentation images and the segmentation labels corresponding to each training segmentation image.

[0273] The generation module is configured to generate a balance score according to the classification accuracy and the recall rate.

[0274] The third iteration module is configured to iteratively train the image segmentation model using the image training set when the balance score is less than a preset score threshold, to obtain a retrained image segmentation model.

[0275] According to an embodiment of the present disclosure, any one or more of the first acquisition module 710, the feature mapping module 720, the attention module 730, the convolution module 740, or the second acquisition module 810, the feature encoding module 820, the second attention module 830, the second convolution module 840, the loss module 850, and the iteration module 860 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 710, the feature mapping module 720, the attention module 730, the convolution module 740, or the second acquisition module 810, the feature encoding module 820, the second attention module 830, the second convolution module 840, the loss module 850, and the iteration module 860 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the first acquisition module 710, the feature mapping module 720, the attention module 730, the convolution module 740, or the second acquisition module 810, the feature encoding module 820, the second attention module 830, the second convolution module 840, the loss module 850, and the iteration module 860 can be at least partially implemented as a computer program module which can perform corresponding functions when the computer program module is run.

[0276] Figure 9 A block diagram of an electronic device suitable for implementing a training method of an image segmentation model or an image segmentation method according to an embodiment of the present disclosure is schematically shown.

[0277] As shown in Figure 9 The electronic device 900 according to an embodiment of the present disclosure includes a processor 901 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage portion 908. The processor 901 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), and / or the like. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0278] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to embodiments of the present disclosure by executing programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to embodiments of the present disclosure by executing programs stored in the one or more memories.

[0279] According to an embodiment of the present disclosure, the electronic device 900 can further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 can further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 908 including a hard disk, and the like; and a communication portion 909 including a network interface card such as a LAN card, a modem, and the like. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 910 as necessary, so that a computer program read therefrom is installed in the storage portion 908 as necessary.

[0280] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist independently without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0281] According to the embodiments of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the present disclosure, the computer readable storage medium can include one or more memories of the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903.

[0282] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the training method of the image segmentation model or the image segmentation method provided by the embodiments of the present disclosure.

[0283] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiments of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0284] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage medium, a magnetic storage medium, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal via a network medium, and be installed and executed through the communication part 909 and / or the detachable medium 911. The program codes contained in the computer program can be transmitted via any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0285] In such embodiments, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the detachable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the system, device, apparatus, module, unit, and the like described above can be implemented by computer program modules.

[0286] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0287] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the figures. For example, two blocks that are shown in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0288] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations fall within the scope of the present disclosure.

[0289] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. An image segmentation method, comprising: obtaining an initial image to be segmented, wherein the initial image comprises at least one object to be segmented; performing feature mapping processing on the initial image by using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each sub-encoder outputs one first target feature map; processing the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one first target feature map and one second target feature map form a feature group; processing the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmented image, wherein the segmented image comprises at least one object to be segmented.

2. The method of claim 1, wherein, The feature mapping processing on the initial image by using the feature encoder in the image segmentation model to obtain n first target feature maps carrying global context information comprises: for each sub-encoder, extracting image pixel features in an input image by using a first convolutional layer, wherein the input image comprises the initial image or a first target feature map output by a sub-encoder connected to the input end of the sub-encoder; processing the image pixel features by using a dense neural network to output a third target feature map; inputting the third target feature map into a random inactivation layer to output a first target feature map corresponding to the sub-encoder.

3. The method of claim 2, wherein, Before the dense neural network processes the image pixel features, further comprising: processing the image pixel features by using a batch normalization layer to obtain normalized image pixel features, so that the dense neural network processes the normalized image pixel features; wherein after inputting the third target feature map into the random inactivation layer, further comprising: inputting the first target feature map output by the random inactivation layer into a max-pooling layer to output a pooled first target feature map, so as to input the pooled first target feature map into the attention mechanism layer or a sub-encoder connected to the output end of the sub-encoder.

4. The method of claim 2, wherein the dense neural network comprises a first dense processing block and at least one second dense processing block and a convolutional pooling layer arranged between any two adjacent dense processing blocks; wherein The processing of the image pixel features by using the dense neural network to output a third target feature map comprises: processing the image pixel features by using the first dense processing block to output a fourth target feature map; processing the fourth target feature map by using the convolutional pooling layer to output a fifth target feature map; processing the fifth target feature map by using the second dense processing block to output the third target feature map.

5. The method of claim 4, wherein the dense processing block comprises i atrous convolutional layers; wherein any one of the first dense processing block and the second dense processing block processes an input feature map in the following manner: In the case of i=1, the input feature map is processed by the ith hollow convolutional layer to obtain a first intermediate feature, wherein the input feature map comprises the image pixel feature or the fifth target feature map; In the case of i≠1, the fusion feature is processed by the ith hollow convolutional layer to obtain a second intermediate feature, wherein the second intermediate feature comprises the fourth target feature map or the third target feature map, and the fusion feature is generated according to the image pixel feature and a plurality of intermediate features output by the previous i-1 hollow convolutional layers.

6. The method of claim 1, wherein, The processing of the first target feature map output by the nth sub-encoder by the attention mechanism layer in the image segmentation model to obtain the second target feature map comprises: A plurality of pooling layers are used to process the first target feature map output by the nth sub-encoder to obtain a plurality of third intermediate features; For each third intermediate feature, a first convolutional layer is used to perform convolutional processing on the third intermediate feature to obtain a fourth intermediate feature; An up-sampling layer is used to perform up-sampling processing on a plurality of fourth intermediate features to obtain a fifth intermediate feature; The fusion intermediate feature is generated according to the fifth intermediate feature and the first feature map output by the nth sub-encoder; The fusion intermediate feature is processed by a compression excitation layer to obtain the second target feature map.

7. The method of claim 6, wherein, The processing of the fusion intermediate feature by the compression excitation layer to obtain the second target feature map comprises: A second convolutional layer is used to process the fusion intermediate feature to obtain a sixth intermediate feature; A squeeze function layer is used to perform global pooling processing on the sixth intermediate feature to obtain an initial global information vector; An excitation layer is used to perform excitation operation on the initial global information vector to obtain a target global information vector, wherein the target global information vector comprises a plurality of channel data and a weight corresponding to each channel data; Based on a plurality of weights, the target global information vector and the sixth intermediate feature are proportionally scaled to obtain a seventh intermediate feature; The seventh intermediate feature is up-sampled to obtain the second target feature map.

8. The method of claim 1, wherein a plurality of convolutional neural networks are connected in sequence, and the number of convolutional neural networks is the same as the number of sub-encoders; wherein The processing of the first target feature map and the second target feature map in each feature group by the convolutional neural network in the image segmentation model to obtain a segmentation image comprises: In the case of m=1, the second target feature map in the feature group and the first target feature map output by the nth sub-encoder are processed by the mth convolutional neural network to obtain an intermediate output feature map; In the case of m≠1, the intermediate output feature map output by the m-1th convolutional neural network and the first target feature map output by the n-m+1th sub-encoder are processed by the mth convolutional neural network to generate an intermediate output feature map, wherein when m is the maximum value, the intermediate output feature map represents the segmentation image.

9. The method of claim 1, wherein, The image segmentation model is obtained by the following method: obtaining an image training set, wherein the image training set comprises a plurality of training images containing objects to be cut and at least one segmentation label corresponding to each of the training images; performing feature mapping processing on the training images by using a feature encoder to obtain n first training feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each of the sub-encoders outputs one of the first training feature maps; processing the first training feature map output by the nth sub-encoder by using an attention mechanism layer to obtain a second training feature map, wherein one of the first training feature maps and one of the second training feature maps form a training feature group; processing the first training feature map and the second training feature map in each of the training feature groups by using a convolutional neural network to obtain a training segmentation image; inputting the training segmentation image and the segmentation label into a loss function to output a loss result; iteratively adjusting network parameters of an initial segmentation model according to the loss result to generate a trained image segmentation model, wherein the initial segmentation model comprises the feature encoder, the attention mechanism layer and the convolutional neural network.

10. The method of claim 9, further comprising: generating segmentation accuracy of the image segmentation model according to a plurality of the training segmentation images and segmentation labels corresponding to each of the training segmentation images; in a case where the segmentation accuracy is less than a preset accuracy threshold, iteratively training the image segmentation model by using the image training set to obtain a retrained image segmentation model.

11. The method of claim 9, further comprising: respectively calculating classification accuracy and recall rate of the image segmentation model according to a plurality of the training segmentation images and segmentation labels corresponding to each of the training segmentation images; generating a balance score according to the classification accuracy and the recall rate; in a case where the balance score is less than a preset score threshold, iteratively training the image segmentation model by using the image training set to obtain a retrained image segmentation model.

12. An image segmentation device, comprising: a first obtaining module configured to obtain an initial image to be segmented, wherein the initial image comprises at least one object to be segmented; a feature mapping module configured to perform feature mapping processing on the initial image by using a feature encoder in an image segmentation model to obtain n first target feature maps carrying global context information, wherein the feature encoder comprises n sub-encoders connected in sequence, and each of the sub-encoders outputs one of the first target feature maps; an attention module configured to process the first target feature map output by the nth sub-encoder by using an attention mechanism layer in the image segmentation model to obtain a second target feature map, wherein one of the first target feature maps and one of the second target feature maps form a feature group; and a training module configured to perform training on the image segmentation model by using a plurality of the initial images and segmentation labels corresponding to each of the initial images to obtain a trained image segmentation model. The convolution module is configured to process the first target feature map and the second target feature map in each feature group by using a convolutional neural network in the image segmentation model to obtain a segmentation image, wherein the segmentation image includes at least one object to be segmented. 13.An electronic device, comprising: one or more processors; a memory device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method according to any one of claims 1-11. 14.A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-11. 15.A computer program product, comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images