Image segmentation method and device, and image segmentation model training method and device

Abstract

Provided are an image segmentation method and device, and an image segmentation model training method and device. The training method may include: obtaining a first image semantic feature of a sample image, the first image semantic feature corresponding to a first resolution; obtaining a semantic coding feature of the sample image based on the first image semantic feature; obtaining a second image semantic feature of the sample image based on the first image semantic feature, the second image semantic feature corresponding to a second resolution, the second resolution being higher than the first resolution; obtaining an image segmentation result of the sample image based on the semantic coding feature and the second image semantic feature; determining a training loss based on the difference between the image segmentation result and a preset image segmentation result, and obtaining an image segmentation model when the training loss meets a preset condition.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image segmentation method and an image segmentation device, and a training method and a training device for an image segmentation model. Background Art

[0002] Image salient object segmentation is a very important computer vision task, which has many applications in image retrieval, visual tracking, image editing, and film and television production. Image salient object segmentation refers to the extraction of salient areas in an image pixel by pixel by simulating human visual characteristics through intelligent algorithms.

[0003] With the rapid development of deep learning, deep neural networks are widely used in image salient object segmentation. The high-level semantic features extracted from deep networks can more accurately distinguish target objects and backgrounds from complex scenes, greatly improving the effect of salient object segmentation. Therefore, deep learning technology has become one of the mainstream technologies in this field. Although segmentation methods based on deep neural networks have made great progress, there is still a problem of low segmentation accuracy. Summary of the invention

[0004] The present disclosure provides an image segmentation method and an image segmentation device as well as an image segmentation model training method and a training device to at least solve the above problems. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for an image segmentation model is provided, and the training method may include: obtaining a first image semantic feature of a sample image, the first image semantic feature corresponding to a first resolution; obtaining a semantic coding feature of the sample image based on the first image semantic feature; obtaining a second image semantic feature of the sample image based on the first image semantic feature, the second image semantic feature corresponding to a second resolution, and the second resolution is higher than the first resolution; obtaining an image segmentation result of the sample image based on the semantic coding feature and the second image semantic feature; determining a training loss based on a difference between the image segmentation result and a preset image segmentation result, and obtaining the image segmentation model when the training loss meets a preset condition.

[0006] As an implementation manner, obtaining the first image semantic feature of the sample image may include: performing convolution processing and downsampling processing on the sample image to obtain the first image semantic feature of the sample image.

[0007] As an implementation mode, obtaining the semantic coding features of the sample image based on the semantic features of the first image may include: performing at least one of pooling processing, full connection processing and feature mapping processing on the semantic features of the first image to obtain the semantic coding features of a preset dimension.

[0008] As an implementation manner, obtaining the second image semantic feature of the sample image based on the first image semantic feature may include: performing convolution processing and upsampling processing on the first image semantic feature to obtain the second image semantic feature of the sample image.

[0009] As an implementation manner, obtaining the image segmentation result of the sample image based on the semantic coding feature and the second image semantic feature may include: performing convolution processing on the second image semantic feature based on the semantic coding feature to obtain the image segmentation result of the sample image.

[0010] As an implementation, performing convolution processing on the semantic features of the second image based on the semantic coding features may include: setting processing parameters corresponding to the sample image based on the semantic coding features; and performing convolution processing on the semantic features of the second image using the set processing parameters.

[0011] As an implementation mode, setting processing parameters corresponding to the sample image based on the semantic coding features may include: determining a preset dimension of the semantic coding features based on a preset number of channels of the semantic features of the second image and a preset number and preset size of convolution kernels used for the convolution processing; and setting each dimensional semantic coding feature of the preset dimensional semantic coding features as a convolution weight in the convolution kernel used for the convolution processing.

[0012] As an implementation method, performing convolution processing on the second image semantic feature using the set processing parameters may include: performing a first convolution processing on the second image semantic feature to obtain an image semantic feature having a first feature channel number, wherein the first feature channel number is less than the channel number of the second image semantic feature; performing a second convolution processing on the image semantic feature having the first feature channel number to obtain an image semantic feature having the first channel number; performing a third convolution processing on the image semantic feature having the first channel number to obtain an image semantic feature having a second channel number, wherein the second channel number is less than the first channel number.

[0013] According to a second aspect of an embodiment of the present disclosure, an image segmentation method is provided, which may include: obtaining an input image; inputting the input image into an image segmentation model to obtain an image segmentation result of the input image, wherein the image segmentation model is trained based on an image segmentation result obtained based on semantic coding features and second image semantic features of a sample image, the semantic coding features and the second image semantic features are obtained based on first image semantic features of the sample image, and a second resolution corresponding to the second image semantic feature is higher than the first resolution corresponding to the first image semantic feature.

[0014] As an implementation method, the input image is input into an image segmentation model to obtain an image segmentation result of the input image, which may include: performing convolution processing and downsampling processing on the input image to obtain the first image semantic feature of the input image; performing at least one of pooling processing, full connection processing and feature mapping processing on the first image semantic feature to obtain the semantic coding feature of a preset dimension; performing convolution processing and upsampling processing on the first image semantic feature to obtain the second image semantic feature of the input image; and performing convolution processing on the second image semantic feature based on the semantic coding feature to obtain the image segmentation result of the input image.

[0015] As an implementation method, performing convolution processing on the semantic features of the second image based on the semantic coding features may include: setting processing parameters corresponding to the input image based on the semantic coding features; and performing convolution processing on the semantic features of the second image using the set processing parameters.

[0016] As an implementation mode, setting processing parameters corresponding to the input image based on the semantic coding features may include: determining a preset dimension of the semantic coding features based on a preset number of channels of the semantic features of the second image and a preset number and preset size of convolution kernels used for the convolution processing; and setting each dimensional semantic coding feature of the preset dimensional semantic coding features as a convolution weight in the convolution kernel used for the convolution processing.

[0017] As an implementation method, performing convolution processing on the second image semantic feature using the set processing parameters may include: performing a first convolution processing on the second image semantic feature to obtain an image semantic feature having a first feature channel number, wherein the first feature channel number is less than the channel number of the second image semantic feature; performing a second convolution processing on the image semantic feature having the first feature channel number to obtain an image semantic feature having the first channel number; performing a third convolution processing on the image semantic feature having the first channel number to obtain an image semantic feature having a second channel number, wherein the second channel number is less than the first channel number.

[0018] According to a third aspect of an embodiment of the present disclosure, an image segmentation model device is provided, which may include: an encoder, configured to obtain a first image semantic feature of an input image, wherein the first image semantic feature corresponds to a first resolution; a first decoder, configured to obtain a second image semantic feature of the input image based on the first image semantic feature, wherein the second image semantic feature corresponds to a second resolution, and the second resolution is higher than the first resolution; an attention module, configured to obtain a semantic coding feature of the input image based on the first image semantic feature; and a second decoder, configured to obtain an image segmentation result of the input image based on the semantic coding feature and the second image semantic feature.

[0019] As an implementation manner, the encoder may be configured to: perform convolution processing and down-sampling processing on the input image to obtain the first image semantic feature of the input image.

[0020] As an implementation, the attention module may be configured to perform at least one of pooling processing, full connection processing, and feature mapping processing on the first image semantic features to obtain the semantic coding features of a preset dimension.

[0021] As an implementation manner, the first decoder may be configured to: perform convolution processing and up-sampling processing on the first image semantic feature to obtain the second image semantic feature of the input image.

[0022] As an implementation manner, the second decoder may be configured to: perform convolution processing on the second image semantic feature based on the semantic coding feature to obtain an image segmentation result of the input image.

[0023] As an implementation, the second decoder may be configured to: set processing parameters corresponding to the input image based on the semantic coding features; and perform convolution processing on the semantic features of the second image using the set processing parameters.

[0024] As an embodiment, the attention module may be configured to determine a preset dimension of the semantic coding feature based on a preset number of channels of the second image semantic feature and a preset number and a preset size of convolution kernels used for the convolution processing; wherein the second decoder may be configured to set each dimensional semantic coding feature of the preset dimensional semantic coding features as a convolution weight in the convolution kernel used for the convolution processing.

[0025] As an embodiment, the second decoder can be configured to: perform a first convolution processing on the second image semantic feature to obtain an image semantic feature with a first feature channel number, wherein the first feature channel number is smaller than the channel number of the second image semantic feature; perform a second convolution processing on the image semantic feature with the first feature channel number to obtain an image semantic feature with the first channel number; perform a third convolution processing on the image semantic feature with the first channel number to obtain an image semantic feature with a second channel number, wherein the second channel number is smaller than the first channel number.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a training device for an image segmentation model is provided, which may include: an acquisition module, configured to acquire a sample image and a preset image segmentation result of the sample image; a training module, configured to: acquire a first image semantic feature of the sample image, the first image semantic feature corresponding to a first resolution; acquire a semantic coding feature of the sample image based on the first image semantic feature; acquire a second image semantic feature of the sample image based on the first image semantic feature, the second image semantic feature corresponding to a second resolution, the second resolution being higher than the first resolution; obtain an image segmentation result of the sample image based on the semantic coding feature and the second image semantic feature; determine a training loss based on the difference between the image segmentation result and the preset image segmentation result, and obtain the image segmentation model when the training loss meets a preset condition.

[0027] As an implementation manner, the training module may be configured to: perform convolution processing and down-sampling processing on the sample image to obtain the first image semantic feature of the sample image.

[0028] As an implementation manner, the training module may be configured to: perform at least one of pooling processing, full connection processing, and feature mapping processing on the first image semantic features to obtain the semantic coding features of a preset dimension.

[0029] As an implementation manner, the training module may be configured to: perform convolution processing and up-sampling processing on the first image semantic feature to obtain the second image semantic feature of the sample image.

[0030] As an implementation manner, the training module may be configured to: perform convolution processing on the second image semantic feature according to the semantic coding feature to obtain an image segmentation result of the sample image.

[0031] As an implementation manner, the training module may be configured to set processing parameters corresponding to the sample image based on the semantic coding features; and perform convolution processing on the semantic features of the second image using the set processing parameters.

[0032] As an embodiment, the training module can be configured to: determine the preset dimension of the semantic coding feature based on the preset number of channels of the second image semantic feature and the preset number and preset size of the convolution kernels used for the convolution processing; and set each dimensional semantic coding feature of the preset dimensional semantic coding feature as the convolution weight in the convolution kernel used for the convolution processing.

[0033] As an embodiment, the training module can be configured to: perform a first convolution processing on the second image semantic feature to obtain an image semantic feature with a first feature channel number, wherein the first feature channel number is smaller than the channel number of the second image semantic feature; perform a second convolution processing on the image semantic feature with the first feature channel number to obtain an image semantic feature with the first channel number; perform a third convolution processing on the image semantic feature with the first channel number to obtain an image semantic feature with a second channel number, wherein the second channel number is smaller than the first channel number.

[0034] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: at least one processor; and at least one memory storing computer-executable instructions, wherein when the computer-executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the image segmentation method and model training method as described above.

[0035] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute the image segmentation method and model training method as described above.

[0036] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the image segmentation method and model training method as described above.

[0037] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0038] By learning the global attention information of the current input image (i.e., the semantic encoding features of the current input image) to influence the training process of the image segmentation model, the accuracy of image segmentation can be improved.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 is a flow chart of an image segmentation method according to an embodiment of the present disclosure;

[0042] Figure 2 is a flowchart of an image segmentation method according to an embodiment of the present disclosure;

[0043] Figure 3 is a flowchart of a method for training an image segmentation model according to an embodiment of the present disclosure;

[0044] Figure 4 is a block diagram of an image segmentation apparatus according to an embodiment of the present disclosure;

[0045] Figure 5 is a block diagram of a training device for an image segmentation model according to an embodiment of the present disclosure;

[0046] Figure 6 is a structural schematic diagram of an image segmentation device according to an embodiment of the present disclosure;

[0047] Figure 7 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.

[0050] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the claims and their equivalents.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0052] Image salient object segmentation methods based on deep learning mainly focus on improving the performance of algorithm models by utilizing fine details, global semantics, convolutional neural network attention mechanisms, and edge information.

[0053] Although existing solutions have made great progress, they are based on a complete labeled training dataset and are also based on the following assumptions: the foreground masks of all data annotations are unambiguous. In other words, the foreground is a very obvious object, and different annotators will annotate the same object mask without interfering with each other.

[0054] However, in actual application scenarios, the training data set annotation accuracy varies more often. The reasons are as follows: 1. The data itself is ambiguous, and different annotators will annotate different foreground masks; 2. There are cases where annotators mislabel the data; 3. Real applications require massive amounts of training data, but if it is completely manual, the manpower and time costs will be very high. Therefore, semi-supervised learning and other methods are used to (semi-)automatically annotate the training data to generate pseudo-labels; however, pseudo-labels will contain a lot of noise. In the above cases, there is a high probability of confusing samples. For example, people in some images represent the foreground, while people in other images represent the background. This will cause serious confusion in the model during the learning process, resulting in the inability to learn a better algorithm model.

[0055] Based on the above problems, the present disclosure proposes a dynamic instance-aware network, which extracts semantic coding features (i.e., attention information) of images from the original data set and applies the semantic coding features to the dynamic instance-aware network, so that the network can implicitly learn such ambiguous information in the data set, so as to more effectively extract valid information from the ambiguous labeled data set, while improving the fault tolerance rate in the model learning process and maximizing the model algorithm performance.

[0056] Hereinafter, according to various embodiments of the present disclosure, the method and apparatus of the present disclosure will be described in detail with reference to the accompanying drawings.

[0057] Figure 1 is a flow chart of an image segmentation method according to an embodiment of the present disclosure. The image segmentation method according to an embodiment of the present disclosure can be implemented in any electronic device having an image processing function. The electronic device may include at least one of the following, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader (e-bookreader), a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a video player, a camera, and a wearable device, etc.

[0058] Reference Figure 1 In step S101, an input image is obtained. The input image is an image to be segmented.

[0059] In step S102, the input image is input into the image segmentation model to obtain the image segmentation result of the input image. The image segmentation model of the present disclosure may include an encoder, a first decoder, an attention module and a second decoder.

[0060] As an example, the input image can be convolved and down-sampled to obtain a first image semantic feature of the input image, the first image semantic feature can be subjected to at least one of pooling, full connection, and feature mapping to obtain a semantic coding feature of a preset dimension, the first image semantic feature can be convolved and up-sampled to obtain a second image semantic feature of the input image, and then the second image semantic feature can be convolved based on the semantic coding feature to obtain an image segmentation result of the input image.

[0061] For example, a convolution operation and a downsampling operation may be performed based on the input image using an encoder of an image segmentation model to obtain a coded feature of the input image (which may be referred to as a first image semantic feature). Here, the encoder may be implemented by any convolutional neural network. For example, a network such as Mobilenet or ResNet may be used to perform a coding operation on the input image to obtain a first image semantic feature of the input image.

[0062] The first decoder of the image segmentation model can be used based on the first image semantic feature to obtain the decoded features of the input image (which can be called the second image semantic feature); and based on the attention module, the semantic coding features of the input image (which can also be called attention information) are obtained. Here, the first decoder can be implemented by any convolutional neural network. For example, a decoding operation (such as a convolution operation and an upsampling operation) is performed on the coded features using a network such as Mobilenet and Resnet to obtain the corresponding decoded features. The attention module can be composed of one or more of a pooling layer, a fully connected layer, and an activation layer in a neural network. For example, the attention module can be composed of a pooling layer, a first fully connected layer, an activation layer, and a second fully connected layer. The first image semantic feature is processed by the pooling layer, the first fully connected layer, the activation layer, and the second fully connected layer to obtain the attention information for the input image.

[0063] Based on the semantic coding features, processing parameters corresponding to the input image may be set, and then convolution processing may be performed on the semantic features of the second image using the set processing parameters.

[0064] For example, the preset dimension of the semantic coding feature can be determined based on the preset number of channels of the second image semantic feature and the preset number and preset size of the convolution kernels used for the convolution process. Then, each dimensional semantic coding feature in the semantic coding feature of the preset dimension is respectively set as the convolution weight in the convolution kernel (here, the convolution kernel included in the second decoder) used for the convolution process.

[0065] According to an embodiment of the present disclosure, the second decoder may be implemented by several convolutional layers, and for the current input image, the parameters of the second decoder may be set by the attention information of the current input image. That is, the parameters of the second decoder may be different according to the attention information of the current input image. That is, the parameters of the second decoder may be set based on the semantic coding features of the current input image. For example, the second decoder is composed of three convolutional layers, and the attention information of the input image is set as the convolution weights of the convolution kernels included in the three convolutional layers. In this case, in order to meet the parameter setting of the second decoder, it is necessary to determine the dimension of the attention information of the input image based on the number of channels of the decoding features of the input image and the number and size of the convolution kernels included in the second decoder. The number of channels of the decoding features can be determined based on the number of output convolution kernels in the first decoder. The number of input convolution kernels of the first decoder and the number and size of the convolution kernels of the second decoder can be preset.

[0066] After setting the parameters of the second decoder, the second decoder can be used to perform a convolution operation on the decoded features of the input image to obtain an image segmentation result. For example, after setting the parameters, the second image semantic feature can be subjected to a first convolution operation to obtain an image semantic feature having a first feature channel number, wherein the first feature channel number is less than the channel number of the second image semantic feature, the image semantic feature having the first feature channel number can be subjected to a second convolution operation to obtain an image semantic feature having the first channel number, and then the image semantic feature having the first channel number can be subjected to a third convolution operation to obtain an image semantic feature having a second channel number, wherein the second channel number is less than the first channel number. Finally, upsampling is activated to output the final segmentation result. The above-mentioned convolution operation is merely exemplary, and the present disclosure is not limited to the above-mentioned mapping of the number of layers and the number of channels.

[0067] According to an embodiment of the present disclosure, the accuracy of image segmentation is improved by learning the global ambiguous semantic information (ie, semantic coding features) of each image and encoding it into the foreground prediction result finally generated.

[0068] Figure 2 It is a flowchart of an image segmentation method according to an embodiment of the present disclosure. Figure 2 The figure shows a network diagram for image segmentation, which can treat each input image as an instance and predict a set of global semantic encoding features r(x i )∈R d (i.e., attention information), where d represents the dimension of semantic encoding, and then the global semantic encoding feature is used as the kernel parameter of the dynamic attention head network (Dynamic Attention Head) convolution operation to generate the final foreground prediction mask. This method can adaptively adjust the convolution parameters according to the input image, implicitly learn the global ambiguous semantic information of the image, and encode it into the final foreground prediction result.

[0069] Reference Figure 2 , input image x i After the encoding network (i.e. Figure 2 The "Encoder" in ) can generate low-resolution high-semantic information encoding features (i.e., the first image semantic features), which are then fed into two branch networks: one branch network is a dynamic attention module (i.e., Figure 2 The encoded features are sequentially passed through the global pooling layer (to perform the global pooling operation) ( Figure 2 "avg pooling" in ), fully connected layers ( Figure 2 "FC" in ), ReLU activation function ( Figure 2The "ReLU" in ) and the fully connected layer ( Figure 2 Another "FC" in ) to generate the global semantic encoding feature r(x i )∈R d (i.e. semantic encoding features); the other branch passes through a decoding convolutional network (i.e. Figure 2 The “Decoder” in ), generates a convolution feature θ (i.e., the second image semantic feature) with a channel number of n (e.g., 10, and the present application does not limit the value of n), and then θ is sent to the dynamic attention head network (i.e., Figure 2 For example, a dynamic attention head network may include three 1×1 convolution operations (i.e. Figure 2 In the example above, the first convolution operation maps θ to a feature with channel m. The second convolution operation has both input and output channels of m. The last convolution operation maps the feature with channel m to the feature with channel 1, and then activates upsampling to output the final segmentation result, i.e., the output mask.

[0070] For example, when n=10, m=8, the dynamic attention head network includes three convolutional layers and the convolution kernel size is 1×1, the dimension of the semantic coding feature d=(10×8×1×1+8)+(8×8×1×1+8)+(8×1×1×1+1)=169, so that each convolution weight in the convolution kernel of the attention head network can be assigned a feature in the global semantic coding feature (i.e., attention information) to achieve encoding the global ambiguous semantic information of the image into the final foreground prediction result. The above examples are only exemplary, and the number of output channels of the decoding convolution network, the number of convolutional layers in the attention head network, and the number and size of convolution kernels in the convolution layer can be set differently according to actual conditions.

[0071] In addition, when assigning convolution weights to the convolution kernels of the attention head network, the elements can be used as convolution weights in sequence according to the order of each element in the attention information, or the respective weights in the attention information can be randomly assigned to the convolution weights.

[0072] Figure 3It is a flowchart of a training method for an image segmentation model according to an embodiment of the present disclosure. The model training method according to an embodiment of the present disclosure can be implemented in any electronic device with an image processing function. The electronic device can include at least one of the following, for example, a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a video player, a camera, and a wearable device, etc.

[0073] The image segmentation model according to the embodiment of the present disclosure may be composed of an encoder, a first decoder, an attention module and a second decoder. The encoder and the first decoder may be implemented by any convolutional neural network, for example, mobilenet, resnet. The attention module is used to generate attention information of the original image. For example, the attention module may be implemented by a pooling layer, an activation layer and a fully connected layer, such as, Figure 2 The dynamic attention module shown, and the pooling mode in the pooling layer, the number of fully connected layers and the activation mode can be set differently based on actual conditions. The second decoder can be implemented by at least one convolutional layer, for example, Figure 2 The dynamic attention head network shown is implemented by three convolutional layers, and the number and size of convolution kernels in each convolutional layer can be set differently based on actual conditions.

[0074] According to an embodiment of the present disclosure, the number of convolution kernels of the first decoder and the number and size of convolution kernels in the second decoder can be set arbitrarily while ensuring that the dimension of the attention information generated by the attention module matches the number of channels output by the first decoder and the number and size of the convolution kernels in the second decoder.

[0075] Reference Figure 3 In step S301, a first image semantic feature of a sample image is obtained, and the first image semantic feature corresponds to a first resolution. Here, the sample image refers to training data for training an image segmentation model. For example, the sample image may be subjected to convolution processing and downsampling processing to obtain the first image semantic feature of the sample image. The first image semantic feature represents a high semantic image feature at a low resolution.

[0076] In step S302, semantic coding features of the sample image are obtained based on the semantic features of the first image. For example, at least one of pooling processing, full connection processing, and feature mapping processing may be performed on the semantic features of the first image to obtain semantic coding features of a preset dimension. Here, the semantic coding features may represent a set of global semantic coding features predicted for the sample image. The preset dimension of the semantic coding features may be determined based on the preset number of channels of the semantic features of the second image and the preset number and preset size of the convolution kernels in the second decoder.

[0077] In step S303, a second image semantic feature of the sample image is obtained based on the first image semantic feature, the second image semantic feature corresponds to the second resolution, and the second resolution is higher than the first resolution. For example, the first image semantic feature may be subjected to convolution processing and upsampling processing to obtain the second image semantic feature of the sample image. The number of output channels of the first decoder may be preset so that the first decoder outputs the second image semantic feature with a preset number of channels.

[0078] In step S304, an image segmentation result of the sample image is obtained according to the semantic coding feature and the second image semantic feature.

[0079] As an example, based on the semantic coding features, processing parameters corresponding to the sample image can be set, and the semantic features of the second image can be convolved using the set processing parameters to obtain the image segmentation result of the sample image. That is, the processing parameters of the second decoder can be set based on the attention information, and then the image segmentation result can be obtained by decoding the decoded features using the set second decoder. For example, the attention information can be set as the convolution weight of the convolution kernel included in the second decoder. The dimension of the attention information can be determined based on the number of channels of the decoded features and the number and size of the convolution kernels included in the second decoder.

[0080] Taking the second decoder including three convolutional layers as an example, for each convolution kernel in each convolutional layer, a semantic coding feature of the sample image can be assigned as the convolution weight of the convolution kernel. After setting the convolution weight of each convolution kernel, the first convolution layer can be used to perform a first convolution process on the second image semantic feature to obtain an image semantic feature with a first feature channel number, for example, the first feature channel number is less than the channel number of the second image semantic feature. The second convolution layer is used to perform a second convolution process on the image semantic feature with the first feature channel number to obtain an image semantic feature with the first channel number. The third convolution layer is used to perform a third convolution process on the image semantic feature with the first channel number to obtain an image semantic feature with a second channel number, for example, the second channel number is less than the first channel number.

[0081] According to an embodiment of the present disclosure, for each image, the processing parameters of the second decoder can be set based on the attention information of the current image. That is, the convolution weights in the convolution kernel of the second decoder are not fixed, but the weights of the convolution kernels in the second decoder are set for the attention information of each current input image.

[0082] In step S305, based on the difference between the predicted image segmentation result and the preset image segmentation result, the training loss is determined, and when the training loss meets the preset condition, the image segmentation model is obtained. Here, the preset image segmentation result can be an annotated image corresponding to the sample image. The annotator can annotate the sample image with significant areas in advance.

[0083] A loss function may be constructed based on the predicted image segmentation result and the corresponding labeled image, and the parameters of the encoder, the first decoder, the attention module, and the second decoder may be adjusted by minimizing the loss value calculated by the loss function.

[0084] In the present disclosure, in the training stage of the image segmentation model, after setting the parameters of the second decoder based on the attention information of the current image, the various network parameters in the image segmentation model can be adjusted by minimizing the value of the loss function constructed by the prediction results and the annotated image. In the prediction stage of the image segmentation model, the parameters of the second decoder can be set according to the attention information of the current input image without changing the parameters of other networks in the model.

[0085] In the present disclosure, aiming at practical application scenarios, the inventors discovered the phenomenon of ambiguous labeled data and gave a clear definition; further, a dynamic instance-aware network was proposed, which enables the network to implicitly learn the ambiguous information in the dataset by extracting the semantic encoding features (i.e., attention information) of the image from the original dataset and applying the semantic encoding features to the dynamic instance-aware network, so as to more effectively extract the semantic features of the image in the ambiguous labeled dataset, thereby obtaining more accurate image segmentation results, while improving the fault tolerance rate in the model learning process and maximizing the model algorithm performance.

[0086] Figure 4 is a block diagram of an image segmentation apparatus according to an embodiment of the present disclosure. The image segmentation apparatus may also be referred to as an image segmentation model apparatus.

[0087] Reference Figure 4, the image segmentation device 400 may include an encoder 401, a first decoder 402, an attention module 403, and a second decoder 404. Each module in the image segmentation device 400 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the image segmentation device 400 may be omitted, or additional modules may be included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus the functions of the corresponding modules / elements before the combination may be equivalently performed.

[0088] The encoder 401 may obtain a first image semantic feature of the input image. The first image semantic feature corresponds to the first resolution. The encoder 401 may perform convolution processing and downsampling processing on the input image to obtain the first image semantic feature of the input image.

[0089] The first decoder 402 may obtain a second image semantic feature of the input image based on the first image semantic feature. The second image semantic feature corresponds to a second resolution, and the second resolution may be higher than the first resolution. The first decoder 402 may perform convolution processing and upsampling processing on the first image semantic feature to obtain the second image semantic feature of the input image.

[0090] The attention module 403 may obtain the semantic coding features of the input image based on the semantic features of the first image. The attention module 403 may perform at least one of pooling processing, full connection processing, and feature mapping processing on the semantic features of the first image to obtain semantic coding features of a preset dimension. The attention module 403 may determine the preset dimension of the semantic coding features based on the preset number of channels of the semantic features of the second image and the preset number and preset size of convolution kernels used for convolution processing. In other words, the dimension of the semantic coding features may be determined based on the number of channels of the semantic features of the second image and the number and size of convolution kernels included in the second decoder.

[0091] The second decoder 404 may obtain an image segmentation result of the input image according to the semantic coding feature and the second image semantic feature. The second decoder 404 may perform convolution processing on the second image semantic feature according to the semantic coding feature to obtain an image segmentation result of the input image.

[0092] As an example, the second decoder 404 may set processing parameters corresponding to the input image based on the semantic coding features, and perform convolution processing on the semantic features of the second image using the set processing parameters.

[0093] The second decoder 404 may respectively set each dimension of the semantic coding features of the preset dimensions as a convolution weight in a convolution kernel for convolution processing.

[0094] Next, the second decoder 404 may perform a first convolution process on the second image semantic feature to obtain an image semantic feature having a first number of feature channels. The first number of feature channels may be smaller than the number of channels of the second image semantic feature. A second convolution process is performed on the image semantic feature having the first number of feature channels to obtain an image semantic feature having the first number of channels. A third convolution process is performed on the image semantic feature having the first number of channels to obtain an image semantic feature having a second number of channels. The second number of channels may be smaller than the first number of channels. Finally, the second decoder 404 outputs the final image segmentation result by activating an upsampling operation.

[0095] Referenced above Figure 1 and Figure 2 The image segmentation process has been described in detail and will not be repeated here.

[0096] Figure 5 is a block diagram of a training device for an image segmentation model according to an embodiment of the present disclosure.

[0097] Reference Figure 5 , the training device 500 may include an acquisition module 501 and a training module 502. Each module in the training device 500 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the training device 500 may be omitted, or additional modules may be further included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules / elements before the combination.

[0098] The acquisition module 501 can acquire a sample image and a preset image segmentation result of the sample image.

[0099] The acquisition module 501 can acquire an original image and an annotated image corresponding to the original image. During the model training process, multiple training images can be acquired, and the region of interest of each training image can be annotated to obtain a corresponding annotated image.

[0100] According to an embodiment of the present disclosure, each training picture (e.g., denoted as T i , i=1,2,3…N, N represents the number of training samples) as an instance, predict the global semantic coding features of each training picture (i.e., the attention information of each training picture), and then use the global semantic coding features as the kernel parameters of the convolution operation for predicting the mask, and then generate the final foreground prediction mask. That is to say, the present disclosure can adaptively adjust the convolution parameters according to each input image, implicitly learn the global ambiguous semantic information of the current picture, and encode it into the final foreground prediction result.

[0101] The training module 502 can obtain the first image semantic feature of the sample image, which may correspond to the first resolution. Based on the first image semantic feature, the semantic coding feature of the sample image is obtained. Based on the first image semantic feature, the second image semantic feature of the sample image is obtained, which may correspond to the second resolution, and the second resolution is higher than the first resolution. Based on the semantic coding feature and the second image semantic feature, an image segmentation result of the sample image is obtained, and the training loss is determined based on the difference between the image segmentation result and the preset image segmentation result. When the training loss meets the preset conditions, an image segmentation model is obtained.

[0102] For example, the training module 502 can use the encoder to obtain the encoding features of the original image based on the original image; use the first decoder and the attention module based on the encoding features to obtain the decoding features of the original image and the attention information of the original image respectively; use the second decoder based on the decoding features and the attention information to obtain the image segmentation result of the original image; update the parameters of the encoder, the first decoder, the attention module and the second decoder based on the annotated image and the image segmentation result, and use the updated encoder, the first decoder, the attention module and the second decoder as the image segmentation model.

[0103] As an embodiment, the training module 502 may perform convolution processing and downsampling processing on the sample image to obtain the first image semantic feature of the sample image. The training module 502 may perform at least one of pooling processing, full connection processing and feature mapping processing on the first image semantic feature to obtain a semantic coding feature of a preset dimension. The training module 502 may perform convolution processing and upsampling processing on the first image semantic feature to obtain the second image semantic feature of the sample image. The training module 502 may perform convolution processing on the second image semantic feature based on the semantic coding feature to obtain an image segmentation result of the sample image.

[0104] For example, the training module 502 may set processing parameters corresponding to the sample image based on the semantic coding features, and perform convolution processing on the semantic features of the second image using the set processing parameters.

[0105] Specifically, the training module 502 may determine the preset dimension of the semantic coding feature based on the preset number of channels of the second image semantic feature and the preset number and preset size of the convolution kernels used for the convolution processing, and set each dimensional semantic coding feature in the preset dimensional semantic coding feature as the convolution weight in the convolution kernel used for the convolution processing. After setting the processing parameters, the training module 502 may perform a first convolution processing on the second image semantic feature to obtain an image semantic feature with a first feature channel number, wherein the first feature channel number is less than the channel number of the second image semantic feature; perform a second convolution processing on the image semantic feature with the first feature channel number to obtain an image semantic feature with the first channel number; perform a third convolution processing on the image semantic feature with the first channel number to obtain an image semantic feature with a second channel number, wherein the second channel number is less than the first channel number.

[0106] Referenced above Figure 3 The model training process is described in detail and will not be repeated here.

[0107] Figure 6 6 is a schematic diagram of the structure of the image segmentation device of the hardware operating environment of the embodiment of the present disclosure. Here, the image segmentation device 600 can realize the above-mentioned image segmentation function.

[0108] like Figure 6 As shown, the image segmentation device 600 may include: a processing component 601, a communication bus 602, a network interface 603, an input / output interface 604, a memory 605, and a power supply component 606. Among them, the communication bus 602 is used to realize the connection and communication signals between these components. The input / output interface 604 may include a video display (such as a liquid crystal display), a microphone and a speaker, and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.), and optionally, the input / output interface 604 may also include a standard wired interface and a wireless interface. The network interface 603 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity interface). The memory 605 may be a high-speed random access memory or a stable non-volatile memory. The memory 605 may also be a storage device independent of the aforementioned processing component 601.

[0109] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the image segmentation device 600, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0110] like Figure 6 As shown, the memory 605 as a storage medium may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, an image segmentation program, a model training program, and a database.

[0111] exist Figure 6 In the image segmentation device 600 shown, the network interface 603 is mainly used for data communication with external devices / terminals; the input and output interface 604 is mainly used for data interaction with users; the processing component 601 and the memory 605 in the image segmentation device 600 can be set in the image segmentation device 600, and the image segmentation device 600 calls the image segmentation program, model training program and various APIs provided by the operating system stored in the memory 605 through the processing component 601 to execute the image segmentation method, model training method, etc. provided in the embodiments of the present disclosure.

[0112] The processing component 601 may include at least one processor, and a computer executable instruction set is stored in the memory 605. When the computer executable instruction set is executed by at least one processor, the image segmentation method and / or model training method according to the embodiment of the present disclosure is executed. In addition, the processing component 601 may perform the model training process, image segmentation, etc. as described above. However, the above examples are only exemplary, and the present disclosure is not limited thereto.

[0113] In addition, the processing component 601 can receive a trained image segmentation model from an external device, and use the image segmentation model to obtain a mask image of the input image.

[0114] As an example, the image segmentation device 600 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned instruction set. Here, the image segmentation device 600 is not necessarily a single electronic device, but can also be any device or circuit that can execute the above-mentioned instructions (or instruction sets) individually or jointly. The image segmentation device 600 can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0115] In the image segmentation device 600, the processing component 601 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processing component 601 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0116] The processing component 601 can run instructions or codes stored in the memory, wherein the memory 605 can also store data. Instructions and data can also be sent and received over the network via the network interface 603, wherein the network interface 603 can use any known transmission protocol.

[0117] The memory 605 may be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 605 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory and processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0118] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 7 It is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 700 may include at least one memory 702 and at least one processor 701. The at least one memory 702 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 701, the image segmentation method or model training method according to the embodiment of the present disclosure is executed.

[0119] Processor 701 may include a central processing unit (CPU), an audio processor, a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, processor 701 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0120] The memory 702 as a storage medium may include an operating system (eg, a MAC operating system), a data storage module, a network communication module, a user interface module, an image segmentation module, and a database.

[0121] The memory 702 may be integrated with the processor 701, for example, RAM or flash memory may be arranged within an integrated circuit microprocessor or the like. In addition, the memory 702 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 702 and the processor 701 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 701 can read files stored in the memory 702.

[0122] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0123] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0124] Those skilled in the art will understand that Figure 7 The structures shown in the drawings do not constitute limitations of the drawings and may include more or less components than shown in the drawings, or combine certain components, or arrange the components differently.

[0125] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the image segmentation method and model training method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0126] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned image segmentation method and model training method.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image segmentation model, characterized in that: include: Acquire a first image semantic feature of the sample image, where the first image semantic feature corresponds to a first resolution; Acquire a semantic coding feature of a preset dimension of the sample image according to the first image semantic feature; Acquire a second image semantic feature of the sample image according to the first image semantic feature, wherein the second image semantic feature corresponds to a second resolution, and the second resolution is higher than the first resolution; Determining, based on the semantic coding feature, a kernel parameter for performing a convolution process on the second image semantic feature; Performing convolution processing on the second image semantic feature based on the kernel parameter to obtain an image segmentation result of the sample image; Based on the difference between the image segmentation result and a preset image segmentation result, a training loss is determined, and when the training loss meets a preset condition, the image segmentation model is obtained.

2. The training method according to claim 1, characterized in that: Acquiring semantic coding features of a preset dimension of the sample image according to the first image semantic features, including: The preset dimension of the semantic coding feature is determined based on the preset number of channels of the second image semantic feature and the preset number and preset size of convolution kernels used for the convolution processing.

3. The training method according to claim 1, characterized in that: Determining a kernel parameter for performing convolution processing on the second image semantic feature based on the semantic coding feature includes: Each dimensional semantic coding feature in the preset dimensional semantic coding features is respectively set as a convolution weight in a convolution kernel used for the convolution processing.

4. The training method according to claim 1, characterized in that: Performing convolution processing on the second image semantic feature based on the kernel parameter includes: Performing a first convolution process on the second image semantic feature to obtain an image semantic feature having a first number of feature channels, wherein the number of the first feature channels is smaller than the number of channels of the second image semantic feature; Performing a second convolution process on the image semantic features having the first number of feature channels to obtain the image semantic features having the first number of channels; A third convolution process is performed on the image semantic features having the first number of channels to obtain image semantic features having a second number of channels, wherein the second number of channels is smaller than the first number of channels.

5. The training method according to claim 1, characterized in that: Obtaining the first image semantic feature of the sample image includes: The sample image is subjected to convolution processing and down-sampling processing to obtain the first image semantic feature of the sample image.

6. The training method according to claim 1, characterized in that: Acquiring semantic coding features of the sample image according to the first image semantic features, including: At least one of pooling processing, full connection processing and feature mapping processing is performed on the first image semantic features to obtain the semantic coding features of the preset dimension.

7. The training method according to claim 1, characterized in that: Acquiring a second image semantic feature of the sample image according to the first image semantic feature includes: The first image semantic feature is subjected to convolution processing and up-sampling processing to obtain the second image semantic feature of the sample image.

8. An image segmentation method, characterized in that: include: Acquire a first image semantic feature of an input image, where the first image semantic feature corresponds to a first resolution; Acquire a semantic coding feature of a preset dimension of the input image according to the first image semantic feature; Acquire a second image semantic feature of the input image according to the first image semantic feature, where the second image semantic feature corresponds to a second resolution, and the second resolution is higher than the first resolution; Determining, based on the semantic coding feature, a kernel parameter for performing a convolution process on the second image semantic feature; A convolution process is performed on the second image semantic feature based on the kernel parameter to obtain an image segmentation result of the input image.

9. The image segmentation method according to claim 8, characterized in that: Obtaining the first image semantic feature of the input image, including: The input image is subjected to convolution processing and down-sampling processing to obtain the first image semantic feature of the input image.

10. The image segmentation method according to claim 8, characterized in that: Acquiring semantic coding features of a preset dimension of the input image according to the first image semantic features, including: At least one of pooling processing, full connection processing and feature mapping processing is performed on the first image semantic features to obtain the semantic coding features of the preset dimension.

11. The image segmentation method according to claim 8, characterized in that: Acquiring a second image semantic feature of the input image according to the first image semantic feature includes: The first image semantic feature is subjected to convolution processing and up-sampling processing to obtain the second image semantic feature of the input image.

12. The image segmentation method according to claim 8, characterized in that: Acquiring semantic coding features of a preset dimension of the input image according to the first image semantic features, including: The preset dimension of the semantic coding feature is determined based on the preset number of channels of the second image semantic feature and the preset number and preset size of convolution kernels used for the convolution processing.

13. The image segmentation method according to claim 8, characterized in that: Determining a kernel parameter for performing convolution processing on the second image semantic feature based on the semantic coding feature includes: Each dimensional semantic coding feature in the preset dimensional semantic coding features is respectively set as a convolution weight in a convolution kernel used for the convolution processing.

14. The image segmentation method according to claim 8, characterized in that: Performing convolution processing on the second image semantic feature based on the kernel parameter to obtain an image segmentation result of the input image includes: Performing a first convolution process on the second image semantic feature to obtain an image semantic feature having a first number of feature channels, wherein the number of the first feature channels is smaller than the number of channels of the second image semantic feature; Performing a second convolution process on the image semantic features having the first number of feature channels to obtain the image semantic features having the first number of channels; A third convolution process is performed on the image semantic features having the first number of channels to obtain image semantic features having a second number of channels, wherein the second number of channels is smaller than the first number of channels.

15. An image segmentation model device, characterized in that: include: An encoder configured to obtain a first image semantic feature of an input image, wherein the first image semantic feature corresponds to a first resolution; A first decoder is configured to obtain a second image semantic feature of the input image according to the first image semantic feature, wherein the second image semantic feature corresponds to a second resolution, and the second resolution is higher than the first resolution; an attention module, configured to obtain a semantic coding feature of a preset dimension of the input image based on the first image semantic feature; The second decoder is configured to determine a kernel parameter for performing convolution processing on the second image semantic feature based on the semantic coding feature; and perform convolution processing on the second image semantic feature based on the kernel parameter to obtain an image segmentation result of the input image.

16. The image segmentation model device according to claim 15, characterized in that: The encoder is configured as: The input image is subjected to convolution processing and down-sampling processing to obtain the first image semantic feature of the input image.

17. The image segmentation model device according to claim 15, characterized in that: The attention module is configured as follows: At least one of pooling processing, full connection processing and feature mapping processing is performed on the first image semantic features to obtain the semantic coding features of the preset dimension.

18. The image segmentation model device according to claim 15, characterized in that: The first decoder is configured to: The first image semantic feature is subjected to convolution processing and up-sampling processing to obtain the second image semantic feature of the input image.

19. The image segmentation model device according to claim 15, characterized in that: The attention module is configured to determine the preset dimension of the semantic coding feature based on a preset number of channels of the second image semantic feature and a preset number and preset size of convolution kernels used for the convolution processing.

20. The image segmentation model device according to claim 15, characterized in that: The second decoder is configured to set each dimensional semantic coding feature of the preset dimensional semantic coding features as a convolution weight in a convolution kernel used for the convolution processing.

21. The image segmentation model device according to claim 15, characterized in that: The second decoder is configured to: Performing a first convolution process on the second image semantic feature to obtain an image semantic feature having a first number of feature channels, wherein the number of the first feature channels is smaller than the number of channels of the second image semantic feature; Performing a second convolution process on the image semantic features having the first number of feature channels to obtain the image semantic features having the first number of channels; A third convolution process is performed on the image semantic features having the first number of channels to obtain image semantic features having a second number of channels, wherein the second number of channels is smaller than the first number of channels.

22. A training device for an image segmentation model, characterized in that: include: An acquisition module, configured to acquire a sample image and a preset image segmentation result of the sample image; The training module is configured as follows: Acquire a first image semantic feature of the sample image, where the first image semantic feature corresponds to a first resolution; Acquire a semantic coding feature of a preset dimension of the sample image according to the first image semantic feature; Acquire a second image semantic feature of the sample image according to the first image semantic feature, wherein the second image semantic feature corresponds to a second resolution, and the second resolution is higher than the first resolution; Determining, based on the semantic coding feature, a kernel parameter for performing a convolution process on the second image semantic feature; Performing convolution processing on the second image semantic feature based on the kernel parameter to obtain an image segmentation result of the sample image; Based on the difference between the image segmentation result and a preset image segmentation result, a training loss is determined, and when the training loss meets a preset condition, the image segmentation model is obtained.

23. The training device according to claim 22, characterized in that The training module is configured as: The sample image is subjected to convolution processing and down-sampling processing to obtain the first image semantic feature of the sample image.

24. The training device according to claim 22, characterized in that The training module is configured as: At least one of pooling processing, full connection processing and feature mapping processing is performed on the first image semantic features to obtain the semantic coding features of the preset dimension.

25. The training device according to claim 22, characterized in that The training module is configured as: The first image semantic feature is subjected to convolution processing and up-sampling processing to obtain the second image semantic feature of the sample image.

26. The training device according to claim 22, characterized in that The training module is configured as: The preset dimension of the semantic coding feature is determined based on the preset number of channels of the second image semantic feature and the preset number and preset size of convolution kernels used for the convolution processing.

27. The training device according to claim 22, characterized in that The training module is configured as: Each dimensional semantic coding feature in the preset dimensional semantic coding features is respectively set as a convolution weight in a convolution kernel used for the convolution processing.

28. The training device according to claim 22, characterized in that The training module is configured as: Performing a first convolution process on the second image semantic feature to obtain an image semantic feature having a first number of feature channels, wherein the number of the first feature channels is smaller than the number of channels of the second image semantic feature; Performing a second convolution process on the image semantic features having the first number of feature channels to obtain the image semantic features having the first number of channels; A third convolution process is performed on the image semantic features having the first number of channels to obtain image semantic features having a second number of channels, wherein the second number of channels is smaller than the first number of channels.

29. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, The processor is configured to execute the instructions to implement the method according to any one of claims 1-14.

30. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 14.

31. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images