Image segmentation and model training method, device, terminal equipment and storage medium

Abstract

Embodiments of the present invention disclose an image segmentation and model training method, apparatus, terminal device, and storage medium. The image segmentation method includes: downsampling an image to be segmented through an encoder to obtain a feature image; and upsampling the feature image through a decoder to obtain a segmented image. The convolution kernel used for upsampling in the decoder is determined based on a spatial transformation matrix between a trained first sample image and a second sample image, wherein the first sample image is the pre-upsampled image corresponding to the convolution kernel, and the second sample image is the post-upsampled image corresponding to the convolution kernel. By using the spatial transformation matrix to determine the convolution kernel used for upsampling, it is possible to achieve structured feature reorganization of features on different channels in a low-resolution feature image based on the convolution kernel, thereby effectively reconstructing a higher-resolution feature image, and achieving rapid segmentation while ensuring segmentation accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision technology, and in particular to an image segmentation and model training method, apparatus, terminal device and storage medium. Background Art

[0002] Image Semantic Segmentation technology uses semantic attributes as the division criteria to achieve pixel-by-pixel classification prediction. It can obtain the semantics and location coordinates of the target object, making it of great practical value in many fields centered around scene understanding.

[0003] In existing technologies, the encoder-decoder architecture based on deep learning is a commonly used framework for image semantic segmentation tasks. In real-time image segmentation scenarios, to ensure segmentation speed, computational complexity is often reduced by sacrificing spatial information in the input image, reducing the number of channels in the feature map, and limiting the receptive field of pixels in the feature map on the input image.

[0004] During the implementation process in related technologies, it was found that there are at least the following technical problems: using the above-mentioned methods of losing spatial information, reducing the number of channels and limiting the receptive field to speed up the segmentation speed will lead to a decrease in segmentation accuracy. Summary of the Invention

[0005] The embodiments of the present invention provide an image segmentation and model training method, apparatus, terminal device and storage medium, which can achieve rapid segmentation while ensuring segmentation accuracy.

[0006] In a first aspect, an embodiment of the present invention provides an image segmentation method, comprising:

[0007] The image to be segmented is downsampled through the encoder to obtain a feature image;

[0008] Upsampling the feature image through a decoder to obtain a segmented image;

[0009] The convolution kernel used for upsampling in the decoder is determined based on the spatial transformation matrix between the trained first sample image and the second sample image, wherein the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0010] In a second aspect, an embodiment of the present invention provides a model training method, comprising:

[0011] A spatial transformation matrix is ​​trained based on a first sample image and a second sample image, and a convolution kernel according to any one of the embodiments of the present invention is determined according to the spatial transformation matrix; wherein the convolution kernel is a convolution kernel used for upsampling in an encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel;

[0012] The convolution kernel in the encoder is fixed, and the image segmentation model including the encoder is trained.

[0013] In a third aspect, an embodiment of the present invention provides an image segmentation device, comprising:

[0014] A downsampling module is used to downsample the image to be segmented through an encoder to obtain a feature image;

[0015] an upsampling module, configured to upsample the feature image through a decoder to obtain a segmented image;

[0016] The convolution kernel used for upsampling in the decoder is determined based on the spatial transformation matrix between the trained first sample image and the second sample image, wherein the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0017] In a fourth aspect, an embodiment of the present invention provides a model training device, comprising:

[0018] a first-stage training module, configured to train a spatial transformation matrix based on a first sample image and a second sample image, and determine a convolution kernel according to any one of the embodiments of the present invention based on the spatial transformation matrix; wherein the convolution kernel is a convolution kernel used for upsampling in an encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel;

[0019] The second stage training module is used to fix the convolution kernel in the encoder and train the image segmentation model including the encoder.

[0020] In a fifth aspect, an embodiment of the present invention provides a terminal device, including:

[0021] one or more processors;

[0022] a memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation or model training method as described in any one of the embodiments of the present invention.

[0024] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image segmentation or model training method as described in any one of the embodiments of the present invention.

[0025] An embodiment of the present invention provides an image segmentation and model training method, apparatus, terminal device, and storage medium, wherein the image segmentation method comprises: downsampling the image to be segmented through an encoder to obtain a feature image; upsampling the feature image through a decoder to obtain a segmented image; wherein the convolution kernel used for upsampling in the decoder is determined based on the spatial transformation matrix between the trained first sample image and the second sample image, wherein the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0026] By using the spatial transformation matrix to determine the convolution kernel used for upsampling, it is possible to achieve structured feature reorganization of features across different channels in the low-resolution feature image based on the convolution kernel, effectively reconstructing a higher-resolution feature image. This not only fully utilizes the spatial information of the low-resolution feature image and avoids the introduction of irrelevant variables to improve upsampling accuracy, but also reduces the number of encoder parameters and computational complexity to improve upsampling efficiency, thereby achieving fast segmentation while ensuring segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A flowchart of an image segmentation method provided by an embodiment of the present invention is shown;

[0029] Figure 2 A flowchart of a convolution kernel determination step in an image segmentation method provided by an embodiment of the present invention is shown;

[0030] Figure 3 A schematic diagram of generating a transformed image in an image segmentation method provided by an embodiment of the present invention is shown;

[0031] Figure 4 A schematic diagram showing an upsampling method in an image segmentation method provided by an embodiment of the present invention is shown;

[0032] Figure 5A flowchart of layer-by-layer upsampling in an image segmentation method provided by an embodiment of the present invention is shown;

[0033] Figure 6 A schematic diagram of an image segmentation model in an image segmentation method provided by an embodiment of the present invention is shown;

[0034] Figure 7 A flowchart showing an image fusion step in an image segmentation method provided by an embodiment of the present invention is shown;

[0035] Figure 8 A schematic diagram of an image fusion framework in an image segmentation method provided by an embodiment of the invention is shown;

[0036] Figure 9 A partial schematic diagram of an image fusion framework in an image segmentation method provided by an embodiment of the invention is shown;

[0037] Figure 10 A flowchart of determining a new current-level image in an image segmentation method provided by an embodiment of the present invention is shown;

[0038] Figure 11 A flowchart of self-attention feature map fusion in an image segmentation method provided by an embodiment of the present invention is shown;

[0039] Figure 12 A flow chart of a model training method provided by an embodiment of the present invention is shown;

[0040] Figure 13 A schematic structural diagram of an image segmentation device provided by an embodiment of the present invention is shown;

[0041] Figure 14 A schematic structural diagram of a model training device provided by an embodiment of the present invention is shown;

[0042] Figure 15 A schematic diagram of the hardware structure of a terminal device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below through implementation methods with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the following embodiments, each embodiment provides optional features and examples at the same time. The various features recorded in the embodiments can be combined to form multiple optional solutions. Each numbered embodiment should not be regarded as just one technical solution.

[0044] Figure 1 A flowchart of an image segmentation method provided by an embodiment of the present invention is shown. The image segmentation method provided by an embodiment of the present invention is applicable to real-time image semantic segmentation, such as semantic segmentation of real-time road images collected in the field of autonomous driving. The method can be performed by an image segmentation device implemented in software and / or hardware, preferably configured in a terminal device, such as a mobile phone, computer, or vehicle-mounted terminal.

[0045] like Figure 1 As shown, the image segmentation method provided in the embodiment of the present invention includes the following steps:

[0046] S110, downsampling the image to be segmented through an encoder to obtain a feature image;

[0047] S120 , upsampling the feature image through a decoder to obtain a segmented image.

[0048] In the embodiment of the present invention, the image segmentation method may refer to an image semantic segmentation method, and the encoder and the decoder may constitute an image segmentation model for performing image semantic segmentation.

[0049] The encoder in this model can be used to perform multi-level downsampling on the input image to be segmented to extract feature images at different levels. Feature images at higher levels have lower resolution and can contain more semantic information, but lack spatial information. Feature images at lower levels have higher resolution and can contain more detailed spatial information, but lack semantic information. Spatial information can be understood as the relative spatial position or orientation of multiple objects in an image, while semantic information can be understood as the semantic attributes of the objects contained in the image.

[0050] The decoder in this model can be used to upsample semantically rich high-level feature images and fuse the spatial details of low-level feature images to restore the high-level feature images to their original resolution and achieve pixel-by-pixel semantic attribute classification. It can be understood that the segmented image is an image with the same resolution as the image to be segmented, obtained by the encoder by upsampling the high-level feature images layer by layer and fusing the low-level feature images. The segmented image also contains the semantic attribute classification of each pixel.

[0051] In traditional image segmentation methods, the decoder introduces a large amount of computational effort in the process of restoring high-level feature images to their original resolution due to the large resolution of the processed feature images. In addition, the traditional bilinear difference upsampling feature calculation method assumes that the center pixel is used as the current pixel. Since the pixel value of the current pixel only depends on the features within a fixed range, it cannot adapt to the scale changes between multiple target objects in the scene, which is a data-independent upsampling method. The traditional transposed convolution upsampling feature calculation method expands the resolution of the input feature map by inserting 0 pixels at the edge of the feature image or inside the feature map, and then uses conventional convolution to achieve feature upsampling. This will introduce a variable value of 0 that is irrelevant to the context, requiring the subsequent neural network layer to learn the features of the corresponding position, which increases the difficulty of learning.

[0052] The present invention provides an upsampling method that can replace traditional difference and transposed convolution. In this embodiment, the convolution kernel used for upsampling in the decoder is determined based on the spatial transformation matrix between the trained first and second sample images, where the first sample image is the pre-upsampling image corresponding to the convolution kernel, and the second sample image is the post-upsampling image corresponding to the convolution kernel.

[0053] The encoder can learn the parameters of the spatial transformation matrix for any convolution kernel required for upsampling the high-level feature image layer by layer. The spatial transformation matrix can be obtained in advance through supervised training based on the first sample image and the second sample image. Since the convolution kernel needs to upsample the low-resolution feature image to a high-resolution feature image, the image before upsampling corresponding to the convolution kernel can be considered as a low-resolution feature image based on the convolution kernel before upsampling, and the image after upsampling corresponding to the convolution kernel can be considered as a high-resolution image based on the convolution kernel after upsampling. The first sample image and the second sample image are sample data labeled in advance for training the spatial transformation matrix corresponding to the convolution kernel.

[0054] By training the spatial transformation matrix using the label information in the first sample image and the second sample image, the spatial transformation matrix can learn the spatial transformation relationship of the features from the low-resolution feature image to the high-resolution feature image. By determining the convolution kernel based on the spatial transformation matrix, the spatial transformation matrix can be implicitly included in the convolution kernel, so that when the encoder performs upsampling based on the convolution kernel, the value of each feature point in the high-resolution feature feature map can be obtained by structured spatial reorganization of each feature point in the low-resolution feature image. Upsampling through a simple structured reorganization of the feature space can greatly reduce the number of parameters and computational complexity in the encoder, thereby improving the efficiency of feature upsampling.

[0055] Because features at the same location in different channel dimensions have a certain degree of semantic correlation, the encoder leverages information from different channels for feature reconstruction. This allows it to learn more refined location-related information without introducing additional variables, enhancing local correlations and mitigating incoherence in the predicted segmentation map. Convolution operations selectively emphasize interdependent features at each location and further fuse transformed spatial features, avoiding data distribution shifts caused by structural transformations.

[0056] Compared to traditional bilinear interpolation upsampling, spatial transformation-based upsampling fully leverages the dataset to adaptively upsample features through self-learning, making it a data-dependent upsampling method. Compared to traditional transposed convolution upsampling, it avoids introducing context-independent variables, leverages the spatial structure of features to aid upsampling, and reduces the learning difficulty of detail recovery.

[0057] In addition, in the traditional upsampling process, it is necessary to lose spatial information, reduce the number of channels and limit the receptive field to speed up the segmentation speed, which will lead to a decrease in segmentation accuracy. The upsampling method based on spatial transformation provided by the embodiment of the present invention not only fully utilizes the spatial information of the low-resolution feature image, but also does not need to limit the number of channels of the low-resolution feature image, and also maintains the multiple receptive fields of the feature image, thereby reducing the detail error of the decoder in the process of restoring the feature map resolution and ensuring the upsampling accuracy. In summary, the image segmentation method provided by the embodiment of the present invention can ensure the upsampling accuracy and improve the upsampling efficiency during the encoder upsampling process, thereby achieving fast segmentation while ensuring segmentation accuracy.

[0058] In some optional implementations, applied to autonomous driving applications, the image to be segmented includes a road image captured in real time. In the field of autonomous driving, real-time, high-precision semantic segmentation of road images can provide smart cars with important road condition information, such as the driving conditions of vehicles on the road, the movement of pedestrians and non-motor vehicles, etc. By utilizing any of the image segmentation methods provided in the embodiments of the present invention, for a real-time captured road image, an encoder is used to obtain a feature image, and a convolution kernel in a decoder is used to upsample the feature image based on spatial transformation to achieve semantic segmentation. This allows accurate and rapid acquisition of road condition information to ensure the safe driving of autonomous vehicles.

[0059] An embodiment of the present invention provides an image segmentation method, which downsamples an image to be segmented through an encoder to obtain a feature image; and upsamples the feature image through a decoder to obtain a segmented image; wherein the convolution kernel used for upsampling in the decoder is determined based on the spatial transformation matrix between a trained first sample image and a second sample image, the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0060] By using the spatial transformation matrix to determine the convolution kernel used for upsampling, it is possible to achieve structured feature reorganization of features across different channels in the low-resolution feature image based on the convolution kernel, effectively reconstructing a higher-resolution feature image. This not only fully utilizes the spatial information of the low-resolution feature image and avoids the introduction of irrelevant variables to improve upsampling accuracy, but also reduces the number of encoder parameters and computational complexity to improve upsampling efficiency, thereby achieving fast segmentation while ensuring segmentation accuracy.

[0061] In this embodiment, the steps for determining the convolution kernel are described in detail. By training the spatial transformation matrix with the goal of ensuring a certain degree of similarity between the first sample image and the transformed image, a more optimal spatial transformation matrix can be learned. Since the spatial transformation matrix in this embodiment is the spatial transformation matrix from a high-resolution feature image to a low-resolution feature image, the inverse matrix of the spatial transformation matrix can be used to determine the convolution kernel to achieve upsampling.

[0062] Figure 2 FIG1 shows a flow chart of the convolution kernel determination step in an image segmentation method provided by an embodiment of the present invention. Figure 2 In the image segmentation method provided in this embodiment, the convolution kernel can be determined based on the following steps:

[0063] S210 : Downsample the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image using a spatial transformation matrix.

[0064] The spatial transformation matrix may be a transformation matrix from the high-resolution second sample image to the low-resolution first sample image, a transformation matrix from the low-resolution first sample image to the high-resolution second sample image, or a transformation matrix from the reconstructed feature image after reconstructing the features of each channel in the low-resolution first sample image to the high-resolution second sample image. Any spatial transformation matrix that can represent the spatial transformation between the first sample image and the second sample image may be included in the embodiments of the present invention as an implementable method.

[0065] In this embodiment, the training of the spatial transformation matrix from the second sample image to the first sample image is specifically described. The images disclosed in the embodiments of the present invention (such as the feature image, the first sample image, and the second sample image) can have several attributes: image height, image width, and number of image channels. The image height can be considered as the number of vertical pixels of the image, the image width can be considered as the number of horizontal pixels of the image, and the image height multiplied by the image width can represent the resolution of the image. The number of image channels can be considered as the number of images of the same resolution that make up the complete image. The concept of channels can be understood with reference to the following examples. For example, a color image can be composed of three channels: red R, green G, and blue B.

[0066] The second sample image has a higher resolution than the first sample image, and typically the width / height of the second sample image is a multiple of the width / height of the first sample image. The number of channels of the second sample image may differ from the number of channels of the first sample image. Using a spatial transformation matrix, the resolution of the second sample image can be compressed to the same level as that of the first sample image, and the channel features of the second sample image can be recombined to produce a transformed image with the same number of channels as the first sample image.

[0067] S220: Training the spatial transformation matrix so that the actual loss value between the first sample image and the transformed image meets a preset condition.

[0068] Among them, the actual loss value between the first sample image and the transformed image can be calculated based on a preset loss function, and the preset loss function may include but is not limited to a cross entropy loss function (Cross Entropy Loss), a dice loss function (Dice Loss) and a focal loss function (focal Loss), etc. In addition, the property of the neural network that can be back-propagated and optimized can be utilized to adjust the parameters of the spatial transformation matrix according to the actual loss value. When the actual loss value reaches the preset condition, it can be considered that the spatial transformation matrix training is completed. The actual loss value reaching the preset condition may include but is not limited to: the actual loss value is less than the preset loss value, and / or the number of training times reaches the preset number of times.

[0069] S230. Determine a convolution kernel according to the inverse matrix of the trained spatial transformation matrix.

[0070] The trained spatial transformation matrix may be the one that minimizes the actual loss value during the training process. Since the spatial transformation matrix in this embodiment is the spatial transformation matrix from the high-resolution second sample image to the low-resolution first sample feature image, the inverse matrix of the trained spatial transformation matrix may be used to determine the convolution kernel for upsampling. Thus, the encoder can achieve upsampling from the low-resolution feature image to the high-resolution feature image using this convolution kernel.

[0071] For example, Figure 3 FIG2 shows a schematic diagram of generating a transformed image in an image segmentation method provided by an embodiment of the present invention. Figure 3 In some optional implementations, downsampling the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image using a spatial transformation matrix may include:

[0072] See also Figure 3 First, the sliding window ω is slid in the second sample image Y, and the current downsampled local image Y' in the second sample image Y is determined in sequence. Where ω∈{0,1} r×r×C , that is, the height, width, and number of channels of the sliding window are r, r, and C respectively; Y∈{0,1} H×W×C , i.e., the height, width, and number of channels of the second sample image are H, W, and C, respectively. In the following text, if the superscripts of the identifiers are also in the format a×b×c, then a can be understood as representing height, b as width, and c as the number of channels, and no further explanation is given. The sliding window ω can be a one-hot window, i.e., the parameters within the window are composed of 0s and 1s; the second sample image Y can be a segmented image, and each channel in the segmented image can correspond to the segmentation result of a semantic attribute.

[0073] The window size r×r of the sliding window is determined according to the resolution compression coefficient from the second sample image to the first sample image E (not shown in the figure). For example, When , the resolution compression coefficient can be used as the height / width of the sliding window, and the resolution compression coefficient can be, for example, r=H / H′ or r=W / W′.

[0074] See also Figure 3 , then the local image Y' is transformed into a feature vector υ, where υ∈{0, 1} K , K=r×r×C; according to the space transformation matrix The eigenvector υ∈{0,1} K Transformed into pixel points of each preset channel The local image can be deformed into a one-dimensional eigenvector, and the number of columns of the spatial transformation matrix can be equal to the number of rows of the eigenvector, for example, both are K. In addition, the number of rows C' of the spatial transformation matrix can be equal to the number of channels of the first sample image, and the number of channels of each corresponding preset channel is the same as the number of channels of the first sample image.

[0075] The spatial transformation matrix can exist in various forms. In this embodiment, a linear form that is easy to embed into the network can be used. Among them, the linear form of the spatial transformation matrix can be understood as the pixel points of each preset channel can be obtained by multiplying the spatial transformation matrix with the eigenvector, that is, t=Pυ.

[0076] See also Figure 3 , when the sliding is completed, the pixel points of each preset channel are Composition Transformation During the sliding process, padding can be used to fill the edges of the second sample image with zero pixels if necessary to ensure that Y is divisible by an integer. Because the sliding window size is determined by the resolution compression factor, the resolution of the resulting transformed image can be the same as that of the first sample image. Furthermore, the feature vectors of the deformed local image can be reorganized in the feature space using a spatial transformation matrix, ensuring that the transformed image has the same number of channels as the first sample image.

[0077] Then, we can maximize the first sample image E and the transformed image The similarity between them can be obtained by using the back propagation optimization process of the neural network to implicitly learn the spatial transformation matrix P in the vector space. For example, the optimization objective function can be: in, Can be the inverse matrix of P, and It can be considered as the trained spatial transformation matrix and its inverse matrix. By using the inverse matrix of the trained spatial transformation matrix, the input E can be upsampled to the same resolution as the output Y while restoring the segmented image as much detail information as possible.

[0078] For example, Figure 4 A schematic diagram of an upsampling method in an image segmentation method provided by an embodiment of the present invention is shown. Figure 4 Without loss of generality, the double upsampling operation at the decoder side is used as an example to illustrate the above. Figure 4 The spatial transformation matrix in the figure can be the transformation matrix from the recombined feature image to the high-resolution second sample image after the channel features of the low-resolution first sample image are recombined. Figure 4 The upsampling process shown may include: for the input feature map E (i.e., a low-resolution feature image), structurally reorganizing the features according to channel C' from 1×C' to 2×2×C' / 4, so that E changes from H'×W'×C' to R of 2H×2W×C' / 4; using the convolution kernel Q to perform a structured feature space reorganization on R, expanding R to Y of 2H×2W×C, thereby achieving a 2x upsampling. In the process of determining Q, the inverse matrix P of Q can be trained, and the trained P is used to determine the convolution kernel Q.

[0079] This embodiment of the present invention, based on the above-described embodiment, describes in detail the steps for determining the convolution kernel. By training the spatial transformation matrix with the goal of ensuring a certain degree of similarity between the first sample image and the transformed image, a more optimal spatial transformation matrix can be learned. Since the spatial transformation matrix in this embodiment is the spatial transformation matrix from a high-resolution feature image to a low-resolution feature image, the inverse of the spatial transformation matrix can be used to determine the convolution kernel to achieve upsampling.

[0080] In addition, the image segmentation and model training methods proposed in the embodiments of the present invention and the above embodiments belong to the same inventive concept. For technical details not described in detail in this embodiment, please refer to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0081] This embodiment describes in detail the process of upsampling at different levels based on convolution kernels. The feature image includes multiple levels of feature images, and upsampling the feature image by the decoder can include upsampling the highest level feature image level by level through the decoder. By using each convolution kernel for level-by-level upsampling, image segmentation efficiency and accuracy can be improved.

[0082] For example, Figure 5 FIG1 shows a flowchart of layer-by-layer upsampling in an image segmentation method provided by an embodiment of the present invention. Figure 5 The image segmentation method provided in this embodiment includes the following steps for any level of upsampling during the level-by-level upsampling process:

[0083] S510, up-sampling the feature image of the current level based on the convolution kernel to obtain a first image to be fused;

[0084] S520: Use the feature image in the encoder that has the same resolution as the first image to be fused as the second image to be fused;

[0085] S530: Fuse the first image to be fused and the second image to be fused to obtain a fused image, and determine a new feature image of the current level according to the fused image.

[0086] For example, Figure 6 FIG1 shows a schematic diagram of an image segmentation model in an image segmentation method provided by an embodiment of the present invention. Figure 6 The image to be segmented may be a road image collected in real time. The semantic segmentation of the road image based on the image segmentation model may include:

[0087] The encoder can extract feature image 1 from the original image through downsampling module 1, and the resolution of feature image 1 can be downsampled to 1 / 2 of the original image; feature image 1 can be spliced ​​with an image of 1 / 2 resolution (indicated by "2×" in the figure) of the original image; after splicing, it can enter downsampling module 2 for feature learning and 2x downsampling; feature image 2 output by downsampling module 2 can be spliced ​​with an image of 1 / 4 resolution (indicated by "4×" in the figure) of the original image; after splicing, it can enter downsampling module 3 for the next round of feature learning and 2x downsampling, thereby outputting feature image 3 with a resolution of 1 / 8 of the original input image (indicated by "8×" in the figure); the levels of feature images 1-3 are gradually improved;

[0088] The decoder can use a skip connection to receive the feature image 3 from the encoder, and can output a segmentation image 1 after passing through the upsampling module 1. The resolution of the segmentation image 1 can be upsampled to twice that of the feature image 3; the segmentation image 1 can be fused with the feature image 2 of the encoder received using the skip connection; the fused image can output a segmentation image 2 after passing through the upsampling module 2, and the resolution of the segmentation image 2 can be upsampled to four times that of the feature image 3; the segmentation image 2 can be fused with the feature image 1 of the encoder received using the skip connection; the fused image can output a segmentation image 3 after passing through the upsampling module 3, thereby outputting a semantic segmentation image with the same resolution as the original input image.

[0089] Among them, during the upsampling process using the upsampling module 1, the feature image 3 can be regarded as the feature image of the current level, the segmentation image 1 can be regarded as the first image to be fused, and the feature image 2 of the encoder received using the jump connection can be regarded as the second image to be fused. During the upsampling process using the upsampling module 2, the image fused with the segmentation image 1 and the feature image 2 can be regarded as the feature image of the current level, the segmentation image 2 can be regarded as the first image to be fused, and the feature image 1 of the encoder received using the jump connection can be regarded as the second image to be fused. During the upsampling process using the upsampling module 3, the image fused with the segmentation image 2 and the feature image 1 can be regarded as the feature image of the current level, and the segmentation image 3 can be regarded as the final semantic segmentation image.

[0090] Among them, at least one module among the upsampling modules 1-3 can be upsampled based on the convolution kernel determined according to the trained spatial transformation matrix. It can be considered that the upsampling process of at least one level in the layer-by-layer upsampling process can be implemented based on the convolution kernel determined according to the trained spatial transformation matrix. In some implementations, the upsampling modules in the encoder can be gradually replaced with modules containing convolution kernels of spatial transformation, which is conducive to analyzing the contribution of convolution kernels containing spatial transformation to upsampling at different levels and is conducive to the construction of an optimal lightweight segmentation model.

[0091] Based on the above embodiments, the embodiments of the present invention provide a detailed description of the upsampling process of different levels based on convolution kernels. The feature image includes feature images of multiple levels; upsampling the feature image through the decoder may include upsampling the feature image of the highest level level by the decoder. By using each convolution kernel for level-by-level upsampling, the image segmentation efficiency and segmentation accuracy can be improved. In addition, the image segmentation and model training methods proposed in the embodiments of the present invention and the above embodiments belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0092] In this embodiment, the steps for fusing the first image to be fused with the second image to be fused are described in detail. In the process of fusing features at both ends of a jump connection, parallel multi-dilation rate dilated convolutions followed by point convolutions are performed in series, effectively alleviating grid artifacts in the fused image.

[0093] Figure 7 FIG1 shows a flow chart of the image fusion step in an image segmentation method provided by an embodiment of the present invention. Figure 7 In the image segmentation method provided in this embodiment, fusing the first image to be fused and the second image to be fused to obtain a fused image includes:

[0094] S710 : stitching images of each channel in the first image to be fused to obtain a first stitched image, and stitching images of each channel in the second image to be fused to obtain a second stitched image.

[0095] For example, Figure 8 FIG2 shows a schematic diagram of an image fusion framework in an image segmentation method provided by an embodiment of the invention. Figure 8 , the first / second image to be fused (the high-level image / low-level image in the figure) can be stitched together according to their respective channel images by a stitching module (the Concat module in the figure), that is, each channel in the high-level image is stitched together, and each channel in the low-level image is stitched together to obtain the first / second stitched image.

[0096] S720 , processing the first stitched image by dilated convolutions with multiple dilation rates to obtain a first convolution image group, and processing the second stitched image by dilated convolutions with multiple dilation rates to obtain a second convolution image group.

[0097] See also Figure 8The first / second spliced ​​image can be input into parallel compensating dilated convolution modules with different dilation rates. For example, the dilated convolution modules with dilation rates (D-Conv rate) of 1, 2, 4, and 8 in the figure respectively obtain the first / second convolution image group. Each group of images can contain convolution images corresponding to dilated convolutions with dilation rates of 1, 2, 4, and 8. Dilated convolutions with multiple dilation rates can cover a variety of effective receptive fields to capture multi-scale contextual information, enhancing the model's ability to analyze complex scenes.

[0098] S730 , performing point convolution on the first convolution image group and the second convolution image group, which correspond to the dilated convolution with the same dilation rate, and fusing the point convolution results to obtain a fused image.

[0099] Grid artifacts occur in traditional dilated convolutions. For image semantic segmentation tasks, grid artifacts can cause the predicted segmentation map to be uneven, thus affecting segmentation accuracy and visual quality.

[0100] See also Figure 8 In this embodiment, after extracting the features of the multi-scale first / second convolution image group using dilated convolution, the first convolution image and the second convolution image corresponding to the dilated convolution with the same dilation rate can be point-convolved through four point convolution modules (P-Conv). By connecting the dilated convolution and the point convolution in series, the channel feature correlation can be fully utilized to compensate for the information loss at each pixel position, so as to alleviate the gridding artifact phenomenon caused by the dilated convolution. Finally, the results of the point convolution can be spliced ​​(Concat), and the convolution can be used to further aggregate the batch normalization layer and the activation layer (Conv+Bn+ReLU) in series to ensure the full fusion of multi-scale features.

[0101] For example, Figure 9 FIG1 shows a partial schematic diagram of an image fusion framework in an image segmentation method provided by an embodiment of the invention. Figure 9, the dilated convolution with a dilation rate of 1 can capture the close contextual relationship in area A, and the dilated convolution with a dilation rate of 2 can capture the relatively long-distance contextual relationship in area B. Since the features extracted by the dilated convolutions with different dilation rates come from irrelevant areas of the input feature map, there will be information gaps in the channels of the features after splicing. This embodiment introduces point convolution (P-Conv) after each dilated convolution branch, which can compensate for the information at each position using channel correlation before feature aggregation. For example, the point convolution acting on area C can not only learn the features of each position independently, but also use channel information to compensate for the gaps in local contextual position information caused by regular dilated convolution. This can promote the extraction and fusion of multiple spatial structural features, which is beneficial for the decoder to learn contextual relationships in multiple modalities.

[0102] Based on the above embodiment, the present embodiment describes in detail the steps for fusing the first image to be fused with the second image to be fused. In the process of fusing the features at both ends of the jump connection, by serially performing point convolutions after parallel multi-dilation rate dilated convolutions, the grid artifact phenomenon in the fused image can be effectively alleviated. Furthermore, the image segmentation and model training methods proposed in the present embodiment and the above embodiment are based on the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment. This embodiment has the same beneficial effects as the above embodiment.

[0103] In this embodiment, the step of determining a new feature image of the current level based on the fused image is described in detail. By capturing the dual information of content attention and position attention in the fused image, the internal features of the fused image can be optimized, thereby improving the accuracy of semantic segmentation.

[0104] Figure 10 FIG1 shows a flow chart of determining a new current level image in an image segmentation method provided by an embodiment of the present invention. Figure 10 In the image segmentation method provided in this embodiment, determining a new feature image of the current level based on the fused image may include:

[0105] S1001. The fused image is subjected to respective preset convolutions to obtain a query matrix, a key matrix, and a value matrix, and a first attention feature map is determined based on the query matrix, the key matrix, and the value matrix.

[0106] For example, Figure 11 FIG1 shows a flow chart of self-attention feature map fusion in an image segmentation method provided by an embodiment of the present invention. Figure 11 (a), fused image X (where ) can obtain the corresponding query matrix Q (where ), key matrix K (where ) and the value matrix V (where ). Among them, d k It can represent the hidden feature dimension, and d k <<C, which can greatly reduce the computational cost of matrix multiplication to adapt to the real-time image semantic segmentation task.

[0107] Among them, the first attention feature map can be considered as a feature map based on content attention, and can be determined by the following steps: the product of the query matrix Q and the key matrix K is normalized into a probability distribution form through the softmax function to obtain a content attention map; the content attention map is applied to the value matrix V to obtain a weighted feature map, namely the first attention feature map.

[0108] For example, the first attention feature map can be calculated by the following formula:

[0109]

[0110] Among them, Att SA can represent the first attention feature map, Q can represent the query matrix, K T It can be expressed as the transposed matrix of the key matrix, d k can represent the hidden feature dimension, and V can represent the value matrix.

[0111] S1002. Construct a column space position matrix according to the column direction parameters and the column direction attention mask in the fused image, and construct a row space position matrix according to the row direction parameters and the row direction attention mask.

[0112] In this embodiment, in addition to leveraging content-level dependencies to enhance the fused image, spatial-level dependencies are also utilized to optimize the fused image. By modeling the self-attention relationships between pixel points in space, position-sensitive information can be incorporated, thereby improving the accuracy of detail prediction in the fused image.

[0113] in, Figure 11 (b) Figure 11 (a) determines the specific expansion of the second attention feature map. Figure 11 (b) The spatial relationship of the fused image can be modeled in two intersecting directions: columns and rows, i.e., a column / row spatial position matrix can be constructed. In some implementations, a hyperparameter L can be introduced to first capture the attention relationship within the L×L neighborhood within the column range, and then concatenate the attention relationship within the L×L neighborhood within the row range. For example, the following can be included:

[0114] First, the column position learning parameters can be defined (i.e. column direction parameters) and row position learning parameters (i.e. row direction parameter); where d k is the hidden layer feature dimension and can be compared with d in the first attention feature image k The value remains consistent.

[0115] Next, for the feature rel_columns on rel_columns (a，b) , we can use the column-wise attention mask I x (where I x ∈{0, 1} H×H×(2L--1) ) limits the information interaction within the L×L domain. For example, the feature (a, b) neighborhood (-L / 2, L / 2) takes the value 1, and the other positions take the value 0. Similarly, for the feature rel_rows on rel_rows (a，b) , we can use the row-wise attention mask I y (where I y ∈{0, 1} W×W×(2L-1) )Limit information interaction within the L×L domain.

[0116] Finally, the column spatial position matrix is ​​constructed based on the column direction parameters and the column direction attention mask, and the row spatial position matrix is ​​constructed based on the row direction parameters and the row direction attention mask. The column spatial position matrix and the row spatial position matrix can be calculated using the following formula:

[0117] P x =einsum(′HHd k , rd k →HHd′,I x ,rel columns );

[0118] P y =einsum(′HHd k , rd k →HHd′,I y ,rel rows );

[0119] r = (2L-1);

[0120] Among them, the einsum function implements matrix multiplication according to the set dimension change method, for example, for H×H×d k Size and r×d k A matrix of size d k Dimensional matrix multiplication is performed to obtain a matrix of size H×H×d. x and rel_columns (a，b) Perform matrix multiplication to obtain the column space position matrix P x , by putting Iy and rel_rows (a，b) Perform matrix multiplication to obtain the row space position matrix P y .

[0121] S1003. Determine a second attention feature map based on the column space position matrix, the row space position matrix, the query matrix, and the value matrix.

[0122] The second attention feature image can be considered as a feature map based on spatial position attention and can be determined by the following steps:

[0123] In P x Based on , the column-wise attention feature map can be obtained based on the following formula: h =Q·P x T V; among them, can represent the column-wise attention feature map, Q can represent the query matrix, P x T can represent the transposed matrix of the column space position matrix, and V can represent the value matrix. See also Figure 11 , you can use Y h After the regularization layer, it is normalized (BN) and compared with P y T And multiply by Q to get the second attention feature image.

[0124] For example, the second attention feature map can be calculated by the following formula:

[0125] Att CA =Q·P y T ·BN(Y h );

[0126] Among them, Att CA can represent the second attention feature map, Q can represent the query matrix, P y T It can represent the transposed matrix of the row space position matrix, BN() can represent the normalization process, Y h It can represent the column-wise attention feature map. In addition, the row-wise attention feature map can be calculated first, normalized, and compared with P x T And multiply by Q to get the second attention feature image.

[0127] Different from content attention, spatial position attention does not directly use key matrix features to construct the attention map, but instead constructs a column / row spatial position matrix based on column / row direction parameters for querying the value matrix.

[0128] S1004. Determine a new feature image of the current level based on the first attention feature map and the second attention feature map.

[0129] Last seen Figure 11 (a) The first attention feature map and the second attention feature map can be summed to obtain a new feature image X' of the current level (where By feeding the fused image into the dual self-attention module, context and position dependencies can be learned. In addition, to enhance information interaction between channels, the new feature image can also be processed by a conventional convolutional layer (conv-1) to further fully fuse the features.

[0130] This embodiment of the present invention, based on the above embodiment, describes in detail the step of determining a new feature image of the current level based on the fused image. By capturing both content attention and position attention in the fused image, the internal features of the fused image can be optimized, thereby improving the accuracy of semantic segmentation.

[0131] It is worth noting that the image semantic segmentation model in any embodiment of the present invention can construct the backbone network of the encoder-decoder based on the Efficient Sub-pixel Convolution (ESP) operator. Among them, the upsampling module in the encoder can include a convolution kernel based on the spatial transformation matrix provided by the embodiment of the present invention. On this basis, the features at both ends of the jump connection can be fused based on the tandem convolution module of the extended convolution and point convolution provided in the embodiment of the present invention; and / or, the internal features of the fused image can be optimized based on the dual attention module based on content attention and spatial position attention provided in the embodiment of the present invention. Thereby, the accuracy of the semantic segmentation model can be further improved.

[0132] In addition, the image segmentation and model training methods proposed in the embodiments of the present invention and the above embodiments belong to the same inventive concept. For technical details not described in detail in this embodiment, please refer to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0133] Figure 12 A flowchart of a model training method provided by an embodiment of the present invention is shown. The model training method provided by an embodiment of the present invention is applicable to the case of training an image segmentation model with an encoder-decoder structure. The method can be performed by a model training device, which is implemented in software and / or hardware and is preferably configured in a terminal device, such as a computer.

[0134] like Figure 12 As shown, the model training method provided in the embodiment of the present invention includes the following steps:

[0135] S1201: Train a spatial transformation matrix based on the first sample image and the second sample image, and determine any convolution kernel in the embodiments of the present invention according to the spatial transformation matrix.

[0136] In this embodiment, the convolution kernel is a convolution kernel used for upsampling in the encoder, the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0137] The convolution kernel is determined based on the following steps: using the spatial transformation matrix to downsample the second sample image to a transformed image with the same resolution and the same number of channels as the first sample image; training the spatial transformation matrix so that the actual loss value between the first sample image and the transformed image meets the preset conditions; and determining the convolution kernel based on the inverse matrix of the trained spatial transformation matrix.

[0138] The method comprises the following steps: downsampling the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image using a spatial transformation matrix, wherein the method comprises: sliding a sliding window in the second sample image to sequentially determine a currently downsampled local image in the second sample image; deforming the local image into a feature vector, and transforming the feature vector into pixel points of each preset channel according to the spatial transformation matrix; generating a transformed image according to the pixel points of each preset channel when the sliding is completed; wherein the window size of the sliding window is determined according to a resolution compression factor from the second sample image to the first sample image; and the number of channels of each preset channel is the same as the number of channels of the first sample image.

[0139] S1202: Fix the convolution kernel in the encoder and train the image segmentation model including the encoder.

[0140] In this embodiment, the image segmentation model can be trained in two stages. The first stage can train the spatial transformation matrix corresponding to the convolution kernels used for upsampling in each upsampling module of the encoder; the second stage can train the overall image segmentation model after the convolution kernel is fixed. The two-stage training allows the performance of the decoder to be tested separately, thereby reducing the impact of the encoder training process on the decoder, improving the efficiency of model training, and helping to complete the experiment efficiently.

[0141] In addition, the image segmentation method proposed in the embodiment of the present invention and the above embodiment belongs to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0142] Figure 13FIG2 is a schematic diagram showing the structure of an image segmentation device provided by an embodiment of the present invention. The image segmentation device provided by an embodiment of the present invention is applicable to the case of real-time image semantic segmentation, for example, the case of semantic segmentation of real-time collected road images in the field of autonomous driving.

[0143] like Figure 13 As shown, the image segmentation device in the embodiment of the present invention includes:

[0144] A downsampling module 1301 is used to downsample the image to be segmented through an encoder to obtain a feature image;

[0145] An upsampling module 1302 is used to upsample the feature image through a decoder to obtain a segmented image;

[0146] Among them, the convolution kernel used for upsampling in the decoder is determined according to the spatial transformation matrix between the trained first sample image and the second sample image. The first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0147] In some optional implementations, the image segmentation device may further include:

[0148] A convolution kernel determination module is used to determine the convolution kernel based on the following steps:

[0149] downsampling the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image using a spatial transformation matrix;

[0150] Training the spatial transformation matrix so that the actual loss value between the first sample image and the transformed image meets a preset condition;

[0151] The convolution kernel is determined according to the inverse matrix of the trained spatial transformation matrix.

[0152] In some optional implementations, the convolution kernel determination module may be specifically configured to:

[0153] Sliding the sliding window in the second sample image, and sequentially determining the currently downsampled local image in the second sample image;

[0154] The local image is deformed into a feature vector, and the feature vector is transformed into pixel points of each preset channel according to the spatial transformation matrix;

[0155] When the sliding is completed, a transformation image is generated according to the pixel points of each preset channel;

[0156] The window size of the sliding window is determined according to a resolution compression factor from the second sample image to the first sample image; and the number of channels of each preset channel is the same as the number of channels of the first sample image.

[0157] In some optional embodiments, the feature image includes feature images of multiple levels; the upsampling module can be used to upsample the feature image of the highest level level by level through the decoder;

[0158] During the layer-by-layer upsampling process, for any layer upsampling process, the upsampling module can be used to:

[0159] Upsampling the feature image of the current level based on the convolution kernel to obtain the first image to be fused;

[0160] The feature image in the encoder with the same resolution as the first image to be fused is used as the second image to be fused;

[0161] The first image to be fused and the second image to be fused are fused to obtain a fused image, and a new feature image of the current level is determined according to the fused image.

[0162] In some optional implementations, the upsampling module may be specifically configured to:

[0163] Stitching the images of each channel in the first image to be fused to obtain a first stitched image, and stitching the images of each channel in the second image to be fused to obtain a second stitched image;

[0164] The first stitched image is processed by dilated convolution with multiple dilation rates to obtain a first convolution image group, and the second stitched image is processed by dilated convolution with multiple dilation rates to obtain a second convolution image group;

[0165] Perform point convolution on the first convolution image group and the second convolution image group, which correspond to the dilated convolution with the same dilation rate, and fuse the convolution results of each point to obtain a fused image.

[0166] In some optional implementations, the upsampling module may be specifically configured to:

[0167] The fused image is convolved with each preset matrix to obtain a query matrix, a key matrix, and a value matrix, and the first attention feature map is determined according to the query matrix, the key matrix, and the value matrix;

[0168] Construct a column space position matrix according to the column direction parameters and the column direction attention mask in the fused image, and construct a row space position matrix according to the row direction parameters and the row direction attention mask;

[0169] determining a second attention feature map according to the column space position matrix, the row space position matrix, the query matrix, and the value matrix;

[0170] Determine a new feature image of the current level according to the first attention feature map and the second attention feature map.

[0171] In some optional implementations, the image segmentation device can be applied to autonomous driving applications, and the image to be segmented includes a road image collected in real time.

[0172] The image segmentation device provided in the embodiment of the present invention belongs to the same inventive concept as the image segmentation method provided in the above embodiment. Technical details not fully described in the embodiment of the present invention can be referred to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.

[0173] Figure 14 FIG2 is a schematic diagram showing the structure of a model training device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of training an image segmentation model with an encoder-decoder structure.

[0174] like Figure 14 As shown, the model training device in the embodiment of the present invention may include:

[0175] The first stage training module 1401 is used to train a spatial transformation matrix based on the first sample image and the second sample image, and determine any convolution kernel in the embodiments of the present invention according to the spatial transformation matrix.

[0176] The convolution kernel is a convolution kernel used for upsampling in the encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel;

[0177] The second stage training module 1402 is used to fix the convolution kernel in the encoder and train the image segmentation model including the encoder.

[0178] The model training device provided in the embodiment of the present invention belongs to the same inventive concept as the model training method provided in the above embodiment. For technical details not fully described in the embodiment of the present invention, please refer to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.

[0179] Figure 15 The figure shows a hardware structure diagram of a terminal device provided by an embodiment of the present invention. The terminal device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 15 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0180] like Figure 15 As shown, terminal device 1500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage device 1508 into a random access memory (RAM) 1503. Various programs and data required for the operation of terminal device 1500 are also stored in RAM 1503. Processing device 1501, ROM 1502, and RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to bus 1504.

[0181] Typically, the following devices may be connected to the I / O interface 1505: an input device 1506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1509. The communication device 1509 may allow the terminal device 1500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 15 The terminal device 1500 is shown as having various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0182] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1509, or installed from the storage device 1508, or installed from the ROM 1502. When the computer program is executed by the processing device 1501, the above-mentioned functions defined in the image segmentation method or model training method provided by the embodiment of the present invention are performed.

[0183] The terminal provided in the embodiment of the present invention and the image segmentation method or model training method provided in the above embodiment belong to the same inventive concept. Technical details not fully described in the embodiment of the present invention can be referred to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.

[0184] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the image segmentation method or model training method provided in the above embodiment is implemented.

[0185] It should be noted that the computer-readable storage medium described above in the embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the embodiments of the present invention, the computer-readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0186] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0187] The computer-readable storage medium may be included in the terminal device, or may exist independently without being incorporated into the terminal device.

[0188] The terminal device stores and carries one or more programs. When the one or more programs are executed by the terminal device, the terminal device:

[0189] The image to be segmented is downsampled through the encoder to obtain a feature image;

[0190] The feature image is upsampled through the decoder to obtain the segmented image;

[0191] Among them, the convolution kernel used for upsampling in the decoder is determined according to the spatial transformation matrix between the trained first sample image and the second sample image. The first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel.

[0192] Alternatively, the terminal device stores and carries one or more programs, and when the one or more programs are executed by the terminal device, the terminal device:

[0193] A spatial transformation matrix is ​​trained based on the first sample image and the second sample image, and a convolution kernel according to any one of the embodiments of the present invention is determined according to the spatial transformation matrix; wherein the convolution kernel is a convolution kernel used for upsampling in the encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel;

[0194] The convolution kernel in the encoder is fixed, and the image segmentation model including the encoder is trained.

[0195] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the function pages marked in the box can occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0197] The units involved in the embodiments of the present invention may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0198] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0199] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image segmentation method, characterized in that: include: The image to be segmented is downsampled through the encoder to obtain a feature image; Upsampling the feature image through a decoder to obtain a segmented image; The feature image includes feature images of multiple levels; upsampling the feature image by the decoder includes upsampling the feature image of the highest level level by level by the decoder; The convolution kernel used for upsampling in the decoder is determined according to the spatial transformation matrix between the trained first sample image and the second sample image, the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel; The convolution kernel is determined based on the following steps: downsampling the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image using the spatial transformation matrix; Training the spatial transformation matrix so that an actual loss value between the first sample image and the transformed image meets a preset condition; The convolution kernel is determined according to the inverse matrix of the trained spatial transformation matrix.

2. The method according to claim 1, characterized in that The downsampling the second sample image to a transformed image having the same resolution and the same number of channels as the first sample image by using the spatial transformation matrix includes: sliding a sliding window in the second sample image to sequentially determine a currently downsampled local image in the second sample image; Deforming the local image into a feature vector, and transforming the feature vector into pixel points of each preset channel according to the spatial transformation matrix; When the sliding is completed, a transformed image is generated according to the pixel points of each preset channel; The window size of the sliding window is determined according to a resolution compression factor from the second sample image to the first sample image; and the number of channels of each preset channel is the same as the number of channels of the first sample image.

3. The method according to claim 1, characterized in that In the layer-by-layer upsampling process, for any layer upsampling process, the process includes: Upsampling the feature image of the current level based on the convolution kernel to obtain a first image to be fused; Using a feature image in the encoder having the same resolution as the first image to be fused as a second image to be fused; The first image to be fused and the second image to be fused are fused to obtain a fused image, and a new feature image of the current level is determined according to the fused image.

4. The method according to claim 3, wherein The fusing the first image to be fused and the second image to be fused to obtain a fused image includes: Stitching the images of each channel in the first image to be fused to obtain a first stitched image, and stitching the images of each channel in the second image to be fused to obtain a second stitched image; Processing the first stitched image by dilated convolutions with multiple dilation rates to obtain a first convolution image group, and processing the second stitched image by dilated convolutions with the multiple dilation rates to obtain a second convolution image group; Performing point convolution on the first convolution image group and the second convolution image group corresponding to the dilated convolution with the same dilation rate, and fusing the point convolution results to obtain a fused image.

5. The method according to claim 3, characterized in that The determining of a new feature image of the current level according to the fused image includes: The fused image is subjected to respective preset convolutions to obtain a query matrix, a key matrix, and a value matrix, and a first attention feature map is determined according to the query matrix, the key matrix, and the value matrix; Constructing a column space position matrix according to the column direction parameters and the column direction attention mask in the fused image, and constructing a row space position matrix according to the row direction parameters and the row direction attention mask; determining a second attention feature map according to the column space position matrix, the row space position matrix, the query matrix, and the value matrix; Determine a new feature image of the current level based on the first attention feature map and the second attention feature map.

6. The method according to any one of claims 1 to 5, characterized in that: Applied to autonomous driving applications, the image to be segmented includes a road image collected in real time.

7. A model training method, characterized in that: include: A spatial transformation matrix is ​​trained based on a first sample image and a second sample image, and a convolution kernel according to any one of claims 1 to 6 is determined according to the spatial transformation matrix; wherein the convolution kernel is a convolution kernel used for upsampling in an encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel; The convolution kernel in the encoder is fixed, and the image segmentation model including the encoder is trained.

8. An image segmentation device, characterized in that: include: A downsampling module is used to downsample the image to be segmented through an encoder to obtain a feature image; An upsampling module is used to upsample the feature image through a decoder to obtain a segmented image; the feature image includes feature images of multiple levels; the upsampling module is used to upsample the feature image of the highest level layer by layer through the decoder; The convolution kernel used for upsampling in the decoder is determined according to a spatial transformation matrix between a first sample image and a second sample image that have been trained, wherein the first sample image is the image before upsampling corresponding to the convolution kernel, and the second sample image is the image after upsampling corresponding to the convolution kernel; A convolution kernel determination module is used to determine the convolution kernel based on the following steps: using a spatial transformation matrix to downsample the second sample image to a transformed image with the same resolution and the same number of channels as the first sample image; training the spatial transformation matrix so that the actual loss value between the first sample image and the transformed image meets a preset condition; and determining the convolution kernel based on the inverse matrix of the trained spatial transformation matrix.

9. A model training device, characterized in that: include: a first-stage training module, configured to train a spatial transformation matrix based on a first sample image and a second sample image, and determine the convolution kernel according to any one of claims 1 to 6 according to the spatial transformation matrix; wherein the convolution kernel is a convolution kernel used for upsampling in an encoder, the first sample image is an image before upsampling corresponding to the convolution kernel, and the second sample image is an image after upsampling corresponding to the convolution kernel; The second stage training module is used to fix the convolution kernel in the encoder and train the image segmentation model including the encoder.

10. A terminal device, characterized in that: The terminal includes: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation method as described in any one of claims 1 to 6, or the model training method as described in claim 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image segmentation method according to any one of claims 1 to 6 or the model training method according to claim 7 is implemented.

Images