Image processing method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, electronic equipment and a computer storage medium, and the method comprises the steps: obtaining a target image, carrying out the feature extraction of the target image based on a feature extraction network, determining an image feature map, carrying out the feature decoding of the image feature map based on a decoder, and generating a feature segmentation map. And performing coordinate conversion on the feature segmentation image to determine a target segmentation image. Therefore, feature extraction and feature decoding are firstly carried out on the target image to determine the feature segmentation image, then coordinate conversion is carried out to determine the target segmentation image, interference factors in the feature segmentation process can be reduced, and the accuracy of an image feature segmentation result is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image processing method, device, electronic equipment and computer storage medium. Background Art

[0002] The bird's-eye view segmentation task, also known as the BEV segmentation task, is a common image processing process that refers to drawing a semantic segmentation map of objects and layouts from a bird's-eye view based on the RGB image in the perspective view to convert the image from a first-person perspective to a bird's-eye view, and assigning objects in the bird's-eye view to predefined semantic categories such as roads, pedestrians, vehicles, etc. to achieve detailed analysis of the scene.

[0003] Existing approaches to BEV segmentation tasks typically involve first converting a polar coordinate perspective image into a bird's-eye view, then converting the features in the polar bird's-eye view into Cartesian features, and finally performing segmentation in Cartesian coordinates to generate the BEV segmentation result. However, because the coordinate conversion process is affected by camera internal and external parameters, the BEV segmentation features and the BEV segmentation map may not align, affecting the accuracy of the segmentation results. Summary of the Invention

[0004] In view of this, an object of an embodiment of the present invention is to provide an image processing method to improve the accuracy of image feature segmentation results.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0006] Acquire the target image;

[0007] Performing feature extraction on the target image based on a feature extraction network to determine an image feature map;

[0008] Performing feature decoding on the image feature map based on a decoder to generate a feature segmentation map;

[0009] Perform coordinate transformation on the feature segmentation map to determine a target segmentation map.

[0010] In a second aspect, an embodiment of the present invention is directed to providing an image processing device, the device comprising:

[0011] An image acquisition unit, configured to acquire a target image;

[0012] A feature extraction unit, configured to extract features from the target image based on a feature extraction network and determine an image feature map;

[0013] A feature segmentation unit, configured to perform feature decoding on the image feature map based on a decoder to generate a feature segmentation map;

[0014] The coordinate conversion unit is used to perform coordinate conversion on the feature segmentation map to determine a target segmentation map.

[0015] In a third aspect, an embodiment of the present invention aims to provide a computer program product, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, implements the method as described in any one of the above items.

[0016] In a fourth aspect, an embodiment of the present invention aims to provide an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of the above items.

[0017] In a fifth aspect, an embodiment of the present invention aims to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps as described in any one of the above items are implemented.

[0018] The technical solution of the embodiment of the present invention obtains a target image, extracts features from the target image using a feature extraction network to determine an image feature map, decodes the image feature map using a decoder to generate a feature segmentation map, and then performs coordinate transformation on the feature segmentation map to determine a target segmentation map. Thus, by first extracting and decoding features from the target image to determine a feature segmentation map, and then performing coordinate transformation to determine the target segmentation map, it is possible to perform view transformation based on accurate feature segmentation results, reduce the influence of interference factors during the segmentation process, and thus improve the accuracy of image feature segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 is a flowchart of an image processing method according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of an image processing process according to an embodiment of the present invention;

[0022] Figure 3 is a flowchart of a method for determining an image feature map according to an embodiment of the present invention;

[0023] Figure 4 is a flow chart of a feature alignment method according to an embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of a feature alignment process according to an embodiment of the present invention;

[0025] Figure 6 is the correspondence between the feature segmentation map and the target image in the embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of coordinate conversion according to an embodiment of the present invention;

[0027] Figure 8 is a flowchart of a method for determining a target segmentation map according to an embodiment of the present invention;

[0028] Figure 9 is a flow chart of a sampling method according to an embodiment of the present invention;

[0029] Figure 10 is a schematic diagram of a model training method according to an embodiment of the present invention;

[0030] Figure 11 Schematic diagram of the working process of the autoencoder according to an embodiment of the present invention;

[0031] Figure 12 is a flowchart of a method for determining an autoencoder according to an embodiment of the present invention;

[0032] Figure 13 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention;

[0033] Figure 14 is another schematic diagram of an image processing apparatus according to an embodiment of the present invention;

[0034] Figure 15 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present application is described below based on the following embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0036] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0037] Unless the context clearly requires otherwise, words like “include”, “comprising” and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0038] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0039] The existing BEV segmentation task processing method generally follows the following steps: based on the feature extractor, the key features in the RGB image are extracted, and then the perspective conversion module converts the perspective image (RGB image) into a bird's-eye view (BEV), and finally the segmentation head (i.e., the segmentation decoder) decodes the bird's-eye view features and generates the segmentation results. At the same time, the existing perspective conversion methods usually include PON and TIM. In the PON method, by adopting multiple MLPs (multi-layer perceptrons), the bird's-eye view features in the polar coordinate system are obtained by first compressing along the height axis and then extending along the depth axis, and then resampled back to the Cartesian coordinate system and segmented in the Cartesian coordinate system. In the TIM method, the perspective features are converted into bird's-eye view features in the polar coordinate system through a transformer, thereby completing the segmentation task.

[0040] Therefore, no matter which method is adopted, it involves first converting the RGB perspective image into a bird's-eye view in a polar coordinate system, and then converting the polar coordinate features in the bird's-eye view into features in a Cartesian coordinate system. However, since the process of resampling the polar coordinate features into Cartesian coordinate features will be affected by the internal and external parameters of the camera, errors will occur, which will in turn cause errors in the segmentation features of the bird's-eye view, which will not align with the actual BEV segmentation map, resulting in inaccurate segmentation results. At the same time, since existing BEV segmentation tasks are end-to-end operations, it is necessary to simultaneously train and optimize the perspective conversion module and the segmentation decoder to achieve task segmentation, which will lead to unclear model optimization goals and greater optimization difficulty. In view of this, an embodiment of the present invention aims to provide an image processing method to improve the accuracy of image feature segmentation results.

[0041] At the same time, this embodiment takes the image processing process in the BEV segmentation task processing in the autonomous driving scenario as an example to illustrate. By processing the target image in the autonomous driving scene to determine the corresponding target segmentation map, it is convenient to perform subsequent tasks such as target detection and path planning based on the target segmentation map.

[0042] However, it should be understood that the image processing in this embodiment can also be applied to other scenarios that require segmentation of image features, such as obstacle detection during delivery by intelligent delivery robots, and the application scenarios of this method are not limited here.

[0043] Figure 1 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 2This is a schematic diagram of the image processing process of an embodiment of the present invention. Figure 1 and Figure 2 The image processing method in this embodiment is described.

[0044] like Figure 1 As shown, the image processing method in this embodiment includes the following steps.

[0045] In step S110 , a target image is acquired.

[0046] In this embodiment, the target image is a view image of the target vehicle's surroundings, which can be a front view, rear view, or side view, among others. Optionally, in this embodiment, an image acquisition device mounted on the target vehicle can be used to acquire the target image in a preset coordinate system. The mounting position of the image acquisition device can be selected based on the angle and field of view of the captured view image. Thus, this arrangement enables acquisition of view images from different directions around the target vehicle, thereby providing assistance in operating the target vehicle through image analysis and ensuring safe and convenient driving.

[0047] Alternatively, as Figure 2 As shown, the target image in this embodiment adopts the front view of the target vehicle, and the target image is usually obtained by a driving recorder installed in the upper center position of the front windshield. Of course, other types of image acquisition devices at other settings can also be used to obtain the target image. This is just an example. Among them, the image format of the front view is an RGB image (when the front view is in YUV or other image format, it can be converted to RGB format before processing). The front view includes information such as traffic participants (such as different types of vehicles, pedestrians, etc.) and traffic environment information (such as road conditions, green belts, buildings, etc.) within the forward field of view of the target vehicle. Therefore, in this embodiment, by performing image processing on the target image to determine the corresponding target segmentation map, it can facilitate the target vehicle to perceive the surrounding environment of the vehicle, provide a reliable basis for the safe and standardized driving of the vehicle, and thus enhance the user experience of the vehicle.

[0048] In step S120, feature extraction is performed on the target image based on the feature extraction network to determine an image feature map.

[0049] In this embodiment, feature extraction is performed on the target image through a feature extraction network to determine the image feature map corresponding to the target image, which can realize the transition of the target image from the image features in the RGB image domain to the implicit spatial features in the implicit spatial domain, thereby facilitating feature segmentation of the target image based on the implicit spatial features. Among them, the implicit space is a representation of compressed data. The role of the implicit space is to learn data features and simplify data representation in order to find patterns. Data compression refers to encoding information with fewer bits than the original representation, with the purpose of learning the more important information in the data. In this embodiment, the dimensionality reduction of data in feature extraction is regarded as a lossy compression. At the same time, since the decoder will be used to reconstruct the data later, the feature extraction network model must learn how to store all relevant information and ignore noise. The advantage of compression (dimensionality reduction) is that it can remove redundant information to focus on the most critical features, and this compressed state is the implicit spatial representation of the data.

[0050] Optionally, in this embodiment, different types of feature extraction networks (e.g., convolutional neural networks (CNNs), autoencoders (Autoencoders), generative adversarial networks (GANs), etc.) can be used to extract features from the target image to determine an image feature map corresponding to the target image. The feature extraction network can be a single, integrated network model or composed of multiple local networks, depending on the actual application scenario and requirements.

[0051] Furthermore, if Figure 2 As shown, the feature extraction network in this embodiment includes an extraction network and an alignment network, so as to determine the feature map through the extraction network and the alignment network, thereby improving the overall performance and generalization ability of the feature extraction network. Figure 3 The steps shown in are implemented.

[0052] In step S310, the target image is input into the extraction network to determine the initial feature map.

[0053] Optionally, the extraction network in this embodiment adopts a network structure that combines Resnet+FPN. Resnet is a residual network. By introducing residual connections, the input features can be directly passed to the subsequent layers through the residual connections, so that the feature extraction network can learn higher-level feature representations, effectively solving the gradient disappearance and representation bottleneck problems in deep neural networks. FPN is a feature pyramid network that can fuse feature maps of different scales at the bottom and top levels based on a feature fusion mechanism, thereby making full use of feature information at each stage. Therefore, in this embodiment, by using the Resnet+FPN network to extract features from the feature maps of each layer in the target image, and performing feature fusion through a series of operations to obtain the initial feature map, better feature extraction can be achieved, so that the initial feature map can better express the feature information in the target image.

[0054] In step S320, feature alignment is performed on the initial feature map based on the alignment network to determine the image feature map.

[0055] In this embodiment, due to differences in the image acquisition device's capture angle of view and acquisition time, objects may be occluded in the acquired target image, resulting in some objects or areas not being fully detected. Therefore, in this embodiment, by performing feature alignment on the initial feature map, different feature points in the image can be matched and aligned, thereby integrating and supplementing information from the occluded areas. This improves the integrity and perceptual quality of the extracted features in the image feature map, facilitating better feature segmentation and subsequent task processing.

[0056] Optionally, in this embodiment, the initial feature map can be divided into multiple feature blocks based on rows, columns or blocks, and feature alignment can be performed separately to simplify the feature alignment complexity while ensuring the feature alignment effect, and improve the feature alignment efficiency and flexibility.

[0057] Furthermore, when the target object is an RGB image and the target segmentation map is a bird's-eye view segmentation map, since the objects in the RGB image follow the characteristic of being larger the closer they are and smaller the farther they are, the objects in the finally generated bird's-eye view segmentation map maintain the same proportion regardless of the distance, and the features of the objects in the target image are more closely correlated in the column direction, therefore, in this embodiment, the initial feature map is aligned based on a column method, which can fully and efficiently utilize the relevant information in the image, thereby improving the accuracy of feature extraction of the feature extraction network.

[0058] Furthermore, in this embodiment, when the initial feature map is aligned based on the alignment network to determine the image feature map, the following steps are specifically included: Figure 4 The following steps are shown.

[0059] In step S410, feature alignment is performed on each sequence graph in the initial feature graph based on the alignment network to determine a corresponding alignment feature graph.

[0060] In this embodiment, each sequence graph is a feature graph determined by partitioning the initial feature graph into columns. Therefore, by converting the initial feature graph into a sequence graph corresponding to the columns, the feature alignment task can be transformed into a column-based sequence-to-sequence translation problem. Feature alignment can be used to complement and restore image loss information in the sequence graph, and then determine the alignment feature graph corresponding to each sequence graph, which helps improve the accuracy and efficiency of feature alignment results.

[0061] Optionally, the alignment network in this embodiment adopts a Transformer model to perform feature alignment on each sequence graph in the initial feature graph through the Transformer model. Among them, the Transformer model is a deep learning model based on the attention mechanism, which is mainly used in fields such as sequence data processing and computer vision (CV) in natural language processing (NLP). The Transformer model mainly consists of two parts: an encoder and a decoder. The encoder encodes the input sequence to obtain a high-dimensional representation vector; the decoder generates an output sequence based on the representation vector. Both the encoder and the decoder contain multiple self-attention mechanisms and feedforward neural networks. These components work together to enable the model to learn the complex relationships in the input sequence, better understand the contextual information of the input data, and can parallelize training to improve training efficiency. In addition, the Transformer model also has global information perception capabilities and can better process long sequence data.

[0062] Furthermore, if Figure 5 As shown, in this embodiment, before the initial feature map is input into the Transformer model, the initial feature map is first reshaped from an arrangement of N×C×H×W to H×(N×W)×C, and the sequence map corresponding to each column is used as the sequence input of the Transformer. The Transformer model then performs feature alignment on each sequence map to obtain the corresponding aligned feature map of the shape H×(N×W)×C. In addition, in this embodiment, learnable position encoding can also be used to enhance the model's ability to capture spatial information, which can further optimize the feature alignment effect.

[0063] In this embodiment, the N×C×H×W form and the H×(N×W)×C form of the feature map are two different dimensional representations used to describe the shape of the feature map. The main difference between these two forms is that they describe the dimensions of the feature map in different ways. In practical applications, depending on different needs and contexts, you may choose to use one of the forms to describe the dimensions of the feature map. For example, in a convolutional neural network, the input feature map is usually in the form of N×C×H×W, while the output feature map may be represented in the form of H×(N×W)×C.

[0064] Specifically, in the form of N×C×H×W, N represents the number of feature maps (number of samples), C represents the number of channels of each feature map, and H and W represent the height and width of the feature map, respectively. This form usually focuses more on the shape and size of a single feature map, especially when flattening the image. In the form of H×(N×W)×C, H represents the height of the feature map, N represents the number of feature maps at each height, W represents the width of each feature map, and C represents the number of channels of each feature map. This form usually focuses more on the organization and arrangement of multi-channel feature maps, and the feature maps of each channel have a different number of channels C. For example, in a convolutional neural network, each channel may represent a color channel (such as RGB).

[0065] In step S420, the aligned feature maps are integrated to determine an image feature map.

[0066] In this embodiment, after the alignment feature graphs corresponding to the sequence graphs are determined by the alignment network, the features of each column are integrated according to their relative position relationship on the target image, and an image feature graph after the integration of the alignment feature graphs is obtained. Specifically, Figure 5 As shown, in the process of integrating the aligned feature maps of H×(N×W)×C shape to obtain the image feature map, the feature map shape will be adjusted from H×(N×W)×C back to N×C×H×W shape, that is, the image feature map determined after integrating the aligned feature maps is in the shape of N×C×H×W.

[0067] Optionally, in this embodiment, the feature alignment maps are integrated through the convolution block Conv to determine the image feature map. Specifically, by inputting the aligned feature map in the form of H×(N×W)×C into the convolution block, and setting the size of the convolution kernel to 1×1, the stride to 1×1, the padding to 0, and the number of output channels to C, the feature maps of multiple channels can be flattened into a feature map of one channel through the convolution operation, resulting in an image feature map in the form of N×C×H×W.

[0068] In step S130, feature decoding is performed on the image feature map based on the decoder to generate a feature segmentation map.

[0069] In this embodiment, after determining the image feature map corresponding to the target image, the image feature map is input to the decoder, and the decoder performs feature decoding on the image feature map, performs feature segmentation on the basis of not changing the original coordinate system expression, and generates a feature segmentation map corresponding to the target object, wherein the correspondence between the feature segmentation map and the target image is as follows: Figure 6 As shown in . In each column area, the feature information such as the object label and object contour identified in the feature segmentation map is consistent with the object existing in the target image. Therefore, the image feature map is feature decoded by the decoder. The decoding process is essentially a process of reconstructing the segmentation label map based on the existing implicit spatial features (i.e., the image feature map corresponding to the target image). In this process, since it does not involve changes in the coordinate system and image perspective, it can retain the original feature information in the target image to the greatest extent, avoid the segmentation process from being affected by internal and external parameters of the camera or other interference factors, ensure that the segmentation features are aligned with the target segmentation map, and thus ensure the accuracy of the image segmentation results.

[0070] Optionally, the decoder in this embodiment can be a decoder based on a convolutional neural network, a decoder based on a specific architecture (such as an autoencoder with a symmetric structure, such as U-Net, etc.), or a decoder based on an attention mechanism, a recurrent neural network (RNN) or a Transformer structure. The specific selection can be made according to the actual application scenario and requirements.

[0071] Furthermore, in this embodiment, the decoder in the autoencoder decodes the image feature map to generate a feature segmentation map. This processing method can preserve the structure and information of the original data as much as possible, helping to identify and extract more precise feature boundaries, and improving the accuracy and robustness of feature segmentation.

[0072] In step S140 , coordinate transformation is performed on the feature segmentation map to determine a target segmentation map.

[0073] In this embodiment, since the target image and the target segmentation map usually use different coordinate systems, it is necessary to perform coordinate transformation on the feature segmentation map obtained after processing the target image in the preset coordinate system to generate the target segmentation map in the target coordinate system.

[0074] Optionally, in this embodiment, the target segmentation map is determined under the BEV segmentation task as an example. The target image to be processed is a front view described by a polar coordinate system (i.e., a perspective view under a front view angle), and the final output of the segmentation (i.e., the target segmentation map) is a bird's-eye view segmentation map described by a Cartesian coordinate system (i.e., a segmentation map under a bird's-eye view angle). Therefore, when determining the target segmentation map, if Figure 7As shown, in this embodiment, the feature segmentation map in the polar coordinate system (i.e., the preset coordinate system) is transformed to determine the target segmentation map in the Cartesian coordinate system (i.e., the target coordinate system). At the same time, the feature segmentation map is the implicit spatial representation of the target image under the front view perspective, and the target segmentation map is the implicit spatial representation of the target image under the bird's-eye view perspective.

[0075] However, it should be noted that in this embodiment, only the coordinate conversion from the polar coordinate system to the Cartesian coordinate system is used as an example to facilitate the understanding of the coordinate conversion process, but there is no restriction on the types of the preset coordinate system and the target coordinate system. The corresponding coordinate conversion process can refer to the coordinate conversion method introduced in this embodiment.

[0076] Furthermore, in this embodiment, Figure 8 The method shown determines the target segmentation map.

[0077] In step S810, a target segmentation region corresponding to the feature segmentation map is determined.

[0078] In this embodiment, Figure 7 As shown in the figure, the image area corresponding to the target segmentation region corresponds to the field of view of the target object corresponding to the feature segmentation map. The difference lies in the perspective used to express the image content and the position coordinate system used. The feature segmentation map is a frontal perspective, while the target segmentation region is an image area from a bird's-eye view. The image area is fan-shaped, and the vertex of the fan is the location of the target vehicle.

[0079] In step S820 , the image features in the feature segmentation map are sampled into the target segmentation region to determine the corresponding target segmentation map.

[0080] In this embodiment, after determining the target segmentation area corresponding to the feature segmentation map, the corresponding target segmentation map is determined by sampling the image features in the feature segmentation map to the target segmentation area, thereby converting the feature segmentation map into the target segmentation map, and then realizing the conversion of the segmentation map from the polar coordinate system to the Cartesian coordinate system and the conversion of the front view segmentation map to the bird's-eye view segmentation map. This can reduce the differences between views under different perspectives and different coordinate systems, and improve the interpretability of image processing tasks and subsequent tasks performed based on the image processing results.

[0081] Optionally, in this embodiment, Figure 9 The sampling method shown in is used to determine the target segmentation map, which specifically includes the following steps.

[0082] In step S910, the feature segmentation map is divided into regions to determine a plurality of first sub-regions.

[0083] In this embodiment, Figure 7As shown, the region division is achieved by dividing the feature segmentation map in width and height directions. For example, the width direction can be evenly divided into 10 parts, and the height direction can be evenly divided into 8 parts, forming a total of 80 first sub-regions of the same size.

[0084] In step S920 , the target segmentation area is divided into regions to determine a plurality of second sub-regions, each of which has a unique corresponding first sub-region.

[0085] In this embodiment, when dividing the target segmentation map area, since the target segmentation area is a fan-shaped area, the area division can be achieved by dividing the top angle (i.e., the angular radian) of the fan-shaped area into 10 equal parts and dividing the radius into 8 equal parts, and forming 80 second sub-areas, and the first sub-area has a unique correspondence with the corresponding second sub-area.

[0086] In step S930 , the image features in the first sub-region are assigned to the corresponding second sub-region.

[0087] In this embodiment, before assigning the image features in the first subregion to the corresponding second subregion, the polar coordinates of each feature position point (a position point having an image feature) in the first subregion are first converted to Cartesian coordinates, and based on the position of the Cartesian coordinates in the target feature map, the position of each feature position point in the feature segmentation map in the target segmentation map is determined. At the same time, in this embodiment, the operation of assigning the image features in the first subregion to the corresponding second subregion is essentially the process of assigning the image features at each feature position point in the polar coordinate system in the first subregion to the corresponding feature position point in the Cartesian coordinate system in the target feature map.

[0088] Optionally, in this embodiment, when converting polar coordinates to Cartesian coordinates, for the non-integer Cartesian coordinates obtained after the conversion, corresponding integer coordinates will be generated based on the rounding method. For example, for a feature position point with polar coordinates (13, 22.6°), the Cartesian coordinates obtained after coordinate conversion are (12.002, 4.996). At this time, the Cartesian coordinates need to be rounded to obtain Cartesian coordinates of (12, 5). That is, the feature position point with polar coordinates (13, 22.6°) in the feature segmentation map corresponds to a feature position point with coordinates (12, 5) in the target segmentation map.

[0089] Therefore, in this embodiment, after completing the coordinate transformation of all feature points based on the above method, there may be a situation where multiple feature position points in the feature segmentation map correspond to one feature position point in the target segmentation map. In this case, when assigning the image features in each first sub-region in the feature segmentation map to the second sub-region in the target segmentation map, there may be a situation where the image features in multiple first sub-regions are assigned to one second sub-region.

[0090] Furthermore, when assigning the image features in the first subregion to the corresponding second subregion, the pixel features in the first subregion and the category labels of the pixel features (category labels include roads, cars, pedestrians, etc., and each category label has a matching probability) are directly added to the corresponding second subregion. At the same time, optionally, for a situation where there are multiple first subregions corresponding to a second subregion, and multiple first subregions corresponding to a second subregion have different category labels, in this embodiment, the category label of the second subregion can be determined based on the category labels of each first subregion corresponding to the second subregion. For example, the category label with the largest number of occurrences or the largest matching probability among all the category labels corresponding to the second subregion is used as the category label of the second subregion. Thus, in this embodiment, the features in the corresponding second subregion can be generated based on the image features in the first subregion, thereby achieving complete and accurate migration of the image features in the first subregion to the corresponding second subregion, and improving the accuracy and interpretability of the segmentation features in the second subregion.

[0091] In step S940 , a corresponding target segmentation map is generated according to the image features of each second sub-region.

[0092] In this embodiment, after assigning the image features in each first sub-region to the corresponding second sub-region, a target segmentation map can be generated, thereby realizing the conversion of the feature segmentation map in the polar coordinate system to the target segmentation map from a bird's-eye view in the Cartesian coordinate system. While ensuring the accuracy of the segmentation results, it can improve the interpretability of the corresponding image processing tasks.

[0093] Therefore, in this embodiment, the sampling method described above directly maps each RGB image column to a segmentation target in a Cartesian coordinate system and performs matching one by one. This not only converts the segmentation map from a polar coordinate system to a Cartesian coordinate system, but also makes the final target segmentation map more accurate to the actual situation, improving the accuracy and interpretability of the image segmentation results. Furthermore, determining the target segmentation map in this way can simplify model training, making overall model training more convenient.

[0094] Optionally, in this embodiment, after determining the target segmentation map, the corresponding image post-processing method further includes: performing target detection on the target segmentation map to determine the target object in the target image. Thus, by processing the target image using the above method to determine the corresponding target segmentation map, and then performing target detection based on the target segmentation map, the accuracy and efficiency of target detection can be improved while determining the target object in the target image.

[0095] It should be noted that, in addition to being applied to target detection processing tasks, the target segmentation map in this embodiment can also be applied to task scenarios such as path planning, and this is not limited here.

[0096] The technical solution of this embodiment reduces interference factors in the image processing process and improves the accuracy of image feature segmentation results by first extracting and decoding the target image after acquiring the target image, and then performing coordinate transformation to determine the target segmentation map. At the same time, by using an extraction network to extract features from the target image to determine the initial feature map, and using an alignment network to align the initial feature map to determine the image feature map, the feature information in the initial feature map can be better captured, making the information in the image feature map more complete and accurate. Furthermore, by using a decoder to decode the features of the image feature map in a preset coordinate system to generate a feature segmentation map, segmentation is achieved in the preset coordinate system. Then, based on a preset sampling method, the feature segmentation map is coordinate transformed to determine the target segmentation map, enabling the conversion of feature segmentation results in different coordinate systems and different viewing angles. Because the feature segmentation map obtained by segmentation can retain the original feature information in the image to the greatest extent, it prevents the segmentation process from being affected by internal and external camera parameters or other interference factors, ensures that the segmentation features are aligned with the target segmentation map, and thus ensures the accuracy of the target segmentation map, i.e., the final image segmentation result.

[0097] Furthermore, in addition to improvements in the image processing process, this embodiment also optimizes the training method of the network model (including the feature extraction network and decoder) used in the image processing process, including first training the decoder and feature extraction network through the generation stage and the perception stage respectively, and then optimizing the decoder through the fine-tuning stage. This can reduce the simultaneous optimization of the feature extraction network and the decoder, and avoid the problems of unclear model optimization goals and high model training complexity caused by the simultaneous optimization of multiple models.

[0098] Correspondingly, the image processing method in this embodiment also includes the following steps: Figure 10 The model training method shown includes the following steps.

[0099] In step S1010, a training set is obtained.

[0100] In this embodiment, the training set includes a first sample set and a second sample set. The first sample set is used at least for training a decoder and includes at least a plurality of sample segmentation maps that do not correspond to the sample images. The second sample set is used at least for training a feature extraction network and includes a plurality of sample images.

[0101] Optionally, the first sample set in this embodiment includes multiple sample segmentation maps that do not correspond to the sample image, and multiple sample segmentation maps corresponding to the sample image. The sample image can be acquired in the same manner as the aforementioned target image. The sample image is a road image in RGB image format, and the pixel position and coordinates in the road image are described by a polar coordinate system. The sample segmentation map corresponding to the sample image adopts the same format as the aforementioned feature segmentation map. The sample segmentation map that does not correspond to the sample image uses a format different from the aforementioned feature segmentation map as much as possible, and the corresponding image can be any image (including non-road images). Therefore, by obtaining the training set through the above method, it is possible to reduce the difficulty of obtaining the training set while ensuring the model training effect, making the model training process more flexible and simple.

[0102] In step S1020 , an autoencoder is determined based on the first sample set training, where the autoencoder includes an encoder and a decoder.

[0103] In this embodiment, the process of training and determining the autoencoder corresponds to the aforementioned generation stage. The purpose of the generation stage is to establish the relationship between the implicit space and the bird's-eye view space (i.e., BEV space) by reconstructing the original segmentation map input to the autoencoder.

[0104] Furthermore, the structure of the autoencoder in this embodiment is as follows Figure 11 As shown in , the autoencoder includes an encoder and a decoder. The input of the autoencoder is the original segmentation map in a given polar coordinate system, and the output is the segmentation map in the polar coordinate system formed after the original segmentation map is reconstructed. Therefore, in this embodiment, the autoencoder is determined by training based on the sample segmentation map (that is, the original segmentation map), and the encoder and decoder in the autoencoder can be trained. At the same time, since the decoder in the autoencoder is essentially trained based on a given implicit feature reconstruction segmentation label map, such a training process can help the decoder learn the prior knowledge of the segmentation map, so that the decoder obtained after training can segment and reconstruct the feature map in the image.

[0105] At the same time, in this embodiment, Figure 12 The method shown in the embodiment determines the autoencoder based on the first sample set training, and specifically includes the following steps.

[0106] In step S1210, the sample segmentation map in the first sample set is processed based on the encoder to determine the corresponding implicit feature map.

[0107] In this embodiment, the implicit feature map corresponding to the sample segmentation map can be obtained by inputting the sample segmentation map in the first sample set into the encoder for processing.

[0108] In step S1220, preset noise information is added to the implicit feature map to determine an intermediate feature map.

[0109] In this embodiment, due to the prior knowledge in the real world traffic bird's-eye view map (i.e., BEV map), streets are usually straight and the size of cars is relatively consistent. However, in previous methods, the generated BEV segmentation map may not match the actual traffic pattern. For example, when the streets are occupied by cars, the unseen streets may be deformed or mislabeled in the generated BEV map. Therefore, during reconstruction, preset noise information is added to the implicit feature map to improve the decoder's overall understanding of global traffic, thereby ensuring that the reconstructed segmentation map is more consistent with the actual desired BEV map.

[0110] Optionally, the preset noise information in this embodiment is randomly generated Gaussian noise, and an intermediate feature map is determined by adding 50% of the preset noise information to the implicit feature map, so as to reconstruct the segmentation map based on the intermediate feature map.

[0111] In step S1230, the intermediate feature map is decoded based on the decoder to determine a reconstructed segmentation map.

[0112] In this embodiment, the implicit feature map after the noise information is introduced, that is, the intermediate feature map, is decoded by a decoder, and the output result of the decoder is determined as the reconstructed segmentation map.

[0113] In step S1240 , the autoencoder is iteratively trained based on the difference between the reconstructed segmentation map and the corresponding sample segmentation map until the training is completed.

[0114] In this embodiment, the difference between the reconstructed segmentation map and the corresponding sample segmentation map is determined by calculating the BCE-Loss (binary cross entropy loss) between the reconstructed segmentation map and the corresponding sample segmentation map. When the BCE-Loss decreases and tends to be stable (that is, the BCE-Loss converges), the training is determined to be completed; conversely, when the BCE-Loss has not reached convergence, the autoencoder continues to be iteratively trained until the training is completed.

[0115] In step S1030 , a feature extraction network is trained based on the second sample set.

[0116] In this embodiment, the process of training and determining the feature extraction network corresponds to the aforementioned perception stage. The purpose of the perception stage is to describe how to establish the relationship between the RGB image and the implicit space, so as to map the RGB image into the implicit space and directly optimize the correlation between the two views at the feature level. This simplifies the complexity of end-to-end training in existing technologies, ensuring that each model has a clear optimization target at each stage.

[0117] Furthermore, in this embodiment, the input of the feature extraction network is the front view, and the output is the image feature map in the implicit space. The corresponding image feature map is determined by inputting the sample image from the second sample set into the feature extraction network. The model parameters in the feature extraction network are adjusted based on the image feature map's ability to express the sample image until the image feature map output by the feature extraction network can well express the corresponding input image.

[0118] It should be understood that there is no strict order in which step S1020 and step S1030 are executed in this embodiment. Step S1020 may be executed first and then step S1030, or step S1030 may be executed first and then step S1020, or step S1020 and step S1030 may be executed simultaneously.

[0119] In step S1040 , the decoder is fine-tuned based on the sample image and the sample segmentation map corresponding to the sample image.

[0120] In this embodiment, the purpose of the fine-tuning stage is to enable the decoder to better adapt to the data distribution of the implicit feature domain and the RGB image domain at the same time, thereby establishing a connection between the data distribution of the BEV segmentation map and the data distribution of the input RGB image, and ensuring the certainty of the segmentation result determined when the trained feature extraction network image and decoder are jointly applied to image processing.

[0121] Optionally, in this embodiment, after the feature extraction network and decoder are trained, the decoder will be fine-tuned based on the sample image and the sample segmentation map corresponding to the sample image. Furthermore, in this embodiment, whether the decoder needs to be fine-tuned can be determined by calculating the difference between the image feature map generated after the feature extraction network processes the sample image and the implicit feature map generated after the corresponding sample segmentation map is encoded by the encoder in the autoencoder. At the same time, in this embodiment, the difference between the image feature map and the corresponding implicit feature map can be determined by calculating the mean square error (MSE) loss between the image feature map generated by the extraction network and the corresponding implicit feature map. When the mean square error (MSE) loss decreases and tends to be stable (that is, the MSE loss converges), it indicates that the current decoder can adapt to the current application scenario and no longer needs to be fine-tuned. On the contrary, when the mean square error loss has not reached convergence, it indicates that the current decoder cannot adapt to the current application scenario and the decoder needs to be fine-tuned to adapt to the current application scenario.

[0122] In this embodiment, after obtaining the training set, the decoder in the autoencoder is determined based on the sample segmentation map in the training set, and the feature extraction network is determined based on the sample image in the training set. This allows the decoder and feature extraction network to be trained separately, reducing the need for simultaneous optimization of the feature extraction network and decoder, and avoiding the problems of unclear model optimization objectives and high model training complexity caused by the simultaneous optimization of multiple models. At the same time, fine-tuning the decoder based on the sample image and the sample segmentation map corresponding to the sample image can enable the decoder to better adapt to the data distribution of both the implicit feature domain and the RGB image domain, thereby establishing a connection between the data distribution of the BEV segmentation map and the data distribution of the input RGB image, and ensuring the certainty of the segmentation results determined when the trained feature extraction network image and decoder are jointly applied to image processing.

[0123] Figure 13 Schematic diagram of an image processing device according to an embodiment of the present invention. Figure 13 As shown, the image processing device in this embodiment includes an image acquisition unit 1, a feature extraction unit 2, a feature segmentation unit 3, and a coordinate conversion unit 4. The image acquisition unit 1 is used to acquire a target image. The feature extraction unit 2 is used to extract features from the target image based on a feature extraction network and determine an image feature map. The feature segmentation unit 3 is used to decode the image feature map based on a decoder and generate a feature segmentation map. The coordinate conversion unit 4 is used to perform coordinate conversion on the feature segmentation map to determine a target segmentation map.

[0124] Optionally, in this embodiment, the polar coordinate system is used as the preset coordinate system, the Cartesian coordinate system is used as the target coordinate system, the target image is an image in the polar coordinate system, and the target segmentation map is a bird's-eye view segmentation map in the Cartesian coordinate system.

[0125] Optionally, the image acquisition unit 1 in this embodiment is further configured to acquire a target image in a preset coordinate system through an image acquisition device installed on the target vehicle.

[0126] Optionally, the feature extraction network in this embodiment includes an extraction network and an alignment network, and the feature extraction unit 2 is further used to input the target image into the extraction network to determine the initial feature map; and perform feature alignment on the initial feature map based on the alignment network to determine the image feature map.

[0127] Optionally, the feature segmentation unit 3 in this embodiment is also used to perform feature alignment on each sequence graph in the initial feature graph based on the alignment network to determine the corresponding alignment feature graph, where the sequence graph is a feature graph determined after dividing the initial feature graph based on columns; and integrate each alignment feature graph to determine the image feature graph.

[0128] Optionally, the coordinate conversion unit 4 in this embodiment is further configured to determine a target segmentation region corresponding to the feature segmentation map; sample image features in the feature segmentation map to the target segmentation region to determine the corresponding target segmentation map. Furthermore, when determining the corresponding target segmentation map, the coordinate conversion unit 4 in this embodiment is further configured to divide the feature segmentation map into regions to determine a plurality of first subregions; divide the target segmentation region into regions to determine a plurality of second subregions, each second subregion having a unique corresponding first subregion; assign image features in the first subregion to the corresponding second subregion; and generate a corresponding target segmentation map based on the image features of each second subregion.

[0129] Furthermore, the coordinate conversion unit 4 in this embodiment performs target detection on the target segmentation map to determine the target object in the target image, so that the target segmentation map can be used for subsequent task processing.

[0130] Figure 14 FIG. 1 is another schematic diagram of an image processing apparatus according to an embodiment of the present invention. Figure 14 As shown, the image processing device in this embodiment is Figure 13 The image processing apparatus shown in further includes a model training unit 5. The model training unit 5 is configured to obtain a training set, the training set including a first sample set and a second sample set, the first sample set including multiple sample images, the second sample set including multiple sample segmentation maps corresponding to and not corresponding to the sample images; determine the feature extraction network based on the sample images; determine the autoencoder based on the sample segmentation maps; and fine-tune the decoder based on the sample images and the sample segmentation maps corresponding to the sample images.

[0131] Optionally, when the model training unit 5 in this embodiment determines the autoencoder based on the sample segmentation map training, it is specifically used to process the sample segmentation map based on the encoder to determine the corresponding implicit feature map; add preset noise information to the implicit feature map to determine the intermediate feature map; decode the intermediate feature map based on the decoder to determine the reconstructed segmentation map; and iteratively train the autoencoder based on the difference between the reconstructed segmentation map and the corresponding sample segmentation map until the training is completed.

[0132] Figure 15 Schematic diagram of an electronic device according to an embodiment of the present invention. Figure 15The electronic device shown is a general-purpose data processing device, which includes a general-purpose computer hardware structure, including at least a processor 61 and a memory 62. The processor 61 and the memory 62 are connected via a bus 63. The memory 62 is suitable for storing instructions or programs executable by the processor 61. The processor 61 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 61 executes the instructions stored in the memory 62, thereby performing the method flow of the embodiment of the present invention described above to process data and control other devices. The bus 63 connects the above-mentioned multiple components together and connects them to the display controller 64 and the display device as well as the input / output (I / O) device 65. The input / output (I / O) device 65 can be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer, and other devices known in the art. Typically, the input / output device 65 is connected to the system via an input / output (I / O) controller 66.

[0133] It will be understood by those skilled in the art that the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present application is described with reference to flowcharts of methods, apparatuses (devices), and computer program products according to embodiments of the present application. It should be understood that each process in the flowcharts can be implemented by computer program instructions.

[0135] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0136] These computer program instructions can also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used to enable a computer to execute part or all of the above method embodiments.

[0137] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by specifying relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0138] The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application are intended to be within the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire the target image; Performing feature extraction on the target image based on a feature extraction network to determine an image feature map; Performing feature decoding on the image feature map based on a decoder to generate a feature segmentation map; Perform coordinate transformation on the feature segmentation map to determine a target segmentation map.

2. The method according to claim 1, characterized in that The feature extraction network includes an extraction network and an alignment network, and extracting features from the target image based on the feature extraction network to determine the image feature map includes: Inputting the target image into the extraction network to determine an initial feature map; The initial feature map is subjected to feature alignment based on the alignment network to determine an image feature map.

3. The method according to claim 2, characterized in that Performing feature alignment on the initial feature map based on the alignment network to determine the image feature map includes: Based on the alignment network, feature alignment is performed on each sequence graph in the initial feature graph to determine a corresponding alignment feature graph, where the sequence graph is a feature graph determined after dividing the initial feature graph based on columns; The alignment feature maps are integrated to determine the image feature map.

4. The method according to claim 1, wherein The performing coordinate transformation on the feature segmentation map to determine the target segmentation map comprises: Determine the target segmentation area corresponding to the feature segmentation map; The image features in the feature segmentation map are sampled into the target segmentation area to determine a corresponding target segmentation map.

5. The method according to claim 4, characterized in that Sampling the image features in the feature segmentation map to the target segmentation area to determine the corresponding target segmentation map includes: Performing region division on the feature segmentation map to determine a plurality of first sub-regions; Dividing the target segmentation area to determine a plurality of second sub-areas, each of the second sub-areas having a unique corresponding first sub-area; Assigning the image features in the first sub-region to the corresponding second sub-region; Generate a corresponding target segmentation map according to the image features of each of the second sub-regions.

6. The method according to claim 1, characterized in that The target image is an image in a polar coordinate system, and the target segmentation map is a bird's-eye view segmentation map in a Cartesian coordinate system.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a training set, where the training set includes a first sample set and a second sample set, the first sample set includes a plurality of sample segmentation maps, the second sample set includes a plurality of sample images, and the first sample set includes at least a plurality of sample segmentation maps that do not correspond to the sample images; Determine an autoencoder based on the first sample set training, where the autoencoder includes an encoder and a decoder; Determine the feature extraction network based on the second sample set training; The decoder is fine-tuned based on the sample image and a sample segmentation map corresponding to the sample image.

8. The method according to claim 7, characterized in that The determining of the autoencoder based on the sample segmentation map training comprises: Processing the sample segmentation map based on the encoder to determine a corresponding implicit feature map; Adding preset noise information to the implicit feature map to determine an intermediate feature map; Decoding the intermediate feature map based on the decoder to determine a reconstructed segmentation map; The autoencoder is iteratively trained based on the difference between the reconstructed segmentation map and the corresponding sample segmentation map until the training is completed.

9. The method according to claim 1, characterized in that The acquiring of the target image comprises: The target image in the preset coordinate system is acquired by an image acquisition device installed on the target vehicle.

10. The method according to claim 1, characterized in that The method further comprises: Target detection is performed on the target segmentation map to determine the target object in the target image.

11. An image processing device, characterized in that: The device comprises: An image acquisition unit, configured to acquire a target image; A feature extraction unit, configured to extract features from the target image based on a feature extraction network and determine an image feature map; A feature segmentation unit, configured to perform feature decoding on the image feature map based on a decoder to generate a feature segmentation map; The coordinate conversion unit is used to perform coordinate conversion on the feature segmentation map to determine a target segmentation map.

12. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. An electronic device comprising a memory and a processor, characterized in that: The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 10 are implemented.

Images