A training method, device, equipment and medium for ground segmentation network model

Abstract

The present invention discloses a training method, device, equipment and medium for a ground segmentation network model. In the method, a ground height channel H image in an HHA image of a horizontal height angle encoding HHA is determined based on a depth image, and H feature vectors of the H image and RGB feature vectors of an RGB image corresponding to multiple convolutional layers in the original ground segmentation network model are extracted. Only the H feature vector of the H image and the RGB feature vector of the RGB image are used for deep attention mechanism fusion processing, thereby reducing the amount of calculation. In addition, the present invention is based on the joint processing of the RGB image and the H image, which solves the problem that objects with similar color and morphology in the image are difficult to segment when only the original image containing traffic road information is used, thereby improving the segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of ground segmentation technology, and in particular to a training method, device, equipment and medium for a ground segmentation network model. Background Art

[0002] The existing methods for ground segmentation include a semantic segmentation method of RGB-D images and a deep learning method for road surface segmentation.

[0003] Figure 1 A flowchart of a semantic segmentation method for RGB-D images provided by the prior art is shown as follows: Figure 1 As shown, the method includes the following steps: S101: extracting RGB coding features and depth coding features of multiple stages of RGB-D images; S102: inputting the RGB coding features and depth coding features of each stage in the multiple stages into the attention model to obtain each multimodal fusion feature corresponding to each stage; S103: using a long short-term memory network to extract the contextual semantic information of the multimodal fusion features of the fifth stage; S104: concatenating the multimodal fusion features of the fifth stage and the contextual semantic information to obtain contextual semantic features; S105: upsampling the contextual semantic features, and fusing them with the multimodal fusion features of the corresponding stage using a jump connection method to obtain a semantic segmentation map and a semantic segmentation model.

[0004] Figure 2 A flowchart of a deep learning method for road segmentation provided by the prior art, such as Figure 2 As shown, the method includes the following steps: S201: using multiple preset filters to filter the original feature image in sequence, at least one of the rows and columns of the filter matrix in each preset filter is greater than 1, and the original feature image is generated after sampling the original image containing traffic road information; S202: splicing the feature image after filtering with the original feature image to generate a target feature image; S203: performing road surface segmentation calculation based on the target feature image.

[0005] Among the above methods, the semantic segmentation method simultaneously performs multi-layer data splicing, resulting in a large amount of calculation when performing ground segmentation, while the deep learning method only uses the original image containing traffic road information, resulting in objects with similar color and shape in the image being difficult to segment and having low segmentation accuracy. Therefore, when performing ground segmentation, how to improve segmentation accuracy and reduce calculations has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides a training method, device, equipment and medium for a ground segmentation network model, which are used to solve the technical problem of how to improve segmentation accuracy and reduce calculation amount in the prior art.

[0007] The present invention provides a training method for a ground segmentation network model, the method comprising:

[0008] Acquire a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model;

[0009] Based on the original ground segmentation network model, a deep attention mechanism is performed on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a fused feature vector;

[0010] Based on the original ground segmentation network model, the fused feature vector corresponding to the preset convolutional layer is convolved and up-sampled to obtain a prototype feature map, and the fused feature vector corresponding to each convolutional layer from the preset convolutional layer to the terminal convolutional layer is convolved to obtain a predicted type, wherein the prototype is an initial contour of the target to be segmented in the RGB image, and the type includes a ground contour type. The prototype feature map and the predicted type are fused to obtain a predicted ground segmentation map, and a trained ground segmentation network model is obtained according to the predicted ground segmentation map.

[0011] Furthermore, extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model includes:

[0012] Extracting the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0013] The H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are subjected to a deep attention mechanism fusion process to obtain a fused feature vector including:

[0014] The H feature vector corresponding to the preset convolution layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer are fused to obtain the fused feature vector corresponding to the preset convolution layer, and the fused feature vector corresponding to the preset convolution layer is used as the RGB feature vector corresponding to the next convolution layer, and the H feature vector and the RGB feature vector corresponding to the next convolution layer are fused to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0015] Furthermore, the performing of a self-attention mechanism on the H feature vector corresponding to the preset convolutional layer to obtain a corresponding self-attention feature vector includes:

[0016] Perform average pooling, convolution, nonlinear activation, and sigmoid activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector;

[0017] The fusing the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer includes:

[0018] The self-attention feature vector and the RGB feature vector corresponding to the preset convolutional layer are concatenated and convolved to obtain a fused feature vector corresponding to the preset convolutional layer.

[0019] Furthermore, the type of prediction obtained by convolving the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer includes:

[0020] For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, convolution is performed on the fused feature vector to obtain each predicted region, category confidence and predicted weight coefficient of each prototype in the prototype feature map corresponding to the fused feature vector;

[0021] Non-maximum suppression processing is performed on each prediction region corresponding to the fused feature vector, the prediction region with the largest category confidence is determined as the target prediction region corresponding to the fused feature vector, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region is determined.

[0022] Furthermore, the fusing of the prototype feature map and the predicted type to obtain a predicted ground segmentation map includes:

[0023] According to the width, height and number of prototypes of each prototype in the prototype feature map, determine the first matrix P identifying the prototype feature map, where P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes;

[0024] Determine a second matrix C of the predicted type according to the number of the target prediction regions of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map, wherein C=k*mask, k is the number of the target prediction regions, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction region;

[0025] According to the first matrix P and the second matrix C, the predicted ground segmentation map M is obtained by fusing them using the S-shaped growth curve sigmoid function, M = σ (P * C T ), where σ is the sigmoid function.

[0026] Furthermore, the ground segmentation network model trained according to the predicted ground segmentation map includes:

[0027] According to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, back propagation is used to train the ground segmentation network model, and the parameters of the ground segmentation network model are updated to obtain a trained ground segmentation network model.

[0028] Further, the determining of the ground height channel H image in the horizontal height angle encoding HHA image based on the depth image comprises:

[0029] Determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image;

[0030] According to the target world coordinates of each pixel point and the pre-saved covariance formula, a target covariance matrix is ​​determined and a singular value decomposition is performed to obtain a target eigenvector corresponding to the minimum eigenvalue;

[0031] According to a preset initial gravity direction vector, a first preset screening condition of an XY plane in the preset camera coordinate system, and a second preset screening condition of an XZ plane, determining each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset screening condition, and a second normal vector of the XZ plane that satisfies the second preset screening condition, and determining a target direction vector of a target gravity direction according to an optimized functional relationship between each first normal vector, each second normal vector, and a pre-saved gravity direction vector;

[0032] According to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, determine a rotation matrix for rotating the direction corresponding to the unit vector toward the direction corresponding to the target direction vector, and determine a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix;

[0033] According to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

[0034] Accordingly, the present invention provides a training device for a ground segmentation network model, the device comprising:

[0035] An extraction module is used to obtain a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model;

[0036] A fusion module is used to perform a deep attention mechanism fusion process on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer based on the original ground segmentation network model to obtain a fused feature vector;

[0037] A prediction module is used to convolve and upsample the fused feature vector corresponding to the preset convolution layer based on the original ground segmentation network model to obtain a prototype feature map, convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the types include human contour type, object contour type and ground contour type, fuse the prototype feature map and the predicted type to obtain a predicted ground segmentation map, and obtain a trained ground segmentation network model based on the predicted ground segmentation map.

[0038] Furthermore, the extraction module is specifically used to extract the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0039] The fusion module is specifically used to perform a self-attention mechanism on the H feature vector corresponding to the preset convolution layer to obtain a corresponding self-attention feature vector, fuse the self-attention feature vector with the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer, and use the fused feature vector corresponding to the preset convolution layer as the RGB feature vector corresponding to the next convolution layer, and fuse the H feature vector and the RGB feature vector corresponding to the next convolution layer to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0040] Furthermore, the fusion module is specifically used to perform average pooling, convolution, nonlinear activation and S-type growth curve sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector; concatenate and convolve the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain the fused feature vector corresponding to the preset convolution layer.

[0041] Furthermore, the prediction module is specifically used to convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, to obtain each prediction region corresponding to the fused feature vector, the category confidence and the weight coefficient of each prototype in the predicted prototype feature map; to perform non-maximum suppression processing on each prediction region corresponding to the fused feature vector, to determine the prediction region with the largest category confidence as the target prediction region corresponding to the fused feature vector, and to determine the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region.

[0042] Furthermore, the prediction module is specifically used to determine a first matrix P that identifies the prototype feature map according to the width, height and number of prototypes of each prototype in the prototype feature map, wherein P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes; according to the number of the target prediction areas of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, and the target weight coefficient of each prototype in the prototype feature map, determine a second matrix C of the predicted type, wherein C=k*mask, k is the number of the target prediction areas, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction area; according to the first matrix P and the second matrix C, the S-type growth curve sigmoid function is used for fusion to obtain the predicted ground segmentation map M, M=σ(P*C T ), where σ is the sigmoid function.

[0043] Furthermore, the prediction module is specifically used to train the ground segmentation network model using back propagation according to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, update the parameters of the ground segmentation network model, and obtain a trained ground segmentation network model.

[0044] Furthermore, the extraction module is specifically used to determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image; determine the target covariance matrix and perform singular value decomposition according to the target world coordinates of each pixel point and the pre-saved covariance formula to obtain the target eigenvalue corresponding to the minimum eigenvalue; determine each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset filtering condition, and each first normal vector of the XY plane that satisfies the first preset filtering condition, according to the preset initial gravity direction vector, the first preset filtering condition of the XY plane in the preset camera coordinate system, and the second preset filtering condition of the XZ plane. The second normal vector of the XZ plane with two preset filtering conditions is used to determine the target direction vector of the target gravity direction according to the optimized functional relationship among each first normal vector, each second normal vector, and the pre-saved gravity direction vector; according to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, a rotation matrix of the unit vector corresponding to the direction toward the target direction vector is determined, and a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix is ​​determined; according to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoded HHA image is determined.

[0045] Accordingly, the present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] The memory stores a computer program, and when the program is executed by the processor, the processor implements the steps of any of the methods described in the above-mentioned ground segmentation network model training method.

[0047] Accordingly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the above-mentioned ground segmentation network model training method.

[0048] The present invention provides a training method, device, equipment and medium for a ground segmentation network model. In the method, a ground height channel H image in an HHA image of a horizontal height angle encoding HHA is determined based on a depth image, H feature vectors of the H image and RGB feature vectors of an RGB image corresponding to a plurality of convolutional layers in the original ground segmentation network model are extracted, and only the H feature vector of the H image and the RGB feature vector of the RGB image are used for deep attention mechanism fusion processing, thereby reducing the amount of calculation; and the present invention is based on the joint processing of the RGB image and the H image, which solves the problem that objects with similar color and morphology in the image are difficult to segment when only the original image containing traffic road information is used, thereby improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flowchart of a semantic segmentation method for RGB-D images provided by the prior art;

[0051] Figure 2 A flowchart of a deep learning method for road segmentation provided by the prior art;

[0052] Figure 3 A schematic diagram of a process of training a ground segmentation network model provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a process for determining a prototype feature map provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a process of performing convolution on a fused feature vector to determine a prediction type provided by an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of a process of performing a deep attention mechanism fusion on the H feature vector and the RGB feature vector of a preset convolutional layer provided by an embodiment of the present invention;

[0056] Figure 7 A schematic diagram of a ground segmentation network model provided by an embodiment of the present invention;

[0057] Figure 8 A schematic diagram of the structure of a training device for a ground segmentation network model provided by an embodiment of the present invention;

[0058] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In order to improve the segmentation accuracy and reduce the amount of calculation when performing ground segmentation, an embodiment of the present invention provides a training method, device, equipment and medium for a ground segmentation network model.

[0061] Embodiment 1:

[0062] Figure 3 A process diagram of a method for training a ground segmentation network model provided by an embodiment of the present invention, the process comprising the following steps:

[0063] S301: Obtain a red, green, blue (RGB) image and a corresponding depth image containing ground information, determine a ground height channel (height above ground, H) image in a horizontal height angle encoding (horizontal disparity height aboveground and the angle the pixel's local surface normal makes with the inferred gravity direction, HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model.

[0064] A training method for a ground segmentation network model provided in an embodiment of the present invention is applied to an electronic device, wherein the electronic device may be a PC, a tablet computer, a smart terminal, a server, etc., wherein the server may be a local server or a cloud server.

[0065] In order to improve the segmentation accuracy and reduce the amount of calculation when performing ground segmentation, in an embodiment of the present invention, the electronic device obtains an RGB image containing ground information and a corresponding depth image, wherein the depth image is a depth image corresponding to the RGB image, and the RGB image can be an RGB image containing ground information obtained by an image acquisition device acquiring a preset area, and the depth image is a single-channel data representation method that represents the depth information in the environment of the preset area in the form of an image, and the depth image can be a depth image converted from the RGB image, or it can be a depth image obtained by acquiring the same preset area by a depth camera.

[0066] The electronic device acquires an RGB image and a depth image collected for a preset area, and can generate an HHA image using an existing HHA encoding method according to the depth image, or can calculate the HHA image according to the pixel coordinates of each pixel point in the depth image. The HHA image includes a horizontal difference channel image, a ground height channel image, and an angle channel image of a surface normal vector. In order to reduce the amount of calculation, the electronic device only uses one of the horizontal difference channel image, the ground height channel image, and the angle channel image of the surface normal vector in the HHA image. In order to ensure the training of the original ground segmentation network model, only the ground height channel H image in the HHA image is used.

[0067] The RGB image and the H image are input into the original ground segmentation network model, and multi-stage feature extraction is performed on the RGB image and the H image respectively, and feature vectors of multiple stages are extracted, that is, feature vectors corresponding to multiple convolutional layers in the original ground segmentation network model are extracted, including H feature vectors of the H image and RGB feature vectors of the RGB image corresponding to multiple convolutional layers. The original ground segmentation network model can be a fully convolutional (Fully Convolutional Networks for Semantic Segmentation, FCN) network. When multi-stage feature extraction is performed based on the original ground segmentation network model, according to the network calculation amount and accuracy requirements, the core network architecture (backbone) of the original ground segmentation network model can be any one of a residual network (rensnet50 / 101) and a lightweight network (mobilenet).

[0068] S302: Based on the original ground segmentation network model, the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are fused using a deep attention mechanism to obtain a fused feature vector.

[0069] In order to implement the training of the original ground segmentation network model, in the embodiment of the present invention, a preset convolutional layer is pre-selected from the initial convolutional layer to the terminal convolutional layer, and the preset convolutional layer can be any convolutional layer from the initial convolutional layer to the terminal convolutional layer. Preferably, in order to reduce the amount of calculation and ensure the training accuracy, the preset convolutional layer is an intermediate convolutional layer from the initial convolutional layer to the terminal convolutional layer.

[0070] For example, when the original ground segmentation network model consists of 5 convolutional layers, the preset convolutional layer is the third convolutional layer in the middle, and the feature vector corresponding to the third convolutional layer is the feature vector of the third stage in the multi-stage feature extraction.

[0071] Based on the original ground segmentation network model, for each convolution layer from the preset convolution layer to the terminal convolution layer, the H feature vector and RGB feature vector corresponding to the convolution layer are determined, and the deep attention mechanism fusion processing is performed to obtain the fused feature vector corresponding to the convolution layer.

[0072] Specifically, a deep attention mechanism fusion process is performed based on the fusion module of the original ground segmentation network model to obtain a fused feature vector corresponding to the convolutional layer.

[0073] S303: Based on the original ground segmentation network model, convolve and upsample the fused feature vector corresponding to the preset convolution layer to obtain a prototype feature map, convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the types include human contour type, object contour type and ground contour type, fuse the prototype feature map and the predicted type to obtain a predicted ground segmentation map, and obtain a trained ground segmentation network model according to the predicted ground segmentation map.

[0074] In order to determine the ground segmentation map predicted for the RGB image, in an embodiment of the present invention, based on the original ground segmentation network model, the prototype feature map of the RGB image is first determined, that is, the feature map of the initial contour of the target to be segmented in the RGB image is determined. Specifically, based on the prototype extraction (ProtoType) module of the original ground segmentation network model, the fused feature vector corresponding to the preset convolution layer is convolved and upsampled to obtain the prototype feature map, and the ProtoType module includes a convolution layer and an upsampling layer.

[0075] Figure 4 A schematic diagram of a process for determining a prototype feature map provided by an embodiment of the present invention is shown in FIG. Figure 4As shown, the fused feature vector corresponding to the preset convolution layer is convolved by a 3*3 convolution layer, upsampled by an upsample layer, and then convolved by a 2*2 convolution layer to obtain a prototype feature map of N*N*numProto, where numProto is the number of prototypes and N represents the dimension of the prototype feature map. For example, Figure 4 A prototype feature map of 138*138*1 is obtained, indicating that the dimension of the prototype feature map is 138*138 and the number of prototypes is 1.

[0076] Based on the original ground segmentation network model, for the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, the fused feature vector is convolved to obtain the predicted type. Specifically, based on the type prediction (PredictionType) module of the original ground segmentation network model, the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer is convolved to obtain the predicted type. The type includes the ground contour type.

[0077] Figure 5 A schematic diagram of a process of performing convolution on a fused feature vector to determine a prediction type provided by an embodiment of the present invention, such as Figure 5 As shown, the fused feature vector is convolved through a 2*2 convolution layer to obtain the predicted type.

[0078] According to the prototype feature map and the predicted type, the prototype feature map and the predicted type are fused to obtain a predicted ground segmentation map. Specifically, the prototype feature map and the predicted type are fused using a sigmoid function to obtain a predicted ground segmentation map.

[0079] According to the predicted ground segmentation map, the loss value between the predicted ground segmentation map and the actual manually annotated ground segmentation map is determined, and iterations are continuously performed until the loss value between the predicted ground segmentation map and the actual manually annotated ground segmentation map is less than a threshold, and the corresponding model parameters are determined to update the parameters of the original ground segmentation network model, thereby obtaining a trained ground segmentation network model.

[0080] In the embodiment of the present invention, the ground height channel H image in the HHA image of the horizontal height angle encoding HHA is determined based on the depth image, the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model are extracted, and only the H feature vector of the H image and the RGB feature vector of the RGB image are used for deep attention mechanism fusion processing, thereby reducing the amount of calculation; and the present invention is based on the joint processing of the RGB image and the H image, which solves the problem that objects with similar color and morphology in the image are difficult to segment when only the original image containing traffic road information is used, thereby improving the segmentation accuracy while reducing the amount of calculation.

[0081] Embodiment 2:

[0082] In order to determine the fused feature vector, based on the above embodiment, in an embodiment of the present invention, extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model includes:

[0083] Extracting the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0084] The H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are subjected to a deep attention mechanism fusion process to obtain a fused feature vector including:

[0085] The H feature vector corresponding to the preset convolution layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer are fused to obtain the fused feature vector corresponding to the preset convolution layer, and the fused feature vector corresponding to the preset convolution layer is used as the RGB feature vector corresponding to the next convolution layer, and the H feature vector and the RGB feature vector corresponding to the next convolution layer are fused to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0086] The H image and the RGB image are input into the original ground segmentation network model. Based on the initial convolution layer to the terminal convolution layer of the original ground segmentation network model, the H feature vector of the H image corresponding to the initial convolution layer to the terminal convolution layer and the RGB feature vector of the RGB image corresponding to the initial convolution layer to the preset convolution layer can be determined.

[0087] Since the H feature vector and RGB feature vector corresponding to the convolution layer before the preset convolution layer are low-level feature vectors, in order to improve the accuracy of training the ground segmentation network model, in an embodiment of the present invention, the electronic device only extracts the feature vector corresponding to the preset convolution layer to the terminal convolution layer, that is, extracts the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer, and the RGB feature vector of the RGB image corresponding to the preset convolution layer.

[0088] In order to determine the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, since RGB images are easily affected by external factors such as illumination, it is difficult to obtain a better self-attention feature vector, and H images are less affected by external environmental factors. Therefore, in the present invention, the H feature vector corresponding to the preset convolution layer is processed by a self-attention mechanism to obtain the self-attention feature vector of the H feature vector corresponding to the preset convolution layer.

[0089] The self-attention feature vector of the H feature vector corresponding to the preset convolution layer is fused with the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer.

[0090] In order to implement the deep attention mechanism fusion processing of the H feature vector and the RGB feature vector corresponding to the preset convolution layer, in an embodiment of the present invention, the self-attention mechanism processing of the H feature vector corresponding to the preset convolution layer to obtain the corresponding self-attention feature vector includes:

[0091] Perform average pooling, convolution, nonlinear activation and sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector;

[0092] The fusing the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer includes:

[0093] The self-attention feature vector and the RGB feature vector corresponding to the preset convolutional layer are concatenated and convolved to obtain a fused feature vector corresponding to the preset convolutional layer.

[0094] The H feature vector corresponding to the preset convolution layer is subjected to average pooling layer, 1*1 convolution layer, nonlinear activation layer, 1*1 convolution layer and sigmoid function activation layer, and the preset convolution layer is subjected to average pooling, 1*1 convolution, nonlinear activation, 1*1 convolution and sigmoid function activation to obtain the weight of each dimension of the H feature vector corresponding to the preset convolution layer.

[0095] According to the H feature vector corresponding to the preset convolution layer and the determined weight of each dimension, the product value of the H feature vector multiplied by the weight is determined, and the product value is the self-attention feature vector of the H feature vector corresponding to the preset convolution layer.

[0096] According to the self-attention feature vector of the H feature vector corresponding to the preset convolution layer and the RGB feature vector corresponding to the preset convolution layer, the self-attention feature vector is concatenated with the RGB feature vector to obtain a concatenated feature vector, and the concatenated feature vector is convolved through a 3*3 convolution layer to obtain a fused feature vector corresponding to the preset convolution layer. When the self-attention feature vector and the RGB feature vector are concatenated, the concatenation is performed in the depth direction.

[0097] After determining the fused feature vector corresponding to the preset convolutional layer, the fused feature vector corresponding to the preset convolutional layer is used as the RGB feature vector corresponding to the next convolutional layer of the preset convolutional layer. According to the H feature vector and RGB feature vector corresponding to the next convolutional layer, the H feature vector corresponding to the next convolutional layer is processed by the self-attention mechanism to obtain the self-attention feature vector of the H feature vector corresponding to the next convolutional layer. The self-attention feature vector corresponding to the next convolutional layer is fused with the RGB feature vector to obtain the fused feature vector corresponding to the next convolutional layer. The fused feature vector corresponding to the next convolutional layer is used as the RGB feature vector corresponding to the next convolutional layer.

[0098] For each convolutional layer after the next convolutional layer, the H feature vector and the RGB feature vector corresponding to the convolutional layer are processed in the same way as the above-mentioned preset convolutional layer until the fused feature vector corresponding to the terminal convolutional layer is determined.

[0099] The process of determining the fused feature vector of the present invention is described below with a specific embodiment:

[0100] The H image and the RGB image are input into the original ground segmentation network model for multi-stage feature extraction. The original ground segmentation network model includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer, and the preset convolution layer is the third convolution layer in the middle.

[0101] Determine the H feature vector H_L1 and RGB feature vector RGB_L1 corresponding to the first convolutional layer of the original ground segmentation network model, the H feature vector H_L2 and RGB feature vector RGB_L2 corresponding to the second convolutional layer, the H feature vector H_L3 and RGB feature vector RGB_L3 corresponding to the third convolutional layer, the H feature vector H_L4 corresponding to the fourth convolutional layer, and the H feature vector H_L5 corresponding to the fifth convolutional layer.

[0102] The H feature vector H_L3 corresponding to the third convolutional layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, and the self-attention feature vector is fused with the RGB feature vector RGB_L3 corresponding to the third convolutional layer to obtain the fused feature vector F3 corresponding to the third convolutional layer. The fused feature vector F3 is the RGB feature vector RGB_L4 corresponding to the fourth convolutional layer.

[0103] Figure 6 A schematic diagram of a process of performing a deep attention mechanism fusion on the H feature vector and the RGB feature vector of a preset convolutional layer provided by an embodiment of the present invention, such as Figure 6 As shown, the preset convolution layer is the third convolution layer, and the H feature vector H_L3 corresponding to the third convolution layer is passed through an average pooling (AvePool) layer, a 1*1 convolution (Conv) layer, a non-linear activation (Relu) layer, a 1*1Conv layer, and a sigmoid function activation layer to obtain the weight of each dimension of the H feature vector H_L3, and the product value of the H feature vector H_L3 multiplied by the weight is concatenated with the RGB feature vector RGB_L3 corresponding to the third convolution layer, and the concatenated feature vector is passed through a 3*3 convolution (Conv) layer, and the convolution processing is performed to obtain the fused feature vector F3 corresponding to the third convolution layer.

[0104] The H feature vector H_L4 corresponding to the fourth convolutional layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, and the self-attention feature vector is fused with the RGB feature vector RGB_L4 corresponding to the fourth convolutional layer to obtain the fused feature vector F4 corresponding to the fourth convolutional layer. The fused feature vector F4 is the RGB feature vector RGB_L5 corresponding to the fifth convolutional layer.

[0105] The H feature vector H_L5 corresponding to the fifth convolutional layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, and the self-attention feature vector is fused with the RGB feature vector RGB_L5 corresponding to the fifth convolutional layer to obtain the fused feature vector F5 corresponding to the fifth convolutional layer.

[0106] Embodiment 3:

[0107] In order to obtain the predicted type, based on the above embodiments, in an embodiment of the present invention, the predicted type obtained by convolving the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer includes:

[0108] For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, convolution is performed on the fused feature vector to obtain each predicted region, category confidence and predicted weight coefficient of each prototype in the prototype feature map corresponding to the fused feature vector;

[0109] Non-maximum suppression processing is performed on each prediction region corresponding to the fused feature vector, the prediction region with the largest category confidence is determined as the target prediction region corresponding to the fused feature vector, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region is determined.

[0110] For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, the fused feature vector is convolved. Specifically, the fused feature vector is convolved through a 2*2 convolution (Conv) layer, and W*H*(class+bbox+mask) is output, where bbox represents each prediction area, class represents the category confidence corresponding to each prediction area, mask represents the identification information of the type corresponding to each prediction area, W represents the width of the feature map obtained by convolution, and H represents the height of the feature map obtained by convolution.

[0111] The predicted area is represented by the bounding box coordinate values ​​[dx, dy, pw, ph], where dx and dy can be the horizontal coordinate and vertical coordinate representing the center point of the bounding box, or the horizontal coordinate and vertical coordinate representing any of the four corner points of the bounding box; pw represents the width of the bounding box, and ph represents the height of the bounding box.

[0112] Each prediction region is subjected to non-maximum suppression processing, that is, the target prediction region when the category confidence is the maximum value is determined. Specifically, according to the category confidence corresponding to each prediction region, the target category confidence with the largest category confidence is determined, the prediction region corresponding to the target category confidence is determined as the target prediction region, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target prediction region is determined, wherein the target weight coefficient of each prototype in the predicted prototype feature map of the target prediction region corresponding to each convolution layer in the preset convolution layer to the terminal convolution layer is the predicted type.

[0113] Embodiment 4:

[0114] In order to determine the predicted ground segmentation map, on the basis of the above embodiments, in an embodiment of the present invention, the fusion of the prototype feature map and the predicted type to obtain the predicted ground segmentation map includes:

[0115] According to the width, height and number of prototypes of each prototype in the prototype feature map, determine the first matrix P identifying the prototype feature map, where P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes;

[0116] Determine a second matrix C of the predicted type according to the number of the target prediction regions of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map, wherein C=k*mask, k is the number of the target prediction regions, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction region;

[0117] According to the first matrix P and the second matrix C, the predicted ground segmentation map M is obtained by fusing them using the S-shaped growth curve sigmoid function, M = σ (P * C T ), where σ is the sigmoid activation function.

[0118] In order to determine the predicted ground segmentation map, in an embodiment of the present invention, a first matrix P identifying the prototype feature map is determined according to the width w, height h and number of prototypes numPtoto of each prototype in the prototype feature map, and the first matrix P, the first matrix P=w*h*numPtoto.

[0119] For each convolution layer from the preset convolution layer to the terminal convolution layer, the number of target prediction areas of the fused feature vector corresponding to the convolution layer is determined, and according to the number k of target prediction areas of the fused feature vector corresponding to each convolution layer and the target weight coefficient mask of each prototype in the prototype feature map corresponding to each target prediction area, a second matrix C of the predicted type is determined, and the second matrix C=k*mask.

[0120] According to the first matrix P and the second matrix C, the sigmoid function is used to fuse the first matrix P and the second matrix C. The fusion formula is σ(P*C T ), where C T is the transposed matrix of the second matrix C, σ is represented by the sigmoid function, and the result M calculated by the above formula is represented as the predicted ground segmentation map.

[0121] Embodiment 5:

[0122] In order to obtain a trained ground segmentation network model, on the basis of the above embodiments, in an embodiment of the present invention, obtaining a trained ground segmentation network model according to the predicted ground segmentation map includes:

[0123] According to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, back propagation is used to train the ground segmentation network model, and the parameters of the ground segmentation network model are updated to obtain a trained ground segmentation network model.

[0124] In order to train the original ground segmentation network model and obtain the trained ground segmentation network model, in the embodiment of the present invention, a ground segmentation map corresponding to the RGB image is also pre-saved, the pre-saved ground segmentation map corresponding to the RGB image is used as the true value, the predicted ground segmentation map is used as the predicted value, and the loss value can be determined according to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image. The loss value includes a classification loss value, a bounding box regression loss value, and a mask loss value.

[0125] According to the determined loss value, the back propagation algorithm is used to train the original ground segmentation network model, the ground segmentation network weight is determined, and the ground segmentation network weight is used to update the parameters of the ground segmentation network model, thereby obtaining a trained ground segmentation network model.

[0126] Embodiment 6:

[0127] In order to determine the ground height channel H image based on the depth image, on the basis of the above embodiments, in an embodiment of the present invention, determining the ground height channel H image in the horizontal height angle encoded HHA image based on the depth image includes:

[0128] Determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image;

[0129] According to the target world coordinates of each pixel point and the pre-saved covariance formula, a target covariance matrix is ​​determined and a singular value decomposition is performed to obtain a target eigenvector corresponding to the minimum eigenvalue;

[0130] According to a preset initial gravity direction vector, a first preset screening condition of an XY plane in the preset camera coordinate system, and a second preset screening condition of an XZ plane, determining each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset screening condition, and a second normal vector of the XZ plane that satisfies the second preset screening condition, and determining a target direction vector of a target gravity direction according to an optimized functional relationship between each first normal vector, each second normal vector, and a pre-saved gravity direction vector;

[0131] According to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, determine a rotation matrix for rotating the direction corresponding to the unit vector toward the direction corresponding to the target direction vector, and determine a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix;

[0132] According to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

[0133] The depth image is a single-channel data representation method that represents the depth information in the environment in the form of an image. The pixel coordinate system is pre-set as The world coordinate system is The homogeneous transformation matrix from the world coordinate system to the pixel coordinate system is Where [R|T] is the rotation and translation matrix of the world coordinate system and the pixel coordinate system. Since the world coordinate system of the depth image is the coordinate system of the image acquisition device, the rotation and translation matrix z c is the z-direction depth information in the camera coordinate system, is the internal parameter camMatrix of the image acquisition device, f x and f y represents the focal length, s represents the coordinate axis tilt parameter, which is generally 0, cx and cx represent the coordinates of the principal point, which is the intersection of the optical axis of the image acquisition device and the image plane, generally the midpoint of the image plane.

[0134] The homogeneous transformation matrix can be simplified to According to the simplified secondary transformation matrix, the transformation relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system can be determined.

[0135] According to the above conversion relationship between pixel coordinates and world coordinates, and the target pixel coordinates of each pixel in the depth image, the target pixel coordinates are substituted into the above conversion relationship to determine the world coordinates of each pixel.

[0136] Based on the world coordinates (X, Y, Z) of each pixel, the pre-saved covariance formula Determine the target covariance matrix C. where x mean represents the mean of the X-axis coordinate values ​​of the i-th column. mean Represents the mean of the Y-axis coordinate values ​​of the i-th column.

[0137] The target covariance matrix C is decomposed by singular value decomposition (SVD) to obtain the target eigenvector corresponding to the minimum eigenvalue.

[0138] In order to determine the ground height channel H image, in the embodiment of the present invention, the actual gravity height direction vector is also determined, and an initial gravity direction vector Gi is preset, where Gi=[0,1,0], and a first preset screening condition of the XY plane in the preset camera coordinate system is preset, where μ1={n:(θ(n,g i-1 )) <d||(θ(n,g i-1 ))>180°-d}, where d is the preset angle threshold, θ(n, g i-1 ) represents the angle between the first normal vector of the XY plane and the gravity direction vector at the i-th iteration, and the first preset screening condition represents each first normal vector n that satisfies the condition that the angle between the first normal vector and the gravity direction vector is less than a preset angle threshold d, or greater than (180°-d).

[0139] A second preset screening condition of the XZ plane in the camera coordinate system is also preset, and the second preset screening condition is μ2={m:90°-d<θ(m, g i-1 )<90°+d}, where d is the preset angle threshold, θ(m, g i-1 ) represents the angle between the second normal vector of the XZ plane and the gravity direction vector at the i-th iteration, and the second preset screening condition represents every second normal vector m that satisfies the angle between the second normal vector and the gravity direction vector that is greater than (90°-d) and less than (90°+d).

[0140] According to the optimized functional relationship between each first normal vector n and each second normal vector m and the pre-saved gravity direction vector Each first normal vector is expressed in matrix form to determine the matrix N1 of each first normal vector; each second normal vector is expressed in matrix form to determine the matrix N1 of each first normal vector; according to the matrix N1 of each first normal vector, the matrix N1 of each first normal vector, and the matrix form of the optimized function relationship of the pre-saved gravity direction vector M=N1N1 T -N2N2 T , determine M and perform eigenvalue decomposition to determine the direction vector of the gravity direction at the i-th iteration. Repeat the above operation N times to determine the target direction vector of the final target gravity direction.

[0141] According to the determined target direction vector and the unit vector of the Y axis in the preset camera coordinate system, the rotation matrix Rotate that rotates the corresponding direction of the unit vector to the corresponding direction of the target direction vector is determined. 3×3 , the target direction vector and the unit vector are both one row and three columns, and the rotation matrix Rotate is three rows and three columns.

[0142] According to the target world coordinates of each pixel, determine the n-row and three-column matrix pc composed of the target world coordinates n×3 , the matrix pcn×3 The three columns in correspond to the X-axis coordinates, Y-axis coordinates, and Z-axis coordinates in the world coordinate system. Multiply the world coordinates of each pixel point with the rotation matrix Rotate to obtain the product matrix pcRotate n×3 .

[0143] According to each pixel value corresponding to the Y axis in the product matrix, that is, the element value of the second column in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

[0144] Embodiment 7:

[0145] The training process of the ground segmentation network model of the present invention is described below through a specific embodiment. Figure 7 A schematic diagram of a ground segmentation network model provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the method comprises the following steps:

[0146] The depth image Depth is encoded by the existing HHA encoding method to obtain the corresponding HHA image, and the ground height channel H image in the HHA image is determined. The H image and the RGB image are input into the original ground segmentation network model, and the H feature vector H_L1 and RGB feature vector RGB_L1 of the H image corresponding to the first convolutional layer, the H feature vector H_L2 and RGB feature vector RGB_L2 corresponding to the second convolutional layer, the H feature vector H_L3 and RGB feature vector RGB_L3 corresponding to the third convolutional layer, the H feature vector H_L4 corresponding to the fourth convolutional layer, and the H feature vector H_L5 corresponding to the fifth convolutional layer are extracted.

[0147] The H feature vector H_L3 and the RGB feature vector RGB_L3 corresponding to the third convolutional layer are input into the fusion module for deep attention mechanism fusion to obtain the fused feature vector F3 corresponding to the third convolutional layer, which is the RGB feature vector RGB_L4 corresponding to the fourth convolutional layer; the H feature vector H_L4 and the RGB feature vector RGB_L4 corresponding to the fourth convolutional layer are input into the FUSION module for deep attention mechanism fusion to obtain the fused feature vector F4 corresponding to the fourth convolutional layer, which is the RGB feature vector RGB_L5 corresponding to the fifth convolutional layer; the H feature vector H_L5 and the RGB feature vector RGB_L5 corresponding to the fifth convolutional layer are input into the FUSION module for deep attention mechanism fusion to obtain the fused feature vector F5 corresponding to the fifth convolutional layer.

[0148] The fused feature vector F3 corresponding to the third convolutional layer is input into the prototype extraction (ProtoType) module, and the prototype extraction is performed to determine the prototype feature map. The fused feature vector F3 corresponding to the third convolutional layer, the fused feature vector F4 corresponding to the fourth convolutional layer, and the fused feature vector F5 corresponding to the fifth convolutional layer are input into the type prediction (PredictionType) module for convolution to obtain the predicted type. The prototype feature map and the type are fused to obtain the predicted ground segmentation map, and non-maximum suppression (NMS) is performed to determine the ground segmentation map with the largest fusion result.

[0149] Embodiment 8:

[0150] Figure 8 A schematic diagram of the structure of a training device for a ground segmentation network model provided by an embodiment of the present invention. Based on the above embodiments, an embodiment of the present invention further provides a training device for a ground segmentation network model, the device comprising:

[0151] Extraction module 801, used to obtain a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model;

[0152] A fusion module 802 is used to perform a deep attention mechanism fusion process on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer based on the original ground segmentation network model to obtain a fused feature vector;

[0153] The prediction module 803 is used to convolve and upsample the fused feature vector corresponding to the preset convolution layer based on the original ground segmentation network model to obtain a prototype feature map, convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the types include human contour type, object contour type and ground contour type, fuse the prototype feature map and the predicted type to obtain a predicted ground segmentation map, and obtain a trained ground segmentation network model based on the predicted ground segmentation map.

[0154] Furthermore, the extraction module is specifically used to extract the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0155] The fusion module is specifically used to perform a self-attention mechanism on the H feature vector corresponding to the preset convolution layer to obtain a corresponding self-attention feature vector, fuse the self-attention feature vector with the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer, and use the fused feature vector corresponding to the preset convolution layer as the RGB feature vector corresponding to the next convolution layer, and fuse the H feature vector and the RGB feature vector corresponding to the next convolution layer to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0156] Furthermore, the fusion module is specifically used to perform average pooling, convolution, nonlinear activation and S-type growth curve sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector; concatenate and convolve the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain the fused feature vector corresponding to the preset convolution layer.

[0157] Furthermore, the prediction module is specifically used to convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, to obtain each prediction region corresponding to the fused feature vector, the category confidence and the weight coefficient of each prototype in the predicted prototype feature map; to perform non-maximum suppression processing on each prediction region corresponding to the fused feature vector, to determine the prediction region with the largest category confidence as the target prediction region corresponding to the fused feature vector, and to determine the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region.

[0158] Furthermore, the prediction module is specifically used to determine a first matrix P that identifies the prototype feature map according to the width, height and number of prototypes of each prototype in the prototype feature map, wherein P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes; according to the number of the target prediction areas of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, and the target weight coefficient of each prototype in the prototype feature map, determine a second matrix C of the predicted type, wherein C=k*mask, k is the number of the target prediction areas, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction area; according to the first matrix P and the second matrix C, the S-type growth curve sigmoid function is used for fusion to obtain the predicted ground segmentation map M, M=σ(P*C T ), where σ is the sigmoid function.

[0159] Furthermore, the prediction module is specifically used to train the ground segmentation network model using back propagation according to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, update the parameters of the ground segmentation network model, and obtain a trained ground segmentation network model.

[0160] Furthermore, the extraction module is specifically used to determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image; determine the target covariance matrix and perform singular value decomposition according to the target world coordinates of each pixel point and the pre-saved covariance formula to obtain the target eigenvalue corresponding to the minimum eigenvalue; determine each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset filtering condition, and each first normal vector of the XY plane that satisfies the first preset filtering condition, according to the preset initial gravity direction vector, the first preset filtering condition of the XY plane in the preset camera coordinate system, and the second preset filtering condition of the XZ plane. The second normal vector of the XZ plane with two preset filtering conditions is used to determine the target direction vector of the target gravity direction according to the optimized functional relationship among each first normal vector, each second normal vector, and the pre-saved gravity direction vector; according to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, a rotation matrix of the unit vector corresponding to the direction toward the target direction vector is determined, and a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix is ​​determined; according to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoded HHA image is determined.

[0161] Embodiment 9:

[0162] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Based on the above embodiments, an electronic device is further provided in an embodiment of the present invention, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904;

[0163] The memory 903 stores a computer program. When the program is executed by the processor 901, the processor 901 performs the following steps:

[0164] Acquire a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model;

[0165] Based on the original ground segmentation network model, a deep attention mechanism is performed on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a fused feature vector;

[0166] Based on the original ground segmentation network model, the fused feature vector corresponding to the preset convolutional layer is convolved and up-sampled to obtain a prototype feature map, and the fused feature vector corresponding to each convolutional layer from the preset convolutional layer to the terminal convolutional layer is convolved to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the types include human contour type, object contour type and ground contour type, the prototype feature map and the predicted type are fused to obtain a predicted ground segmentation map, and the trained ground segmentation network model is obtained according to the predicted ground segmentation map.

[0167] Further, the processor 901 is specifically used for extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model, including:

[0168] Extracting the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0169] The H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are subjected to a deep attention mechanism fusion process to obtain a fused feature vector including:

[0170] The H feature vector corresponding to the preset convolution layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer are fused to obtain the fused feature vector corresponding to the preset convolution layer, and the fused feature vector corresponding to the preset convolution layer is used as the RGB feature vector corresponding to the next convolution layer, and the H feature vector and the RGB feature vector corresponding to the next convolution layer are fused to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0171] Further, the processor 901 is specifically used for performing a self-attention mechanism on the H feature vector corresponding to the preset convolutional layer to obtain a corresponding self-attention feature vector, including:

[0172] Perform average pooling, convolution, nonlinear activation, and S-shaped growth curve sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector;

[0173] The fusing the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer includes:

[0174] The self-attention feature vector and the RGB feature vector corresponding to the preset convolutional layer are concatenated and convolved to obtain a fused feature vector corresponding to the preset convolutional layer.

[0175] Furthermore, the processor 901 is specifically used to convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain the predicted type, which includes:

[0176] For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, convolution is performed on the fused feature vector to obtain each predicted region, category confidence and predicted weight coefficient of each prototype in the prototype feature map corresponding to the fused feature vector;

[0177] Non-maximum suppression processing is performed on each prediction region corresponding to the fused feature vector, the prediction region with the largest category confidence is determined as the target prediction region corresponding to the fused feature vector, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region is determined.

[0178] Further, the processor 901 is specifically used for fusing the prototype feature map and the predicted type to obtain the predicted ground segmentation map, including:

[0179] According to the width, height and number of prototypes of each prototype in the prototype feature map, determine the first matrix P identifying the prototype feature map, where P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes;

[0180] Determine a second matrix C of the predicted type according to the number of the target prediction regions of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map, wherein C=k*mask, k is the number of the target prediction regions, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction region;

[0181] According to the first matrix P and the second matrix C, the predicted ground segmentation map M is obtained by fusing them using the S-shaped growth curve sigmoid function, M = σ (P * C T ), where σ is the sigmoid function.

[0182] Further, the processor 901 is specifically used to obtain the trained ground segmentation network model according to the predicted ground segmentation map, including:

[0183] According to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, back propagation is used to train the ground segmentation network model, and the parameters of the ground segmentation network model are updated to obtain a trained ground segmentation network model.

[0184] Further, the processor 901 is specifically used for determining the ground height channel H image in the horizontal height angle encoding HHA image based on the depth image, including:

[0185] Determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image;

[0186] According to the target world coordinates of each pixel point and the pre-saved covariance formula, a target covariance matrix is ​​determined and a singular value decomposition is performed to obtain a target eigenvector corresponding to the minimum eigenvalue;

[0187] According to a preset initial gravity direction vector, a first preset screening condition of an XY plane in the preset camera coordinate system, and a second preset screening condition of an XZ plane, determining each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset screening condition, and a second normal vector of the XZ plane that satisfies the second preset screening condition, and determining a target direction vector of a target gravity direction according to an optimized functional relationship between each first normal vector, each second normal vector, and a pre-saved gravity direction vector;

[0188] According to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, determine a rotation matrix for rotating the direction corresponding to the unit vector toward the direction corresponding to the target direction vector, and determine a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix;

[0189] According to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

[0190] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0191] The communication interface 902 is used for communication between the electronic device and other devices.

[0192] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0194] Embodiment 10:

[0195] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the following steps:

[0196] Acquire a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model;

[0197] Based on the original ground segmentation network model, a deep attention mechanism is performed on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a fused feature vector;

[0198] Based on the original ground segmentation network model, the fused feature vector corresponding to the preset convolutional layer is convolved and up-sampled to obtain a prototype feature map, and the fused feature vector corresponding to each convolutional layer from the preset convolutional layer to the terminal convolutional layer is convolved to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the types include human contour type, object contour type and ground contour type, the prototype feature map and the predicted type are fused to obtain a predicted ground segmentation map, and the trained ground segmentation network model is obtained according to the predicted ground segmentation map.

[0199] Furthermore, extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model includes:

[0200] Extracting the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer;

[0201] The H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are subjected to a deep attention mechanism fusion process to obtain a fused feature vector including:

[0202] The H feature vector corresponding to the preset convolution layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer are fused to obtain the fused feature vector corresponding to the preset convolution layer, and the fused feature vector corresponding to the preset convolution layer is used as the RGB feature vector corresponding to the next convolution layer, and the H feature vector and the RGB feature vector corresponding to the next convolution layer are fused to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

[0203] Furthermore, the performing of a self-attention mechanism on the H feature vector corresponding to the preset convolutional layer to obtain a corresponding self-attention feature vector includes:

[0204] Perform average pooling, convolution, nonlinear activation, and S-shaped growth curve sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector;

[0205] The fusing the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer includes:

[0206] The self-attention feature vector and the RGB feature vector corresponding to the preset convolutional layer are concatenated and convolved to obtain a fused feature vector corresponding to the preset convolutional layer.

[0207] Furthermore, the type of prediction obtained by convolving the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer includes:

[0208] For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, convolution is performed on the fused feature vector to obtain each predicted region, category confidence and predicted weight coefficient of each prototype in the prototype feature map corresponding to the fused feature vector;

[0209] Non-maximum suppression processing is performed on each prediction region corresponding to the fused feature vector, the prediction region with the largest category confidence is determined as the target prediction region corresponding to the fused feature vector, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region is determined.

[0210] Furthermore, the fusing of the prototype feature map and the predicted type to obtain a predicted ground segmentation map includes:

[0211] According to the width, height and number of prototypes of each prototype in the prototype feature map, determine the first matrix P identifying the prototype feature map, where P=w*h*numPtoto, w is the width of the prototype, h is the height of the prototype, and numPtoto is the number of prototypes;

[0212] Determine a second matrix C of the predicted type according to the number of the target prediction regions of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map, wherein C=k*mask, k is the number of the target prediction regions, and mask is the target weight coefficient of each prototype in the predicted prototype feature map corresponding to each target prediction region;

[0213] According to the first matrix P and the second matrix C, the predicted ground segmentation map M is obtained by fusing them using the S-shaped growth curve sigmoid function, M = σ (P * C T ), where σ is the sigmoid function.

[0214] Furthermore, the ground segmentation network model trained according to the predicted ground segmentation map includes:

[0215] According to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, back propagation is used to train the ground segmentation network model, and the parameters of the ground segmentation network model are updated to obtain a trained ground segmentation network model.

[0216] Further, the determining of the ground height channel H image in the horizontal height angle encoding HHA image based on the depth image comprises:

[0217] Determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image;

[0218] According to the target world coordinates of each pixel point and the pre-saved covariance formula, a target covariance matrix is ​​determined and a singular value decomposition is performed to obtain a target eigenvector corresponding to the minimum eigenvalue;

[0219] According to a preset initial gravity direction vector, a first preset screening condition of an XY plane in the preset camera coordinate system, and a second preset screening condition of an XZ plane, determining each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset screening condition, and a second normal vector of the XZ plane that satisfies the second preset screening condition, and determining a target direction vector of a target gravity direction according to an optimized functional relationship between each first normal vector, each second normal vector, and a pre-saved gravity direction vector;

[0220] According to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, determine a rotation matrix for rotating the direction corresponding to the unit vector toward the direction corresponding to the target direction vector, and determine a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix;

[0221] According to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

[0222] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0223] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0224] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0225] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0226] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A training method for a ground segmentation network model, characterized in that: The method comprises: Acquire a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model; Based on the original ground segmentation network model, a deep attention mechanism is performed on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a fused feature vector; Based on the original ground segmentation network model, convolve and upsample the fused feature vector corresponding to the preset convolution layer to obtain a prototype feature map, convolve the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the type includes a ground contour type, fuse the prototype feature map and the predicted type to obtain a predicted ground segmentation map, and obtain a trained ground segmentation network model according to the predicted ground segmentation map; The predicted ground segmentation map obtained by fusing the prototype feature map and the predicted type includes: Determine a first matrix identifying the prototype feature map according to the width, height and number of prototypes of each prototype in the prototype feature map; Determine a second matrix of the predicted type according to the number of target prediction areas of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map; According to the first matrix and the second matrix, a predicted ground segmentation map is obtained by fusing them using an S-shaped growth curve sigmoid function; The ground segmentation network model trained according to the predicted ground segmentation map includes: According to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, back propagation is used to train the ground segmentation network model, and the parameters of the ground segmentation network model are updated to obtain a trained ground segmentation network model.

2. The method according to claim 1, characterized in that The step of extracting the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to the multiple convolutional layers in the original ground segmentation network model comprises: Extracting the H feature vector of the H image corresponding to the preset convolution layer to the terminal convolution layer in the original ground segmentation network model, and the RGB feature vector of the RGB image corresponding to the preset convolution layer; The H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer are subjected to a deep attention mechanism fusion process to obtain a fused feature vector including: The H feature vector corresponding to the preset convolution layer is processed by the self-attention mechanism to obtain the corresponding self-attention feature vector, the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer are fused to obtain the fused feature vector corresponding to the preset convolution layer, and the fused feature vector corresponding to the preset convolution layer is used as the RGB feature vector corresponding to the next convolution layer, and the H feature vector and the RGB feature vector corresponding to the next convolution layer are fused to determine the fused feature vector corresponding to the next convolution layer, until the fused feature vector corresponding to the terminal convolution layer is determined.

3. The method according to claim 2, characterized in that The performing of self-attention mechanism processing on the H feature vector corresponding to the preset convolutional layer to obtain the corresponding self-attention feature vector includes: Perform average pooling, convolution, nonlinear activation, and S-shaped growth curve sigmoid function activation on the H feature vector corresponding to the preset convolution layer to obtain the weight of each dimension of the H feature vector, and multiply the weight by the H feature vector to obtain the corresponding self-attention feature vector; The fusing the self-attention feature vector and the RGB feature vector corresponding to the preset convolution layer to obtain a fused feature vector corresponding to the preset convolution layer includes: The self-attention feature vector and the RGB feature vector corresponding to the preset convolutional layer are concatenated and convolved to obtain a fused feature vector corresponding to the preset convolutional layer.

4. The method according to claim 1, characterized in that: The type of prediction obtained by convolving the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer includes: For the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer, convolution is performed on the fused feature vector to obtain each prediction region, category confidence and predicted weight coefficient of each prototype in the prototype feature map corresponding to the fused feature vector; Non-maximum suppression processing is performed on each prediction region corresponding to the fused feature vector, the prediction region with the largest category confidence is determined as the target prediction region corresponding to the fused feature vector, and the target weight coefficient of each prototype in the predicted prototype feature map corresponding to the target preset region is determined.

5. The method according to claim 1, characterized in that The method of determining the ground height channel H image in the horizontal height angle encoding HHA image based on the depth image comprises: Determine the target world coordinates of each pixel point according to the conversion relationship between the pixel coordinates in the pre-saved pixel coordinate system and the world coordinates in the preset world coordinate system, and the target pixel coordinates of each pixel point in the depth image; According to the target world coordinates of each pixel point and the pre-saved covariance formula, a target covariance matrix is ​​determined and a singular value decomposition is performed to obtain a target eigenvector corresponding to the minimum eigenvalue; According to a preset initial gravity direction vector, a first preset screening condition of an XY plane in a preset camera coordinate system, and a second preset screening condition of an XZ plane, determining each first normal vector of the XY plane whose angle with the initial gravity direction vector satisfies the first preset screening condition, and a second normal vector of the XZ plane that satisfies the second preset screening condition, and determining a target direction vector of a target gravity direction according to an optimized functional relationship between each first normal vector, each second normal vector, and a pre-saved gravity direction vector; According to the target direction vector and the unit vector of the Y axis in the preset camera coordinate system, determine a rotation matrix for rotating the target direction vector in the direction corresponding to the unit vector, and determine a product matrix of the matrix composed of the target world coordinates of each pixel point and the rotation matrix; According to each element value corresponding to the Y axis in the product matrix, the minimum value of the element value is determined, and according to the difference between each element value and the minimum value, the ground height channel H image in the horizontal height angle encoding HHA image is determined.

6. A training device for a ground segmentation network model, characterized in that: The device comprises: An extraction module is used to obtain a red, green, and blue (RGB) image containing ground information and a corresponding depth image, determine a ground height channel H image in a horizontal height angle encoding (HHA) image based on the depth image, and input the image into an original ground segmentation network model for multi-stage feature extraction, and extract the H feature vector of the H image and the RGB feature vector of the RGB image corresponding to multiple convolutional layers in the original ground segmentation network model; A fusion module is used to perform a deep attention mechanism fusion process on the H feature vector and the RGB feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer based on the original ground segmentation network model to obtain a fused feature vector; A prediction module, for convolving and upsampling the fused feature vector corresponding to the preset convolution layer to obtain a prototype feature map based on the original ground segmentation network model, convolving the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer to obtain a predicted type, wherein the prototype is the initial contour of the target to be segmented in the RGB image, and the type includes a human contour type, an object contour type, and a ground contour type, fusing the prototype feature map and the predicted type to obtain a predicted ground segmentation map, and obtaining a trained ground segmentation network model according to the predicted ground segmentation map; The prediction module is specifically used to determine a first matrix identifying the prototype feature map according to the width, height and number of prototypes of each prototype in the prototype feature map; determine a second matrix of the predicted type according to the number of target prediction areas of the fused feature vector corresponding to each convolution layer from the preset convolution layer to the terminal convolution layer and the target weight coefficient of each prototype in the prototype feature map; and obtain a predicted ground segmentation map by fusing the first matrix and the second matrix using the S-type growth curve sigmoid function; The prediction module is specifically used to train the ground segmentation network model using back propagation according to the predicted ground segmentation map and the pre-saved ground segmentation map corresponding to the RGB image, update the parameters of the ground segmentation network model, and obtain a trained ground segmentation network model.

7. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; A computer program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer program executable by a processor is stored therein, and when the program runs on the processor, the processor is enabled to execute the method according to any one of claims 1 to 5.

Images