Scene perception method, training method, program product, medium, and electronic device

Abstract

The application relates to the field of electronic equipment, in particular to a scene perception method, a training method, a program product, a medium and an electronic device. The scene perception method comprises the following steps: acquiring multi-view images of a target scene; performing feature extraction on the multi-view images to obtain a first feature map corresponding to the multi-view images; initializing a plurality of Gaussian balls and determining initial query vectors of the plurality of Gaussian balls according to initial attributes of the plurality of Gaussian balls; inputting the first feature map and the plurality of initial query vectors into a target Gaussian decoder to predict first target attributes of the plurality of Gaussian balls corresponding to the first feature map through the target Gaussian decoder, wherein the first target attributes comprise a first predicted position and a first predicted semantic of the Gaussian balls; and obtaining a perception result of the target scene according to the first target attributes of the plurality of Gaussian balls, wherein the perception result comprises semantic information of a spatial position of the target scene determined according to the first predicted position and the first predicted semantic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic devices, and in particular to a scene perception method, a training method, a program product, a medium and an electronic device. BACKGROUND

[0002] With the continuous development of the field of assisted driving, the environment perception method based on the BEV feature map plays an increasingly important role in obstacle perception.

[0003] Currently, the BEV feature map is usually a dense feature map, which results in a huge calculation cost for perceiving the dense feature map.

[0004] In addition, in a scene such as assisted driving, the computing resources are limited and the real-time requirement of scene perception is high, and the perception method based on the dense feature map cannot effectively meet the needs of the scene.

[0005] In actual application, the obstacles in the environment to be perceived usually have a sparse feature, and using the dense feature map to make obstacle prediction will waste a lot of computing resources. SUMMARY

[0006] The embodiments of the present application provide a scene perception method, a training method, a program product, a medium and an electronic device, which are used to solve the problem of high consumption of computing resources in scene perception in the prior art. The following introduces the present application from multiple aspects, and the embodiments and advantages of the following multiple aspects can be mutually referenced.

[0007] In a first aspect, the present application provides a scene perception method applied to an electronic device. The method comprises:

[0008] obtaining a multi-view image of a target scene; performing feature extraction on the multi-view image to obtain a first feature map corresponding to the multi-view image; initializing a plurality of Gaussian balls and determining initial query vectors of the plurality of Gaussian balls according to initial attributes of the plurality of Gaussian balls; inputting the first feature map and the plurality of initial query vectors into a target Gaussian decoder to predict first target attributes of the plurality of Gaussian balls corresponding to the first feature map through the target Gaussian decoder, the first target attributes including first predicted positions and first predicted semantics of the Gaussian balls; and obtaining a perception result of the target scene according to the first target attributes of the plurality of Gaussian balls, the perception result including semantic information of a spatial position of the target scene determined according to the first predicted positions and the first predicted semantics.

[0009] According to the embodiments of the present application, a target scene perception method based on Gaussian balls is provided. Through a single path of multi-view feature-Gaussian decoder-space perception result, the reconstruction and semantic labeling of the target scene perception result are completed at one time. Without using a dense feature map, the perception process is simplified and the demand for computing resources is reduced.

[0010] In some embodiments, inputting the first feature map and the plurality of initial query vectors into the target Gaussian decoder to predict, by the target Gaussian decoder, the first target attribute of the plurality of Gaussian spheres corresponding to the first feature map comprises: generating a plurality of sampling points in each Gaussian sphere according to the initial query vectors, and sampling the first feature map according to the plurality of sampling points to obtain a second feature map; performing feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian spheres to obtain a third feature map; updating the initial query vectors according to an adaptive attention mechanism to obtain first query vectors; performing first processing on the third feature map and the first query vectors to obtain second query vectors, the first processing at least comprising linear processing; superimposing the second query vectors and third query vectors to obtain fourth query vectors, the third query vectors being obtained by embedding the initial attribute; and decoding the fourth query vectors to obtain the first target attribute of the plurality of Gaussian spheres.

[0011] According to embodiments of the present application, a Gaussian decoder is provided. First, sampling points are dynamically generated in the sphere for the initial query vectors of each Gaussian sphere; local features are extracted from the first feature map to obtain a second feature map; then feature fusion is performed to obtain a third feature map, and the query vectors are updated through an adaptive attention mechanism, and the updated query vectors are superimposed with the attribute embedding vectors after linear transformation, and finally decoded into the position and semantic attribute of the Gaussian sphere at one time.

[0012] The Gaussian decoder realizes the step-by-step coupling of local to global features, uses the sampling points in the Gaussian sphere to collect features in the dense feature map, obtains a sparse feature map, and reduces the computational overhead.

[0013] In some embodiments, the number of sampling points is determined according to the computing resources of the electronic device.

[0014] In some embodiments, performing feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian spheres to obtain a third feature map comprises: performing adaptive feature fusion on the second feature map to obtain a fourth feature map; and performing residual connection on the fourth feature map and the initial query vectors of the plurality of Gaussian spheres to obtain the third feature map.

[0015] In some embodiments, performing adaptive feature fusion on the second feature map comprises: obtaining a first weight for adaptive fusion according to the second feature map, and performing adaptive fusion on the second feature map according to the feature channels and / or spatial positions of the second feature map.

[0016] In some embodiments, the first feature map is a multi-layer feature map; and the residual linking of the fourth feature map and the initial query vectors of the plurality of Gaussian spheres to obtain the third feature map comprises: residual linking the fourth feature map and the initial query vectors of the plurality of Gaussian spheres to obtain a fifth feature map, the fifth feature map being a multi-layer feature map, and the number of layers of the fifth feature map being the same as the number of layers of the first feature map; and performing layer normalization on the fifth feature map to obtain the third feature map.

[0017] According to the embodiment of the present application, a feature fusion method is provided. The method integrates feature information of feature maps at different scales, different feature channels or different positions through dynamic weight fusion, suppresses redundancy and highlights key features. Residual linking is used to retain original query semantics, alleviate gradient disappearance and accelerate training convergence.

[0018] In some embodiments, the initial query vectors are updated according to the adaptive attention mechanism to obtain the first query vectors, comprising: inputting the plurality of initial query vectors into a linear layer to obtain receptive field weights; obtaining attention weights according to distances between the initial query vectors and the receptive field weights, the attention weights decreasing with the increase of the receptive field weights; and updating the initial query vectors according to the attention weights to obtain the first query vectors.

[0019] According to the embodiment of the present application, a calculation method of adaptive attention is provided. Each initial query vector is mapped to a receptive field weight through a linear layer, and then attention weights are calculated according to distances between two initial query vectors, and the attention weights monotonically decrease with the increase of the receptive field weights, realizing spatial-semantic double sparsification.

[0020] The self-mechanism adaptively suppresses queries with long distances or large receptive fields, and strengthens queries with short distances and small receptive fields. The mechanism automatically prunes redundant interactions while maintaining local precision, significantly reduces computational complexity and memory occupation, and improves the accuracy and convergence stability of subsequent Gaussian attribute decoding.

[0021] In some embodiments, the first processing further comprises an activation processing after the linear processing.

[0022] In some embodiments, the perception result of the target scene is obtained according to the first target attributes of the plurality of Gaussian spheres, comprising: determining fifth query vectors of the plurality of Gaussian spheres according to the first target attributes of the plurality of Gaussian spheres; inputting the fifth query vectors and the first feature map into a target Gaussian decoder to obtain second target attributes of the plurality of Gaussian spheres; and obtaining the perception result according to the second target attributes of the plurality of Gaussian spheres.

[0023] According to the embodiment of the present application, the target attributes of the Gaussian spheres are more close to the target scene through multiple cyclic predictions of the Gaussian decoder.

[0024] In a second aspect, the present application provides a training method applied to an electronic device. The method comprises: obtaining training sample data, the training sample data comprising a first feature map corresponding to multi-view images of a target scene and initial query vectors of a plurality of Gaussian spheres; inputting the training sample data into a Gaussian decoder to be trained to obtain predicted attributes of the plurality of Gaussian spheres, the predicted attributes comprising predicted positions and predicted semantics of the plurality of Gaussian spheres; and training the Gaussian decoder to be trained according to the predicted attributes of the plurality of Gaussian spheres to obtain a target Gaussian decoder.

[0025] In some embodiments, inputting the training sample data into the Gaussian decoder to be trained to obtain the predicted attributes of the plurality of Gaussian spheres comprises: generating a plurality of sampling points in each Gaussian sphere according to the initial query vectors, and sampling the first feature map according to the plurality of sampling points to obtain a second feature map; performing feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian spheres to obtain a third feature map; updating the initial query vectors according to an adaptive attention mechanism to obtain a first query vector; performing first processing on the third feature map and the first query vector to obtain a second query vector, the first processing at least comprising linear processing; superimposing the second query vector and a third query vector to obtain a fourth query vector, the third query vector being obtained by embedding initial attributes; and decoding the fourth query vector to obtain the predicted attributes of the plurality of Gaussian spheres.

[0026] In some embodiments, training the Gaussian decoder to be trained according to the predicted attributes of the plurality of Gaussian spheres to obtain the target Gaussian decoder comprises: obtaining label data, the label data comprising target position information and target semantic information of a plurality of positions of the target scene; calculating a first loss between the predicted positions of the plurality of Gaussian spheres and the target position information; calculating a second loss between the predicted semantics of the plurality of Gaussian spheres and the target semantic information; and adjusting the Gaussian decoder to be trained according to the first loss and the second loss to obtain the target Gaussian decoder.

[0027] According to the embodiments of the present application, a training method of a Gaussian decoder is provided, which comprises four steps of input-output-loss calculation-back propagation. The target position information and the target semantic information in the target attributes of the Gaussian spheres output by the Gaussian decoder are taken as label values, a loss value between the label values and predicted values is calculated, and the Gaussian decoder is trained through supervised learning.

[0028] The training method outputs the predicted values of the target attributes of the Gaussian spheres in an end-to-end manner through the prior first feature map and the target attributes of the learnable Gaussian spheres, thereby avoiding error accumulation in traditional multi-stage training.

[0029] In addition, the training method improves the prediction accuracy of the Gaussian decoder by constraining the predicted position and semantic information double label values, thereby improving the prediction accuracy of the Gaussian decoder.

[0030] In a third aspect, the present application provides a computer program product, which comprises instructions, which, when executed by an electronic device, cause the electronic device to perform the scene-aware method or the training method of the Gaussian decoder according to any one of the embodiments of the first aspect or the second aspect.

[0031] In a fourth aspect, the present application provides a computer medium, which stores instructions, which, when executed on a computer, cause the computer to perform the scene-aware method or the training method of the Gaussian decoder according to any one of the embodiments of the first aspect or the second aspect.

[0032] In a fifth aspect, the present application provides an electronic device, which comprises a memory for storing instructions executed by one or more processors of the electronic device, and the processor, when executing the instructions in the memory, causes the electronic device to perform the scene-aware method or the training method of the Gaussian decoder according to any one of the embodiments of the first aspect or the second aspect.

[0033] It can be understood that the beneficial effects achieved by the third aspect to the fifth aspect described above can refer to the beneficial effects of any one of the embodiments of the first aspect or the second aspect, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 An exemplary application scenario of the scene-aware method provided by the embodiments of the present application is shown in the following figure;

[0035] Figure 2 An exemplary flowchart of the scene-aware method provided by the embodiments of the present application is shown in the following figure;

[0036] Figure 3 An exemplary flowchart of the Gaussian decoder provided by the embodiments of the present application is shown in the following figure;

[0037] Figure 4 An exemplary flowchart of the training method of the Gaussian decoder provided by the embodiments of the present application is shown in the following figure;

[0038] Figure 5 An exemplary diagram of the occupancy grid provided by the embodiments of the present application is shown in the following figure;

[0039] Figure 6 An exemplary modular flowchart provided by the embodiments of the present application is shown in the following figure;

[0040] Figure 7 A block diagram of an electronic device provided by the embodiments of the present application is shown in the following figure;

[0041] Figure 8 A block diagram of a system on chip provided by the embodiments of the present application is shown in the following figure. DETAILED DESCRIPTION

[0042] Embodiments of the present application are used to provide a scene perception method. Through the method provided by the embodiments of the present application, the problem of high consumption of computing resources in the target scene perception process can be improved.

[0043] Figure 1 An exemplary application scenario of the present application is shown, specifically an assisted driving scenario of a vehicle. Referring to Figure 1 , there are roadblocks, bicycles and pedestrians and other objects on the road where the vehicle travels. In the assisted driving process, the vehicle needs to perceive the objects existing on the road to avoid them.

[0044] That is, in the example shown in Figure 1 , the road scene is the target scene that needs to be perceived by the vehicle in the assisted driving process, and the roadblock 110, bicycle 120, pedestrian 130 and the like are the objects to be perceived.

[0045] In some technical solutions, the target scene is perceived based on a Bird’s-Eye-View (BEV) scene perception technology. In the BEV scene perception technology, the data collected by multiple sensors (such as cameras, lidar, millimeter wave radar) mounted on the vehicle is converted to a unified, overhead two-dimensional plane coordinate system, and then feature extraction, fusion, and subsequent perception tasks (such as target detection, map segmentation, prediction, etc.) are performed in this unified BEV space.

[0046] Currently, the BEV feature map is usually a dense feature map, which results in a huge computational cost for perceiving the dense feature map.

[0047] In addition, in scenarios such as assisted driving, the computing resources are limited and the real-time requirement of scene perception is high, and the perception method based on the dense feature map cannot effectively meet the needs of the scenario.

[0048] In addition, prediction based on the dense feature map results in a prediction result that includes a large number of downstream parts that are not of interest. For example, in the assisted driving field shown in Figure 1 , path planning based on the currently perceived target scene only needs information such as obstacles or road edges in the target scene. However, when using a dense feature map for prediction, the prediction result will include redundant information such as the sky, road texture or background buildings. This irrelevant information not only wastes valuable computing resources, but can also cause false detection and other problems.

[0049] To this end, the present application proposes a scene perception method. In the embodiment of the present application, the electronic device obtains multi-view images of a target scene (for example, images collected by multiple cameras on a vehicle). The electronic device extracts features from the multi-view images (for example, using a multi-scale feature extraction network) to obtain a first feature map (for example, a multi-scale feature map) corresponding to the multi-view images. The electronic device initializes multiple Gaussian balls (for example, initializes the Gaussian balls with random data at random positions) and determines initial query vectors of the multiple Gaussian balls according to initial attributes of the multiple Gaussian balls (for example, center position coordinates, rotation matrices, scaling matrices, opacity, or semantics of the Gaussian balls) (for example, encoding the initial attributes into high-dimensional vectors).

[0050] The electronic device inputs the first feature map and the multiple initial query vectors into a target Gaussian decoder to predict first target attributes of the multiple Gaussian balls corresponding to the first feature map through the target Gaussian decoder (for example, using the Gaussian decoder to predict the semantics or positions of objects perceived by each Gaussian ball), the first target attributes including first predicted positions and first predicted semantics of the Gaussian balls. The electronic device obtains a perception result of the target scene according to the first target attributes of the multiple Gaussian balls (for example, converting the predicted positions and predicted results of the Gaussian balls into a point cloud or a voxel network), the perception result including semantic information of spatial positions of the target scene determined according to the first predicted positions and the first predicted semantics.

[0051] In the embodiment of the present application, the dense multi-view feature map can be replaced by the Gaussian ball feature to obtain a sparse feature map. The corresponding query vector is obtained through the Gaussian ball to construct a sparse occupancy grid.

[0052] Through the above scheme, the electronic device can reduce the amount of calculation, improve the perception effect, and improve the general obstacle perception capability. The scene perception method provided in the embodiment of the present application is specifically introduced as follows.

[0053] It should be noted that, Figure 1 Although the assisted driving scene is taken as an example of an application scenario, the present application is not limited thereto. In other embodiments, the present application can be applied to scenes such as unmanned aerial vehicle inspection, smart city management, or industrial equipment monitoring. For example, in the unmanned aerial vehicle inspection scene, the scene perception method proposed in the present application can be used to quickly perceive target objects in the working scene of the unmanned aerial vehicle, and the target objects include key components of equipment such as power transmission towers and photovoltaic panels.

[0054] In addition, Figure 1 Although the car machine is taken as an example of an electronic device, the present application is not limited thereto. In other embodiments, the electronic device can be a smart phone, a tablet, or a controller of a logistics robot, and the present application is not limited thereto.

[0055] The following auxiliary driving scene is taken as an example of a target scene, and the car machine is taken as an example of an electronic device, to exemplarily introduce the scene perception method provided in the application.

[0056] Figure 2 An exemplary flowchart of the scene perception method provided in the embodiments of the application is shown in FIG. 1. Figure 2 The execution subject of each step can be an electronic device such as a car machine.

[0057] Reference Figure 2 The scene perception method provided in the embodiments of the application can include:

[0058] Step S110: Obtain multi-view images of a target scene.

[0059] Exemplarily, the target scene can be an environment where the current vehicle is located, for example, a city road, an expressway, a tunnel, an underground parking lot, a port container yard, a bridge or an elevated bridge, etc., which are not limited herein.

[0060] Exemplarily, the vehicle can include multiple cameras, which can have different shooting angles, so as to shoot images of different views. That is, by shooting the target scene through multiple cameras, multi-view images of the target scene can be obtained. In the embodiments of the application, through the cameras with different shooting angles, the perception range of the car machine to the target scene can be increased, so that the car machine can perceive objects in the target scene as much as possible to reduce the blind area. After obtaining the multi-view images of the target scene, the multiple cameras can transmit the shot multi-view images to the car machine through a data transmission interface or a data transmission bus, so that the car machine can perceive the multi-view images of the target scene.

[0061] Exemplarily, the car machine can obtain the shot multi-view images from the multiple cameras in step S110, which can be represented by formula (1):

[0062] (1)

[0063] Wherein, I can represent a set of multi-view images shot by the multiple cameras in step S110. Each element in the set can represent an image of the target scene shot by the i-th camera.

[0064] In some embodiments of the application, the shooting of the multiple cameras is time-sequential. For example, when the auxiliary driving function of the car machine is turned on, the multiple cameras of the vehicle can collect data of the target scene around the vehicle at all times. Therefore, I can also represent a set of multi-view images shot by the multiple cameras at the current time t.

[0065] ​In some embodiments of the present application, N can represent the total number of cameras. Since different cameras have different shooting angles, N different cameras can also represent N different shooting angles. It can be understood that in some embodiments of the present application, N can be 4, which respectively shoots the front view, rear view, left view and right view of the vehicle, thereby forming a 360-degree ring-shaped field of view centered on the vehicle. In some other embodiments of the present application, N can also be 4, 5, 6, 7 or 8, that is, 4, 5, 6, 7 or 8 cameras are configured on the vehicle.

[0066] W and H respectively represent the width and height of the image shot by each camera in the plurality of cameras, and in some embodiments of the present application, the number of pixels corresponding to the width or height of the image can be directly used for representation.

[0067] Step S120: performing feature extraction on the multi-view image to obtain a first feature map corresponding to the multi-view image.

[0068] In some embodiments of the present application, the first feature map obtained through feature extraction includes a plurality of features serving subsequent scene perception tasks. For example, the geometric and semantic features of the road surface include the feature texture and category information of ground elements such as lane lines, speed bumps or parking spaces.

[0069] In some embodiments of the present application, the first feature map can include multiple layers. Each layer can represent the features extracted by the vehicle machine at a certain scale of the multi-view image. For example, large-scale features can represent rough semantic information in a relatively far range (for example, 20m x 20m) around the vehicle. At this scale, the features extracted by this layer can better reflect large structures such as lane lines, road edges or buildings.

[0070] The vehicle machine can obtain a plurality of features of the multi-view image at a certain scale through the feature extraction network (for example, using a plurality of convolution kernels for down-sampling). In some embodiments of the present application, a pixel point containing a plurality of feature channels can be used to represent a plurality of features of the multi-view feature map at this position. The plurality of feature channels are used to store a plurality of feature values at this position.

[0071] For example, before performing feature extraction on the multi-view image, the intrinsic and extrinsic parameters of the camera are first obtained. The intrinsic and extrinsic parameters can make each pixel in the multi-view image be accurately projected into the same world coordinate system. This facilitates subsequent learning of the features of the multi-view image.

[0072] The intrinsic parameter can be represented by formula (2):

[0073] (2)

[0074] Here, K can represent a set of intrinsic parameters from multiple cameras, where the intrinsic parameters are a 3×3 matrix. For each element in the set... , which represents the intrinsic parameters corresponding to the i-th camera. N represents the total number of cameras.

[0075] The role of intrinsic parameters is to define the mapping relationship between the camera coordinate system and the image coordinate system, and to realize the mutual transformation of coordinates between the camera coordinate system and the image coordinate system.

[0076] The external parameters can be expressed using formula (3):

[0077] (3)

[0078] Here, T represents the set of extrinsic parameters from multiple cameras, which can be a 4×4 matrix. For each element in the set... , which represents the extrinsic parameter corresponding to the i-th camera. N represents the total number of cameras.

[0079] The role of extrinsic parameters is to determine the mapping relationship between the camera coordinate system and the world coordinate system, and to realize the mutual transformation of coordinates between the camera coordinate system and the world coordinate system.

[0080] For example, after obtaining a multi-view image in the world coordinate system using intrinsic and extrinsic parameters, a feature extraction network can be used to extract features from this multi-view image. The feature extraction network can extract various features from the multi-view image at different scales. These different features can be stored in multiple feature channels at each location in the multi-view image.

[0081] In some embodiments of this application, a backbone network and a feature pyramid network (FPN) can be used as multi-scale feature extraction networks to extract features from multi-view images and obtain first feature maps at different scales.

[0082] The first feature map extracted by the feature extraction network can be a multi-layered feature map, where each layer represents a feature map of a multi-view image at a certain scale. For example, the resolution of the first-scale feature map can be... The resolution of the feature map at the second scale can be In the matrix representation, H can represent the width of the multi-view image, and W can represent the length of the multi-view image.

[0083] Furthermore, each feature map layer includes a feature vector at each location point. For example, the feature map at the first scale could be... That is, each location point corresponds to a C-dimensional feature vector. Similarly, the feature map at the second scale can be derived from... .

[0084] In some embodiments of the present application, C can be referred to as the number of channels of the feature map, and the present application does not limit the value of C. In some embodiments, C can be 128. In other embodiments of the present application, C can also be 64, 96, 192, 256 or 512, which are not limited herein.

[0085] The reason why the resolution of the feature map of the second scale is smaller than that of the first scale is that a larger receptive field convolution kernel is used for downsampling when feature sampling is performed on the multi-view image according to the second scale, so that a feature map with smaller resolution is obtained. For example, under the second scale, a 5*5 receptive field convolution is used to downsample the multi-view image. Since the calculation amount of the larger receptive field convolution kernel is larger, in some embodiments of the present application, the larger the receptive field of the convolution kernel, the larger the moving step when the convolution is performed (for example, the moving step is 1 when using a 3*3 convolution kernel, and the moving step is 2 when using a 5*5 convolution kernel).

[0086] In some embodiments of the present application, in addition to using the backbone network and the feature pyramid, the multi-scale feature extraction network can also be based on the multi-scale attention structure of the Transformer. The present application does not limit this.

[0087] Step S130: initializing a plurality of Gaussian balls and determining initial query vectors of the plurality of Gaussian balls according to initial attributes of the plurality of Gaussian balls.

[0088] The car machine can randomly initialize a plurality of Gaussian balls according to the multi-view image corresponding to the target scene. Each Gaussian ball can be uniquely identified by a set of initial attributes including a plurality of parameters. According to the initial attributes of the initialized Gaussian balls, the car machine can encode a query vector corresponding to the Gaussian ball for interaction with the first feature map.

[0089] For ease of understanding, the present application first briefly describes the role of the Gaussian ball, and how to initialize the Gaussian ball and generate the initial query vector of the Gaussian ball will be described below.

[0090] For example, the Gaussian ball can be determined by a set of initial attributes including a plurality of parameters, such as: the center position (mean value) of the Gaussian ball, the rotation matrix (quaternion), the semantic logits and / or the scaling matrix. In some embodiments of the present application, the initial attributes can also include the opacity of the Gaussian ball and other attributes.

[0091] wherein the center position (mean) of the Gaussian sphere is used to represent the 3D position of the Gaussian sphere in the world coordinate system. The rotation matrix is used to represent the angle between the Gaussian sphere and the coordinate axis. The semantic logits are used to represent the category of the object corresponding to the Gaussian sphere in the target scene, such as a road shoulder or an obstacle. The scaling matrix is used to control the scaling ratio of the Gaussian sphere along the coordinate axis.

[0092] Exemplarily, the initial attribute can be represented by formula (4):

[0093] (4)

[0094] wherein g represents a set of initial attribute compositions of the Gaussian sphere. For each of the initial attributes g , which represents the initial attribute of the i-th Gaussian sphere. The value range of i is 1 to P, wherein P represents the number of initialized Gaussian spheres. In some embodiments of the present application, the value of P can be 100 to 1000. In some other embodiments of the present application, P can be set according to the computing resources possessed by the electronic device on which the perception method runs, which is not limited in the present application.

[0095] Exemplarily, g can be used to represent the initial attribute of the Gaussian sphere. In the initialization process of the Gaussian sphere, g can be set as a random value to complete the initialization.

[0096] Subsequently, according to g0and formula (5), the vehicle machine can encode the corresponding query vector for the Gaussian sphere:

[0097] (5)

[0098] wherein q represents a set of initial query vectors of all Gaussian spheres. For each of the initial query vectors q , which represents the initial query vector of the i-th Gaussian sphere. is an m-dimensional vector, and the vehicle machine can implicitly encode g into q when encoding the initial query vector. The value range of i is 1 to P, wherein P represents the number of initialized Gaussian spheres.

[0099] Exemplarily, q can be used to represent the initial query vector of the Gaussian sphere. In some embodiments of the present application, for each of the initial attributes g , the initial attribute g of the Gaussian sphere can be encoded into the Gaussian query q by a dense embedding method or the like. In this process, the initial query vector of the Gaussian sphere The attribute values in g are filled into a one-dimensional vector of length L in a fixed order. When the dimension of the attribute is not enough to make the length of the one-dimensional vector L, the one-dimensional vector is randomly filled to make its length meet the requirement of L. In some embodiments of the present application, the length of L is 11. In some other embodiments of the present application, the length of L can also be any integer between 11 and 30, which is not limited in the present application.

[0100] In addition to the dense embedding, the embodiments provided by the present application can also encode the initial attribute g to obtain the corresponding query vector q through attribute embedding or continuous value embedding, which is not limited in the present application.

[0101] Step S140: input the first feature map and the plurality of initial query vectors into the target Gaussian decoder to predict the first target attribute of the plurality of Gaussian spheres corresponding to the first feature map through the target Gaussian decoder, the first target attribute including the first predicted position and the first predicted semantics of the Gaussian spheres.

[0102] Exemplarily, the Gaussian decoder functions to simultaneously receive the first feature map and the plurality of initial query vectors q, and predict the first target attribute corresponding to q according to the input data. Wherein, the first feature map carries the multi-scale features of the target scene, and each query vector in q corresponds to a Gaussian sphere to be optimized.

[0103] Subsequently, the Gaussian decoder starts the decoding process, and according to the center position of the Gaussian sphere corresponding to each query vector in q, obtains the position in the first feature map at which it should be perceived. The q performs the relevant perception operation at the corresponding position. Through the result (for example, the semantics and position of the object at the query position) perceived by q at the query position, the q is decoded to obtain the first target attribute that the Gaussian sphere corresponding to q should be corrected.

[0104] In some embodiments of the present application, the first target attribute can include the predicted center position (or “first predicted position”) of the plurality of Gaussian spheres, and the semantics (or “first predicted semantics”) of the perceived object that each Gaussian sphere should represent in the corresponding target scene. For ease of understanding, the specific process of perceiving the first feature map by q is described below.

[0105] In addition, in some embodiments, the first target attribute can also include the rotation matrix (or “quaternion”), the scaling matrix and the opacity attribute.

[0106] Step S150: obtaining a perception result of the target scene according to the first target attributes of the plurality of Gaussian spheres, the perception result including semantic information of a spatial position of the target scene determined according to the first predicted position and the first predicted semantics.

[0107] After the vehicle machine completes decoding through the Gaussian decoder, the vehicle machine can obtain semantic information of an object at an arbitrary position in the target scene and a confidence degree of the semantic information in combination with the predicted first target attributes. By aggregating these information, the vehicle machine can obtain a perception result of a panorama of the target scene. In order to obtain a more accurate perception result, the vehicle machine can also decode the Gaussian spheres multiple times to improve the accuracy of the perception result.

[0108] For ease of understanding, the perception result is described briefly in the present application, and how to obtain the perception result will be described below.

[0109] In some embodiments of the present application, the perception result of the target scene can be represented by a semantic point cloud. The semantic point cloud can be composed of a plurality of discrete points in the three-dimensional space of the target scene. Each point can be represented by a set of first description information. The first description information can include the coordinates of the current point in the three-dimensional space of the target scene, and can also include the semantics of the target scene at the current point.

[0110] For example, according to the first predicted position in the first target attributes of each Gaussian sphere, the covered points of each Gaussian sphere in the target scene can be determined. Finally, the first description information of the points covered by each Gaussian sphere is assigned using the first predicted semantics in the first target attributes of the Gaussian sphere. For example, the semantics of the points covered by each Gaussian sphere is assigned as the first predicted semantics of the Gaussian sphere.

[0111] In some embodiments of the present application, the perception result of the target scene can also be represented by an occupancy grid. Specifically, the three-dimensional space where the target scene is located can be represented as a plurality of voxels in the occupancy grid, and each voxel can be represented by a set of second description information. The second description information can include the coordinates of the current voxel in the occupancy grid, and can also include the semantics of the current voxel. It can be understood that the coordinates corresponding to different voxels are different.

[0112] First, the occupancy grid corresponding to the target scene is established. The space range occupied by the occupancy grid can be greater than or equal to the three-dimensional space of the target scene. Then, according to the first predicted position in the first target attributes of each Gaussian sphere, the range of the voxels covered by each Gaussian sphere in the target scene can be determined. Finally, the second description information of the voxels covered by each Gaussian sphere is assigned using the first predicted semantics in the first target attributes of the Gaussian sphere. For example, the semantics of the voxels covered by each Gaussian sphere is assigned as the first predicted semantics of the Gaussian sphere.

[0113] The above steps are repeated so that all Gaussian balls complete the assignment of their covering voxel description information to obtain the perception result of the target scene.

[0114] Exemplarily, to ensure that the predicted first target attribute of the plurality of Gaussian balls is closer to the target scene, the target scene can be perceived multiple times: a fifth query vector of the plurality of Gaussian balls is determined according to the first target attribute of the plurality of Gaussian balls. The fifth query vector and the first feature map are input into the target Gaussian decoder to obtain a second target attribute of the plurality of Gaussian balls. The perception result is obtained according to the second target attribute of the plurality of Gaussian balls. The method of obtaining the perception result of the target scene according to the second target attribute is substantially the same as the method of obtaining the perception result of the target scene according to the first target attribute, which will not be repeated here.

[0115] Exemplarily, the predicted first target attribute of the plurality of Gaussian balls output by the Gaussian decoder is re-encoded into a fifth query vector. The re-encoded fifth query vector is input into the Gaussian decoder to re-execute step S140 to obtain a second target attribute that is re-predicted. The perception result corresponding to the second target attribute is calculated according to the second target attribute.

[0116] In some embodiments of the present application, after obtaining the re-predicted second target attribute, the second target attribute can be encoded into a sixth query vector again. The re-encoded sixth query vector is input into the Gaussian decoder to re-execute step S140 to obtain a third target attribute that is re-predicted. Then, the perception result of the target scene is obtained according to the third target attribute. The method of obtaining the perception result of the target scene according to the third target attribute is substantially the same as the method of obtaining the perception result of the target scene according to the first target attribute, which will not be repeated here.

[0117] In some embodiments of the present application, the target attribute predicted by the Gaussian decoder can be used for multiple times of loop prediction, and the target attribute of the Gaussian ball obtained last time is used to determine the perception result of the target scene by the car machine. Since the number of loops directly affects the resource demand of calculation, those skilled in the art can determine the number of loops according to the actual scene, for example, three times, four times or five times, which will not be limited here.

[0118] Figure 3 An exemplary flowchart of the Gaussian decoder provided in the embodiments of the present application. Figure 3 The execution subject of each step can be an electronic device such as a car machine.

[0119] Exemplarily, referring to Figure 3 , step S140 can further include the following substeps:

[0120] Step S141: generating a plurality of sampling points in each Gaussian ball according to the initial query vector, and sampling in the first feature map according to the plurality of sampling points to obtain a second feature map.

[0121] The car machine selects a plurality of sampling points in the 3D region range of each Gaussian ball based on the initial query vector of the Gaussian ball. The selection manner can be uniform selection, so that the sampling points are uniformly distributed in each Gaussian ball. Then, the car machine samples the positions of the sampling points in the first feature map to obtain a second feature map, so that the originally dense feature map becomes sparse.

[0122] For ease of understanding, the application first describes the role of the sampling points, and how to use the sampling points for sampling will be described below.

[0123] For example, the sampling points can be represented by the position coordinates of the points used for feature sampling in the Gaussian ball. After determining the center position and the included range of each Gaussian ball, the car machine can generate a plurality of uniformly spaced sampling points in the Gaussian ball according to the initial attributes of the remaining Gaussian balls. For example, X sampling points are generated in the Gaussian ball, and the number X can be 10. Since the required computing resources of the Gaussian decoder are positively correlated with the number of sampling points in the Gaussian ball, those skilled in the art can determine the number of sampling points in the Gaussian ball according to actual needs, which is not limited in the application.

[0124] After generating a plurality of sampling points in each Gaussian ball, in order to sample the first feature map, the coordinates of the sampling points need to be projected into the first feature map according to the intrinsic and extrinsic parameters of the multi-view image for sampling to obtain a second feature map.

[0125] In some embodiments of the application, the second feature map and the first feature map have the same storage structure except for the resolution. Since the number of feature points in the Gaussian ball is much smaller than the number of pixels in the first feature map, this sampling based on a plurality of sampling points of the Gaussian ball is a kind of down-sampling. Down-sampling makes the resolution of the second feature map much smaller than that of the first feature map. Since the Gaussian ball samples the positions of the first feature map, the second feature map is a kind of sparse feature map.

[0126] Step S142: performing feature fusion on the second feature map and the initial query vector of the plurality of Gaussian balls to obtain a third feature map.

[0127] For example, feature fusion refers to integrating feature vectors from different sources, different scales or different modalities into a unified feature map, so that it has more rich semantics and more complete spatial details.

[0128] In some embodiments of the application, adaptive feature fusion can be used to perform feature fusion on the sampled second feature map.

[0129] Specifically, the fusion manner of adaptive feature fusion can be performed from two dimensions of the dimension of sampling points or the dimension of feature channels.

[0130] In some embodiments of the present application, adaptive feature fusion first requires stacking a plurality of second feature maps in time sequence. For example, first, T second feature maps in time sequence are collected, each of which has S sampling points, and the second feature maps are stacked to obtain P=T*S sampling points.

[0131] When adaptive feature fusion is performed from the dimension of feature channels, the stacked second feature maps are fused from the dimension of feature channels according to formula (6) and formula (7).

[0132] (6)

[0133] (7)

[0134] wherein, represents a dynamic weight for feature fusion from the dimension of feature channels, which is shared between all frames and sampling points. Linear function represents a linear layer. q represents an initial query vector of a Gaussian sphere, LayerNorm function is a layer normalization function, ReLU function represents an activation function, and output represents a third feature map after feature fusion from the dimension of feature channels.

[0135] When adaptive feature fusion is performed from the dimension of sampling points, the stacked second feature maps are fused from the dimension of sampling points according to formula (8) and formula (9).

[0136] (8)

[0137] (9)

[0138] wherein, represents a dynamic weight for feature fusion from the dimension of sampling points, which is shared between all frames and sampling points. Linear function represents a linear layer. q represents an initial query vector of a Gaussian sphere, LayerNorm function is a layer normalization function, ReLU function represents an activation function, and output represents a third feature map after feature fusion from the dimension of sampling points.

[0139] Exemplarily, in some embodiments of the present application, the feature map obtained after adaptive feature fusion is a fourth feature map, and the output fourth feature map can also be linked with the initial query vectors of the plurality of Gaussian balls in residual to obtain a third feature map after feature fusion. The residual link can reduce gradient explosion or gradient disappearance by superimposing the fourth feature map and the initial query vectors of the plurality of Gaussian balls, so that the training of the Gaussian decoder is more stable.

[0140] Step S143: updating the initial query vectors according to the adaptive attention mechanism to obtain first query vectors.

[0141] For the convenience of understanding, the function of the attention mechanism is described first. How to use the attention mechanism to update the initial vectors will be described below.

[0142] Exemplarily, the car machine can use the adaptive attention mechanism to automatically adjust the size of the receptive field of each Gaussian query vector representing a potential obstacle, so as to pay more attention to the most relevant visual area in the current frame. For example, for a Gaussian ball with a long distance from the center position to the vehicle, the receptive field can be automatically increased to pay more attention to the obstacle far away.

[0143] Specifically, the adaptive attention mechanism can be realized by adaptive weights. The updating of the initial query vectors based on the attention weights can be obtained by performing the following steps: inputting the plurality of initial query vectors into a linear layer to obtain receptive field weights. According to the distance between the receptive field weights and the initial query vectors, the attention weights are obtained, and the attention weights decrease with the increase of the receptive field weights. The initial query vectors are updated according to the attention weights to obtain first query vectors.

[0144] Specifically, in the process of inputting the plurality of initial query vectors into the linear layer to obtain the receptive field weights, the receptive field refers to the area where each initial query vector interacts with the third feature map. Its range and shape adaptively change with the third feature map.

[0145] Specifically, the receptive field weights of the Gaussian decoder can be obtained by the following formula (10).

[0146] (10)

[0147] wherein d represents the dimension of the initial query vector, H represents the number of attention heads used to generate the receptive field weights, q represents the initial query vector, represents a vector composed of the generated receptive field weights.

[0148] In some embodiments of the present application, the Euclidean distance between the center positions of two Gaussian balls can be represented by the formula as shown in formula (11):

[0149] (11)

[0150] wherein, denotes the Euclidean distance between the center positions of two Gaussian spheres, and , and respectively represent the coordinates of the center positions of the Gaussian spheres.

[0151] According to the Euclidean distance between the center positions of the two Gaussian spheres and the receptive field weight, the attention weight can be obtained according to formula (12):

[0152] (12)

[0153] wherein, denotes the weight of adaptive attention, is an activation function, Q denotes an initial query vector of a Gaussian sphere, K denotes a set of features of all query vectors to be queried in all receptive fields, V denotes a feature dimension of a vector to be actually aggregated or weightedly summed, and d denotes a dimension of the initial query vector.

[0154] When the receptive field weight increases, the attention weight of the initial query vector of the Gaussian sphere with a farther distance decreases, which will cause the receptive field to be correspondingly reduced.

[0155] In some embodiments of the present application, when the receptive field weight decreases to 0, the adaptive attention mechanism degenerates into an attention mechanism with a global receptive field.

[0156] Through the adaptive attention mechanism, the car machine can dynamically adjust the attention degree to different Gaussian spheres, so that the model is more flexible and efficient when processing complex scenes.

[0157] In some embodiments of the present application, the car machine determines the size of the receptive field of the Gaussian decoder according to the distance between the initial query vectors of the two Gaussian spheres, and then updates the initial query vectors to obtain the first query vectors.

[0158] Step S144: performing first processing on the third feature map and the first query vector to obtain a second query vector, the first processing at least including linear processing.

[0159] The car machine can obtain the second query vector by performing one-time fast mapping on the third feature map and the first query vector associated through the attention mechanism via a linear network. In some embodiments of the present application, the first processing can further include activation processing after linear processing.

[0160] Exemplarily, the linear processing functions to integrate the third feature map and the first query vector in a unified spatial dimension. In some embodiments of the present application, the linear processing and the activation processing can be collectively used as a feedforward network to process the first query vector to obtain the second query vector.

[0161] In some embodiments of the present application, the linear processing can be performed using a fully connected layer. The activation processing can be performed using an activation function, including ReLu, GELU or Swish, etc., which functions to increase the nonlinearity of the network and realize the abstraction of deep features.

[0162] Those skilled in the art can determine which linear network or activation function to use to form a feedforward network to perform the first processing, which is not limited herein.

[0163] Step S145: superimposing the second query vector and the third query vector to obtain a fourth query vector, the third query vector being obtained by embedding the initial attributes.

[0164] The vehicle machine adds the elements of the two vectors with the same dimension at each dimension of the vector to obtain the fourth query vector. Since the third query vector is obtained by embedding the initial attributes of each Gaussian sphere, the vehicle machine adjusts the third query vector to obtain the fourth query vector corresponding to the predicted Gaussian attributes of the Gaussian sphere through superimposition.

[0165] Exemplarily, embedding refers to converting the initial attributes such as the center position of the Gaussian sphere, the rotation matrix, the scaling matrix, the opacity and the semantic logits, etc. into the third query vector in a high-dimensional space through a mapping process. This process enhances the expression ability of the attributes, so that they can more effectively participate in subsequent feature fusion and information processing.

[0166] In some embodiments of the present application, the embedding can be obtained by decoding the initial Gaussian attributes of a plurality of Gaussian spheres through a multi-layer perceptron.

[0167] By superimposing the second query vector and the third query vector, the fourth query vector corresponding to the predicted Gaussian sphere is obtained, which is the fusion of the information carried by the second query vector and the third query vector, forming a new vector containing the information of both.

[0168] Step S146: decoding the fourth query vector to obtain the first target attributes of the plurality of Gaussian spheres.

[0169] The vehicle machine converts the high-dimensional query vector into the first target attributes that can be used to specifically describe the Gaussian sphere features by decoding the fourth query vector.

[0170] In some embodiments of the present application, the fourth query vector can be decoded into the first target attribute of the plurality of Gaussian spheres using a multi-layer perception. The present application does not limit this.

[0171] Figure 4 A schematic diagram of the Gaussian decoder training method provided by the embodiments of the present application, Figure 4 The execution subject of each step in the method can be an electronic device such as a car machine, or a server.

[0172] Reference Figure 4 The Gaussian decoder training method provided by the embodiments of the present application can include:

[0173] Step S210: Obtain training sample data, the training sample data including a first feature map corresponding to a multi-view image of a target scene, and an initial query vector of a plurality of Gaussian spheres.

[0174] Illustratively, the initial query vector of the plurality of Gaussian spheres included in the training sample can be obtained from the predicted value of the Gaussian sphere target attribute output by the previous round target Gaussian decoder.

[0175] Step S220: Input the training sample data into the Gaussian decoder to be trained to obtain predicted attributes of the plurality of Gaussian spheres, the predicted attributes including predicted positions and predicted semantics of the plurality of Gaussian spheres.

[0176] In some embodiments of the present application, the parameters of each model in the Gaussian decoder can be randomly initialized, or can be initialized according to prior knowledge, which is not limited by the present application.

[0177] Step S230: Train the Gaussian decoder to be trained according to the predicted attributes of the plurality of Gaussian spheres to obtain a target Gaussian decoder.

[0178] In some embodiments of the present application, the training method of the Gaussian decoder can be supervised learning, and the loss value between the supervision value of the target scene and the output of each round of Gaussian decoder is obtained to perform back propagation on the Gaussian decoder to adjust the weights of each network in the Gaussian decoder to obtain the target Gaussian decoder.

[0179] Illustratively, step S230 can further include the following sub-steps:

[0180] Step S231: Obtain label data, the label data including target position information and target semantic information of a plurality of positions of the target scene.

[0181] Illustratively, the label data can include semantic occupancy or point cloud of each pixel position of the multi-view image of the target scene. The label data can be generated by artificial generation or obtained by artificial annotation, which is not limited by the present application.

[0182] Step S232: Calculate the first loss between the predicted positions of the plurality of Gaussian spheres and the target position information.

[0183] In some embodiments of the present application, the chamfer distance between the non-empty point cloud can be used as the first loss between the predicted positions of the plurality of Gaussian spheres and the target positions, and the calculation method of the first loss can be represented by formula (13) to formula (15):

[0184] (13)

[0185] (14)

[0186] (15)

[0187] Wherein, P represents the set of the center positions of the Gaussian spheres predicted by the Gaussian decoder in the present application, that is, the point cloud composed of the center positions of the Gaussian spheres. represents the point cloud composed of the set of label values (Ground Truth) of the target positions. represents the sum of the chamfer distances between each predicted center position of the Gaussian sphere and the point cloud represents the sum of the chamfer distances between each label value at the target position and the point cloud P.

[0188] W(d) is a reweighting function realized by a step function, and the output result of the function increases with the increase of the independent variable. The purpose of the function is to punish the prediction result of the Gaussian sphere farthest from the label value (Ground Truth), so that it has a larger chamfer distance. For example, in some embodiments of the present application, if d ≥ 0.2, then the step function of W(d) can be 5, otherwise it can be 1.

[0189] represents the symmetric chamfer distance, which is represented in quantity as the sum of the above two chamfer distances, and is used to measure the difference between the predicted positions of the Gaussian spheres predicted by the Gaussian decoder and the true positions in the whole, and therefore can be used as a representation of the first loss value in the embodiments of the present application. When the first loss is smaller, the difference between the predicted positions and the true positions is smaller, that is, the center positions of the Gaussian spheres predicted by the Gaussian decoder are more accurate.

[0190] Step S233: Calculate the second loss between the predicted semantics of the plurality of Gaussian spheres and the target semantic information.

[0191] ​Exemplarily, Figure 5 A schematic diagram of a semantic occupancy grid is shown. In some embodiments of the present application, a semantic occupancy grid can be used to predict the semantic and the second loss between the semantic information of the Gauss sphere.

[0192] Since the scaling matrix of the Gauss sphere can be used to control the scaling ratio of the Gauss sphere along the coordinate axis, in the present application, the scaling matrix can be used to convert the Gauss sphere into a semantic occupancy grid including a plurality of voxels. For each Gauss sphere, the position of each voxel included by the Gauss sphere and the semantic corresponding to each voxel form a tuple. Since the position of each voxel can uniquely represent a tuple, a list can be indexed by the position of the voxel, and a list of voxels of the Gauss sphere can be established according to the tuples corresponding to all voxels of the Gauss sphere.

[0193] Therefore, in the occupancy grid, each voxel can correspond to one or more lists, i.e., each voxel can correspond to one or more Gauss spheres.

[0194] Since the semantic of the voxel is more affected by the semantic of the Gauss sphere with a shorter distance, the weight of the influence of each Gauss sphere on the voxel can be obtained according to the distance between the voxel and the center of the Gauss sphere.

[0195] Specifically, the calculation formula of the weight can be represented by formula (16):

[0196] (16)

[0197] Formula (16) uses the Mahalanobis distance between the voxel and the center of the Gauss sphere. When the voxel is closer to the center of the Gauss sphere, the value of the weight is closer to 1, otherwise, it is closer to 0. By multiplying the weight with the semantic of all Gauss spheres corresponding to the voxel and accumulating the sum, the final semantic occupancy representation value of the voxel can be obtained.

[0198] According to the semantic occupancy representation value of each voxel in the semantic occupancy grid, the second loss between the predicted semantic and the semantic information of the plurality of Gauss spheres can be obtained.

[0199] Exemplarily, the focal loss with distance weight, the Dice loss, the Scene-Class Affinity Loss or the Lovász-Softmax loss between the semantic occupancy grids can be used as the second loss between the predicted semantic and the semantic information of the plurality of Gauss spheres.

[0200] The distance weight in the focal loss with distance weight can be represented by formula (17):

[0201] (17)

[0202] Equation (17) is a focal loss function with distance weight, where BEVCenterness represents the distance weight, and represent the coordinates of the center position of the Gaussian ball. Under this weight, the sample prediction error far from the center position of the Gaussian ball will be more punished.

[0203] The Dice loss can be represented by equation (18):

[0204] (18)

[0205] where and represent the predicted voxel semantics and the true semantics, respectively, and the Dice loss can alleviate class imbalance.

[0206] The Scene-Class Affinity Loss can be represented by equation (19):

[0207] (19)

[0208] where, represents the second loss, can represent the proportion of the number of voxels with the same predicted and labeled values in the total number of voxels, i.e., the precision. can represent the proportion of all voxels that are truly of class C that are correctly identified by the model, i.e., the recall. can represent the ability of the model to identify voxels that are not of the class, i.e., the proportion of voxels that are not of the class but are correctly classified as the class.

[0209] The Scene-Class Affinity Loss obtains the second loss by calculating the average of the above three indicators, and has the characteristics of strong global optimization ability.

[0210] The Lovász-Softmax loss can be represented by equation (20) and equation (21):

[0211] (20)

[0212] (21)

[0213] Lovász-Softmax loss is a loss function based on IoU. IoU represents the degree of overlap between the predicted value and the label value of class c, i.e., the Intersection over Union (IoU).

[0214] in, This represents the second loss. This represents the set of voxels of category c in the label values. The set of semantics representing category c among the predicted voxels.

[0215] when The closer to 0, the higher the overlap between the prediction and the label; conversely, the lower the overlap, the lower the overlap.

[0216] Step S234: Adjust the Gaussian decoder to be trained according to the first loss and the second loss to obtain the target Gaussian decoder.

[0217] In some embodiments of this application, the backpropagation method can be used to adjust the parameters of each network in the Gaussian decoder according to the first loss and the second loss. The specific backpropagation method is not limited here.

[0218] Figure 6 A modular flowchart provided in an embodiment of this application is shown, such as Figure 6 As shown, Figure 6 It includes a graph encoder training module 101, a Gaussian decoder training module 102, a graph encoder 103, and a Gaussian decoder 104.

[0219] The Gaussian decoder training module 102 corresponds to step S230 and its sub-steps S231 to S234 provided in the embodiments of this application. The Gaussian decoder training module 102 includes the calculation of the center position of the predicted Gaussian sphere based on the label value pair and the chamfer distance between the label values, as well as the calculation of the loss value between the voxel semantic prediction value and the label value of the semantic occupancy grid.

[0220] The image encoder 103 corresponds to steps S120 and S130, which are used to obtain the corresponding first feature map based on the input multi-view image, and initialize multiple Gaussian spheres to obtain the initial Gaussian query vector corresponding to the initial attributes of each Gaussian sphere.

[0221] The Gaussian decoder 104 corresponds to step S140 and its sub-steps S141 to S146 in the embodiments of this application.

[0222] Exemplarily, the image encoder training module 101 firstly obtains the depth data of the encoder, and secondly calculates the label value of the depth information of the target scene according to the Lidar data. Finally, the loss value of the depth information is obtained according to formula (22) and formula (23):

[0223] (22)

[0224] (23)

[0225] Formula (22) and formula (23) describe the depth supervision loss of the image. Specifically, formula (22) and formula (23) use the logarithmic loss of a single image to represent the depth supervision loss. Wherein, the label value is calculated from the depth map projected by the lidar point cloud, is the logarithmic difference value between the label value and the predicted value of the feature extracted by the multi-scale feature extraction network, λ is a weight parameter, and the value range is between [0, 1]. It is used to balance the influence of the two items in the loss function. When λ=0.5, it means using the absolute scale error. When λ=1, it means using the scale invariant error. n represents the total number of pixels of the image, which is used for normalization processing.

[0226] The image encoder training module 101 can improve the accuracy of the multi-scale feature extraction network based on the loss value obtained by formula (22) and formula (23), and perform sparse depth supervision on the label value calculated from the depth map projected by the lidar point cloud.

[0227] In some embodiments of the present application, the Gaussian decoder training module 102 can also use the cross-entropy loss function (Cross-Entropy Loss) to evaluate the semantic loss of the output of the Gaussian decoder. The remaining loss functions that can be used are not limited herein.

[0228] The embodiments of the present application also provide a computer program product. The computer program product can be software or a program product including instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device implements the scene perception method and the training method of the Gaussian decoder provided by the embodiments of the present application.

[0229] The embodiments of the present application also provide a computer medium. The computer medium can be any storage medium (such as a magnetic medium, an optical medium, a semiconductor medium, etc.) that a computing device can store and / or read data. The computer medium includes instructions that instruct the computing device to implement the scene perception method and the training method of the Gaussian decoder provided by the embodiments of the present application.

[0230] The embodiment of the present application further provides an electronic device, including one or more processors and one or more memories, and the one or more processors can be used to execute instructions to implement the scene perception method and the training method of the Gaussian decoder provided by the embodiment of the present application.

[0231] Exemplarily, Figure 7 A block diagram of an electronic device 400 is shown. The electronic device 400 can include one or more processors 401 coupled to a controller hub 403. For at least one embodiment, the controller hub 403 communicates with the processor(s) 401 via a multi-limb bus, such as a Front Side Bus (FSB), a point-to-point interface such as the QuickPath Interconnect (QPI), or a similar connection. The processor(s) 401 execute instructions to perform a general type of processing operation. In one embodiment, the controller hub 403 can include, but is not limited to, a Graphics & Memory Controller Hub (GMCH) (not shown) which can include a memory and graphics controller, and an Input / Output Hub (IOH) (which can be on a separate chip) (not shown) coupled to the GMCH.

[0232] The electronic device 400 can also include a coprocessor 402 and a memory 404 coupled to the controller hub 403. Alternatively, one or both of the memory and the GMCH can be integrated within the processor (as described in the embodiments of the present application), and the memory 404 and the coprocessor 402 are directly coupled to the processor 401 and the controller hub 403, with the controller hub 403 in a single chip with the IOH.

[0233] The memory 404 can be, for example, a Dynamic Random Access Memory (DRAM), a Phase Change Memory (PCM), or a combination of both. The memory 404 can include one or more tangible, non-transitory computer-readable media for storage of data and / or instructions, for example. The computer-readable media can have instructions stored thereon, in particular, a transitory and a non-transitory copy of the instructions. The instructions may Figures 1 to 6 include instructions that, when executed by at least one of the processors, cause the electronic device 400 to perform the method as shown. When the instructions are run on the computer, the computer is caused to perform the scene perception method and the training method of the Gaussian decoder disclosed by the embodiments of the present application.

[0234] In one embodiment, coprocessor 402 is a special-purpose processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processing unit, a compression engine, a GPGPU, embedded processor, or the like. The optional nature of coprocessor 402 is denoted in Figure 7

[0235] In one embodiment, electronic device 400 can further include a network interface controller (NIC) 406. Network interface 406 can include a transceiver to provide a radio interface for electronic device 400 to communicate to any other suitable device (e.g., a front end module, an antenna, etc.). In various embodiments, network interface 406 can be integrated with other components of electronic device 400. Network interface 406 can implement the functionality of the communication unit in the above-described embodiments.

[0236] Electronic device 400 can further include input / output (I / O) device 405. I / O 405 can include a user interface designed to enable a user to interact with electronic device 400, a peripheral component interface designed to enable peripheral components to interact with electronic device 400, and / or a sensor designed to determine environmental conditions and / or location information related to electronic device 400.

[0237] Notably, Figure 7 are merely exemplary. That is, although Figure 7 Although electronic device 400 is shown in Figure 7 to include processor 401, controller hub 403, memory 404, and the like, in actual applications, a device using the methods of embodiments of the present application can include only a portion of the components of electronic device 400, such as, for example, processor 401 and network interface 406.

[0238] Figure 8 A block diagram of a system on chip (SoC) 500 for use in an electronic device in accordance with an embodiment of the present application is shown in Figure 8 Figure 8 ​​In particular embodiments, the SoC 500 includes: an interconnect unit 550 coupled to the processor(s) 510; a system agent unit 580; a bus controller unit(s) 590; an integrated memory controller(s) unit 540; a set or one or more coprocessors 520A-N which can include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 530; a direct memory access (DMA) unit 560. In one embodiment, the coprocessors 520A-N include a special-purpose processor, such as, for example, a network or communication processor, compression engine, GPGPU, a high-throughput MIC processor, embedded processor, or the like.

[0239] The static random access memory (SRAM) unit 530 can include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable media has stored therein instructions that, when executed by at least one of the processors, cause the SoC to perform the methods described herein. When the instructions are executed on the computer, the computer is caused to perform the scene-aware method and the training method of the Gaussian decoder disclosed herein.

[0240] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in this paper is only a description of the association relationship of the associated obstacles, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, "at least one" and "one or more" means one, two or more than two.

[0241] It should be noted that the terms "first", "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0242] In this specification, the phrase "one embodiment" or "some embodiments" etc. means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments" etc. in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms "comprising", "having", "including" and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted.

[0243] In the embodiments described above, all or some of the methods can be implemented by software, hardware, firmware or any combination thereof. When implemented in software, all or some of the embodiments can be implemented in the form of a computer program product storing computer program instructions. The computer program instructions can be stored in a computer readable medium. The computer program instructions can be loaded onto a computer to cause the computer to perform a series of operations to generate all or some of the procedures or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable medium, or transmitted by a computer readable medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable medium can be any available medium that can be accessed by a computer or data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium, etc.

[0244] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The aforementioned storage medium can include read-only memory or random access memory, magnetic disk or optical disk, and various program code storage media.

[0245] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A scene perception method applied to an electronic device, comprising: The method comprises: obtaining a multi-view image of a target scene; performing feature extraction on the multi-view image to obtain a first feature map corresponding to the multi-view image; initializing a plurality of Gaussian balls and determining initial query vectors of the plurality of Gaussian balls according to initial attributes of the plurality of Gaussian balls; inputting the first feature map and the plurality of initial query vectors into a target Gaussian decoder to predict first target attributes of the plurality of Gaussian balls corresponding to the first feature map through the target Gaussian decoder, the first target attributes comprising first predicted positions and first predicted semantics of the Gaussian balls; obtaining a perception result of the target scene according to the first target attributes of the plurality of Gaussian balls, the perception result comprising semantic information of a spatial position of the target scene determined according to the first predicted positions and the first predicted semantics; wherein the inputting the first feature map and the plurality of initial query vectors into the target Gaussian decoder to predict the first target attributes of the plurality of Gaussian balls corresponding to the first feature map through the target Gaussian decoder comprises: generating a plurality of sampling points in each of the Gaussian balls according to the initial query vectors, and sampling the first feature map according to the plurality of sampling points to obtain a second feature map; performing feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian balls to obtain a third feature map; updating the initial query vectors according to an adaptive attention mechanism to obtain first query vectors; performing first processing on the third feature map and the first query vectors to obtain second query vectors, the first processing comprising at least linear processing; stacking the second query vectors and third query vectors to obtain fourth query vectors, the third query vectors being obtained by embedding the initial attributes; decoding the fourth query vectors to obtain the first target attributes of the plurality of Gaussian balls.

2. The method of claim 1, wherein, The number of sampling points is determined according to the computing resources of the electronic device.

3. The method of claim 1, wherein, The feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian balls to obtain the third feature map comprises: performing adaptive feature fusion on the second feature map to obtain a fourth feature map; performing residual linking of the fourth feature map and the initial query vectors of the plurality of Gaussian balls to obtain the third feature map.

4. The method of claim 3, wherein, The adaptive feature fusion on the second feature map comprises: performing adaptive fusion on the second feature map according to feature channels and / or spatial positions of the second feature map.

5. The method of claim 3, wherein, The first feature map is a multi-layer feature map; The residual linking of the fourth feature map and the initial query vectors of the plurality of Gaussian balls to obtain the third feature map comprises: The residual linking of the fourth feature map and the initial query vectors of the plurality of Gaussian balls to obtain a fifth feature map, the fifth feature map being a multi-layer feature map, the number of layers of the fifth feature map being the same as the number of layers of the first feature map; performing layer normalization on the fifth feature map to obtain the third feature map.

6. The method of claim 1, wherein, The updating of the initial query vectors according to the adaptive attention mechanism to obtain the first query vectors comprises: Inputting a plurality of the initial query vectors into a linear layer to obtain receptive field weights; Obtaining attention weights according to distances between the receptive field weights and the initial query vectors, the attention weights decreasing with the increase of the receptive field weights; Updating the initial query vectors according to the attention weights to obtain first query vectors.

7. The method of claim 1, wherein, The first processing further comprises an activation processing after the linear processing.

8. The method of claim 1, wherein, The obtaining of the perception result of the target scene according to the first target attributes of the plurality of Gaussian balls comprises: Determining fifth query vectors of the plurality of Gaussian balls according to the first target attributes of the plurality of Gaussian balls; Inputting the fifth query vectors and the first feature map into the target Gaussian decoder to obtain second target attributes of the plurality of Gaussian balls; Obtaining the perception result according to the second target attributes of the plurality of Gaussian balls. 9.A method for training a Gaussian decoder, applied to an electronic device, the method comprising: The method comprises: Obtaining training sample data, the training sample data comprising a first feature map corresponding to multi-view images of a target scene, and initial query vectors of a plurality of Gaussian balls; Inputting the training sample data into a Gaussian decoder to be trained to obtain predicted attributes of the plurality of Gaussian balls, the predicted attributes comprising predicted positions and predicted semantics of the plurality of Gaussian balls; Training the Gaussian decoder to be trained according to the predicted attributes of the plurality of Gaussian balls to obtain a target Gaussian decoder; The inputting of the training sample data into the Gaussian decoder to be trained to obtain the predicted attributes of the plurality of Gaussian balls comprises: Generating a plurality of sampling points in each of the Gaussian balls according to the initial query vectors, and sampling the first feature map according to the plurality of sampling points to obtain a second feature map; Performing feature fusion on the second feature map and the initial query vectors of the plurality of Gaussian balls to obtain a third feature map; Updating the initial query vectors according to an adaptive attention mechanism to obtain first query vectors; Performing first processing on the third feature map and the first query vectors to obtain second query vectors, the first processing at least comprising linear processing; Superimposing the second query vectors and third query vectors to obtain fourth query vectors, the third query vectors being obtained by embedding initial attributes of the Gaussian balls; Decoding the fourth query vectors to obtain the predicted attributes of the plurality of Gaussian balls.

10. The method of claim 9, wherein, The training of the Gaussian decoder to be trained according to the predicted attributes of the plurality of Gaussian balls to obtain a target Gaussian decoder comprises: Obtaining label data, the label data comprising target position information and target semantic information of a plurality of positions of the target scene; Calculating a first loss between the predicted positions of the plurality of Gaussian balls and the target position information; Calculating a second loss between the predicted semantics of the plurality of Gaussian balls and the target semantic information; Adjusting the Gaussian decoder to be trained according to the first loss and the second loss to obtain the target Gaussian decoder.

11. A computer medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the scene-aware method of any one of claims 1 to 8 or the training method of a Gaussian decoder of any one of claims 9 to 10.

12. A computer program product, characterised in that, The computer program product, when running on an electronic device, causes the electronic device to implement the scene-aware method of any one of claims 1 to 8 or the training method of a Gaussian decoder of any one of claims 9 to 10.

13. An electronic device, comprising: comprising: a memory for storing one or more programs; a processor for executing the one or more programs to cause the electronic device to implement the scene-aware method of any one of claims 1 to 8 or the training method of a Gaussian decoder of any one of claims 9 to 10.

Images