Environment sensing method, device, equipment, medium and product

Abstract

The invention provides an environment sensing method and device, equipment, a medium and a product. According to one embodiment of the invention, the method comprises the following steps: acquiring environment data around a vehicle, and generating environment characteristics of different actual areas in an environment where the vehicle is located according to the environment data; mapping the environment features of the different areas to corresponding grid units in a BEV grid chart to obtain a BEV feature map; wherein the resolutions of each layer of grid units from the center to the outside in the BEV grid chart are gradually reduced in sequence; the environment perception task is performed according to the BEV feature map, and the environment features are mapped to the BEV grid map of which the resolution is gradually reduced from the center to the outside, so that the calculation amount of a long-distance area can be reduced while high-precision perception of a close-distance area around a vehicle is ensured, and the balance of calculation resources and perception precision is realized.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an environment perception method, device, equipment, medium and product. Background Art

[0002] With the rapid development of autonomous driving technology, environmental perception, as one of the core components of the autonomous driving system, plays a vital role in achieving safe and reliable autonomous driving. Bird's Eye View (BEV) technology has become the mainstream method in the field of environmental perception due to its ability to intuitively present environmental information.

[0003] Currently, BEV-based environmental perception methods usually use BEV feature maps with fixed resolution, which leads to the problem of difficult balance between computing resources and perception accuracy. Summary of the invention

[0004] To overcome the problems existing in the related art, this specification provides an environment perception method, device, equipment, medium and product.

[0005] According to a first aspect of any embodiment of the present application, there is provided an environment perception method, the method comprising: Acquire environmental data around the vehicle, and generate environmental features of different actual areas in the environment where the vehicle is located according to the environmental data; The environmental characteristics of the different areas are respectively mapped to the corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside; An environmental perception task is performed according to the BEV characteristic map.

[0006] According to a second aspect of any embodiment of the present application, there is provided an environment perception device, the device comprising: A generating module, used for acquiring environmental data around the vehicle, and generating environmental features of different actual areas in the environment where the vehicle is located according to the environmental data; A mapping module, used for mapping the environmental features of the different areas to the corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside; A perception module is used to perform an environmental perception task according to the BEV characteristic map.

[0007] According to a third aspect of any embodiment of the present application, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor implements the method described in any embodiment of the present application by running the executable instructions.

[0008] According to a fourth aspect of any embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the method described in any embodiment of the present application is implemented.

[0009] According to a fifth aspect of any embodiment of the present application, a computer program product is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in any embodiment of the present application is implemented.

[0010] The technical solution provided by this application may have the following beneficial effects: According to the above embodiments, by acquiring environmental data around the vehicle and generating environmental features of different actual areas in the vehicle's environment based on the environmental data, the environmental features of different areas are respectively mapped to corresponding grid cells in the BEV grid map to obtain a BEV feature map, and environmental perception tasks are performed based on the BEV feature map. By mapping the environmental features to a BEV grid map with decreasing resolution from the center to the outside, it is possible to ensure high-precision perception of the close-range area around the vehicle while reducing the amount of calculation in the long-range area, thereby achieving a balance between computing resources and perception accuracy.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0013] Figure 1 is a flow chart of an environment perception method according to an exemplary embodiment of the present application; Figure 2 is a schematic diagram of a vehicle body coordinate system according to an exemplary embodiment of the present application; Figure 3 is a schematic diagram of a visual field grid diagram according to an exemplary embodiment of the present application; Figure 4 is a schematic diagram of a BEV grid diagram according to an exemplary embodiment of the present application; Figure 5 is a schematic diagram of a division effect of a visual field grid map according to an exemplary embodiment of the present application; Figure 6 is a flow chart of another environment perception method according to an exemplary embodiment of the present application; Figure 7 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application; Figure 8 It is a block diagram of an environment perception device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0017] The BEV-based environmental perception method converts environmental data collected by multiple cameras into a bird's-eye view to achieve an overall perception of the surrounding environment. However, the current BEV feature map uses a fixed-resolution feature representation method, which has the problem of difficult balance between computing resources and perception accuracy.

[0018] In autonomous driving scenarios, close-range areas usually require more refined perception to ensure safety, while long-range areas require relatively lower accuracy. Fixed-resolution BEV feature maps not only waste computing resources in long-range areas, but also lead to insufficient perception accuracy in close-range areas.

[0019] In order to solve the above problems, the present application proposes an environment perception method. To further illustrate the present application, the following embodiments are provided: See also Figure 1 , Figure 1 This is a flow chart of an environment perception method according to an exemplary embodiment of the present application. The environment perception method can be executed by a perception system, which can be applied to a vehicle or to a single server, a cluster server, a cloud server or other service end. The environment perception method can also be executed by other systems or devices in different application scenarios, and the embodiments of the present application do not limit this.

[0020] like Figure 1 As shown, the environment perception method may include the following steps: Step 101: Acquire environmental data around the vehicle, and generate environmental features of different actual areas in the vehicle's environment based on the environmental data.

[0021] In this step, the perception system can obtain environmental data around the vehicle based on the vehicle camera, radar or other sensor devices. Environmental data is the original information about the vehicle's surroundings obtained by various sensors, providing basic information for subsequent environmental perception.

[0022] In one embodiment, the environmental data may include visual data collected by a visual sensor and / or point cloud data collected by a radar. The visual sensor may be a panoramic camera located at various positions of the vehicle, which may cover a 360° field of view of the vehicle. The visual data is a sequence of raw images collected by a camera, which may include RGB or grayscale pixel information. Exemplarily, the visual data may be collected by six panoramic cameras located at the front / rear / left / right / left front / right front positions of the vehicle, respectively.

[0023] As mentioned above, by fusing the multimodal data of visual data and point cloud data, the advantages of different sensors can be complemented, the generalization ability of environmental perception in scenes with changing lighting or occlusion can be enhanced, and the accuracy of environmental perception can be improved.

[0024] The environmental data around the vehicle can be feature extracted to generate environmental features of different actual areas in the vehicle's environment. Environmental features are information extracted from environmental data and are used to describe specific properties of the vehicle's environment. They can be two-dimensional (2D) features, three-dimensional (3D) features, etc. Different actual areas are different spatial areas into which the vehicle's environment is divided.

[0025] For example, a gradually expanding geometric division method can be used to divide different actual areas in the environment where the vehicle is located, forming an outwardly expanding nested structure. For example, the nested square area can be expanded outward with the origin of the vehicle body coordinate system as the center to divide multiple concentric square areas to obtain different actual areas, each of which corresponds to a different distance range.

[0026] It is understandable that other division methods such as polar coordinates and dynamic topology can also be used to divide the different actual areas in the vehicle's environment, as long as the resolution of each layer of grid cells from the center to the outside in the mapped BEV grid map can be reduced successively. The embodiment of the present application does not limit this.

[0027] In one embodiment, the environmental data may include visual data collected by a visual sensor, and the environmental features may include visual features, which are two-dimensional features extracted from the visual data and used to capture abstract information such as target contours and textures.

[0028] Visual data collected by a visual sensor may be obtained. The visual data is input into a feature extraction model to obtain visual features output by the feature extraction model. The feature extraction model may include a resnet50 model, and the resnet50 model has a deep residual structure. Visual features of the visual data may be extracted based on the resnet50 model.

[0029] As mentioned above, by extracting visual features of visual data based on the ResNet50 model and utilizing the deep residual structure of ResNet50 itself, the expression ability of complex environmental features can be enhanced and the robustness of subsequent environmental perception tasks can be improved.

[0030] Step 102: Mapping the environmental features of different actual areas to corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside.

[0031] In this step, the perception system can map the environmental characteristics of different actual areas to the corresponding grid cells in the BEV grid map respectively. Each actual area corresponds to each layer of grid cells. The environmental characteristics of each actual area can be mapped to the grid cells of the corresponding layer in the BEV grid map to obtain a BEV feature map.

[0032] Among them, the BEV grid map is a structured spatial representation centered on the vehicle and describing the surrounding environment from a bird's-eye view. It is a two-dimensional plane map composed of multiple layers of grid units, and each grid unit corresponds to an area in the physical world.

[0033] The BEV grid map includes multiple layers of grid cells from the center to the outside, and the resolution of each layer of grid cells (i.e., the physical size represented by each grid cell) decreases from the center to the outside. The number of different actual regions is the same as the number of layers of grid cells in the BEV grid map.

[0034] Step 103: Perform an environmental perception task according to the BEV feature map.

[0035] In this step, the perception system can perform environmental perception tasks based on the BEV feature map. The environmental perception tasks can be target detection tasks, space occupancy prediction tasks, etc.

[0036] For example, the BEV feature map can be input into a target detection model to obtain a target detection result output by the target detection model; for another example, the BEV feature map can be input into a space occupancy prediction model to obtain a space occupancy prediction result output by the space occupancy prediction model, etc. The space occupancy prediction model can be a multi-layer perceptron model, and the space occupancy prediction result can include the probability of occupation or non-occupancy of each occupied grid.

[0037] The environmental perception method of this embodiment obtains environmental data around the vehicle, generates environmental features of different actual areas in the vehicle's environment based on the environmental data, maps the environmental features of different areas to corresponding grid cells in the BEV grid map to obtain a BEV feature map, and performs environmental perception tasks based on the BEV feature map. By mapping the environmental features to a BEV grid map with decreasing resolution from the center to the outside, it is possible to reduce the amount of calculation in long-distance areas while ensuring high-precision perception of short-range areas around the vehicle, thereby achieving a balance between computing resources and perception accuracy, and is more suitable for deployment and application in actual autonomous driving scenarios.

[0038] Moreover, through the multi-resolution BEV mapping mechanism based on the spatial representation angle, a more intuitive and stable resolution adjustment mechanism can be achieved, without the need to dynamically switch the network depth according to the resolution, thereby simplifying the system structure and improving the robustness of environmental perception.

[0039] In the above-mentioned embodiments, it is introduced that by mapping the environmental features of different actual areas to each layer of grid units with decreasing resolution from the center to the outside in the BEV grid map, the computing resource consumption for the distant area is reduced while ensuring high-precision perception of the near area. In the following embodiments, the mapping process of environmental features will be described in more detail and can be applied to any of the above embodiments.

[0040] In one embodiment, a visual field grid map can be set in a preset coordinate system. The visual field grid map is a two-dimensional grid structure established in the preset coordinate system, which is used to divide the physical space into multi-level areas, and can be used as an intermediate mapping carrier from environmental features to BEV space, and is used to establish a spatial correspondence between environmental data and BEV feature maps. The preset coordinate system can be a world coordinate system, a vehicle body coordinate system, etc.

[0041] The field of view grid map is divided into n field of view areas from the center to the outside, and the area of ​​the field of view area gradually increases, and n is not less than 2. Among them, the number of n can be equal to the number of layers of grid units in the BEV grid map. The field of view area is a continuous spatial range divided from the center to the outside in the field of view grid map. Each field of view area corresponds to a different distance level and resolution requirement, which is used for progressive zoning management of physical space.

[0042] In one embodiment, the visual field grid map may include: n square frames nested in sequence from the center to the outside. The side length of the outermost square frame is greater than or equal to the side length of the visual field grid map, and the difference between the side lengths of every two adjacent square frames increases from the center to the outside. If the side length of the square frame is greater than the side length of the visual field grid map, the corresponding area in the visual field grid map that exceeds the length may be filled with zero values ​​and regarded as an invalid area.

[0043] The n viewing areas may include: for the 1st viewing area, the viewing area is the area of ​​the 1st square; for the ith viewing area, the ith viewing area is the area between the ith square and the i-1th square, i is not less than 2 and i is not greater than n.

[0044] See also Figure 2 , Figure 2 A schematic diagram of a vehicle body coordinate system is shown. The origin of the vehicle body coordinate system is located at the center of mass of the vehicle, that is, the geometric center of the vehicle. The x-axis, y-axis, and z-axis correspond to the front and rear, left and right, and vertical directions of the vehicle, respectively. The x-axis points to the forward direction of the vehicle, the y-axis points to the left of the driver of the vehicle, and the z-axis is perpendicular to the vehicle and points upward. For example, the environment perception range can be taken as the center of the origin of the vehicle body coordinate system, with 10m in front, back, left, and right, 2m up, and 1m down.

[0045] Exemplarily, the resolution of each layer of grid cells from the center to the outside in the BEV grid map can be determined according to the following formula 1: Formula 1 Where d represents the side length of the i-th square border in the visual field grid map, S represents the real-world side length corresponding to the i-th grid unit in the BEV grid map, Represents the gradual attenuation of the resolution of each layer of grid cells from the center to the outside.

[0046] Exemplarily, the side length of the i-th square border may be calculated according to the following formula 2: Formula 2 in, Represents the side length of the i-th square border.

[0047] See also Figure 3 , Figure 3 A schematic diagram of a vision grid diagram is shown. A vision grid diagram of a square with a side length of 20 meters can be established based on the origin of the vehicle coordinate system. The side length of each grid unit in the vision grid diagram is 1 meter. The vision grid diagram is a grid top view in the world coordinate system. The vision grid diagram can include 4 square borders (red square borders) nested in sequence from the center to the outside. =2 meters as an example, the side length of the first square border (i=1) is 2 meters, the side length of the second square border (i=2) is 6 meters, the side length of the third square border (i=3) is 12 meters, and the side length of the fourth square border (i=4) is 20 meters.

[0048] Using nested square borders, the visual field grid is divided into four visual field areas (i=1, 2, 3, 4) from the center to the outside, and the area of ​​each of the four visual field areas gradually increases. The first visual field area is the area of ​​the first square, the second visual field area is the area between the second square and the first square, the third visual field area is the area between the third square and the second square, and the fourth visual field area is the area between the fourth square and the third square.

[0049] As described above, by determining n nested square borders from the center to the outside in the field of view grid diagram set under a preset coordinate system, using nested squares to divide the field of view area, and simplifying the feature space mapping logic through geometric symmetry and hierarchical progressive relationship, the computational efficiency of grid division and feature mapping can be improved.

[0050] For the environmental features of each actual area, the environmental features can be mapped to the positions of corresponding grid cells in the BEV grid map according to the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map to obtain a BEV feature map.

[0051] Among them, the coordinate mapping relationship can describe the coordinate transformation relationship between the coordinate position of a grid cell in the field of view grid map and the coordinate position of the corresponding grid cell in the BEV grid map, which can ensure the geometric consistency of environmental features in spatial transformation and realize lossless transmission of multi-resolution features.

[0052] See also Figure 4 , Figure 4 A schematic diagram of a BEV grid diagram is shown. Exemplarily, the BEV grid diagram may include four layers of grid units. The feature dimensions of the BEV feature in the BEV grid are B×C×H×W, where B is the number of image frames of the environmental data, C is the number of channels, W is the width of the BEV feature, and H is the height of the BEV feature. Exemplarily, Figure 4 The feature dimensions of the BEV grid map shown are B=1, C=256, W=H=10.

[0053] Please continue reading Figure 3 , the different actual areas in the vehicle environment can be divided into four actual areas, and the environmental characteristics of each actual area can be calculated according to Figure 3 The i-th area in the visual field grid diagram shown is Figure 4 The coordinate mapping relationship of the i-th layer in the BEV grid map shown maps the environmental features to the positions of the corresponding grid cells in the BEV grid map to obtain a BEV feature map, so that the resolution of the BEV features in the close range of the center of the field of view is higher than that in the long range.

[0054] As described above, for the environmental characteristics of each actual area, according to the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map, the environmental characteristics are mapped to the positions of the corresponding grid cells in the BEV grid map to obtain a BEV feature map. By utilizing the coordinate mapping relationship, the BEV mapping of environmental characteristics can be quickly realized without relying on complex models or algorithms, thereby further reducing computational overhead.

[0055] See also Figure 5 , Figure 5 A schematic diagram of a grid division effect of a field of view is shown. For example, the side length of the grid unit in the i-th square frame can be calculated according to the following formula 3: Formula 3 by =1 meter as an example, in the first field of view (i=1), the side length of each grid cell is 1 meter, in the second field of view (i=2), the side length of each grid cell is 2 meters, in the third field of view (i=3), the side length of each grid cell is 3 meters, and in the fourth field of view (i=4), the side length of each grid cell is 4 meters.

[0056] Each grid cell in the first layer of grid cells corresponds to a 1m×1m grid in the first field of view area in the field of view grid map, covering a smaller physical range and having a higher resolution, and each grid cell in the fourth layer of grid cells corresponds to a 4m×4m grid in the fourth field of view area in the field of view grid map, covering a larger physical range and having a lower resolution.

[0057] It can be understood that the above-mentioned division method of the field of view range in the field of view grid diagram is only an example, and other division methods can also be used, and the embodiments of the present application are not limited to this.

[0058] In one embodiment, the coordinate position of the grid unit in the field of view grid map can be obtained by camera parameter conversion or the like. The coordinate position of the real world in the vehicle coordinate system The projection mapping relationship between them.

[0059] The coordinate mapping relationship between the ith region in the visual field grid map and the ith layer in the BEV grid map can be determined according to the linear interpolation algorithm. For example, the coordinate position of the grid unit in the BEV grid map can be , by linear interpolation, calculate the coordinate position of the real world in the corresponding vehicle coordinate system , determine the coordinate position of the grid cell in the BEV grid map The coordinate position of the real world in the vehicle coordinate system The interpolation mapping relationship between .

[0060] Combining the interpolation mapping relationship and the projection mapping relationship, the coordinate positions of the grid cells in the BEV grid map are associated with the coordinate positions of the grid cells in the field of view grid map according to the real-world coordinate positions in the vehicle body coordinate system. For each grid cell in the BEV grid map corresponding to the real-world coordinate position in the vehicle body coordinate system, the coordinate position of the grid cell in the field of view grid map corresponding to the coordinate position can be determined, and the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map is obtained.

[0061] As described above, by respectively obtaining the projection mapping relationship between the coordinate position of the grid cells in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system, and the interpolation mapping relationship between the coordinate position of the grid cells in the BEV grid map and the coordinate position of the real world in the vehicle body coordinate system, based on the projection mapping relationship and the interpolation mapping relationship, the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map is determined, which can avoid the feature dislocation problem caused by coordinate discretization and improve the spatial continuity of the BEV feature map.

[0062] In one embodiment, the projection mapping relationship may be determined according to the camera parameters of the camera used to collect environmental data. For example, the camera parameters may include camera intrinsic parameters and camera extrinsic parameters, and the coordinate position of the grid unit in the ith region in the field of view grid map corresponding to the real-world coordinate position in the vehicle body coordinate system may be determined according to the camera intrinsic parameters and extrinsic parameters, so as to obtain the projection mapping relationship between the coordinate position of the grid unit in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system.

[0063] According to the determined coordinate mapping relationship, environmental features can be extracted from the coordinate position of the i-th area in the field of view grid map, and the environmental features can be assigned to corresponding grid cells in the BEV grid map to obtain a BEV feature map.

[0064] Among them, the camera intrinsic parameters can describe the matrix of the internal optical properties of the camera, which may include focal length, distortion coefficient, etc. The camera extrinsic parameters can describe the position and posture of the camera in the world coordinate system, which may include rotation matrix, translation vector, etc.

[0065] Exemplarily, the coordinate position of the grid unit of the i-th area of ​​the environmental feature in the field of view grid map may be determined according to the following formula 4: Formula 4 in, represents the coordinate position of the grid unit in the i-th region of the field of view grid map, K represents the camera intrinsic parameter, T represents the camera extrinsic parameter, Represents the real-world coordinate position in the vehicle coordinate system.

[0066] As described above, by determining the projection mapping relationship based on the camera parameters of the camera used to collect environmental data, extracting environmental features from the coordinate position of the i-th area in the field of view grid map, assigning the environmental features to the corresponding grid cells in the BEV grid map to obtain a BEV feature map, and using the camera parameters to determine the coordinate position and directly assign features, the coordinate conversion error can be reduced and the calculation process can be simplified, thereby improving the reliability of BEV feature map generation.

[0067] To further introduce the environmental perception process, Figure 6 A flow chart of another environment perception method is shown. The environment perception method may include the following steps: Step 601: Acquire visual data around the vehicle.

[0068] In this step, the perception system can obtain visual data around the vehicle by using six panoramic cameras located in front / rear / left / right / left front / right front of the vehicle to synchronously capture images with a length and width size of 1920×1080.

[0069] Step 602: Extract visual features of visual data based on the resnet50 model.

[0070] In this step, the environmental data can be input into the resnet50 model to obtain the visual features output by the resnet50 model.

[0071] Step 603: Determine n square frames that are nested in sequence from the center outward in the visual field grid map.

[0072] In this step, n sequentially nested square frames from the center to the outside can be determined in the visual field grid map set in the preset coordinate system to obtain n visual field areas sequentially divided from the center to the outside in the visual field grid map.

[0073] Step 604: According to the camera parameters of the camera, determine the coordinate position of the grid unit in the field of view grid map corresponding to the coordinate position of the real world in the vehicle body coordinate system, and obtain the projection mapping relationship.

[0074] In this step, for the visual features of each actual area, the coordinate positions of the grid cells in the field of view grid map corresponding to the real-world coordinate positions in the vehicle body coordinate system are determined based on the camera intrinsic parameters and camera extrinsic parameters, and the projection mapping relationship between the coordinate positions of the grid cells in the field of view grid map and the real-world coordinate positions in the vehicle body coordinate system is obtained.

[0075] Step 605: According to the linear interpolation algorithm, the coordinate position of the grid unit in the BEV grid map corresponding to the coordinate position of the real world in the vehicle body coordinate system is determined to obtain an interpolation mapping relationship.

[0076] In this step, the coordinate position of the grid unit in the BEV grid map corresponding to the real-world coordinate position in the vehicle body coordinate system is determined by linear interpolation, and the interpolation mapping relationship between the coordinate position of the grid unit in the BEV grid map and the real-world coordinate position in the vehicle body coordinate system is obtained.

[0077] Step 606: Determine the coordinate mapping relationship between the ith region in the field of view grid map and the ith layer in the BEV grid map based on the projection mapping relationship and the interpolation mapping relationship.

[0078] In this step, the interpolation mapping relationship and the projection mapping relationship are combined, and according to the coordinate position of the real world in the vehicle body coordinate system, the coordinate position of the grid cell in the BEV grid map is associated with the coordinate position of the grid cell in the field of view grid map to obtain the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map.

[0079] Step 607: Based on the coordinate mapping relationship, the visual features are assigned to the corresponding grid cells in the BEV grid map to obtain a BEV feature map.

[0080] In this step, based on the coordinate mapping relationship, for the coordinate position of each grid cell in the BEV grid map, visual features can be extracted from the coordinate position of the grid cell in the corresponding field of view grid map, and the visual features can be assigned to the corresponding grid cells in the BEV grid map to obtain a BEV feature map with a BEV feature dimension of B×C×H×W.

[0081] Through the coordinate mapping relationship, the two-dimensional visual features are directly assigned to the BEV grid map, which can quickly convert the two-dimensional features into BEV features of different resolutions, further reducing the computational overhead and making it more suitable for vehicle platforms with limited computing resources.

[0082] Step 607: Perform an environmental perception task according to the BEV characteristic map.

[0083] In this step, the environmental perception task can be performed based on the BEV feature map. For example, the BEV feature map can be input into the space occupancy prediction model, which predicts the space occupancy state of the vehicle's environment and obtains the occupancy probability map output by the space occupancy prediction model, which is convenient for subsequent support of autonomous driving decision-making and other tasks.

[0084] The dimensions of the occupancy probability map may be B×Z×H×W. For example, the dimensions of the occupancy probability map may be B=1, H=W=8, and Z=6. The occupancy probability map includes a plurality of occupancy grids, each of which has a height value of 0.5m in the height direction, covering a height range of -1m to 2m. The grid value of each occupancy grid represents the occupancy probability (0~1) of the corresponding cubic space.

[0085] It is understandable that the zoom BEV model can be used to implement the above steps 602 to 606, and the visual data around the vehicle can be obtained, and the visual data can be input into the zoom BEV model to obtain the BEV feature map output by the zoom BEV model. The BEV feature map is then input into the downstream target detection model, space occupancy prediction model, etc., and the model is used to perform environmental perception tasks.

[0086] Figure 7 1 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. The electronic device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, a car computer, etc. Figure 7 At the hardware level, the electronic device includes a processor 701, an internal bus 702, a network interface 703, a memory 704, and a non-volatile memory 705, and may also include hardware required for other services. The processor 701 reads the corresponding computer program from the non-volatile memory 705 into the memory 704 and then runs it, forming an environmental perception device at the logical level. Of course, in addition to software implementations, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0087] Figure 8 is a block diagram of an environment perception device according to an exemplary embodiment of the present application. Figure 8 , the device may include: a generating module 801, a mapping module 802 and a sensing module 803, wherein: The generating module 801 is used to obtain environmental data around the vehicle and generate environmental features of different actual areas in the environment where the vehicle is located according to the environmental data; The mapping module 802 is used to map the environmental features of the different actual areas to the corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside; The perception module 803 is used to perform an environmental perception task according to the BEV characteristic map.

[0088] In one example, a field of view grid map set in a preset coordinate system is divided into n field of view areas from the center to the outside, and the area of ​​the field of view area gradually increases, and n is not less than 2; the mapping module 802, when used to map the environmental characteristics of the different actual areas to the corresponding grid cells in the BEV grid map to obtain a BEV feature map, includes: for the environmental characteristics of each actual area, according to the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map, mapping the environmental characteristics to the position of the corresponding grid unit in the BEV grid map to obtain a BEV feature map.

[0089] In one example, the field of vision grid map includes: n square borders nested in sequence from the center to the outside; wherein the side length of the outermost square border is greater than or equal to the side length of the field of vision grid map; the n field of vision areas include: for the first field of vision area, the field of vision area is the area of ​​the first square; for the i-th field of vision area, the i-th field of vision area is the area between the i-th square and the i-1-th square, i is not less than 2 and i is not greater than n.

[0090] In one example, the mapping module 802 is also used to obtain the projection mapping relationship between the coordinate position of the grid unit in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system; determine the interpolation mapping relationship between the coordinate position of the grid unit in the BEV grid map and the coordinate position of the real world in the vehicle body coordinate system according to a linear interpolation algorithm; and determine the coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map based on the projection mapping relationship and the interpolation mapping relationship.

[0091] In one example, the mapping module 802, when used to obtain the projection mapping relationship between the coordinate position of the grid unit in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system, includes: determining the projection mapping relationship according to the camera parameters of the camera used to collect the environmental data; the mapping module 802, when used to map the environmental features to the corresponding grid units in the BEV grid map to obtain the BEV feature map, includes: extracting the environmental features from the coordinate position of the i-th area in the field of view grid map, and assigning the environmental features to the corresponding grid units in the BEV grid map to obtain the BEV feature map.

[0092] In one example, the environmental data includes visual data collected by a visual sensor; the generation module 801, when used to generate environmental features of different actual areas in the vehicle's environment, includes: extracting visual features of the visual data based on a resnet50 model.

[0093] In one example, the environmental data includes visual data collected by a visual sensor and / or point cloud data collected by a radar.

[0094] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0095] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0096] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions. The instructions can be executed by a processor of an environment sensing device to implement any of the methods described in the above embodiments.

[0097] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0098] In an exemplary embodiment, a computer program product including a computer program / instruction is also provided. The computer program / instruction can be executed by a processor of an environment sensing device to implement any of the methods described in the above embodiments.

[0099] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0100] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0101] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for environmental perception, characterized in that: The method comprises: Acquire environmental data around the vehicle, and generate environmental features of different actual areas in the environment where the vehicle is located according to the environmental data; The environmental features of the different actual areas are respectively mapped to the corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside; An environmental perception task is performed according to the BEV characteristic map.

2. The method according to claim 1, characterized in that The visual field grid map set in the preset coordinate system is divided into n visual field regions from the center to the outside, and the areas of the visual field regions gradually increase, and n is not less than 2; The step of mapping the environmental features of the different actual areas to the positions of the corresponding grid cells in the BEV grid map to obtain the BEV feature map comprises: For the environmental features of each actual area, according to the coordinate mapping relationship between the ith area in the field of view grid map and the ith layer in the BEV grid map, the environmental features are mapped to the positions of the corresponding grid cells in the BEV grid map to obtain a BEV feature map.

3. The method according to claim 2, characterized in that The visual field grid map includes: n square frames nested in sequence from the center to the outside; wherein the side length of the outermost square frame is greater than or equal to the side length of the visual field grid map; The n visual field areas include: For the first visual field area, the visual field area is the area of ​​the first square; For the i-th field of view area, the i-th field of view area is an area between the i-th square and the i-1-th square, i is not less than 2 and i is not greater than n.

4. The method according to claim 2, characterized in that: The method further comprises: Acquire a projection mapping relationship between the coordinate position of the grid unit in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system; Determine, according to a linear interpolation algorithm, an interpolation mapping relationship between a coordinate position of a grid cell in the BEV grid map and a coordinate position of a real world in the vehicle body coordinate system; Based on the projection mapping relationship and the interpolation mapping relationship, a coordinate mapping relationship between the i-th area in the field of view grid map and the i-th layer in the BEV grid map is determined.

5. The method according to claim 4, characterized in that The obtaining of the projection mapping relationship between the coordinate position of the grid unit in the field of view grid map and the coordinate position of the real world in the vehicle body coordinate system includes: Determining the projection mapping relationship according to camera parameters of a camera used to collect the environmental data; Mapping the environmental features to corresponding grid cells in the BEV grid map to obtain a BEV feature map includes: The environmental features are extracted from the coordinate position of the i-th area in the field of view grid map, and the environmental features are assigned to the corresponding grid cells in the BEV grid map to obtain a BEV feature map.

6. The method according to claim 1, characterized in that The environmental data includes visual data collected by a visual sensor; The generating of environmental features of different actual areas in the environment where the vehicle is located includes: The visual features of the visual data are extracted based on the resnet50 model.

7. The method according to claim 1, characterized in that The environmental data includes visual data collected by a visual sensor and / or point cloud data collected by a radar.

8. An environment sensing device, characterized in that: The device comprises: A generating module, used for acquiring environmental data around the vehicle, and generating environmental features of different actual areas in the environment where the vehicle is located according to the environmental data; A mapping module, used for mapping the environmental features of the different actual areas to the corresponding grid cells in the BEV grid map to obtain a BEV feature map; wherein the resolution of each layer of grid cells in the BEV grid map decreases from the center to the outside; A perception module is used to perform an environmental perception task according to the BEV characteristic map.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images