Environment perception method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an environment perception method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a training environment image collected by a look-around fisheye camera, carrying out the semantic recognition processing of the training environment image, and obtaining a training semantic recognition result corresponding to the training environment image, acquiring point cloud data acquired in a target environment through a radar, and performing image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image acquisition environment of the training environment image; and performing at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model, and sending the target environment perception model to the drivable device, and enabling the drivable equipment to collect an environment visual angle image, inputting the environment visual angle image into the target environment perception model for environment perception recognition, and outputting a target depth map and a target semantic recognition result corresponding to the environment visual angle image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an environment perception method and device, a storage medium, and an electronic device. BACKGROUND

[0002] With the development of autonomous driving technology, the environment perception system has become a core component of autonomous vehicles and other mobile robots. Environment perception involves using sensors and perception algorithms to acquire and understand information about the surrounding environment, so that the robot can accurately perceive, understand and respond to various complex road and traffic situations.

[0003] In related technologies, the robot near-field perception technology usually adopts a multi-sensor fusion method for detecting obstacles, identifying roads or pedestrians, measuring distances and speeds, etc. The multi-sensor fusion method refers to fusing data from multiple sensors, such as lidar, cameras, ultrasonic sensors, etc., to obtain surrounding environmental information. SUMMARY

[0004] The embodiments of the present application provide an environment perception method, device, computer storage medium and electronic device, and the technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide an environment perception method, which comprises:

[0006] Obtaining a training environment image collected by a surround-view fisheye camera, performing semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image;

[0007] Obtaining point cloud data collected by a radar in a target environment, performing image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image collection environment of the training environment image;

[0008] Performing at least one round of model training on an initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model;

[0009] Sending the target environment perception model to a drivable device, so that the drivable device inputs an environmental perspective image into the target environment perception model for environment perception after collecting the environmental perspective image, and outputs a target depth map and a target semantic recognition result corresponding to the environmental perspective image.

[0010] In a second aspect, the embodiments of the present application provide an environment perception device, which comprises:

[0011] The semantic acquisition module is configured to acquire a training environment image collected by the surround-view fisheye camera, and perform semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image.

[0012] The depth acquisition module is configured to acquire point cloud data collected by the radar in a target environment, and perform image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image acquisition environment of the training environment image.

[0013] The model training module is configured to perform at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model.

[0014] The model processing module is configured to send the target environment perception model to the drivable device, so that the drivable device inputs an environment perspective image into the target environment perception model for environment perception after collecting the environment perspective image, and outputs a target depth map and a target semantic recognition result corresponding to the environment perspective image.

[0015] In a third aspect, an embodiment of the present application provides a computer storage medium, which has a plurality of instructions, the instructions being suitable for being loaded and executed by a processor to perform the method described above.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, which can include a memory and a processor, wherein the memory stores a computer program, and the computer program is suitable for being loaded and executed by the memory to perform the method described above.

[0017] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0018] In the embodiment of the present application, a training environment image collected by a surround-view fisheye camera is obtained, semantic recognition processing is performed on the training environment image to obtain a training semantic recognition result corresponding to the training environment image, point cloud data collected by a radar in a target environment is obtained, image projection processing is performed on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment is an image collection environment of the training environment image, at least one round of model training is performed on an initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model, and the target environment perception model is sent to a drivable device, so that the drivable device inputs an environmental perspective image into the target environment perception model for environment perception after collecting the environmental perspective image, and outputs a target depth map and a target semantic recognition result corresponding to the environmental perspective image. By adopting the technical solution, the embodiment of the present application introduces the point cloud data collected by the radar as auxiliary supervision of a depth estimation branch to train the environment perception model, so that the environment perception model can learn not only the semantics of the training environment image but also the depth corresponding to the training environment image in the training process, and thus the target environment perception model finally trained can accurately recognize the semantics of the environmental image and accurately estimate the depth information corresponding to the environmental image, thereby achieving the effect of strengthening the near-field semantic understanding ability and spatial perception ability of the pure-vision environment perception model. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flowchart of an environment perception method provided by the embodiment of the present application;

[0021] Figure 2 is a scene diagram of an environment perception method provided by the embodiment of the present application;

[0022] Figure 3 is a flowchart of another environment perception method provided by the embodiment of the present application;

[0023] Figure 4 is a structural diagram of an environment perception device provided by the embodiment of the present application;

[0024] Figure 5 is a structural diagram of a depth acquisition module provided by the embodiment of the present application;

[0025] Figure 6Fig. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the application purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0027] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for the purpose of description, and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units not listed, or optionally includes other steps or units inherent to the process, method, product or device. Those skilled in the art can understand the specific meaning of the above terms in the present application according to the specific circumstances. In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. "And / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects.

[0028] In related technologies, the robot near field perception technology usually adopts the way of fusing data of multiple sensors such as lidar, camera, ultrasonic sensor, etc. to obtain the surrounding environment information. However, the multi-sensor data fusion method has many problems: first, the sensor configuration is complex, the use of multiple sensors requires a complex sensor configuration and calibration process, which increases the complexity and cost of the system, and requires additional hardware support; second, data fusion is difficult, the data of different sensors has different characteristics and noise, such as lidar point cloud data and camera image data, which cannot effectively fuse these data; third, the multi-sensor fusion method may have high data processing and calculation complexity, which cannot meet the real-time requirement.

[0029] In view of the problems existing in the above multi-sensor data fusion mode, the pure visual perception scheme has the advantages of simple hardware configuration and strong real-time compared with the multi-sensor scheme. However, the current pure visual perception scheme is only applicable to identifying a specific task, such as single target detection, segmentation, depth estimation and the like. Therefore, the pure visual perception scheme in the robot near-field perception technology can only identify a specific single task, resulting in the problem of poor environmental perception capability in the environmental perception scene.

[0030] To solve the above technical problems, the environmental perception method of the present application is further described in combination with specific embodiments.

[0031] In the following method embodiments, each step is described in detail with the execution subject of each step being an electronic device.

[0032] Please refer to Figure 1 , a flowchart of an environmental perception method provided by the present application. As Figure 1 indicated, the method of the present application can include the following steps:

[0033] S101, acquiring a training environment image collected by a surround-view fisheye camera, and performing semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image.

[0034] As can be easily understood, the surround-view fisheye camera refers to a fisheye camera arranged around the electronic device. The fisheye camera refers to a camera with a fisheye lens, which is a lens with extremely short focal length and nearly or equal to 180° viewing angle. In the present application, the electronic device can be a movable device, which refers to a mobile robot, a load-carrying robot, a cleaning robot, an autonomous driving vehicle device and the like. In a possible implementation, the surround-view fisheye camera can refer to a surround-view fisheye camera arranged on the front, back, left and right sides of the movable device to collect panoramic images around the movable device, which can include images of the front, back, left and right sides.

[0035] The training environment image refers to an environment image collected by each surround-view fisheye camera under its own viewing angle. In the present application, the training environment image is used as a sample image to train the environmental perception model, so that the environmental perception model can learn pixel-level semantic information from the sample image. The pixel-level semantic information can be understood as the meaning or category of the object represented by at least one pixel in the image.

[0036] The training semantic recognition result refers to the name or category of the object contained in each training environment image.

[0037] In some embodiments, the training environment image collected by the surround-view fisheye camera is obtained, and one implementation can be: obtaining the training environment image from the movable device, wherein the training environment image is obtained by the movable device equipped with the surround-view fisheye camera collecting the training environment image at different angles. Specifically, the training environment image can be obtained from the movable device by a removable storage device or by a wireless communication transmission mode. The wireless communication transmission mode can include wireless local area network, Bluetooth, mobile communication, etc.

[0038] The training environment image collected by the surround-view fisheye camera is obtained, and another implementation can be: the electronic device collects the training environment image at different angles by the surround-view fisheye camera, and the electronic device itself is installed with the surround-view fisheye camera at different shooting angles.

[0039] The training environment image is subjected to semantic recognition processing to obtain the training semantic recognition result corresponding to the training environment image, and one implementation can be: inputting the training environment image into the pre-trained semantic segmentation model to output the training semantic recognition result corresponding to the training environment image. The semantic segmentation model can be obtained by training based on the sample training semantic recognition result label corresponding to the sample training environment image.

[0040] In S102, the point cloud data collected by the radar in the target environment is obtained, and the training environment image corresponding to the depth map is obtained by image projection processing of the point cloud data. The target environment is the image collection environment of the training environment image.

[0041] The target environment can be understood as the environment in which the training environment image is collected. Specifically, the intelligent collection device is installed with the surround-view fisheye camera and the radar, and in the target environment, the surround-view fisheye camera collects the training environment image and the radar collects the point cloud data within a certain time period. The intelligent collection device can be a movable device with driving function, or an electronic device without driving function.

[0042] The point cloud data refers to the data set of points in the space of the environment in which the training environment image is located, which is collected by the radar at the same time as the training environment image is collected. The point cloud data includes the position of the points in the three-dimensional space.

[0043] The depth map refers to an image containing the depth values of part or all of the pixels in the training environment image. The depth value refers to the distance from the radar of the object represented by the pixel, since the radar is a sensor installed on the device, the depth value can also be understood as the distance from the device of the object represented by the pixel.

[0044] In some embodiments, the point cloud data collected in the target environment by the radar is obtained from the movable device, and an implementation manner can be as follows: the point cloud data is obtained from the movable device, and the point cloud data is obtained by the movable device equipped with the surround-view fisheye camera and the radar when the movable device collects the training environment images at different view angles in the target environment by the surround-view fisheye camera and collects the point cloud data in the target environment by the radar. Specifically, the point cloud data is obtained from the movable device, which can be obtained from the movable device by a removable storage device or obtained from the movable device by a wireless communication transmission manner. The wireless communication transmission manner can include a wireless local area network, Bluetooth, mobile communication, and the like.

[0045] The point cloud data collected in the target environment by the radar is obtained, and another implementation manner can be as follows: the electronic device collects the training environment images at different view angles in the target environment by the surround-view fisheye camera, and the electronic device also collects the point cloud data in the target environment by the radar, and the electronic device is installed with the radar and the surround-view fisheye camera at different shooting view angles.

[0046] The point cloud data is subjected to image projection processing to obtain the depth map corresponding to the training environment image, and an implementation manner can be as follows: the extrinsic matrix of the radar is used to perform projection calculation on the point cloud data to obtain the pixel coordinates of the spatial point on the image plane, and the depth map corresponding to the training environment image is determined according to the pixel coordinates of the spatial point on the pixel plane and the depth value of the spatial point. It should be noted that, before the depth map corresponding to the training environment image is determined, the point cloud data with the same collection time as that of each frame of training environment image needs to be obtained, and then the projection is performed on the point cloud data collected at the same time, so as to determine the depth map corresponding to the point cloud data collected at the same time, thereby obtaining the depth map corresponding to the frame of training environment image.

[0047] In S103, at least one round of model training is performed on the initial environment perception model based on the training environment image, the training semantic recognition result, and the depth map to obtain the target environment perception model.

[0048] The initial environment perception model can be understood as a preliminary environment perception model obtained by setting model parameters of the machine learning model based on expert experience. The target environment perception model can be understood as a senior environment perception model with better environment perception capability compared with the initial environment perception model. The expert experience can be the expert experience related to model parameter calibration in the semantic recognition scene and the depth recognition scene. The initial environment perception model is a model that has not been trained by a large amount of sample data in the environment perception scene, and the target environment perception model is a model obtained by training the initial environment perception model by a large amount of sample data in the environment perception scene.

[0049] In some embodiments, step S103 is performed, which can specifically be: obtaining an initial environment perception model; obtaining sample semantic recognition result labels and sample depth labels for each training environment image according to the training semantic recognition result and the depth map label; inputting the training environment image into the initial environment perception model for at least one round of model training to obtain a predicted semantic recognition result and a predicted depth; determining a model loss value based on the predicted semantic recognition result, the predicted depth, and the sample semantic recognition result label and the sample depth label, and adjusting the model parameters of the initial environment perception model based on the model loss value until a model training end condition is met to obtain a target environment perception model.

[0050] The initial environment perception model can be understood as an initial environment perception model for semantic recognition and depth recognition scenarios based on a machine learning model. The machine learning model can include, but is not limited to, one or more of a convolutional neural network (CNN) model, a deep neural network (DNN) model, a recurrent neural network (RNN), an embedding model, a gradient boosting decision tree (GBDT) model, a logistic regression (LR) model, and the like.

[0051] The model loss function can be a regression loss function, such as a mean square error loss, a mean absolute error loss, a quantile loss, or the like.

[0052] The model training end condition can be a training end condition determined based on the model loss value, or a training end condition determined based on the test result of the model. The training end condition determined based on the model loss value can be that the loss value of the model reaches a certain value and no longer decreases, at which point the model is considered to have reached the model training end condition. The training end condition determined based on the test result of the model can be that the test result of the model is basically similar after testing the model of several rounds of iterations using the data of the test set, at which point the model is considered to have reached the model training end condition.

[0053] S104, the target environment perception model is sent to the drivable device, so that the drivable device collects an environment perspective image and inputs the environment perspective image into the target environment perception model for environment perception, and outputs a target depth map and a target semantic recognition result corresponding to the environment perspective image.

[0054] Before describing this step, please refer to Figure 2A scene diagram of an environment perception method is shown in the embodiments of the present application. As shown in Figure 2 As shown, the environment perception method includes an electronic device and a drivable device. The electronic device is configured to train an environment perception model, and the drivable device is configured to use the environment perception model to perform environment perception. When the drivable device performs environment perception, a surround-view fisheye camera installed on the drivable device collects environment images at different viewing angles, and then uses the environment perception model to perform environment perception on the environment images to obtain semantic recognition results and depth information corresponding to the environment images. For the drivable device, the semantic recognition results and the depth information obtained can be used for subsequent driving route planning control and other processing.

[0055] In the embodiments of the present application, after the electronic device trains a target environment perception model, the electronic device can perform format conversion and model structure adjustment on the target environment perception model, and then send the adjusted target environment perception model to the drivable device. The format conversion of the target environment perception model can be understood as converting the target environment perception model into a corresponding format according to the computing power chip equipped in the drivable device. The structure adjustment of the target environment perception model can be understood as performing channel, structure pruning, and quantization processing on the target environment perception model, so that the adjusted model can adapt to the computing power chip of the drivable device. Therefore, by adjusting the target environment perception model, the adjusted model can be applied to the computing power chip of the drivable device, so that the drivable device does not need to be equipped with a computing power chip with strong computing power, and the target environment perception model using pure vision can still achieve good semantic perception ability and spatial perception ability.

[0056] For the drivable device, after collecting environment view images at different viewing angles through the surround-view fisheye camera, the drivable device can input the environment view images into the target environment perception model to perform environment perception, and output a target depth map and a target training semantic recognition result corresponding to the environment view images. The target depth map contains the depth value of each object in the environment view images. The target semantic recognition result contains the semantic information of each object in the environment view images. The semantic information can be the name of the object, or the category of the object.

[0057] In the embodiment of the present application, the training environment image collected by the surround-view fisheye camera is obtained, semantic recognition processing is performed on the training environment image to obtain the training semantic recognition result corresponding to the training environment image, the point cloud data in the environment where the training environment image is collected by the radar is obtained, image projection processing is performed on the point cloud data to obtain the depth map corresponding to the training environment image, at least one round of model training is performed on the initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain the target environment perception model, and the target environment perception model is sent to the drivable device, so that the drivable device inputs the environment perspective image into the target environment perception model for environment perception after collecting the environment perspective image, and outputs the target depth map and the target semantic recognition result corresponding to the environment perspective image. By adopting the technical solution, the point cloud data collected by the radar is introduced as auxiliary supervision of the depth estimation branch to train the environment perception model, so that the environment perception model can learn not only the semantics of the training environment image but also the depth corresponding to the training environment image in the training process, so that the target environment perception model finally trained can accurately identify the semantics of the environment image and accurately estimate the depth information corresponding to the environment image, and thus the effect of strengthening the near-field semantic understanding ability and spatial perception ability of the pure-vision environment perception model is achieved.

[0058] Please refer to Figure 3 The flowchart of an environment perception method provided by the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method of the embodiment of the present application can include the following steps: Figure 3

[0059] S301, a training environment image collected by a surround-view fisheye camera is obtained, and semantic recognition processing is performed on the training environment image to obtain a training semantic recognition result corresponding to the training environment image.

[0060] The description of the surround-view fisheye camera, the training environment image and the training semantic recognition result can be referred to the description of S101 in the embodiment shown in FIG. 1, which will not be repeated here. Figure 1

[0061] In the embodiment of the present application, the training environment image collected by the surround-view fisheye camera can be obtained from the drivable device, and the training environment image is obtained by the drivable device carrying the surround-view fisheye camera from different perspectives. Specifically, the training environment image can be obtained from the drivable device by a removable storage device or by a wireless communication transmission method. The wireless communication transmission method can include wireless local area network, Bluetooth, mobile communication and the like.

[0062] ​​The step of performing semantic recognition processing on the training environment images to obtain training semantic recognition results corresponding to the training environment images can be: inputting one or more training environment images into a semantic segmentation model to output a training semantic recognition result corresponding to each training environment image. The semantic segmentation model is a model trained in advance and having image semantic recognition capability. In a possible implementation, the training step of the semantic segmentation model can include: creating a model, specifically, creating an initial semantic segmentation model for an image semantic recognition scenario based on a machine learning model; obtaining sample data, specifically, obtaining a large number of sample environment images collected by the surround-view fisheye camera; labeling the sample data, specifically, introducing expert-side services to manually label sample labels corresponding to the sample data based on the requirements of the image semantic segmentation scenario, the sample labels including semantic recognition labels for each sample data, and each sample data semantic recognition label can include the name or category of each object present in the sample environment image; training the model, specifically, inputting the sample data into the initial semantic segmentation model for at least one round of model training to obtain a predicted semantic recognition result, determining a model loss value based on the predicted semantic recognition result and the semantic recognition label using a model loss function, and adjusting the model parameters of the initial semantic segmentation model based on the model loss value until a training end condition is met to obtain the semantic segmentation model. Optionally, the model end training condition of the model can include, for example, that the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, and the like. The specific model end training condition can be determined based on actual conditions, which is not limited here.

[0063] In S302, point cloud data collected by a radar in a target environment is obtained, and the target environment is an image collection environment of the training environment images.

[0064] In the embodiments of the present application, the collected point cloud data can be at least one frame of point cloud data. For each frame of point cloud data, there is a training environment image in different perspectives with the same collection time. That is, at the collection time of each frame of point cloud data, there is a training environment image in different perspectives collected at the collection time.

[0065] The implementation of step S302 can be: obtaining the point cloud data from the movable device, which is obtained by the movable device equipped with the surround-view fisheye camera and the radar while collecting the training environment images in different perspectives in the target environment by the surround-view fisheye camera. Specifically, obtaining the point cloud data from the movable device can be obtaining the point cloud data from the movable device through a movable storage device, or obtaining the point cloud data from the movable device through a wireless communication transmission mode. The wireless communication transmission mode can include wireless local area network, Bluetooth, mobile communication, etc.

[0066] S303, obtain the previous frame point cloud data of the current frame point cloud data, and perform point cloud superposition processing on the current frame point cloud data based on the previous frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data.

[0067] As can be easily understood, since the point cloud data includes at least one frame of point cloud data, each frame of point cloud data can be referred to as current frame point cloud data. The previous frame point cloud data refers to the point cloud data of the previous frame of the current frame point cloud data. The target frame point cloud data refers to the point cloud data obtained by superimposing the previous frame point cloud data of the current frame point cloud data on the current frame point cloud data. For the first frame point cloud data, there is no previous frame point cloud data, and the target frame point cloud data corresponding to the first frame point cloud data is the first frame point cloud data.

[0068] In some embodiments, performing step S303 can include the following steps: A1, obtaining a radar extrinsic matrix of the radar, and performing world coordinate system conversion processing on the previous frame point cloud data based on the radar extrinsic matrix to obtain first point cloud data corresponding to the previous frame point cloud data; A2, performing radar coordinate system conversion processing on the first point cloud data based on the radar extrinsic matrix to obtain second point cloud data corresponding to the previous frame point cloud data; and A3, performing point cloud merging processing on the second point cloud data and the current frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data.

[0069] The radar extrinsic matrix refers to an extrinsic matrix including a rotation matrix and a translation vector. Assuming that the rotation matrix is represented by R and the translation vector is represented by t, the radar extrinsic matrix can be represented as Specifically, the radar extrinsic matrix can include, but is not limited to, an extrinsic matrix for converting from the radar coordinate system to the world coordinate system and an extrinsic matrix for converting from the world coordinate system to the radar coordinate system. For the extrinsic matrices for conversion between different coordinate systems, the values of R and t in the extrinsic matrices are different.

[0070] The first point cloud data refers to the point cloud data of the previous frame point cloud data in the world coordinate system after the previous frame point cloud data is converted from the radar coordinate system at the previous frame time to the world coordinate system.

[0071] The second point cloud data refers to the point cloud data of the previous frame point cloud data in the radar coordinate system at the current frame time after the first point cloud data is converted from the world coordinate system to the radar coordinate system at the current frame time.

[0072] In the execution of step A1, specifically, the radar extrinsic matrix of the radar can be obtained, which can include an extrinsic matrix converted from the radar coordinate system to the world coordinate system, and an extrinsic matrix converted from the world coordinate system to the radar coordinate system. Further, the first coordinate calculation formula is used to calculate the world coordinate conversion of the previous frame of point cloud data based on the radar extrinsic matrix, to obtain the first point cloud data corresponding to the previous frame of point cloud data. The first coordinate calculation formula satisfies the following formula:

[0073]

[0074] wherein, represents the first point cloud data corresponding to the previous frame of point cloud data, represents the extrinsic matrix converted from the radar coordinate system at the previous frame time to the world coordinate system, represents the previous frame of point cloud data, and x represents the matrix multiplication operation.

[0075] In the execution of step A2, specifically, the second coordinate calculation formula is used to calculate the radar coordinate conversion of the first point cloud data based on the radar extrinsic matrix, to obtain the second point cloud data corresponding to the previous frame of point cloud data. The second coordinate calculation formula satisfies the following formula:

[0076]

[0077] wherein, represents the second point cloud data corresponding to the previous frame of point cloud data, represents the extrinsic matrix converted from the world coordinate system to the radar coordinate system at the current frame time, represents the first point cloud data corresponding to the previous frame of point cloud data, and x represents the matrix multiplication operation.

[0078] In the execution of step A3, specifically, the target frame point cloud data corresponding to the current frame of point cloud data can be obtained by using the merging calculation formula to perform point cloud merging calculation on the second point cloud data and the current frame of point cloud data. The merging calculation formula satisfies the following formula:

[0079]

[0080] wherein, represents the target frame point cloud data corresponding to the current frame of point cloud data, represents the second point cloud data corresponding to the previous frame of point cloud data, represents the current frame of point cloud data, and concat() represents a merging function for merging the second point cloud data and the current frame of point cloud data.

[0081] The sparse point cloud data can be changed into dense point cloud data by means of point cloud superposition, and the accuracy of subsequent point cloud processing is ensured.

[0082] In S304, the distance from the radar to the ground is obtained, and the ground point cloud data corresponding to the current frame training environment image is randomly generated based on the distance.

[0083] The current frame time refers to the collection time of the current frame point cloud data.

[0084] The current frame training environment image refers to the training environment image collected at the collection time of the current frame point cloud data.

[0085] In some embodiments, for the device on which the radar is installed, the distance from the radar to the ground can be easily measured, a fixed value can be obtained by taking the negative value of the distance from the radar to the ground, the fixed value is taken as the z-axis coordinate value of the ground point, a large number of coordinate value combinations of x-axis coordinate values and y-axis coordinate values are randomly generated, each coordinate value combination is combined with the z-axis coordinate value, a large number of three-dimensional coordinates can be obtained, each three-dimensional coordinate can be taken as the three-dimensional coordinate of a ground point, and the set of these ground points is the ground point cloud data.

[0086] The embodiments of the present application can complete the ground point cloud data, and achieve the effect of densifying the point cloud data.

[0087] In S305, the target frame point cloud data and the ground point cloud data are subjected to image projection processing to obtain the depth map corresponding to the current frame training environment image.

[0088] In some embodiments, step S305 can be specifically implemented as follows: B1, obtaining the radar extrinsic matrix of the radar; B2, performing image projection processing on the target frame point cloud data based on the radar extrinsic matrix to obtain the first pixel coordinates corresponding to the target frame point cloud data; B3, performing image projection processing on the ground point cloud data based on the radar extrinsic matrix to obtain the second pixel coordinates corresponding to the ground point cloud data; and B4, obtaining the first depth value corresponding to the first pixel coordinates and the second depth value corresponding to the second pixel coordinates, and determining the depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates and the second depth value.

[0089] The radar extrinsic matrix includes but is not limited to the extrinsic matrix for converting from the radar coordinate system to the world coordinate system, the extrinsic matrix for converting from the world coordinate system to the radar coordinate system, and the extrinsic matrix for converting from the radar coordinate system to the camera coordinate system. The rotation matrix and the translation vector in the extrinsic matrix for conversion between different coordinate systems have different values.

[0090] The first pixel coordinates refer to the pixel coordinates of the pixel points in the image plane when the spatial points in the target frame point cloud data are converted into the pixel points.

[0091] The second pixel coordinate refers to a pixel coordinate of a pixel obtained by converting a ground point in the ground point cloud data into a pixel in an image plane.

[0092] In the execution of step B2, specifically, a first projection formula can be used to perform image projection calculation on the target frame point cloud data based on the radar extrinsic matrix, to obtain a first pixel coordinate corresponding to the target frame point cloud data. The first projection formula satisfies the following formula:

[0093]

[0094] In the formula, (u, v) represents the first pixel coordinate corresponding to the target frame point cloud data, represents an extrinsic matrix for converting from the radar coordinate system to the camera coordinate system, and (x, y, z) represents the target frame point cloud data. l represents a target frame point cloud data, x represents a matrix multiplication operation, and proj() represents a projection conversion function.

[0095] Optionally, in the projection calculation process, the target frame point cloud data can also be filtered by point cloud inflation to obtain accurate point cloud data.

[0096] In the execution of step B3, specifically, a second projection formula can be used to perform image projection calculation on the ground point cloud data based on the radar extrinsic matrix, to obtain a second pixel coordinate corresponding to the ground point cloud data. The second pixel coordinate satisfies the following formula:

[0097]

[0098] In the formula, (u, v) represents the second pixel coordinate corresponding to the ground point cloud data, represents an extrinsic matrix for converting from the radar coordinate system to the camera coordinate system, and (x, y, z) represents the ground point cloud data.

[0099] In the execution of step B4, a first depth value corresponding to the first pixel coordinate and a second depth value corresponding to the second pixel coordinate are obtained. Since the point cloud data can be obtained by the radar, the radar itself can measure the distance from a point in space to the radar, that is, the radar can measure the depth value of the point in space, so the depth value of the space point represented by the first pixel coordinate and the depth value of the space point represented by the second pixel coordinate are easily obtained. The first depth value is the depth value of the space point represented by the first pixel coordinate, and the second depth value is the depth value of the space point represented by the second pixel coordinate.

[0100] Further, the first pixel coordinate, the first depth value, the second pixel coordinate and the second depth value are used to determine a depth map corresponding to the current frame training environment image. In one implementation, the third pixel coordinate and the fourth pixel coordinate corresponding to the point cloud data collected at the collection time of the current frame training environment image are determined from the first pixel coordinate and the second pixel coordinate, the third depth value corresponding to the third pixel coordinate is filled at the third pixel coordinate, and the fourth depth value corresponding to the fourth pixel coordinate is filled at the fourth pixel coordinate, so as to obtain the depth map corresponding to the current frame training environment image.

[0101] Further, the first pixel coordinate, the first depth value, the second pixel coordinate and the second depth value are used to determine a depth map corresponding to the current frame training environment image. In one implementation, the third pixel coordinate and the fourth pixel coordinate corresponding to the point cloud data collected at the collection time of the current frame training environment image are determined from the first pixel coordinate and the second pixel coordinate, the third depth value corresponding to the third pixel coordinate is filled at the third pixel coordinate, and the fourth depth value corresponding to the fourth pixel coordinate is filled at the fourth pixel coordinate, so as to obtain the depth map corresponding to the current frame training environment image.

[0102] In this way, the reference depth map is filtered by using the distance median filtering method, and a more accurate depth map can be obtained. This is because the distance median filtering method can filter out the penetrating pixel points in the depth map. The penetrating pixel points are obtained based on the projection of the point cloud collected by the radar. Since there is a visual difference between the radar and the camera, there are penetrating point clouds in the point cloud data collected by the radar. The penetrating point clouds should not exist in the depth map, and therefore the data obtained by projecting the penetrating point clouds needs to be filtered out.

[0103] The embodiment of the present application determines the depth map by using the superimposed point cloud data and the completed ground point cloud data, and ensures the accuracy of the depth map.

[0104] S306, the training semantic recognition result is encoded to obtain a training semantic recognition code corresponding to the training environment image.

[0105] In some embodiments, the semantics of each object in the training environment image can be determined according to the training semantic recognition result, the semantics can refer to the name or category of the object, and each object semantics is encoded to obtain a training semantic recognition code corresponding to the training environment image. Specifically, the semantics of each object can be encoded by using an encoding formula to obtain the training semantic recognition code of each object in the training environment image. The encoding formula can satisfy the following formula:

[0106] ∑encode_pixel i,j =seg*1000+id

[0107] wherein, ∑encode_pixel i,j represents each object, seg represents the semantics of each object, and id represents the number of each object, which refers to the object number of the object in the category to which the object belongs. For example, the training environment image includes 10 cars, and id represents the number of each car in the cars, and the value of id can be 1-10.

[0108] The embodiments of the present application encode the semantic recognition result of the training environment image, which facilitates subsequent calculation of quotient and remainder values according to the output data of the model to quickly obtain the semantic result of the model output when training the model.

[0109] S307, based on the training environment image, the training semantic recognition code, and the depth map, at least one round of model training is performed on the initial environment perception model to obtain a target environment perception model.

[0110] In some embodiments, step S307 is performed, specifically, an initial environment perception model can be created, which can be an initial environment perception model for semantic recognition and depth recognition scenes based on a machine learning model; the training environment image is used as sample data; sample labels of the sample data are labeled, which can be, for each object in the training environment image, the corresponding sample label is labeled according to the training semantic recognition code and the depth map, the sample label can include the semantics (name or category) and the depth value of each object; the initial environment perception model is trained, which can be, the sample data (training environment image) is input into the initial environment perception model for at least one round of model training to obtain a prediction result, the prediction result includes a predicted semantic recognition result and a predicted depth value corresponding to the training environment image, the predicted semantic recognition result and the predicted depth value are the predicted semantic result and the predicted depth value of each object in the training environment image, a model loss value is determined by using a model loss function based on the prediction result and the sample label, the model parameters of the initial environment perception model are adjusted based on the model loss value, until a training end condition is met, and a target environment perception model is obtained.

[0111] Optionally, the model training end condition can be a training end condition determined according to a model loss value, and can also be a training end condition determined according to a test result of the model. The training end condition determined according to the model loss value can be that the loss value of the model reaches a certain value and no longer decreases, at which time the model can be considered to reach the model training end condition. The training end condition determined according to the test result of the model can be that the model is tested by using the data of the test set for several rounds of iterations, and the test results obtained are basically similar, at which time the model can be considered to reach the model training end condition.

[0112] Optionally, the model loss function can be a regression loss function, such as a mean square error loss, a mean absolute error loss, a quantile loss, or the like.

[0113] Optionally, the implementation of step S307 can also be: performing at least one round of model training on the initial environment perception model based on the training environment image and the training semantic recognition code to obtain a reference environment perception model, and performing at least one round of model training on the reference environment perception model based on the training environment image, the training semantic recognition result, and the depth map to obtain the target environment perception model. The method of training the reference environment perception model and the method of training the target environment perception model can refer to the training process of the above-mentioned implementation, which will not be described here.

[0114] That is, in the process of training the initial environment perception model to obtain the target environment perception model, the embodiment of the present application can adopt a training method of grouping components, that is, first training the semantic perception capability of the model, and then training the depth perception capability of the model. Such a training method of grouping components enables the model to first learn the simple semantic recognition capability, and then learn the comprehensive recognition capability, that is, to comprehensively learn the semantic recognition capability and the depth recognition capability, which can improve the model training efficiency.

[0115] S308, sending the target environment perception model to the drivable device, so that the drivable device collects an environment perspective image and inputs the environment perspective image into the target environment perception model for environment perception, and outputs a target depth map corresponding to the environment perspective image and a target semantic recognition result.

[0116] In the embodiments of the present application, after the electronic device trains the target environment perception model, the electronic device can perform format conversion and model structure adjustment on the target environment perception model, so as to send the adjusted target environment perception model to the drivable device. The format conversion of the target environment perception model can be understood as converting the target environment perception model into an environment perception model of a corresponding format according to the computing power chip equipped by the drivable device. The structure adjustment of the target environment perception model can be understood as performing channel, structure pruning and quantization processing on the target environment perception model, so that the adjusted model can adapt to the computing power chip of the drivable device. Therefore, by adjusting the target environment perception model, the adjusted model can be applied to the computing power chip of the drivable device, so that the drivable device does not need to be equipped with a computing power chip with strong computing power, and the target environment perception model using pure vision can still achieve good semantic perception ability and spatial perception ability.

[0117] For the drivable device, after collecting environment perspective images of different perspectives through the surround-view fisheye camera, the environment perspective images can be input into the target environment perception model for environment perception identification, and the target depth map corresponding to the environment perspective images and the target training semantic recognition result are output. The target depth map contains the depth value of each object in the environment perspective image. The target semantic recognition result contains the semantic information of each object in the environment perspective image, which can be the name of the object or the category of the object.

[0118] In the embodiment of the present application, a training environment image collected by the surround-view fisheye camera is obtained, semantic recognition processing is performed on the training environment image to obtain a training semantic recognition result corresponding to the training environment image, so as to obtain sample data and a semantic label of the sample data. Point cloud data in an environment where the training environment image is located is obtained by the radar, the current frame point cloud data is subjected to point cloud superposition processing according to previous frame point cloud data of the current frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data, the distance from the radar to the ground is obtained, ground point cloud data corresponding to the current frame training environment image is randomly generated based on the distance, and the target frame point cloud data and the ground point cloud data are subjected to image projection processing to obtain a depth map corresponding to the current frame training environment image. In the embodiment of the present application, the point cloud is superimposed and the ground point cloud is completed to obtain dense point cloud data, and then the dense point cloud data can be used to obtain an accurate depth map, so that a depth label of the sample data is obtained according to the depth map. In addition, the training semantic recognition result is encoded in the embodiment of the present application, so that the subsequent data processing process is simplified by using the encoding. The environment perception model is trained by the semantic encoding, the accurate depth label and the sample data, so that the trained environment perception model has rich perception ability, that is, the semantic perception ability and the depth perception ability. Finally, the trained target environment perception model is sent to the movable device, so that the movable device also has rich perception ability, that is, the movable device has the semantic perception ability and the depth perception ability through the pure visual environment perception model.

[0119] The following will be combined with Figure 4 The environment perception device provided by the embodiment of the present application will be described in detail. It should be noted that Figure 4 The environment perception device shown in the embodiment of the present application is used to execute the method of the embodiment of the present application Figure 1 and Figure 3 The method shown in the embodiment of the present application is only used to show the part related to the embodiment of the present application, and the specific technical details are not disclosed, please refer to the embodiment shown in the present application Figure 1 and Figure 3 The embodiment.

[0120] Please refer to Figure 4 , which shows the structure schematic diagram of the environment perception device of the embodiment of the present application. The environment perception device 1 can be realized by software, hardware or combination of the two to become all or part of the device. According to some embodiments, the environment perception device 1 includes a semantic acquisition module 11, a depth acquisition module 12, a model training module 13 and a model processing module 14, which are specifically used for:

[0121] The semantic acquisition module 11 is configured to acquire a training environment image collected by the surround-view fisheye camera, and perform semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image.

[0122] The depth acquisition module 12 is configured to acquire point cloud data collected by the radar in a target environment, and perform image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image collection environment of the training environment image.

[0123] The model training module 13 is configured to perform at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model.

[0124] The model processing module 14 is configured to send the target environment perception model to the drivable device, so that the drivable device collects an environment perspective image and inputs the environment perspective image into the target environment perception model for environment perception, and outputs a target depth map and a target semantic recognition result corresponding to the environment perspective image.

[0125] Optionally, the model training module 13 comprises:

[0126] The data encoding module is configured to perform encoding processing on the training semantic recognition result to obtain a training semantic recognition code corresponding to the training environment image.

[0127] The data training module is configured to perform at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition code and the depth map to obtain the target environment perception model.

[0128] Optionally, the data training module is specifically configured to:

[0129] perform at least one round of model training on the initial environment perception model based on the training environment image and the training semantic recognition code to obtain a reference environment perception model;

[0130] perform at least one round of model training on the reference environment perception model based on the training environment image, the training semantic recognition code and the depth map to obtain the target environment perception model.

[0131] Optionally, please refer to Figure 5 The depth acquisition module 12 comprises a point cloud superposition unit 121, a point cloud completion unit 122 and a depth acquisition unit 123, and is specifically configured to:

[0132] The point cloud superposition unit 121 is configured to acquire previous frame point cloud data of current frame point cloud data, and perform point cloud superposition processing on the current frame point cloud data based on the previous frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data.

[0133] The point cloud completion unit 122 is configured to acquire a distance from the radar to the ground, and generate ground point cloud data corresponding to the current frame training environment image based on the distance;

[0134] The depth acquisition unit 123 is configured to perform image projection processing on the target frame point cloud data and the ground point cloud data to obtain a depth map corresponding to the current frame training environment image.

[0135] Optionally, the point cloud superposition unit 121 is specifically configured to:

[0136] acquire a radar extrinsic matrix of the radar, and perform world coordinate system conversion processing on the previous frame point cloud data based on the radar extrinsic matrix to obtain first point cloud data corresponding to the previous frame point cloud data;

[0137] perform radar coordinate system conversion processing on the first point cloud data based on the radar extrinsic matrix to obtain second point cloud data corresponding to the previous frame point cloud data;

[0138] perform point cloud merging processing on the second point cloud data and the current frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data.

[0139] Optionally, the depth acquisition unit 123 includes:

[0140] a first acquisition unit configured to acquire a radar extrinsic matrix of the radar;

[0141] a second acquisition unit configured to perform image projection processing on the target frame point cloud data based on the radar extrinsic matrix to obtain first pixel coordinates corresponding to the target frame point cloud data;

[0142] a third acquisition unit configured to perform image projection processing on the ground point cloud data based on the radar extrinsic data to obtain second pixel coordinates corresponding to the ground point cloud data;

[0143] a fourth acquisition unit configured to acquire a first depth value corresponding to the first pixel coordinates and a second depth value corresponding to the second pixel coordinates, and determine a depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates, and the second depth value.

[0144] Optionally, the fourth acquisition unit is specifically configured to:

[0145] determine a reference depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates, and the second depth value;

[0146] perform filtering processing on the reference depth map based on a semantic recognition result corresponding to the current frame training image using a distance median filtering method to obtain a depth map corresponding to the current frame training environment image.

[0147] Please refer toFigure 6 A structural block diagram of an electronic device is provided for an embodiment of the present application. The electronic device in the embodiment of the present application can include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 can be connected through the bus 150.

[0148] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the entire electronic device by using various interfaces and lines, performs various functions of the electronic device 100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 110 can integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but can be implemented by a separate communication chip.

[0149] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). Alternatively, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 can be used to store instructions, programs, codes, code sets, or instruction sets.

[0150] The input device 130 is configured to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is configured to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In the embodiments of the present application, the input device 130 can be a temperature sensor configured to obtain the operating temperature of the electronic device. The output device 140 can be a speaker configured to output an audio signal.

[0151] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-described drawings does not constitute a limitation on the terminal, and the electronic device can include more or fewer components than those shown in the drawings, or combine certain components, or different component arrangements. For example, the electronic device further includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WIFI) module, a power supply, a Bluetooth module, and the like, which are not described herein.

[0152] In the embodiments of the present application, the execution subject of each step can be the electronic device introduced above. Alternatively, the execution subject of each step is an operating system of the electronic device. The operating system can be an Android system, an IOS system, or other operating systems, which are not limited in the embodiments of the present application.

[0153] In the electronic device, Figure 6 In the electronic device, the processor 110 can be configured to call a program of the environment perception method stored in the memory 120, and the processor 110 can be configured to call the program of the environment perception method stored in the memory 120 and specifically perform the following operations:

[0154] Obtain a training environment image collected by a surround-view fisheye camera, and perform semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image;

[0155] Obtain point cloud data collected by a radar in a target environment, and perform image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image collection environment of the training environment image;

[0156] Perform at least one round of model training on an initial environment perception model based on the training environment image, the training semantic recognition result, and the depth map to obtain a target environment perception model;

[0157] Send the target environment perception model to a drivable device, so that the drivable device collects an environment perspective image and inputs the environment perspective image into the target environment perception model for environment perception, and outputs a target depth map and a target semantic recognition result corresponding to the environment perspective image.

[0158] In an embodiment, the processor 110, in performing the step of performing at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition result, and the depth map to obtain the target environment perception model, specifically performs the following operations:

[0159] encoding the training semantic recognition result to obtain a training semantic recognition code corresponding to the training environment image;

[0160] performing at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition code, and the depth map to obtain the target environment perception model.

[0161] In an embodiment, the processor 110, in performing the step of performing at least one round of model training on the initial environment perception model based on the training environment image, the training semantic recognition code, and the depth map to obtain the target environment perception model, specifically performs the following operations:

[0162] performing at least one round of model training on the initial environment perception model based on the training environment image and the training semantic recognition code to obtain a reference environment perception model;

[0163] performing at least one round of model training on the reference environment perception model based on the training environment image, the training semantic recognition code, and the depth map to obtain the target environment perception model.

[0164] In an embodiment, the processor 110, in performing the step of performing image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, specifically performs the following operations:

[0165] obtaining previous frame point cloud data of the current frame point cloud data, and performing point cloud superimposition processing on the current frame point cloud data based on the previous frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data;

[0166] obtaining a distance from the radar to the ground, and randomly generating ground point cloud data corresponding to the current frame training environment image based on the distance;

[0167] performing image projection processing on the target frame point cloud data and the ground point cloud data to obtain a depth map corresponding to the current frame training environment image.

[0168] In an embodiment, the processor 110, in performing the step of performing point cloud superimposition processing on the current frame point cloud data based on the previous frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data, specifically performs the following operations:

[0169] obtaining a radar extrinsic matrix of the radar, and performing world coordinate system conversion processing on the previous frame point cloud data based on the radar extrinsic matrix to obtain first point cloud data corresponding to the previous frame point cloud data;

[0170] The first point cloud data is subjected to radar coordinate system conversion processing based on the radar extrinsic parameter matrix to obtain second point cloud data corresponding to the previous frame of point cloud data.

[0171] The second point cloud data and the current frame of point cloud data are subjected to point cloud merging processing to obtain target frame point cloud data corresponding to the current frame of point cloud data.

[0172] In one embodiment, when the processor 110 performs the step of obtaining a depth map corresponding to the current frame of training environment image by performing image projection processing on the target frame point cloud data and the ground point cloud data, the processor 110 specifically performs the following operations:

[0173] obtaining a radar extrinsic parameter matrix of the radar;

[0174] performing image projection processing on the target frame point cloud data based on the radar extrinsic parameter matrix to obtain first pixel coordinates corresponding to the target frame point cloud data;

[0175] performing image projection processing on the ground point cloud data based on the radar extrinsic parameter data to obtain second pixel coordinates corresponding to the ground point cloud data;

[0176] obtaining a first depth value corresponding to the first pixel coordinates and a second depth value corresponding to the second pixel coordinates, and determining a depth map corresponding to the current frame of training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates, and the second depth value.

[0177] In one embodiment, when the processor 110 performs the step of determining a depth map corresponding to the current frame of training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates, and the second depth value, the processor 110 specifically performs the following operations:

[0178] determining a reference depth map corresponding to the current frame of training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates, and the second depth value;

[0179] performing filtering processing on the reference depth map based on the semantic recognition result corresponding to the current frame of training image by using a distance median filtering method to obtain a depth map corresponding to the current frame of training environment image.

[0180] The embodiments of the present application also provide a computer readable storage medium, which stores at least one instruction, and the at least one instruction is used for being executed by a processor to implement the environment perception method of the above embodiments.

[0181] The embodiments of the present application also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the environment perception method of the above embodiments.

[0182] Those skilled in the art should be aware that, in the above one or more examples, the functions described in the embodiments of the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on the computer readable medium. The computer readable medium includes computer storage medium and communication medium, and the communication medium includes any medium that facilitates transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

[0183] The above merely provides the optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An environmental perception method, characterized in that, The method comprises: acquiring a training environment image collected by a surround-view fisheye camera, performing semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image; acquiring point cloud data collected by a radar in a target environment, performing image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image collection environment of the training environment image; based on the training environment image, the training semantic recognition result, and the depth map, performing at least one round of model training on an initial environment perception model to obtain a target environment perception model; sending the target environment perception model to a drivable device, so that the drivable device collects an environmental perspective image and inputs the environmental perspective image into the target environment perception model for environment perception identification, and outputs a target depth map and a target semantic recognition result corresponding to the environmental perspective image.

2. The method of claim 1, wherein, The method comprises: performing encoding processing on the training semantic recognition result to obtain a training semantic recognition code corresponding to the training environment image; based on the training environment image, the training semantic recognition code, and the depth map, performing at least one round of model training on an initial environment perception model to obtain a target environment perception model.

3. The method of claim 2, wherein, The method comprises: based on the training environment image and the training semantic recognition code, performing at least one round of model training on an initial environment perception model to obtain a reference environment perception model; based on the training environment image, the training semantic recognition code, and the depth map, performing at least one round of model training on the reference environment perception model to obtain a target environment perception model.

4. The method of claim 1, wherein, The method comprises: acquiring previous frame point cloud data of current frame point cloud data, and performing point cloud superposition processing on the current frame point cloud data based on the previous frame point cloud data to obtain target frame point cloud data corresponding to the current frame point cloud data; acquiring a distance from the radar to the ground, and randomly generating ground point cloud data corresponding to the current frame training environment image based on the distance; performing image projection processing on the target frame point cloud data and the ground point cloud data to obtain a depth map corresponding to the current frame training environment image.

5. The method of claim 4, wherein, The method comprises: acquiring a radar extrinsic matrix of the radar, and performing world coordinate system conversion processing on the previous frame point cloud data based on the radar extrinsic matrix to obtain first point cloud data corresponding to the previous frame point cloud data; performing radar coordinate system conversion processing on the first point cloud data based on the radar extrinsic matrix to obtain second point cloud data corresponding to the previous frame point cloud data; The second point cloud data and the current frame point cloud data are subjected to point cloud merging processing to obtain target frame point cloud data corresponding to the current frame point cloud data.

6. The method of claim 4, wherein, The image projection processing on the target frame point cloud data and the ground point cloud data to obtain a depth map corresponding to the current frame training environment image comprises: obtaining a radar extrinsic matrix of the radar; performing image projection processing on the target frame point cloud data based on the radar extrinsic matrix to obtain first pixel coordinates corresponding to the target frame point cloud data; performing image projection processing on the ground point cloud data based on the radar extrinsic data to obtain second pixel coordinates corresponding to the ground point cloud data; obtaining a first depth value corresponding to the first pixel coordinates and a second depth value corresponding to the second pixel coordinates, and determining a depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates and the second depth value.

7. The method of claim 6, wherein, The determination of the depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates and the second depth value comprises: determining a reference depth map corresponding to the current frame training environment image based on the first pixel coordinates, the first depth value, the second pixel coordinates and the second depth value; performing filtering processing on the reference depth map based on the semantic recognition result corresponding to the current frame training image by using a distance median filtering method to obtain a depth map corresponding to the current frame training environment image.

8. An environmental perception apparatus, characterized by, The device comprises: a semantic acquisition module configured to acquire a training environment image collected by a surround-view fisheye camera, and perform semantic recognition processing on the training environment image to obtain a training semantic recognition result corresponding to the training environment image; a depth acquisition module configured to acquire point cloud data collected by a radar in a target environment, and perform image projection processing on the point cloud data to obtain a depth map corresponding to the training environment image, the target environment being an image collection environment of the training environment image; a model training module configured to perform at least one round of model training on an initial environment perception model based on the training environment image, the training semantic recognition result and the depth map to obtain a target environment perception model; a model processing module configured to send the target environment perception model to a drivable device, so that the drivable device collects an environmental perspective image and inputs the environmental perspective image into the target environment perception model for environment perception identification, and outputs a target depth map and a target semantic recognition result corresponding to the environmental perspective image.

9. A computer storage medium, characterized in that The computer storage medium stores instructions adapted to be loaded and executed by the processor to implement the method of any one of claims 1-7.

10. An electronic device, comprising: comprise: a processor and a memory; wherein the memory stores a computer program adapted to be loaded and executed by the processor to implement the method of any one of claims 1-7.

Images