Vehicle sensing method and device

Abstract

The invention provides a vehicle sensing method and device. The method comprises the following steps: acquiring a two-dimensional original image around a vehicle; in the current frame, obstacle tracking is carried out based on the obstacle information in the three-dimensional space of the historical frame, the tracked obstacle is removed from the current frame, and a two-dimensional target image of the current frame is obtained; identifying the two-dimensional target image by using a lane line identification model and an obstacle identification model, and determining lane line information and obstacle information in a three-dimensional space; determining a drivable area of the vehicle according to the lane line information; and generating vehicle environment information of a bird's-eye view according to the tracked obstacle, the obstacle information and the travelable area of the vehicle. Compared with the prior art, the method has the advantages that the obstacle information in the three-dimensional space can be determined from the two-dimensional image acquired by the image sensor on the premise that the precision and the speed are guaranteed, the target position sensing capability is improved, and more and more accurate decision data are provided for auxiliary driving.

Description

Technical Field

[0001] The present application relates to the field of vehicle perception technology, and in particular to a vehicle perception method and device. Background Art

[0002] At present, assisted driving hopes that vehicles can perceive through visual algorithms, judge the road conditions during driving, and make reasonable driving decisions like human drivers. Human drivers make driving decisions based on the human brain's perception of the distance of the target through the parallax information of the two eyes, the movement of the vehicle and the relative movement of the scenery. Therefore, the perception of the surrounding environment is the basis for realizing vehicle assisted driving. Cameras, or image sensors, are currently commonly used on-board sensors for sensing the surrounding environment. They are low-cost and can capture information on road signs (such as lane lines, traffic lights, road signs, etc.) to obtain a large amount of environmental information required for assisted driving. However, the two-dimensional images generated by the camera lack depth information and have low perception of the target's position, which is not conducive to assisted driving making accurate driving decisions. Summary of the invention

[0003] The embodiments of the present application provide a vehicle perception method, device, equipment and storage medium to solve the problems existing in the related technologies. The technical solutions are as follows:

[0004] In a first aspect, an embodiment of the present application provides a vehicle perception method, including:

[0005] Acquire a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround image of the vehicle's surroundings and includes multiple frames of images that are continuous in time sequence;

[0006] In the two-dimensional original image of the current frame, obstacle tracking is performed based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame; the two-dimensional target image of the current frame is respectively recognized by using a lane line recognition model and an obstacle recognition model to determine the lane line information and the obstacle information in the three-dimensional space;

[0007] Determine a drivable area of ​​the vehicle according to the lane line information;

[0008] Vehicle environment information from a bird's-eye view is generated according to the tracked obstacle, the obstacle information, and the drivable area of ​​the vehicle.

[0009] This application uses a model recognition method to obtain obstacle information and lane line information in three-dimensional space from two-dimensional image recognition, and determines the three-dimensional vehicle environment information from a bird's-eye view by combining obstacles and lane lines. In the bird's-eye view mode, the driver can see the vehicle's surroundings at a glance. Compared with the prior art, this application can directly determine obstacle information in three-dimensional space from two-dimensional images collected by image sensors, increase the ability to perceive the position of the target, and provide more and more accurate decision-making data for assisted driving by combining lane line information. By identifying obstacles and lane lines separately through different models, the recognition accuracy of both is improved.

[0010] Moreover, in the present application, based on the high-precision recognition results of the historical frame images by the obstacle recognition model (i.e., the three-dimensional obstacle information corresponding to the historical frame), the relevant obstacles are tracked in the two-dimensional original image of the current frame. Compared with the obstacle recognition model, the tracking improves the recognition speed of at least some obstacles in the current frame. Based on this tracking, the obstacle recognition model only needs to identify the image of the tracked obstacle removed from the two-dimensional original image of the current frame, i.e., the two-dimensional target image, which simplifies the image information and improves the recognition speed of the obstacle recognition model. This allows the present application to improve the recognition speed while ensuring the recognition accuracy.

[0011] In one implementation, the lane line information is lane line information in a two-dimensional space and the drivable area of ​​the vehicle is a drivable area of ​​the vehicle in a two-dimensional space;

[0012] The generating of the vehicle environment information from a bird's-eye view according to the tracked obstacle, the obstacle information and the drivable area of ​​the vehicle comprises:

[0013] Generating preliminary environmental information from a bird's-eye view according to the tracked obstacles and the obstacle information;

[0014] The drivable area of ​​the vehicle in the two-dimensional space is projected onto the ground in the bird's-eye view; the preliminary environmental information and the projection of the drivable area together form the vehicle environmental information in the bird's-eye view.

[0015] Unlike obstacle information, in some scenarios, such as urban road scenarios, the ground is basically flat and there are no large potholes. The difference in the height direction of the lane lines on it can be ignored. In this case, only two-dimensional recognition of the lane lines can be performed to reduce the workload of model recognition. From a bird's-eye view, the ground is used as the reference in the height direction, and the lane line information is projected on the ground.

[0016] In one embodiment,

[0017] When it is determined that the current driving surface of the vehicle is flat, a two-dimensional lane line recognition model is selected from a preset two-dimensional lane line recognition model and a three-dimensional lane line recognition model;

[0018] The using of the lane line recognition model and the obstacle recognition model to respectively recognize the two-dimensional target image and determine the lane line information and the obstacle information in the three-dimensional space includes:

[0019] The two-dimensional lane line recognition model and the obstacle recognition model are used to respectively recognize the two-dimensional target image to determine the lane line information in the two-dimensional space and the obstacle information in the three-dimensional space.

[0020] Based on this, the appropriate lane line recognition model can be selected according to the actual driving scenario to balance the relationship between recognition speed and accuracy.

[0021] In one embodiment,

[0022] The vehicle perception method further includes:

[0023] Pre-recognize the two-dimensional original image of the current frame, and generate a two-dimensional pre-recognition image according to the pre-recognition result; the pre-recognition result includes an obstacle recognition result and / or a lane line recognition result;

[0024] In the two-dimensional original image of the current frame, obstacle tracking is performed based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame, including:

[0025] In the two-dimensional pre-recognition image of the current frame, obstacle tracking is performed based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is eliminated from the two-dimensional pre-recognition image of the current frame to obtain the two-dimensional target image of the current frame.

[0026] In a vehicle driving environment, it is necessary to quickly perceive and identify the surrounding environment. Once there is a long delay, the perception algorithm is meaningless and may even lead to dangerous driving decisions. The three-dimensional recognition algorithm for two-dimensional images is complex, computationally intensive, and time-consuming. For this reason, the present application first performs a two-dimensional pre-recognition of the two-dimensional target image with a relatively small amount of computation, and can quickly identify a two-dimensional pre-recognition image with approximate obstacle information and / or approximate lane line information. Subsequent obstacle tracking and obstacle recognition models can be identified based on the two-dimensional pre-recognition image. Since the two-dimensional pre-recognition image reduces a lot of background information compared to the two-dimensional original image, the computational complexity of the subsequent model is correspondingly reduced, thereby improving the overall recognition speed.

[0027] In one embodiment,

[0028] The pre-recognition of the two-dimensional original image and the generation of a two-dimensional pre-recognition image according to the pre-recognition result include:

[0029] Preprocessing the two-dimensional original image; the preprocessing includes at least one of mean filtering, grayscale processing, and color level balancing;

[0030] The pre-processed two-dimensional original image is subjected to edge detection to determine the obstacle area and / or lane line area therein, and the area is binarized to generate a two-dimensional pre-recognition image.

[0031] In this application, edge detection is performed after a series of preprocessing of the two-dimensional original image, which reduces the complexity of edge detection and improves the detection speed. Binarization after edge detection can further reduce image information and facilitate the improvement of subsequent recognition speed.

[0032] In one embodiment,

[0033] The obstacle tracking in the two-dimensional original image of the current frame based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame includes:

[0034] In the two-dimensional original image of the current frame, based on the obstacle information of the two-dimensional original image of the historical frame, the optical flow method is used to track the obstacle. The present application uses the optical flow method, which can identify moving objects with high precision.

[0035] In one embodiment,

[0036] The obstacle recognition model is pre-trained in the following way:

[0037] Setting an image sensor and a radar sensor on the target vehicle;

[0038] During the movement of the target vehicle, a plurality of surround two-dimensional images are collected by the image sensor, and three-dimensional point cloud data are obtained by the radar sensor;

[0039] Annotating obstacles in three dimensions using the three-dimensional point cloud data, and performing spatiotemporal alignment of the annotated obstacle data with the collected two-dimensional image to obtain a training sample set;

[0040] The two-dimensional image in the sample set is used as input, the annotated obstacle data is used as output, and the initially established model is trained to obtain the obstacle recognition model.

[0041] This application realizes data association from two-dimensional images to three-dimensional obstacle information by simultaneously setting up image sensors and radar sensors, thereby providing a data basis for model training; in addition, due to the large differences in obstacle characteristics in different regions, such as different overpasses and traffic lights in different countries, the model trained using sample data collected in one region has poor accuracy when used in other regions. In this application, image sensors and radar sensors are set on the vehicle to collect local sample data to train the model. Compared with directly using open source data, the accuracy of vehicle perception in local scenarios can be improved.

[0042] In one embodiment,

[0043] The image sensor used to obtain the two-dimensional target image is of the same model as the image sensor used to collect sample data, so that the trained model can recognize the image obtained by the corresponding image sensor with higher accuracy.

[0044] In one embodiment,

[0045] Sample sets are collected for various regions and corresponding models are trained based on the sample sets. When the vehicle perceives, the model of the corresponding region is selected for perception and recognition. In this way, the trained model can recognize images of the corresponding region with higher accuracy.

[0046] In one embodiment,

[0047] The obstacle recognition model is Bevformer, DETR3D, BEVDet or PETR; and / or, the lane line recognition model is MapTR or Yolov5.

[0048] In one embodiment,

[0049] The obstacle information includes obstacle location information, obstacle category information, obstacle size information and obstacle posture information;

[0050] and / or,

[0051] The lane line information includes lane line feature point position information and lane line direction information.

[0052] In a second aspect, an embodiment of the present application provides a vehicle sensing device, including:

[0053] A first acquisition module is used to acquire a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround view image and includes multiple frames of images that are continuous in time sequence;

[0054] An obstacle tracking module is used to perform obstacle tracking in the two-dimensional original image of the current frame based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and remove the tracked obstacle from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame;

[0055] A recognition module, used to recognize the two-dimensional target image of the current frame by using a lane recognition model and an obstacle recognition model, and determine lane information and obstacle information in three-dimensional space;

[0056] A determination module, used to determine a drivable area of ​​the vehicle according to the lane line information;

[0057] The bird's-eye view generating module is used to generate the vehicle environment information of the bird's-eye view according to the tracked obstacle, the obstacle information and the drivable area of ​​the vehicle.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a memory and a processor. The memory and the processor communicate with each other through an internal connection path, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor executes the method in any one of the above-mentioned embodiments.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is executed.

[0060] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

[0061] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0063] Figure 1This is a flow chart of a vehicle perception method according to Embodiment 1 of the present application;

[0064] Figure 2 This is a flow chart of a vehicle perception method according to Embodiment 2 of the present application;

[0065] Figure 3 This is the model training flow chart in Example 3 of the present application;

[0066] Figure 4 This is a structural block diagram of the vehicle sensing device of embodiment 4 of the present application. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0068] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this application and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The terms "and / or" or "and / or" used in this application include any and all combinations of one or more related listed items.

[0069] As described in the background technology, cameras, as vehicle-mounted sensors, can obtain a large amount of environmental information required for assisted driving. However, the two-dimensional images generated by the cameras lack depth information and have low ability to perceive the position of targets. Such perception errors can cause assisted driving to make wrong decisions, posing a threat to the lives of drivers and pedestrians.

[0070] In order to make up for the shortcomings of pure camera solutions, the industry has also proposed solutions such as multimodal fusion and map + perception. However, these solutions either require the fusion of radar sensors or the construction of city models, which are costly and highly dependent on the outside world. The camera-assisted driving solution has the advantages of low cost, low dependence on the outside world, and easy popularization, and is still the preferred method currently hoped by the industry. For this reason, this application creatively proposes a new vehicle perception method, which uses an obstacle recognition model to identify three-dimensional obstacle information from a two-dimensional image, and uses a lane line recognition model to identify lane line information from a two-dimensional image to determine the vehicle's drivable area, so that the surrounding environment information containing three-dimensional obstacles and the vehicle's drivable area can be formed from a bird's-eye view. Compared with the prior art, under the premise of using only low-cost image sensors, a variety of information from a bird's-eye view is obtained, which increases the driver's ability to perceive the target's position, and provides more and more accurate decision-making data for assisted driving.

[0071] Some embodiments of the present application are described below. It should be noted that the following description is for illustrative purposes and is not intended to limit the scope of protection of the present application.

[0072] Embodiment 1

[0073] The following is a vehicle perception method based on an image sensor proposed in Example 1 of the present application. Figure 1 As shown, the method comprises the following steps:

[0074] S11, acquiring a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround image of the vehicle's surroundings and includes a plurality of frames of images that are continuous in time sequence.

[0075] In step S11, the two-dimensional original image is obtained by an image sensor installed on the vehicle. In a specific embodiment, multiple image sensors can be installed on the vehicle to obtain images of different fields of view around the vehicle. In a preferred embodiment, these images can be spliced ​​to form a 360° panoramic surround image to facilitate subsequent 360° obstacle recognition of the vehicle.

[0076] To facilitate subsequent image processing, the multiple image sensors on the vehicle are preferably image sensors of the same model.

[0077] S13. In the two-dimensional original image of the current frame, obstacle tracking is performed based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame.

[0078] The present application has several characteristics: First, the recognition of images has temporal continuity, and the images are transmitted to the model frame by frame in the form of image streams for recognition, and most of the obstacles and lane lines in adjacent frame images overlap. The obstacles recognized in the previous frame or the previous frames have a high probability of appearing in the next frame, and the position of the obstacle is near the obstacle in the previous frame or the previous frames; second, the obstacle recognition model used for three-dimensional obstacle recognition is highly accurate but time-consuming. Utilizing these two characteristics, the embodiment of the present application executes step S13, which utilizes the high accuracy of the obstacle recognition model and the fast target tracking and recognition speed. With the help of the high-precision three-dimensional obstacle information obtained by the obstacle recognition model for the historical frame image recognition, the obstacle is quickly and accurately tracked in the two-dimensional image of the current frame, and the image to be recognized by the obstacle recognition model is simplified accordingly. This method greatly improves the recognition speed while ensuring the overall recognition accuracy.

[0079] In a specific embodiment of the present application, the historical frame may be a previous frame or previous frames. The tracking may be performed using an existing optical flow method.

[0080] When tracking, only one or part of the feature points of the obstacle can be tracked to improve the tracking speed, such as the center of mass or corner point, so as to determine the moving direction angle of the obstacle based on the movement of the center of mass or corner point relative to the historical frame. The current frame and the historical frame are obtained when the vehicle moves to different positions, so these images constitute different perspective images of the same obstacle feature point. By reconstructing the image using the geometric relationship between images of different perspectives, the depth information of the tracked feature point can be determined.

[0081] Then, based on the newly tracked feature points, the coordinates corresponding to the three-dimensional obstacle information in the historical frame image are transformed according to the depth information of the feature points and the direction angle of the obstacle movement, so as to obtain the three-dimensional information of the tracked obstacle.

[0082] S15. Use the lane line recognition model and the obstacle recognition model to respectively recognize the two-dimensional target image, and determine the lane line information and the obstacle information in the three-dimensional space.

[0083] In this application, the obstacle recognition model refers to a model that can recognize a two-dimensional image and output obstacle information in three-dimensional space. In some embodiments, the obstacle recognition model can be selected as Bevformer, DETR3D, BEVDet or PETR. Among them, Bevformer is the model selected in the preferred embodiment of this application, and Bevformer will be discussed in detail below.

[0084] The obstacle information described in the embodiments of the present application may include obstacle position information, obstacle category information, obstacle size information, and obstacle posture information. After the obstacle is identified, it will be indicated in the form of a rectangular frame, where the obstacle position information is its coordinate information, specifically expressed as the three-dimensional coordinates of the vertices of the corresponding rectangular frame; the obstacle category information is the obstacle type, such as the obstacle is other vehicles, the obstacle is a traffic light, the obstacle is an overpass, etc.; the obstacle size information is the size of the obstacle, which can be determined by the size of the corresponding rectangular frame (or quantitatively determined according to the coordinates of each vertex of the rectangular frame); the obstacle posture information is the rotation direction of the obstacle. The obstacle position information, obstacle size information, and obstacle posture information can all be determined by the vertex coordinates of the corresponding rectangular frame.

[0085] The lane line recognition model refers to a model that can recognize two-dimensional images and output lane line information. It should be noted that the lane line recognition model in this application can select those models that recognize two-dimensional images and output lane line information in two-dimensional space, such as Yolov5, or models that recognize two-dimensional images and output lane line information in three-dimensional space, such as MapTR. In addition, traditional image vision algorithms can be used in this application to perform edge detection, semantic segmentation, etc. on the image to identify lane line information. The lane line information in this application may specifically include lane line feature point location information and lane line direction information.

[0086] Although lane line information in three-dimensional space can provide more accurate lane line information, the corresponding recognition model is complex to operate, which reduces the recognition speed. Considering that in some scenarios, such as most urban roads, the ground is roughly flat and the difference in lane line height direction can be ignored, the recognition speed can be prioritized at this time, and only two-dimensional recognition of lane lines can be performed to reduce the model workload.

[0087] In a further preferred embodiment, a variety of lane line recognition models can be provided for selection. When it is detected that the current driving ground is relatively flat, a two-dimensional lane line recognition model is selected from the preset two-dimensional lane line recognition model and the three-dimensional lane line recognition model to obtain lane line information in two-dimensional space based on the two-dimensional lane line recognition model. Conversely, a three-dimensional lane line recognition model is selected to obtain lane line information in three-dimensional space. The determination of the flatness of the ground can be carried out with the help of existing technologies, such as determining the flatness of the ground based on a laser sensor, or determining the height of each part of the ground based on a combination of multiple ground images to determine whether it is flat, or determining whether the ground is flat based on ground height data when the ground is laid, existing navigation data, etc.

[0088] Although the present application does not limit the specific lane line recognition model and obstacle recognition model. However, in the embodiment of the present application, the obstacle recognition model is different from the lane line recognition model. This is because obstacles and lane lines have different characteristics, so it is difficult for a model to provide high accuracy in obstacle recognition and high accuracy in lane line recognition at the same time. For this reason, the present application selects an adapted model to identify obstacles and lane lines respectively to improve the accuracy of both recognition. Such a choice is also conducive to the subsequent collection of obstacle-related sample sets and lane line-related sample sets for separate model training.

[0089] S17: Determine a drivable area for the vehicle based on the lane line information.

[0090] The assisted driving of a vehicle not only needs to know the surrounding obstacle information, but also needs to comply with traffic rules and drive within the permitted area. Therefore, the solution of this application identifies and determines the drivable area of ​​the vehicle based on the lane line information, thereby providing more decision-making data for assisted driving, which is convenient for improving the accuracy of assisted driving decisions.

[0091] S19: Generate vehicle environment information from a bird's-eye view according to the tracked obstacles, obstacle information, and the drivable area of ​​the vehicle.

[0092] In the present application, the vehicle environment information from a bird's-eye view is three-dimensional vehicle environment information.

[0093] In step S15, the process of model recognition of obstacles and lane lines is the process of true value annotation of obstacles and lane lines. The result of the true value annotation is a true value rectangular frame. For the driver's intuitive observation, the obstacle information and lane line information presented in the rectangular frame need to be rendered in step S19. Specifically, visualization is performed according to the preset rendering template of each obstacle. For example, if the obstacle information is marked as a car, the corresponding rectangular frame is rendered into the shape of a car, and the lane line information is also visualized according to the preset rendering template.

[0094] In this application, the tracked obstacles, obstacle information, and lane line information are obtained separately and need to be fused into the same bird's-eye view space in this step. The specific fusion process varies depending on the lane line information:

[0095] When the lane line information is lane line information in a two-dimensional space, the vehicle environment information from a bird's-eye view is generated according to the tracked obstacles, obstacle information, and the drivable area of ​​the vehicle, including:

[0096] Generate preliminary environmental information from a bird's-eye view based on the tracked obstacles and obstacle information;

[0097] The drivable area of ​​the vehicle in the two-dimensional space is projected onto the ground in the bird's-eye view; the preliminary environment information and the projection of the drivable area together form the vehicle environment information in the bird's-eye view. Specifically, the height coordinate of the drivable area is set to be the same as the height coordinate of the ground, so that the drivable area is projected onto the ground in the bird's-eye view.

[0098] When the lane line information is lane line information in three-dimensional space, it is only necessary to place the lane line information and obstacle information in the bird's-eye view space respectively.

[0099] In some application scenarios, in order to improve the perception speed, the above perception method can be further simplified. In one embodiment, the bird's-eye view is specifically a bird's-eye view, that is, the obtained three-dimensional information (including obstacle information in three-dimensional space and lane line information in three-dimensional space) is removed from the height direction to form a bird's-eye view.

[0100] In this embodiment, the vehicle perception method can be implemented through a cloud server and / or a vehicle system. Where it is implemented through a cloud server, the two-dimensional target image can be transmitted to the cloud server, and after the cloud server executes the above method, the result is fed back to the vehicle system, and then the vehicle surrounding environment from a bird's-eye view is displayed through the vehicle system display screen, so that the driver can adjust the driving in time. In addition, the vehicle perception method of this embodiment can also be implemented on a mobile terminal such as a mobile phone, tablet or laptop computer. The mobile terminal can be connected to the vehicle system to obtain the two-dimensional target image from the vehicle system for processing.

[0101] This application uses a model recognition method to obtain obstacle information in three-dimensional space from two-dimensional image recognition, and obtains lane line information from two-dimensional image recognition, and combines the obstacle information and lane line information to determine the vehicle environment information from a bird's-eye view. In the bird's-eye view mode, the driver can see the vehicle's surrounding environment at a glance. Compared with the prior art, this application can directly determine obstacle information in three-dimensional space from two-dimensional images collected by image sensors, increase the ability to perceive the position of the target, and provide more and more accurate decision data for assisted driving by combining lane line information.

[0102] Moreover, in the present application, based on the high-precision recognition results of the historical frame images by the obstacle recognition model (i.e., the three-dimensional obstacle information corresponding to the historical frame), the relevant obstacles are tracked in the two-dimensional original image of the current frame. Compared with the obstacle recognition model, the tracking improves the recognition speed of at least some obstacles in the current frame. Based on this tracking, the obstacle recognition model only needs to identify the image of the tracked obstacle removed from the two-dimensional original image of the current frame, i.e., the two-dimensional target image, which simplifies the image information and improves the recognition speed of the obstacle recognition model. This allows the present application to improve the recognition speed while ensuring the recognition accuracy.

[0103] Embodiment 2

[0104] In a vehicle driving environment, it is necessary to quickly perceive and recognize the surrounding environment. Once there is a long delay, the perception algorithm is meaningless and may even lead to dangerous driving decisions. This application performs three-dimensional recognition on a two-dimensional image. The algorithm is relatively complex and takes a long time. In order to further shorten the time for vehicle recognition perception, Example 2 of this application provides the following Figure 2 The sensing method shown comprises the following steps:

[0105] S21, acquiring a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround image of the vehicle's surroundings and includes a plurality of frames of images that are continuous in time sequence.

[0106] S22, pre-recognize the two-dimensional original image of the current frame, and generate a two-dimensional pre-recognition image according to the pre-recognition result; the pre-recognition result includes an obstacle recognition result and / or a lane line recognition result.

[0107] S23. In the two-dimensional pre-recognition image of the current frame, obstacle tracking is performed based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional pre-recognition image of the current frame to obtain the two-dimensional target image of the current frame.

[0108] S25. Use the lane line recognition model and the obstacle recognition model to respectively recognize the two-dimensional target image, and determine the lane line information and the obstacle information in the three-dimensional space.

[0109] S27: Determine a drivable area for the vehicle based on the lane line information.

[0110] S29: Generate vehicle environment information from a bird's-eye view according to the tracked obstacles, obstacle information, and the drivable area of ​​the vehicle.

[0111] The above steps S21, S25, S27, and S29 of the second embodiment are the same or substantially similar to the steps S11, S15, S17, and S19 of the first embodiment. The main difference from the first embodiment is that a pre-identification step S22 is added, and accordingly the tracked object is adjusted from the two-dimensional original image of S13 to the two-dimensional pre-identified image in S23.

[0112] In the above embodiment, the pre-recognition of the two-dimensional original image can be the pre-recognition of obstacles alone, or the pre-recognition of lane lines, or of course, the pre-recognition of both obstacles and lane lines. The pre-recognition of lane lines is applicable to the scenario where a three-dimensional lane line recognition model is selected.

[0113] In a feasible embodiment, the same algorithm may be used to pre-recognize obstacles and lane lines in the two-dimensional original image to form a two-dimensional pre-recognition image.

[0114] Based on this embodiment, the present application first performs two-dimensional pre-recognition on the two-dimensional original image with a relatively small amount of computation, and can quickly identify and obtain a two-dimensional pre-recognition image with approximate obstacle information and approximate lane line information. Subsequent tracking and three-dimensional recognition can be performed on the basis of the two-dimensional pre-recognition image. Since the two-dimensional pre-recognition image reduces a lot of background information compared to the two-dimensional original image, the computational amount of subsequent tracking and three-dimensional recognition models is correspondingly reduced, thereby improving the overall recognition speed.

[0115] In step S22, the two-dimensional original image is pre-recognized to obtain a two-dimensional pre-recognized image in a variety of possible ways. In some embodiments, step S22 specifically includes:

[0116] Preprocessing the two-dimensional original image; the preprocessing includes at least one of mean filtering, grayscale processing, and color level balancing;

[0117] The pre-processed two-dimensional original image is subjected to edge detection to determine the obstacle area and / or lane line area therein, and the area is binarized to obtain the two-dimensional pre-recognition image.

[0118] In this application, edge detection is performed after a series of preprocessing of the two-dimensional original image, which reduces the complexity of edge detection and improves the detection speed. Binarization after edge detection can further reduce image information and facilitate the improvement of subsequent recognition speed.

[0119] Embodiment 3

[0120] In some embodiments, the obstacle recognition model can be implemented by using an existing model; in an alternative embodiment, the obstacle recognition model is obtained by training a sample set. At present, there are some open source data for training obstacle recognition models, but due to the large differences in obstacle characteristics in different regions, such as the differences in overpasses and traffic lights in different countries, the model trained using open source samples may have poor recognition accuracy due to the large differences between the open source samples and the local scenes. For this reason, the third embodiment of the present application provides a method for obtaining a sample set to train an obstacle recognition model, such as Figure 3 As shown, the method specifically includes the following steps:

[0121] S31. Install an image sensor and a radar sensor on the target vehicle.

[0122] S33. When the target vehicle is moving, a plurality of surround two-dimensional images are collected by an image sensor, and three-dimensional point cloud data is obtained by a radar sensor.

[0123] Since the image sensor and radar sensor are installed on the same vehicle, the two-dimensional image and three-dimensional point cloud data obtained by the two sensors at the same time correspond to the same surrounding environment of the vehicle, and the two types of data are therefore associated.

[0124] S35, performing three-dimensional annotation of obstacles through the three-dimensional point cloud data, and performing spatiotemporal alignment of the annotated obstacle data with the collected two-dimensional image to obtain a training sample set.

[0125] Obstacles are three-dimensionally annotated using the three-dimensional point cloud data, and the annotated obstacle data is spatially and temporally aligned with the collected two-dimensional image to obtain a sample set with the two-dimensional image as input and the annotated obstacle data as output.

[0126] The three-dimensional annotation of obstacles in the three-dimensional point cloud data can be done manually or with the existing three-dimensional recognition model. The annotated obstacles and the two-dimensional image can be aligned in time and space according to the existing technology, and a sample set is formed after alignment. The time alignment can be done by aligning the acquisition time; the spatial alignment can be done by projecting the three-dimensional annotation frame of the obstacle onto the two-dimensional image through the internal and external parameters of the camera. After projection, the accuracy of the spatial alignment can also be manually reviewed.

[0127] S37, taking the two-dimensional image in the sample set as input, taking the labeled obstacle data as output, training the initially established model, and obtaining an obstacle recognition model.

[0128] This application realizes data association from two-dimensional images to three-dimensional obstacle information by simultaneously setting up image sensors and radar sensors, providing a data basis for model training; and in this application, image sensors and radar sensors are set on the vehicle to collect local sample data to train the model, which can improve the accuracy of vehicle perception and recognition in local scenarios compared to directly using open source data.

[0129] In a further embodiment, the image sensor used to obtain the two-dimensional original image and the image sensor used to collect the sample data are of the same model. The image sensor may be a surround view sensor.

[0130] In a further embodiment, sample sets are collected for each of the various regions and corresponding models are trained based on the respective sample sets. During vehicle perception, an obstacle recognition model of the corresponding region is selected for perception and recognition.

[0131] The obstacle recognition model mentioned above has multiple options. Considering the recognition accuracy and speed, this application prefers BEVForme. BEVForme is an algorithm model that uses pure vision (camera) for perception tasks. It extracts image features from two-dimensional surround images and converts the extracted image features into BEV space within the model, thereby realizing three-dimensional target recognition in two-dimensional images.

[0132] BEVForme mainly includes feature extraction layer, encoding layer Encoder and decoding layer Decoder. After the image is input, the feature extraction layer extracts the multi-scale features of the surround view image; then the multi-scale features of the surround view image are converted into BEV features at the Encoder; and then the classification and positioning of 3D target detection are completed at the Decoder. The feature extraction layer includes Backbone+Neck (ResNet-101-DCN+

[0133] FPN); Encoder includes Temporal Self-Attention (self-attention module) and Spatial Cross-Attention (cross attention module).

[0134] BEVForme's self-attention module and cross-attention module both use multi-scale deformable attention modules to transform the model's global attention into local attention, thereby improving the model's recognition speed.

[0135] The self-attention mechanism established in time by the self-attention module of BEVForme can comprehensively infer the probability of the existence of obstacles around the vehicle at the current moment by utilizing the three-dimensional position information of the obstacle features remaining in the previous frames, thereby having a memory function for some obstacles that are temporarily occluded, which can improve the recognition accuracy.

[0136] The cross-attention mechanism established in the space by the cross-attention module of BEVForme can integrate all the cameras on the vehicle to generate the location of obstacles in the bird's-eye view space around the vehicle. Because the surround view function of the vehicle is designed with safety redundancy, it will lead to the situation that only one camera can observe certain angles among the adjacent cameras, and the adjacent cameras will have overlapping fields of view. The cross-attention mechanism can greatly increase the speed and accuracy of obstacle recognition in the overlapping fields of view.

[0137] Embodiment 4

[0138] Figure 4 FIG. 2 shows a structural block diagram of a vehicle sensing device according to an embodiment of the present application. Figure 4 As shown, the vehicle sensing device includes:

[0139] The first acquisition module 41 is used to acquire a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround view image and includes multiple frames of images that are continuous in time sequence.

[0140] The obstacle tracking module 43 is used to track obstacles in the two-dimensional original image of the current frame based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacles, and eliminate the tracked obstacles from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame.

[0141] The recognition module 45 is used to recognize the two-dimensional target image of the current frame by using the lane recognition model and the obstacle recognition model, and determine the lane information and the obstacle information in the three-dimensional space.

[0142] The determination module 47 is used to determine the drivable area of ​​the vehicle according to the lane line information.

[0143] The bird's-eye view generating module 49 is used to generate the vehicle environment information in a bird's-eye view according to the tracked obstacle, the obstacle information and the drivable area of ​​the vehicle.

[0144] In one embodiment, the lane line information is lane line information in a two-dimensional space and the drivable area of ​​the vehicle is a drivable area of ​​the vehicle in a two-dimensional space;

[0145] The bird's-eye view generating module 49 is specifically used to: generate preliminary environment information of a bird's-eye view according to the tracked obstacle and the obstacle information;

[0146] The drivable area of ​​the vehicle in the two-dimensional space is projected onto the ground in the bird's-eye view; the preliminary environmental information and the projection of the drivable area together form the vehicle environmental information in the bird's-eye view.

[0147] In one embodiment, the vehicle sensing device further includes:

[0148] A model selection module, for selecting a two-dimensional lane line recognition model from a preset two-dimensional lane line recognition model and a three-dimensional lane line recognition model when it is determined that the current driving ground of the vehicle is flat ground;

[0149] The recognition module 45 is specifically used to use the two-dimensional lane recognition model and the obstacle recognition model to recognize the two-dimensional target image respectively, and determine the lane information in the two-dimensional space and the obstacle information in the three-dimensional space.

[0150] In one embodiment, the vehicle sensing device further includes:

[0151] The pre-recognition module 42 performs pre-recognition on the two-dimensional original image of the current frame, and generates a two-dimensional pre-recognition image according to the pre-recognition result; the pre-recognition result includes an obstacle recognition result and / or a lane line recognition result.

[0152] Specifically, the pre-recognition module 42 is a pre-processing module for pre-processing the two-dimensional original image; and performing edge detection on the pre-processed two-dimensional original image to determine the obstacle area and / or lane line area therein, and binarizing the area to generate a two-dimensional pre-recognition image. The pre-processing includes at least one of mean filtering, grayscale processing, and color level balancing.

[0153] In one embodiment, the vehicle perception device further includes a model training module, which specifically includes:

[0154] The data acquisition module is used to collect multiple two-dimensional surround images through the image sensor on the vehicle during the movement of the target vehicle, and obtain three-dimensional point cloud data through the radar sensor on the vehicle.

[0155] The data annotation module performs three-dimensional annotation of obstacles through the three-dimensional point cloud data, and performs spatiotemporal alignment of the annotated obstacle data with the collected two-dimensional image to obtain a training sample set.

[0156] The model training submodule takes the two-dimensional image in the sample set as input and the annotated obstacle data as output, trains the initially established model, and obtains the obstacle recognition model.

[0157] This embodiment corresponds to the above method embodiments one to three. The relevant contents can be referred to the discussion of the above method embodiments and will not be repeated here.

[0158] Embodiment 5

[0159] Embodiment 5 of the present application discloses an electronic device. The electronic device includes: a memory and a processor, wherein the memory stores a computer program that can be run on the processor. When the processor executes the computer program, the vehicle perception method in the above embodiment is implemented. The number of the memory and the processor can be one or more.

[0160] The electronic device / terminal / server also includes:

[0161] Communication interface, used to communicate with external devices and perform data exchange transmission.

[0162] If the memory, processor and communication interface are implemented independently, the memory, processor and communication interface can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0163] Optionally, in a specific implementation, if the memory, processor, and communication interface are integrated on a chip, the memory, processor, and communication interface can communicate with each other through an internal interface.

[0164] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0165] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0166] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0167] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0168] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0171] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0172] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0173] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0174] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0175] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A vehicle perception method, characterized in that: include: Acquire a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround image of the vehicle's surroundings and includes multiple frames of images that are continuous in time sequence; In the two-dimensional original image of the current frame, obstacle tracking is performed based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame; Using a lane recognition model and an obstacle recognition model to respectively recognize the two-dimensional target image of the current frame, and determine lane information and obstacle information in three-dimensional space; Determine a drivable area of ​​the vehicle according to the lane line information; Vehicle environment information from a bird's-eye view is generated according to the tracked obstacle, the obstacle information, and the drivable area of ​​the vehicle.

2. The vehicle perception method according to claim 1, characterized in that: The lane line information is lane line information in two-dimensional space and the drivable area of ​​the vehicle is a drivable area of ​​the vehicle in two-dimensional space; The generating of the vehicle environment information from a bird's-eye view according to the tracked obstacle, the obstacle information and the drivable area of ​​the vehicle comprises: Generating preliminary environmental information from a bird's-eye view according to the tracked obstacles and the obstacle information; The drivable area of ​​the vehicle in the two-dimensional space is projected onto the ground in the bird's-eye view; the preliminary environmental information and the projection of the drivable area together form the vehicle environmental information in the bird's-eye view.

3. The vehicle perception method according to claim 1, characterized in that: When it is determined that the current driving surface of the vehicle is flat, a two-dimensional lane line recognition model is selected from a preset two-dimensional lane line recognition model and a three-dimensional lane line recognition model; The using of the lane line recognition model and the obstacle recognition model to respectively recognize the two-dimensional target image and determine the lane line information and the obstacle information in the three-dimensional space includes: The two-dimensional lane line recognition model and the obstacle recognition model are used to respectively recognize the two-dimensional target image to determine the lane line information in the two-dimensional space and the obstacle information in the three-dimensional space.

4. The vehicle perception method according to claim 1, characterized in that: The vehicle perception method further includes: Pre-recognize the two-dimensional original image of the current frame, and generate a two-dimensional pre-recognition image according to the pre-recognition result; the pre-recognition result includes an obstacle recognition result and / or a lane line recognition result; In the two-dimensional original image of the current frame, obstacle tracking is performed based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is removed from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame, including: In the two-dimensional pre-recognition image of the current frame, obstacle tracking is performed based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and the tracked obstacle is eliminated from the two-dimensional pre-recognition image of the current frame to obtain the two-dimensional target image of the current frame.

5. The vehicle perception method according to claim 4, characterized in that: The pre-recognition of the two-dimensional original image and the generation of a two-dimensional pre-recognition image according to the pre-recognition result include: Preprocessing the two-dimensional original image; the preprocessing includes at least one of mean filtering, grayscale processing, and color level balancing; The pre-processed two-dimensional original image is subjected to edge detection to determine the obstacle area and / or lane line area therein, and the area is binarized to generate a two-dimensional pre-recognition image.

6. The vehicle perception method according to claim 1, characterized in that: The performing obstacle tracking in the two-dimensional original image of the current frame based on the obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame includes: In the two-dimensional original image of the current frame, the obstacle tracking is performed using the optical flow method based on the obstacle information of the two-dimensional original image of the historical frame.

7. The vehicle perception method according to any one of claims 1 to 6, characterized in that: The obstacle recognition model is pre-trained in the following way: Setting an image sensor and a radar sensor on the target vehicle; During the movement of the target vehicle, a plurality of surround two-dimensional images are collected by the image sensor, and three-dimensional point cloud data are obtained by the radar sensor; Annotating obstacles in three dimensions using the three-dimensional point cloud data, and performing spatiotemporal alignment of the annotated obstacle data with the collected two-dimensional image to obtain a training sample set; The two-dimensional image in the sample set is used as input, the annotated obstacle data is used as output, and the initially established model is trained to obtain the obstacle recognition model.

8. The vehicle perception method according to any one of claims 1 to 6, characterized in that: The obstacle recognition model is Bevformer, DETR3D, BEVDet or PETR; and / or, the lane line recognition model is MapTR or Yolov5.

9. The vehicle perception method according to any one of claims 1 to 6, characterized in that: The obstacle information includes obstacle location information, obstacle category information, obstacle size information and obstacle posture information; and / or, The lane line information includes lane line feature point position information and lane line direction information.

10. A vehicle sensing device, characterized in that: include: A first acquisition module is used to acquire a two-dimensional original image of the vehicle's surroundings; the two-dimensional original image is a surround view image and includes multiple frames of images that are continuous in time sequence; An obstacle tracking module is used to perform obstacle tracking in the two-dimensional original image of the current frame based on obstacle information in the three-dimensional space of the two-dimensional original image of the historical frame to determine the tracked obstacle, and remove the tracked obstacle from the two-dimensional original image of the current frame to obtain the two-dimensional target image of the current frame; A recognition module, used to recognize the two-dimensional target image of the current frame by using a lane recognition model and an obstacle recognition model, and determine lane information and obstacle information in three-dimensional space; A determination module, used to determine a drivable area of ​​the vehicle according to the lane line information; The bird's-eye view generating module is used to generate the vehicle environment information of the bird's-eye view according to the tracked obstacle, the obstacle information and the drivable area of ​​the vehicle.

Images