Travelable area detection method, device, equipment, medium, product and vehicle

Abstract

The invention relates to a drivable area detection method, device and equipment, a medium, a product and a vehicle, a point cloud two-dimensional image is obtained by projecting a three-dimensional point cloud of an environment where a target vehicle is located to an overlook plane of the target vehicle, and the three-dimensional point cloud is obtained by detection equipment of the target vehicle; and determining a drivable area of the target vehicle according to the point cloud two-dimensional image. In this way, the three-dimensional point cloud for the environment where the vehicle is located is projected to the overlook plane for the vehicle, the drivable area of the vehicle is obtained through detection in a point cloud two-dimensional image mode, the detection accuracy of the drivable area is improved, the collision risk is reduced, the safety is improved, and meanwhile the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a drivable area detection method, device, equipment, medium, product and vehicle. Background Art

[0002] Existing vehicles need to detect the drivable area when parking automatically. The current mainstream drivable area detection method is a semantic segmentation method based on deep learning that relies on visual sensors, which is also a drivable area detection model. Visual sensor data mainly includes original images from on-board cameras or spliced ​​and synthesized overhead panoramic images.

[0003] However, the data perceived by the visual sensor data is easily affected by the installation position and detection accuracy of the vehicle-mounted camera, resulting in the problem of low accuracy of the drivable area detected by the drivable area detection model that relies on the visual sensor. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, storage medium, computer program product and vehicle for detecting a drivable area. The method projects a three-dimensional point cloud of the vehicle's environment onto a bird's-eye view plane of the vehicle, and detects the vehicle's drivable area in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, and improving safety while also improving the user experience.

[0005] The present application provides a method for detecting a drivable area, including:

[0006] Projecting a three-dimensional point cloud of the environment in which the target vehicle is located onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle;

[0007] The drivable area of ​​the target vehicle is determined according to the point cloud two-dimensional image.

[0008] Accordingly, an embodiment of the present application provides a drivable area detection device, comprising:

[0009] A projection module, used to project a three-dimensional point cloud of the environment in which the target vehicle is located onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle;

[0010] The drivable area detection module is used to determine the drivable area of ​​the target vehicle based on the point cloud two-dimensional image.

[0011] In addition, an embodiment of the present application also provides an electronic device, including one or more processors and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the drivable area detection method provided in the embodiment of the present application.

[0012] In addition, an embodiment of the present application further provides a storage medium, which stores a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute any one of the drivable area detection methods provided in the embodiments of the present application.

[0013] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements any of the drivable area detection methods provided in the embodiments of the present application.

[0014] In addition, an embodiment of the present application also provides a vehicle, comprising the above-mentioned electronic device.

[0015] In the embodiment of the present application, a two-dimensional image of the point cloud is obtained by projecting the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle, wherein the three-dimensional point cloud is acquired by the detection equipment of the target vehicle; and the drivable area of ​​the target vehicle is determined according to the two-dimensional image of the point cloud. In this way, the three-dimensional point cloud of the environment in which the vehicle is located is projected onto the top-view plane of the vehicle, and the drivable area of ​​the vehicle is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of an implementation environment scenario of the drivable area detection method provided in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a drivable area detection method provided by an embodiment of the present application;

[0019] Figure 3 It is a flowchart of a specific embodiment of a drivable area detection method provided by an embodiment of the present application;

[0020] Figure 4is a schematic diagram of the structure of a drivable area detection device provided by an embodiment of the present application;

[0021] Figure 5 is another structural schematic diagram of a drivable area detection device provided by an embodiment of the present application;

[0022] Figure 6 It is a structural schematic diagram of a clock synchronization module in a drivable area detection device provided in one embodiment of the present application;

[0023] Figure 7 It is a structural schematic diagram of a multi-sensor data acquisition module in a drivable area detection device provided in one embodiment of the present application;

[0024] Figure 8 It is a structural schematic diagram of a data acquisition control submodule in a drivable area detection device provided in an embodiment of the present application;

[0025] Fig. 9 It is a structural schematic diagram of a vehicle-mounted computing power platform in a drivable area detection device provided in one embodiment of the present application;

[0026] Fig.10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0028] In addition, the term "plurality" in the embodiments of the present application refers to two or more than two. The terms "first" and "second" in the embodiments of the present application are used to distinguish descriptions and should not be understood as implying relative importance.

[0029] The embodiments of the present application provide a drivable area detection method, device, system, electronic device, storage medium, computer program product and vehicle. The drivable area detection device can be integrated in an electronic device, which can be a server or a terminal.

[0030] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0031] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0032] See also Figure 1 , taking the integration of the drivable area detection device into an electronic device as an example, Figure 1 A schematic diagram of an implementation scenario of the drivable area detection method provided in an embodiment of the present application, wherein the electronic device may be a vehicle-mounted computing platform, which projects a three-dimensional point cloud of the target vehicle's environment onto a top-down plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle; and the drivable area of ​​the target vehicle is determined based on the two-dimensional image of the point cloud.

[0033] It should be noted that Figure 1 The schematic diagram of the implementation environment scenario of the drivable area detection method shown is only an example. The implementation environment scenario of the drivable area detection method described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of data processing and the emergence of new business scenarios, the technical solution provided by the present application is also applicable to similar technical problems.

[0034] The solutions provided by the embodiments of the present application are specifically described by the following embodiments. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0035] This embodiment will be described from the perspective of a drivable area detection device. The drivable area detection device may be specifically integrated into an electronic device, which may be a terminal and / or a server, and this application does not limit this.

[0036] Research has found that the current mainstream method for detecting drivable areas is a semantic segmentation method based on deep learning that relies on visual sensors. Visual sensor data mainly includes original images from on-board cameras or stitched and synthesized overhead panoramic images. Directly detecting the drivable area in the original images from on-board cameras that include the road surface may result in the calculated drivable area being too large due to the influence of the camera installation angle; the drivable area detection based on overhead panoramic images is easily affected by uneven ground, image distortion, stitching errors, etc., resulting in inaccurate or even erroneous detection of the drivable area; both of these will bring greater collision risks to automatic parking, seriously threatening the safety of the automatic parking function.

[0037] In order to solve the above problems, the present application projects the three-dimensional point cloud of the vehicle's environment onto a top-down plane of the vehicle, and detects the vehicle's drivable area in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0038] Specifically, see Figure 2 , Figure 2 : is a flowchart of a drivable area detection method provided in an embodiment of the present application. The drivable area detection method may include the following steps S101 to S102:

[0039] S101. Projecting a three-dimensional point cloud of the environment in which the target vehicle is located onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle.

[0040] The target vehicle refers to a vehicle that needs to be detected in the drivable area.

[0041] The target vehicle's environment is used to indicate the physical environment where the target vehicle is located.

[0042] A three-dimensional point cloud refers to a data structure of objects or scenes in the environment where the target vehicle is located in three-dimensional space. Among them, the objects or scenes in the environment where the target vehicle is located include but are not limited to obstacles and the ground.

[0043] The bird's-eye view plane refers to a plane that can fully display the vehicle and its surroundings from a vertical downward perspective from above the vehicle. The bird's-eye view plane can be centered on the target vehicle or can be determined with the target vehicle at its target position, which is adjusted according to actual conditions.

[0044] The point cloud two-dimensional image refers to the projection image formed on the top-view plane after the three-dimensional point cloud is projected onto the top-view plane.

[0045] The point cloud two-dimensional image contains the two-dimensional information of the three-dimensional point cloud on the top-down plane.

[0046] There are many forms of expression of point cloud two-dimensional images. For example, a point cloud two-dimensional image is a point cloud binary image in which the pixel points are binary (such as 0 or 1). For another example, a point cloud two-dimensional image can also be an image in which the pixel value of the pixel point corresponds to the height information of its projection point.

[0047] S102: Determine a drivable area of ​​the target vehicle according to the point cloud two-dimensional image.

[0048] A drivable area is a road or area where a vehicle can be driven safely and legally.

[0049] The type of drivable area is related to the scene the vehicle is in. For example, in a vehicle parking scene, the drivable area may be a drivable area for parking the vehicle. For another example, in an autonomous driving scene, the drivable area may be a drivable area for the vehicle to perform path planning and obstacle avoidance.

[0050] It can be seen that the drivable area detection method provided in the embodiment of the present application obtains a two-dimensional image of the point cloud by projecting the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle, wherein the three-dimensional point cloud is obtained by the detection equipment of the target vehicle; and the drivable area of ​​the target vehicle is determined according to the two-dimensional image of the point cloud. In this way, the three-dimensional point cloud of the environment in which the vehicle is located is projected onto the top-view plane of the vehicle, and the drivable area of ​​the vehicle is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0051] In some embodiments, the three-dimensional point cloud may be acquired by the detection device of the target vehicle in the following manner: acquiring detection data from a plurality of detection devices; and obtaining a three-dimensional point cloud for the environment in which the target vehicle is located based on the detection data.

[0052] The detection device refers to a device used to detect a target vehicle.

[0053] The detection device may be installed on the vehicle or not. For example, the detection device includes a camera installed on the vehicle and a motion detection device. For another example, the detection device includes a motion detection device installed on the vehicle and a camera installed in the environment where the vehicle is located.

[0054] There can be many types of detection equipment. For example, the detection equipment includes shooting equipment such as fisheye cameras, visible light cameras, infrared cameras, and motion detection equipment such as wheel speed meters, inertial measurement units (IMU), steering wheels, etc.

[0055] There can be multiple detection devices. For example, the detection device may include multiple cameras and multiple motion sensors installed on the vehicle, such as cameras installed around the vehicle body, on the top, on the bottom and inside the vehicle, as well as sensors such as wheel speed meters, inertial measurement units (IMUs), and steering wheels.

[0056] The specific content of the detection device can be adjusted according to actual conditions and is not limited in the embodiments of the present application.

[0057] The detection data refers to the three-dimensional point cloud data that can be used to determine the environment in which the target vehicle is located.

[0058] The detection data is obtained based on detection of the environment in which the target vehicle is located by the detection device.

[0059] The specific content of the detection data is adaptively adjusted according to the detection device, and the embodiments of the present application do not limit it. For example, when the detection device is a camera, the detection data is an image, and when the detection device is a steering wheel, the detection data is the motion-related data recorded by the steering wheel.

[0060] By acquiring detection data and constructing a three-dimensional point cloud, it is convenient for subsequent projection processing based on the three-dimensional point cloud. The vehicle's drivable area is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0061] In some embodiments, before acquiring detection data from a plurality of detection devices, the drivable area detection method further includes: performing time synchronization processing on the plurality of detection devices.

[0062] Time synchronization processing refers to the processing of ensuring that the timestamps of all detection devices are accurately synchronized.

[0063] There are many ways to process time synchronization, which can be adjusted according to actual conditions and are not limited in the embodiments of this application.

[0064] For example, multiple detection devices use the same reference clock provided by the clock synchronization module.

[0065] The clock synchronization module is used to ensure that the timestamps of all detection devices are accurately synchronized.

[0066] Each detection device calibrates its clock time based on the reference clock, so that the timestamps of the data detected by each detection device are synchronized.

[0067] It should be noted that since the clock sources of different detection devices may be different, and each detection device operates according to its reference clock, its sampling frequency is often different. In addition, problems such as data transmission and camera exposure will cause uncontrollable delays. Therefore, multiple detection devices can use the same reference clock provided by the clock synchronization module for hardware synchronization and software synchronization to avoid affecting the accuracy of drivable area detection due to asynchrony of timestamps.

[0068] For another example, multiple detection devices are synchronously triggered by the data acquisition control module to acquire detection data.

[0069] The data acquisition control module is used to synchronously trigger and synchronously stop multiple detection device acquisition threads. That is, the data acquisition control module is used to synchronously trigger multiple detection devices to acquire detection data, and the data acquisition control module is used to synchronously stop multiple detection devices from acquiring detection data.

[0070] The specific content of the data acquisition control module can be adjusted according to actual conditions, and the embodiments of the present application do not limit it. For example, the data acquisition control module may include at least one of a keyboard for inputting data acquisition instructions, a display for displaying data acquisition logs, and a terminal multiplexer for realizing synchronous triggering of multi-sensor acquisition threads.

[0071] In some embodiments, the detection device includes a photographing device and a motion detection device, and the detection data includes an image of the environment of the target vehicle taken by the photographing device and motion information of the target vehicle acquired by the motion detection device.

[0072] The photographing device is used to take images of the environment in which the target vehicle is located. The motion detection device is used to detect the motion information of the target vehicle.

[0073] The installation location, device type, and quantity of the shooting equipment, as well as the installation location, device type, and quantity of the motion detection equipment can be set according to actual conditions and are not limited in the embodiments of the present application.

[0074] For example, the photographing device is installed on the target vehicle, or the photographing device is installed in the environment where the target vehicle is located. The motion detection device is installed on the target vehicle, or the motion detection device is installed in the environment where the target vehicle is located.

[0075] For example, photographing equipment includes fisheye cameras, visible light cameras, infrared cameras, etc. Motion detection equipment includes wheel speed meters, IMUs, steering wheels, etc.

[0076] For example, the number of photographing devices may be one, such as a camera mounted on the roof. Alternatively, the number of photographing devices may be multiple, such as cameras mounted around the vehicle body, on the top, on the bottom, and inside the vehicle. For example, the number of motion detection devices may be one, such as a wheel speed meter. Alternatively, the number of motion detection devices may be multiple, such as a wheel speed meter and an IMU.

[0077] By introducing the detection data of multiple detection devices for three-dimensional reconstruction, the vehicle's drivable area is detected in the form of a point cloud two-dimensional image, which improves the detection accuracy of the drivable area, reduces the risk of collision, improves safety, and improves the user experience. The above model can effectively solve the collision risk caused by the missed detection of special-shaped obstacles, low obstacles, etc. and inaccurate distance and height measurement of the drivable area detection model.

[0078] In some embodiments, when the detection device includes a shooting device and a motion detection device, the above process of obtaining a three-dimensional point cloud for the environment in which the target vehicle is located based on the detection data may include: determining the target extrinsic parameter information of each shooting device based on the motion information of the motion detection device and the image of each shooting device; performing three-dimensional reconstruction based on the target extrinsic parameter information and the image to obtain a three-dimensional point cloud for the environment in which the target vehicle is located.

[0079] The target external parameter information refers to the external parameter information of the shooting device when performing three-dimensional reconstruction. Three-dimensional reconstruction refers to the process of determining the three-dimensional coordinates corresponding to each pixel point on the image based on the image.

[0080] It should be noted that the external parameter information is a parameter used to describe the position and posture of the camera in the world coordinate system. The external parameter information includes a rotation matrix and a translation vector, which are used to realize the conversion of three-dimensional coordinates and two-dimensional coordinates between the world coordinate system and the camera coordinate system.

[0081] In some embodiments, the above-mentioned process of determining the target external parameter information of each shooting device based on the motion information of the motion detection device and the images of each shooting device includes: performing feature point matching on the images acquired by the same shooting device at adjacent times to determine multiple groups of feature point pairs; determining the target external parameter information of the shooting device based on the multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs.

[0082] The feature point pair refers to the matching feature points between two frames of images acquired by the same shooting device at adjacent moments.

[0083] Specifically, multiple groups of feature point pairs may be obtained by performing feature tracking on images acquired by the same photographing device at adjacent moments.

[0084] There are many ways to match feature points, which can be adjusted according to actual conditions and are not limited in the embodiments of the present application. For example, the images acquired by the same shooting device at adjacent times are tracked by optical flow method, deep learning method, feature matching algorithm (such as scale-invariant feature conversion algorithm), feature matching model, etc. to obtain multiple sets of feature point pairs.

[0085] It should be noted that, since the feature points extracted from the images captured by multiple shooting devices have at least scale invariance, rotation invariance, brightness invariance and affine invariance, and are evenly distributed in the image plane, etc. Therefore, the embodiment of the present application preferably uses the optical flow method to track the features of the images acquired by the same shooting device at adjacent times, and obtains multiple groups of feature point pairs. Among them, the optical flow method may include a sparse optical flow method and a dense optical flow method. The embodiment of the present application preferentially uses the sparse optical flow method (Kanade Lucas Tomasi, KLT) to track the features of the images acquired by the same shooting device at adjacent times, and obtains multiple groups of feature point pairs.

[0086] Furthermore, multiple constraints can be used to eliminate mismatched feature point pairs. Specifically, based on epipolar constraints, random sampling consistency constraints based on essential matrices or homography matrices, parallax constraints, etc., mismatched feature point pairs of multiple feature point pairs are determined and mismatched feature point pairs are eliminated.

[0087] In some embodiments, the process of determining the target external parameter information of the shooting device based on multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs may include: determining multiple candidate external parameter information based on multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs; determining the target external parameter information of the shooting device from the multiple candidate external parameter information.

[0088] The external parameter information of each candidate includes a rotation matrix and a translation matrix.

[0089] There are many ways to determine multiple candidate external parameter information based on feature point pairs and motion information corresponding to the feature point pairs, which are not limited in the embodiments of the present application.

[0090] For example, multiple rotation matrices between two frames of images acquired at adjacent moments are first calculated through different feature point pairs, and then the corresponding translation matrices are determined based on the motion information corresponding to the feature point pairs. For another example, multiple candidate external parameter information is determined through multiple feature point pairs, and multiple candidate external parameter information is determined through the motion information corresponding to the feature point pairs. Among them, the corresponding translation matrix determined based on the motion information corresponding to the feature point pairs can be selected according to the specific type of motion detection equipment. For example, when the motion detection equipment is a wheel speed meter, multiple candidate external parameter information can be determined through the two-wheel differential geometry model and the wheel speed meter's track estimation method.

[0091] In some embodiments, the process of determining target external parameter information of a shooting device from multiple candidate external parameter information includes: determining first residual information of the shooting device for each candidate external parameter information, and determining second residual information of the motion detection device; determining target residual information corresponding to each external parameter information based on the first residual information and the second residual information; and selecting target external parameter information from multiple candidate external parameter information based on the target residual information, wherein the target residual information corresponding to the target external parameter information is the smallest.

[0092] The residual information is used to indicate the error between the actual value detected by the detection device and the theoretical value.

[0093] The target residual information is used to characterize the accuracy of three-dimensional reconstruction based on the rotation matrix and translation matrix contained in the external parameter information. The smaller the residual information, the higher the accuracy.

[0094] Specifically, in order to further improve the reconstruction accuracy of the 3D point cloud, the residuals of multiple cameras and multiple motion sensors are constructed to form the objective function shown in the following formula (1), and a sliding window is used for local nonlinear optimization:

[0095]

[0096] Among them, r wheel refers to the residual information of the wheel speed meter, r IMU Refers to the residual information of IMU, r camera refers to the residual information of the camera, that is, the visual reprojection error, ρ refers to the Huber norm, which is used to suppress the influence of noise or mismatching to obtain higher optimization accuracy, i is used to index the camera, j is used to index the feature point pair, P represents the feature point pair, R represents the rotation matrix, t represents the translation matrix, and min is used to indicate that the external parameter information with the smallest residual information is selected from multiple candidate external parameter information as the target external parameter information.

[0097] In some embodiments, each feature point pair includes a first feature point and a second feature point, and the shooting time of the image where the first feature point is located is earlier than the shooting time of the image where the second feature point is located.

[0098] Based on this, the above-mentioned process of performing three-dimensional reconstruction based on the target external parameter information and the image to obtain a three-dimensional point cloud for the environment in which the target vehicle is located may include: obtaining a first association relationship and a second association relationship based on the target external parameter information, the first association relationship representing the relationship between the three-dimensional coordinates of the first feature point in the feature point pair, the target external parameter information, and the second feature point, and the second association relationship representing the relationship between the second feature point, the unit external parameter information, and the three-dimensional coordinates of the second feature point; calculating the three-dimensional coordinates of the second feature point in the target feature point pair of the image based on the first association relationship and the second association relationship; and obtaining a three-dimensional point cloud for the environment in which the target vehicle is located based on the three-dimensional coordinates of the second feature point in the target feature point pair.

[0099] Specifically, the first association relationship and the second association relationship are combined, and the least square method is used to realize three-dimensional reconstruction of multiple feature point pairs to obtain a three-dimensional point cloud for the environment in which the target vehicle is located.

[0100] The first association relationship is used to indicate the relationship between the three-dimensional coordinates corresponding to the feature points in the image with the earlier shooting time and the feature points in the image with the later shooting time in the adjacent images. The second association relationship is used to indicate the relationship between the three-dimensional coordinates corresponding to the feature points in the image with the later shooting time and the feature points in the image.

[0101] Specifically, the first association relationship can be expressed according to the following formula (2), and the second association relationship can be expressed according to the following formula (3):

[0102]

[0103] Among them, K is the intrinsic parameter matrix of the shooting device, λ t1 is the scale factor at time t1, λ t2 is the scale factor at time t2, (u t1 , v t1 ) is the feature point at time t1, that is, the first feature point, (u t2 , v t2 ) is the feature point at time t2, that is, the second feature point, (u t1 , v t1 ) and (u t2 , v t2 ) is a pair of feature points in the image at adjacent moments, (X t2 , Y t2 , Z t2 ) refers to the three-dimensional coordinates of the feature point pair in the observation coordinate system (ie, the camera coordinate system) at time t2, that is, the three-dimensional coordinates corresponding to the second feature point.

[0104] It should be noted that I in formula (3) is a 3×3 unit matrix, 0 is a 3D column vector whose elements are all 0, and I and 0 represent the external parameter information of the current frame image, indicating that the reference system of the restored three-dimensional coordinates is the camera coordinate system at time t2.

[0105] It should be noted that the scale factor is equal to the coordinate of the feature point on the coordinate axis that coincides with the optical axis in the respective camera coordinate system. In formula (2), the scale factor needs to use the external parameter information to transform the coordinates first, and then transform the three-dimensional coordinates to the current camera coordinate system. In formula (3), the scale factor is equal to the coordinate of the z axis in the three-dimensional coordinates.

[0106] Specifically, when the target extrinsic parameter information is determined, the target feature point pair of the calculated image is input into formula (2) and formula (3), and the above formula (2) and formula (3) are combined to realize the three-dimensional reconstruction of multiple feature point pairs using least squares to obtain the three-dimensional coordinates of the second feature point in the target feature point pair, and further obtain the three-dimensional point cloud of the environment in which the target vehicle is located.

[0107] If there are multiple shooting devices, it is necessary to construct a three-dimensional point cloud for each shooting device in the above manner, and aggregate the three-dimensional point clouds of multiple shooting devices to obtain a three-dimensional point cloud for the environment in which the target vehicle is located.

[0108] In some embodiments, the process of projecting the three-dimensional point cloud of the target vehicle's environment onto the target vehicle's top-down plane to obtain a point cloud two-dimensional image includes: determining the top-down plane based on the target vehicle's top-down panoramic image; and projecting the three-dimensional point cloud onto the top-down plane to obtain a point cloud two-dimensional image.

[0109] The bird's-eye view panoramic image refers to a panoramic image on a bird's-eye view plane centered on the vehicle, which is synthesized from images captured by multiple cameras on the vehicle at the same time.

[0110] Specifically, images taken by multiple cameras are acquired, and multiple images taken at the same time are stitched and fused to generate a bird's-eye view panoramic image.

[0111] The overhead plane indicates the plane where the overhead panoramic image is located.

[0112] By projecting the three-dimensional point cloud onto a top-down plane, a two-dimensional image of the point cloud is obtained, so that the drivable area can be detected based on the two-dimensional image of the point cloud to obtain the detection accuracy.

[0113] In some embodiments, the process of projecting a three-dimensional point cloud onto a top-down plane to obtain a two-dimensional image of the point cloud includes: projecting three-dimensional points in the three-dimensional point cloud whose height values ​​are not less than a predetermined height onto the top-down plane to determine the two-dimensional image of the point cloud.

[0114] The predetermined height refers to a height value used to distinguish obstacles from non-obstacles, and can be set specifically according to actual conditions.

[0115] The above-mentioned point cloud two-dimensional image can be expressed in a variety of forms, which can be adjusted according to actual conditions, and the embodiments of the present application do not limit it. For example, the point cloud two-dimensional image is a point cloud binary image (such as a two-dimensional image with pixel values ​​of 0 or 1). For another example, the point cloud two-dimensional image is a point cloud multi-value image (such as the depth information of the pixel value corresponding to the pixel point on the three-dimensional point cloud).

[0116] In some embodiments, the point cloud two-dimensional image is a point cloud binary image, the pixel value of the pixel point obtained by the three-dimensional point projection in the point cloud binary image is the first pixel value, the pixel value of other pixel points is the second pixel value, and the second pixel value is different from the first pixel value. The point cloud binary image represents a binary image generated according to the three-dimensional point cloud, wherein the binary value represents whether the pixel point is a three-dimensional point greater than a specified height, thereby determining whether it is drivable. The second pixel value indicates a drivable area. The first pixel value indicates a non-drivable area.

[0117] Exemplarily, if there are one or more three-dimensional points projected onto a certain pixel on the top-view plane, the pixel value at this pixel is set to 1, otherwise it is set to 0, and a point cloud binary image is obtained.

[0118] The drivable area can be directly determined through the point cloud binary image, which improves the efficiency of drivable area detection.

[0119] In some embodiments, the process of determining the drivable area of ​​the target vehicle based on the point cloud two-dimensional image includes: determining the drivable area of ​​the target vehicle based on the environment image of the target vehicle and the point cloud two-dimensional image.

[0120] Specifically, an environmental image of the environment in which the target vehicle is located is obtained; and a drivable area of ​​the target vehicle is determined based on the environmental image and the point cloud binary image.

[0121] The environmental image includes but is not limited to a depth image, a binocular parallax image, and a bird's-eye view panoramic image, which are adjusted according to actual conditions.

[0122] The drivable area is determined by combining the environment image and the point cloud binary image, thereby improving the reliability, accuracy and robustness of the detected drivable area.

[0123] In some embodiments, the above-mentioned environment image includes a bird's-eye view panoramic image of the target vehicle, and the bird's-eye view plane indicates the plane where the bird's-eye view panoramic image is located.

[0124] The above-mentioned overhead panoramic image can be provided via a panoramic image synthesis module. After multiple shooting devices acquire images, the panoramic image synthesis module synthesizes the images acquired by the multiple shooting devices and outputs a panoramic image.

[0125] Based on this, the process of determining the drivable area of ​​the target vehicle according to the environment image and the point cloud two-dimensional image of the target vehicle includes: determining the drivable area of ​​the target vehicle according to the overhead panoramic image and the point cloud two-dimensional image of the target vehicle.

[0126] Specifically, the process of determining the drivable area of ​​the target vehicle based on the overhead panoramic image and the point cloud two-dimensional image of the target vehicle includes: fusing the overhead panoramic image and the point cloud two-dimensional image to obtain a fused image; and analyzing the fused image through a drivable area detection model to obtain the drivable area of ​​the target vehicle.

[0127] The fused image refers to an image generated by fusing a bird's-eye view panoramic image and a point cloud two-dimensional image.

[0128] The drivable area detection model is used to indicate a model used for drivable area detection.

[0129] There can be many types of drivable area detection models, which can be adjusted according to actual conditions, and the embodiments of the present application do not limit this. For example, the drivable area detection model includes a neural network model. The neural network model can be a convolutional neural network model, or other types of neural network models, and the embodiments of the present application do not limit this. For example, the neural network model is a lightweight convolutional neural network, and the network consists of N encoders and decoders.

[0130] In some embodiments, the point cloud two-dimensional image can also be identified through the drivable area detection model to obtain a preliminary drivable area. Combined with the overhead panoramic image, the preliminary drivable area is adjusted to obtain a drivable area with higher accuracy. In this way, the detection accuracy of the drivable area is improved.

[0131] In some embodiments, the image resolution of the point cloud two-dimensional image is the same as the image resolution (i.e., spatial resolution) of the overhead panoramic image. It should be noted that the spatial resolution of the overhead panoramic image has been set when the multi-camera image is synthesized. According to the parameters set during synthesis, three-dimensional points greater than a specified height can be easily projected onto the two-dimensional image plane, thereby improving the efficiency of projection.

[0132] It should be noted that each pixel in the overhead panoramic image represents the information of the chromaticity and brightness of the object. Each pixel in the point cloud binary image indicates whether there is a three-dimensional point or obstacle greater than the specified height. Since the point cloud binary image is sparse, it is also necessary to fill the missing pixel values ​​by interpolation. This application uses the nearest neighbor interpolation algorithm. Then, the overhead panoramic image and the point cloud binary image are fused into a 4-channel image and input into a lightweight convolutional neural network. The network consists of N encoders and decoders. Among them, the encoder uses a series of convolutional layers, activation functions, and pooling layers for image downsampling and feature extraction; the decoder uses multiple deconvolutions to upsample the output of the encoder, and uses multiple convolutional layers and activation functions to fuse each upsampled output with the output of the corresponding encoder, which can effectively improve the detection accuracy. Finally, the model outputs a drivable area detection result that is consistent with the spatial resolution of the overhead panoramic image.

[0133] By fusing overhead panoramic images, original camera images and data from multiple motion sensors, the problem of low accuracy and robustness of the current algorithm model can be effectively solved, further ensuring the safety of the vehicle during automatic parking. In addition, it can fully utilize traditional multi-sensor 3D reconstruction to extract the prior initial drivable area, and fully combine the modern deep learning-based semantic segmentation network to further fuse multi-sensor data, thereby improving the reliability, accuracy and robustness of drivable area detection.

[0134] The following is an explanation with a specific embodiment. Figure 3 , Figure 3 is a flow chart of a drivable area detection method provided in an embodiment of the present application, combined with Figure 3 It can be seen that the drivable area detection method provided in the embodiment of the present application may include the following steps S201 to S205:

[0135] S201: Acquire a bird's-eye view panoramic image centered on the vehicle, images from multiple cameras installed on the vehicle, and data from motion sensors.

[0136] S202 , extracting feature points of multiple camera images respectively, and performing feature tracking in images of the same camera at adjacent moments to obtain multiple groups of feature point pairs.

[0137] S203 , performing motion estimation in combination with the motion sensor data, and performing three-dimensional reconstruction on multiple groups of feature point pairs to obtain a three-dimensional point cloud.

[0138] S204 , projecting the three-dimensional point cloud onto the two-dimensional image plane where the overhead panoramic image is located, to obtain a point cloud binary image with consistent spatial resolution.

[0139] S205, the overhead panoramic image and the point cloud binary image are fused and input into a pre-trained lightweight convolutional neural network to obtain a drivable area.

[0140] By inputting the overhead panoramic image, multi-camera images and motion sensor data through the above embodiment, it is possible to quickly detect the drivable area in the automatic parking environment. The three-dimensional reconstruction based on multi-sensor fusion can provide a priori sparse three-dimensional point clouds similar to pseudo-lidar. These point clouds have very accurate height and distance information. After fusing the overhead panoramic image and the point cloud binary image as the input of the lightweight convolutional neural network, a more accurate drivable area detection result can be obtained.

[0141] In order to facilitate better implementation of the drivable area detection method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the drivable area detection method. The meanings of the terms are the same as those in the drivable area detection method, and the specific implementation details can refer to the description in the method embodiment.

[0142] For example, Figure 4 As shown, the drivable area detection device may include a projection module 301 and a drivable area detection module 302, as follows:

[0143] The projection module 301 is used to project the three-dimensional point cloud of the environment where the target vehicle is located onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is obtained by the detection device of the target vehicle;

[0144] The drivable area detection module 302 is used to determine the drivable area of ​​the target vehicle according to the point cloud two-dimensional image.

[0145] In some embodiments, the projection module 301 includes:

[0146] Acquire detection data from multiple detection devices;

[0147] Based on the detection data, a three-dimensional point cloud of the target vehicle's environment is obtained.

[0148] In some embodiments, before acquiring the detection data from the plurality of detection devices, the drivable area detection device further includes:

[0149] Perform time synchronization for multiple detection devices.

[0150] In some embodiments, the detection device includes a shooting device and a motion detection device; the detection data includes: images shot by the shooting device of the target vehicle's environment, and motion information acquired by the motion detection device of the target vehicle.

[0151] In some embodiments, the three-dimensional point cloud of the target vehicle's environment is obtained based on the detection data, specifically including:

[0152] Determine target extrinsic information of each shooting device based on the motion information of the motion detection device and the images of each shooting device;

[0153] Three-dimensional reconstruction is performed based on the target external parameter information and image to obtain a three-dimensional point cloud of the environment in which the target vehicle is located.

[0154] In some embodiments, the above-mentioned determination of target extrinsic parameter information of each shooting device based on the motion information of the motion detection device and the image of each shooting device specifically includes:

[0155] Perform feature point matching on images acquired by the same shooting device at adjacent times to determine multiple groups of feature point pairs;

[0156] According to the multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs, target extrinsic parameter information of the shooting device is determined.

[0157] In some embodiments, each feature point pair includes a first feature point and a second feature point, and the shooting time of the image where the first feature point is located is earlier than the shooting time of the image where the second feature point is located.

[0158] Based on this, the above three-dimensional reconstruction is performed based on the target external parameter information and the image to obtain a three-dimensional point cloud for the environment in which the target vehicle is located, specifically including:

[0159] Acquire a first association relationship and a second association relationship based on the target external parameter information, the first association relationship representing the relationship between the first feature point in the feature point pair, the target external parameter information, and the three-dimensional coordinates of the second feature point, and the second association relationship representing the relationship between the second feature point, the unit external parameter information, and the three-dimensional coordinates of the second feature point;

[0160] Calculating the three-dimensional coordinates of the second feature point in the target feature point pair of the image according to the first association relationship and the second association relationship;

[0161] Based on the three-dimensional coordinates of the second feature point in the target feature point pair, a three-dimensional point cloud for the environment in which the target vehicle is located is obtained.

[0162] In some embodiments, the above-mentioned determining the target extrinsic parameter information of the shooting device according to the multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs specifically includes:

[0163] Determine multiple candidate external parameter information according to multiple groups of feature point pairs and motion information corresponding to the multiple groups of feature point pairs;

[0164] Determine target external parameter information of the shooting device from multiple candidate external parameter information.

[0165] In some embodiments, determining the target external parameter information of the shooting device from the plurality of candidate external parameter information specifically includes:

[0166] For each candidate external parameter information, determining first residual information of the shooting device and determining second residual information of the motion detection device;

[0167] Determine target residual information corresponding to each extrinsic parameter information based on the first residual information and the second residual information;

[0168] Based on the target residual information, the target external parameter information is selected from a plurality of candidate external parameter information, wherein the target residual information corresponding to the target external parameter information is the smallest.

[0169] In some embodiments, the above-mentioned projecting of the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud specifically includes:

[0170] Determine a bird's-eye view plane according to the bird's-eye view panoramic image of the target vehicle;

[0171] Project the three-dimensional point cloud onto the top-down plane to obtain a two-dimensional image of the point cloud.

[0172] In some embodiments, the above-mentioned projecting of the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud specifically includes:

[0173] A two-dimensional image of the point cloud is determined by projecting three-dimensional points whose height values ​​in the three-dimensional point cloud are not less than a predetermined height onto a top-view plane.

[0174] In some embodiments, the point cloud two-dimensional image is a point cloud binary image, the pixel values ​​of the pixels obtained by projecting the three-dimensional points in the point cloud binary image are first pixel values, and the pixel values ​​of other pixels are second pixel values, and the second pixel value is different from the first pixel value.

[0175] In some embodiments, determining the drivable area of ​​the target vehicle based on the point cloud two-dimensional image specifically includes:

[0176] The drivable area of ​​the target vehicle is determined based on the environment image of the target vehicle and the point cloud two-dimensional image.

[0177] In some embodiments, the above-mentioned environment image includes a bird's-eye view panoramic image.

[0178] Based on this, the above-mentioned determination of the drivable area of ​​the target vehicle based on the environment image of the target vehicle and the point cloud two-dimensional image specifically includes:

[0179] The drivable area of ​​the target vehicle is determined based on the overhead panoramic image and the point cloud two-dimensional image of the target vehicle.

[0180] In some embodiments, the above-mentioned determining the drivable area of ​​the target vehicle based on the overhead panoramic image and the point cloud two-dimensional image of the target vehicle specifically includes:

[0181] The bird's-eye view panoramic image and the point cloud two-dimensional image are fused to obtain a fused image;

[0182] The fused image is analyzed through the drivable area detection model to obtain the drivable area of ​​the target vehicle.

[0183] In some embodiments, the image resolution of the point cloud binary image is the same as the image resolution of the overhead panoramic image.

[0184] It can be seen that the drivable area detection device provided in the embodiment of the present application projects the three-dimensional point cloud of the target vehicle's environment onto the top-view plane of the target vehicle through the projection module 301 to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is obtained by the detection equipment of the target vehicle; the drivable area detection module 302 determines the drivable area of ​​the target vehicle based on the two-dimensional image of the point cloud. In this way, the three-dimensional point cloud of the vehicle's environment is projected onto the top-view plane of the vehicle, and the drivable area of ​​the vehicle is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0185] For another example, see Figure 5 The implementation of this application also provides a drivable area detection device, which includes a clock synchronization module 401, a multi-sensor data acquisition module 402, a communication module 403 and a vehicle-mounted computing power platform 404.

[0186] The clock synchronization module 401 is used to provide the same reference clock for multiple detection devices.

[0187] The clock synchronization module is used to ensure that the timestamps of all detection devices are accurately synchronized.

[0188] Each detection device calibrates its clock time based on the reference clock, so that the timestamps of the data detected by each detection device are synchronized.

[0189] It should be noted that since the clock sources of different detection devices may be different, and each detection device runs according to its own reference clock, their sampling frequencies are often different. In addition, problems such as data transmission and camera exposure will cause uncontrollable delays. Therefore, multiple detection devices can use the same reference clock provided by the clock synchronization module for hardware synchronization and software synchronization to avoid affecting the accuracy of drivable area detection due to asynchronous timestamps. For example, see Figure 6The clock synchronization module 401 includes hardware synchronization and software synchronization.

[0190] The multi-sensor data acquisition module 402 is used to acquire sensor data for realizing drivable area detection.

[0191] The multi-sensor data acquisition module 402 mainly includes four sub-modules. Figure 7 The multi-sensor data acquisition module 402 includes a data acquisition control submodule, a bird's-eye view panoramic image synthesis submodule, a multi-camera image acquisition submodule, and a motion sensor acquisition submodule. Among them, the data acquisition control submodule is used to synchronously trigger and pause the multi-sensor acquisition thread. That is, the data acquisition control submodule is used to synchronously trigger and pause multiple detection devices to acquire detection data.

[0192] The specific content of the data acquisition control submodule can be adjusted according to actual conditions, and the present application embodiment does not limit it. Figure 8 The data acquisition control submodule may include at least one of a keyboard for inputting data acquisition instructions, a display for displaying data acquisition logs, and a terminal multiplexer for realizing synchronous triggering of multi-sensor acquisition threads.

[0193] The multi-camera image acquisition submodule is used to acquire images captured by the shooting device, and the motion sensor acquisition submodule is used to acquire the motion information detected by the motion detection device. The shooting devices include fisheye cameras, visible light cameras, infrared cameras, etc., and the motion detection devices include wheel speed meters, IMUs, steering wheels, etc. The bird's-eye view panoramic image synthesis submodule is used to stitch the images captured by multiple cameras at the same time to synthesize a bird's-eye view panoramic image.

[0194] The communication module 403 is used for transmission of detection data, system status information, control instructions, etc.

[0195] The vehicle-mounted computing platform 404 is used to receive, store and process multi-sensor data. Fig. 9 As shown, the vehicle computing platform 404 mainly includes a memory, a processor, a communication interface, etc. The memory is also used to store programs, instructions, and system operation status information.

[0196] The vehicle-mounted computing platform is used to execute any of the above-mentioned drivable area detection methods. For example, the vehicle-mounted computing platform projects the three-dimensional point cloud of the target vehicle's environment onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is obtained by the detection equipment of the target vehicle; and the drivable area of ​​the target vehicle is determined based on the two-dimensional image of the point cloud.

[0197] Therefore, the drivable area detection device provided in the embodiment of the present application projects the three-dimensional point cloud of the vehicle's environment onto the bird's-eye view plane of the vehicle through a clock synchronization module, a multi-sensor data acquisition module, a communication module and a vehicle-mounted computing power platform, and detects the vehicle's drivable area in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety and improving the user experience.

[0198] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0199] The present application also provides an electronic device, such as Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0200] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more storage media, a power supply 503, and an input unit 504. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0201] in:

[0202] The processor 501 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing computer programs and / or modules stored in the memory 502, and calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0203] The memory 502 can be used to store computer programs and modules. The processor 501 executes various functional applications and drivable area detection by running the computer programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0204] The electronic device also includes a power supply 503 for supplying power to each component. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 503 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0205] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0206] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 will run the computer programs stored in the memory 502, thereby realizing various functions, such as:

[0207] Projecting a three-dimensional point cloud of the target vehicle's environment onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle;

[0208] According to the point cloud two-dimensional image, the drivable area of ​​the target vehicle is determined.

[0209] It can be seen that the electronic device provided in the embodiment of the present application obtains a two-dimensional image of the point cloud by projecting the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle, wherein the three-dimensional point cloud is obtained by the detection equipment of the target vehicle; and the drivable area of ​​the target vehicle is determined according to the two-dimensional image of the point cloud. In this way, the three-dimensional point cloud of the environment in which the vehicle is located is projected onto the top-view plane of the vehicle, and the drivable area of ​​the vehicle is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0210] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the drivable area detection method above, which will not be elaborated here.

[0211] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a storage medium and loaded and executed by a processor.

[0212] To this end, an embodiment of the present application provides a storage medium in which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any of the drivable area detection methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:

[0213] Projecting a three-dimensional point cloud of the target vehicle's environment onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle;

[0214] According to the point cloud two-dimensional image, the drivable area of ​​the target vehicle is determined.

[0215] It can be seen that the storage medium provided in the embodiment of the present application obtains a two-dimensional image of the point cloud by projecting the three-dimensional point cloud of the target vehicle's environment onto the top-down plane of the target vehicle, wherein the three-dimensional point cloud is acquired by the detection equipment of the target vehicle; based on the two-dimensional image of the point cloud, the drivable area of ​​the target vehicle is determined, and the drivable area of ​​the vehicle is detected in the form of a two-dimensional image of the point cloud, thereby improving the detection accuracy of the drivable area, reducing the risk of collision, improving safety, and improving the user experience.

[0216] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the previous embodiments and will not be described in detail here.

[0217] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0218] Since the computer program stored in the storage medium can execute the steps in any of the drivable area detection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the drivable area detection methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0219] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned drivable area detection method.

[0220] The embodiment of the present application also provides a vehicle, which includes the above-mentioned electronic device. The specific structure of the vehicle is not limited in this application. The specific implementation methods and corresponding beneficial effects of the above-mentioned operations of the electronic device are also applicable to the vehicle. For details, please refer to the detailed description of the drivable area detection method above, which will not be repeated here.

[0221] The above is a detailed introduction to a drivable area detection method, device, electronic device, storage medium and vehicle provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting a drivable area, characterized in that: The method comprises: Projecting a three-dimensional point cloud of the environment in which the target vehicle is located onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle; The drivable area of ​​the target vehicle is determined according to the point cloud two-dimensional image.

2. The drivable area detection method according to claim 1, characterized in that: The three-dimensional point cloud is obtained by a detection device of the target vehicle, including: Acquiring detection data from a plurality of detection devices; A three-dimensional point cloud of the environment in which the target vehicle is located is obtained according to the detection data.

3. The drivable area detection method according to claim 2, characterized in that: Before acquiring detection data from a plurality of detection devices, the method comprises: Time synchronization is performed on the multiple detection devices.

4. The method for detecting a drivable area according to claim 2, characterized in that: The detection device includes a shooting device and a motion detection device; The detection data includes: an image captured by the shooting device of the environment where the target vehicle is located, and motion information acquired by the motion detection device of the target vehicle.

5. The drivable area detection method according to claim 4, characterized in that: The step of obtaining a three-dimensional point cloud of the environment in which the target vehicle is located according to the detection data includes: Based on the motion information of the motion detection device and the images of each shooting device, determining target extrinsic parameter information of each shooting device; A three-dimensional reconstruction is performed based on the target external parameter information and the image to obtain a three-dimensional point cloud of the environment in which the target vehicle is located.

6. The method for detecting a drivable area according to claim 5, characterized in that: The step of determining target extrinsic information of each shooting device based on the motion information of the motion detection device and the image of each shooting device includes: Perform feature point matching on images acquired by the same shooting device at adjacent times to determine multiple groups of feature point pairs; According to the multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs, target extrinsic parameter information of the shooting device is determined.

7. The method for detecting a drivable area according to claim 6, characterized in that: Each feature point pair includes a first feature point and a second feature point, and the shooting time of the image where the first feature point is located is earlier than the shooting time of the image where the second feature point is located; the three-dimensional reconstruction is performed according to the target external parameter information and the image to obtain a three-dimensional point cloud for the environment where the target vehicle is located, including: Based on the target external parameter information, a first association relationship and a second association relationship are acquired, wherein the first association relationship represents a relationship between a first feature point in a feature point pair, the target external parameter information, and the three-dimensional coordinates of a second feature point, and the second association relationship represents a relationship between the second feature point, the unit external parameter information, and the three-dimensional coordinates of the second feature point; Calculating the three-dimensional coordinates of the second feature point in the target feature point pair of the image according to the first association relationship and the second association relationship; Based on the three-dimensional coordinates of the second feature point in the target feature point pair, a three-dimensional point cloud for the environment in which the target vehicle is located is obtained.

8. The method for detecting a drivable area according to claim 6, characterized in that: The step of determining target extrinsic parameter information of the shooting device according to the plurality of groups of feature point pairs and the motion information corresponding to the plurality of groups of feature point pairs includes: Determine multiple candidate extrinsic parameter information according to the multiple groups of feature point pairs and the motion information corresponding to the multiple groups of feature point pairs; Determine target external parameter information of the shooting device from a plurality of candidates of the external parameter information.

9. The method for detecting a drivable area according to claim 8, characterized in that: The step of determining target external parameter information of the shooting device from a plurality of candidate external parameter information includes: For each candidate external parameter information, determining first residual information of the shooting device and determining second residual information of the motion detection device; Determining target residual information corresponding to each of the extrinsic parameter information based on the first residual information and the second residual information; Based on the target residual information, target external parameter information is selected from a plurality of candidate external parameter information, wherein the target residual information corresponding to the target external parameter information is the smallest.

10. The drivable area detection method according to claim 1, characterized in that: The step of projecting the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud includes: Determining the top-view plane according to the top-view panoramic image of the target vehicle; The three-dimensional point cloud is projected onto the top-view plane to obtain a two-dimensional image of the point cloud.

11. The method for detecting a drivable area according to any one of claims 1 to 10, characterized in that: The step of projecting the three-dimensional point cloud of the environment in which the target vehicle is located onto the top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud includes: A two-dimensional image of the point cloud is determined by projecting three-dimensional points in the three-dimensional point cloud whose height values ​​are not less than a predetermined height onto the top-view plane.

12. The drivable area detection method according to claim 11, characterized in that: The point cloud two-dimensional image is a point cloud binary image, the pixel values ​​of the pixel points obtained by projecting the three-dimensional points in the point cloud binary image are first pixel values, and the pixel values ​​of other pixel points are second pixel values, and the second pixel value is different from the first pixel value.

13. The drivable area detection method according to claim 1, characterized in that: Determining the drivable area of ​​the target vehicle according to the point cloud two-dimensional image includes: The drivable area of ​​the target vehicle is determined according to the environment image of the target vehicle and the point cloud two-dimensional image.

14. The method for detecting a drivable area according to claim 13, characterized in that: The environmental image includes the overhead panoramic image, and determining the drivable area of ​​the target vehicle according to the environmental image of the target vehicle and the point cloud two-dimensional image includes: The drivable area of ​​the target vehicle is determined according to the overhead panoramic image of the target vehicle and the point cloud two-dimensional image.

15. The method for detecting a drivable area according to claim 14, characterized in that: The step of determining the drivable area of ​​the target vehicle according to the overhead panoramic image of the target vehicle and the point cloud two-dimensional image includes: Fusing the overhead panoramic image and the point cloud two-dimensional image to obtain a fused image; The fused image is analyzed by a drivable area detection model to obtain the drivable area of ​​the target vehicle.

16. The drivable area detection method according to claim 14 or 15, characterized in that: The image resolution of the point cloud two-dimensional image is the same as the image resolution of the overhead panoramic image.

17. A drivable area detection device, characterized in that: The device comprises: A projection module, used to project a three-dimensional point cloud of the environment in which the target vehicle is located onto a top-view plane of the target vehicle to obtain a two-dimensional image of the point cloud, wherein the three-dimensional point cloud is acquired by a detection device of the target vehicle; The drivable area detection module is used to determine the drivable area of ​​the target vehicle based on the point cloud two-dimensional image.

18. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the drivable area detection method according to any one of claims 1 to 16.

19. A storage medium, characterized in that: It comprises a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the drivable area detection method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the drivable area detection method according to any one of claims 1 to 16.

21. A vehicle, characterized in that: The vehicle comprises the electronic device according to claim 18, or the storage medium according to claim 19, or the computer program product according to claim 20.

Images