Automatic driving visual perception method and device, electronic equipment and readable storage medium

Abstract

The application provides an automatic driving visual perception method and device, electronic equipment and readable storage medium, wherein the method comprises: performing semantic segmentation on each surrounding environment object in the fisheye image containing the surrounding environment of the automatic driving vehicle to obtain contour points for representing the contour of each surrounding environment object; according to the camera parameters of the fisheye camera used for shooting the fisheye image, performing inverse perspective change on the contour points of each surrounding environment object to remove the contour points above the horizon in the fisheye image; for the contour points below the horizon in the fisheye image, removing the non-ground contour points in the contour points to obtain the ground contour points; according to the positions of each ground contour point in the vehicle body physical coordinate system, determining the distance information between the corresponding surrounding environment object of each ground contour point and the automatic driving vehicle. Through the method, it is beneficial to reduce the missed detection and blind area and improve the accuracy of perception.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving visual perception method, device, electronic device, and readable storage medium. Background Technology

[0002] During operation, autonomous vehicles need to perceive their surroundings to measure the distance between the vehicle and objects in the environment. Currently, autonomous driving perception often relies on LiDAR sensors, which provide distance information between the vehicle and obstacles and are often used as the primary perception sensor for autonomous driving. However, LiDAR sensors have limitations, such as the tendency to miss small objects and potential blind spots at close range. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide an autonomous driving visual perception method, device, electronic device and readable storage medium to reduce missed detections and blind spots and improve the accuracy of perception.

[0004] In a first aspect, embodiments of this application provide an autonomous driving visual perception method, the method comprising:

[0005] Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points that characterize the contours of each surrounding environment object.

[0006] Based on the camera parameters of the fisheye camera used to capture the fisheye image, inverse perspective transformation is performed on the contour points of each of the surrounding environment objects to remove contour points above the horizon in the fisheye image.

[0007] For the contour points located below the horizon in the fisheye image, remove the non-ground contour points to obtain the ground contour points.

[0008] Based on the position of each of the ground contact points in the vehicle's physical coordinate system, the distance information between the surrounding environmental objects corresponding to each ground contact point and the autonomous vehicle is determined.

[0009] In conjunction with the first aspect, embodiments of this application provide a first possible implementation of the first aspect, wherein the semantic segmentation of various surrounding environment objects in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points for characterizing the contours of each of the surrounding environment objects includes:

[0010] Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the environment around an autonomous vehicle to obtain the area occupied by each surrounding environment object in the fisheye image.

[0011] Based on the area occupied by each of the surrounding environment objects in the fisheye image, contour points are determined to characterize the contours of each of the surrounding environment objects.

[0012] In conjunction with the first aspect, this application provides a second possible implementation of the first aspect, wherein the semantic segmentation of various surrounding environment objects in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points for characterizing the contours of each of the surrounding environment objects includes:

[0013] Using an image semantic segmentation model, semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, to obtain contour points that characterize the contours of each surrounding environment object and the object category to which each surrounding environment object belongs.

[0014] After determining the distance information between the surrounding environmental objects and the autonomous vehicle corresponding to each of the landing contour points based on their positions in the vehicle's physical coordinate system, the method further includes:

[0015] Based on the distance information between the surrounding environment objects corresponding to each landing contour point and the autonomous vehicle, and the object category to which the surrounding environment objects corresponding to each landing contour point belong, the autonomous vehicle is controlled to perform obstacle avoidance driving.

[0016] In conjunction with the first aspect, this application provides a third possible implementation of the first aspect, wherein performing inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to remove contour points above the horizon in the fisheye image, includes:

[0017] Based on the camera parameters of the fisheye camera used to capture the fisheye image, the contour points of each of the surrounding environment objects are subjected to inverse perspective transformation to obtain the first coordinates of each contour point in the vehicle body physical coordinate system.

[0018] Based on the first coordinates corresponding to each of the contour points, remove the contour points located above the horizon in the fisheye image.

[0019] In conjunction with the third possible implementation of the first aspect, this application provides a fourth possible implementation of the first aspect, wherein the step of performing inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to obtain the first coordinates of each contour point in the vehicle body physical coordinate system, includes:

[0020] The following formula is used to perform inverse perspective transformation on the outline points of each of the surrounding environment objects:

[0021]

[0022] Where (u,v) represents the coordinates of the contour point in the fisheye image; (X,Y) represents the first coordinates of the contour point in the vehicle's physical coordinate system; K is the camera parameter; R cv T cv Represent the rotation matrix and translation vector from the autonomous vehicle to the fisheye camera; [] col:1,2,4 This indicates taking the 1st, 2nd, and 4th columns of the matrix; λ is a scalar in the calculation process;

[0023] Each first coordinate corresponds to a λ; the step of removing contour points above the horizon in the fisheye image based on the first coordinates corresponding to each contour point includes:

[0024] Based on the λ corresponding to the first coordinate of each contour point, remove the contour points corresponding to the first coordinate where λ is negative.

[0025] In conjunction with the third or fourth possible implementation of the first aspect, this application provides a fifth possible implementation of the first aspect, wherein removing non-ground-contact contour points from the contour points located below the horizon in the fisheye image to obtain ground-contact contour points includes:

[0026] For each contour point located below the horizon in the fisheye image, the first coordinate of the contour point in the vehicle body physical coordinate system is transformed into the second coordinate in the polar coordinate system.

[0027] The target polar angle is determined from all polar angles according to a preset angle interval; wherein, two adjacent target polar angles differ by the preset angle interval.

[0028] For each target polar angle, the polar angle is selected from all the second coordinates as the third coordinate of the target polar angle, and the third coordinate with the smallest polar radius is selected from all the third coordinates as the target coordinate corresponding to the target polar angle;

[0029] Among the contour points located below the horizon in the fisheye image, the contour point corresponding to the target coordinates is determined as the landing contour point.

[0030] In conjunction with the first aspect, this application provides a sixth possible implementation of the first aspect, wherein the autonomous vehicle is equipped with a plurality of fisheye cameras; different fisheye cameras are respectively arranged around the autonomous vehicle.

[0031] Secondly, embodiments of this application also provide an autonomous driving visual perception device, comprising:

[0032] The semantic segmentation module is used to perform semantic segmentation on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, and to obtain contour points that characterize the contours of each of the surrounding environment objects.

[0033] The transformation module is used to perform inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, so as to remove the contour points above the horizon in the fisheye image.

[0034] The removal module is used to remove non-ground contour points from contour points located below the horizon in the fisheye image to obtain ground contour points.

[0035] The determination module is used to determine the distance information between the surrounding environmental objects corresponding to each of the ground contact points and the autonomous vehicle based on the position of each ground contact point in the vehicle's physical coordinate system.

[0036] In conjunction with the second aspect, this application provides a first possible implementation of the second aspect, wherein the semantic segmentation module, when performing semantic segmentation on various surrounding environment objects in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points characterizing the contours of each of the surrounding environment objects, is specifically used for:

[0037] Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the environment around an autonomous vehicle to obtain the area occupied by each surrounding environment object in the fisheye image.

[0038] Based on the area occupied by each of the surrounding environment objects in the fisheye image, contour points are determined to characterize the contours of each of the surrounding environment objects.

[0039] In conjunction with the second aspect, this application provides a second possible implementation of the second aspect, wherein the semantic segmentation module, when performing semantic segmentation on various surrounding environment objects in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points characterizing the contours of each of the surrounding environment objects, is specifically used for:

[0040] Using an image semantic segmentation model, semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, to obtain contour points that characterize the contours of each surrounding environment object and the object category to which each surrounding environment object belongs.

[0041] Also includes:

[0042] The control module is used to control the autonomous vehicle to perform obstacle avoidance driving based on the distance information between the surrounding environmental objects corresponding to each landing contour point and the autonomous vehicle, after the determining module determines the distance information between the surrounding environmental objects corresponding to each landing contour point and the autonomous vehicle, and the object category to which the surrounding environmental objects corresponding to each landing contour point belong.

[0043] In conjunction with the second aspect, this application provides a third possible implementation of the second aspect, wherein the transformation module, when performing inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to remove contour points above the horizon in the fisheye image, is specifically used for:

[0044] Based on the camera parameters of the fisheye camera used to capture the fisheye image, the contour points of each of the surrounding environment objects are subjected to inverse perspective transformation to obtain the first coordinates of each contour point in the vehicle body physical coordinate system.

[0045] Based on the first coordinates corresponding to each of the contour points, remove the contour points located above the horizon in the fisheye image.

[0046] In conjunction with the third possible implementation of the second aspect, this application provides a fourth possible implementation of the second aspect, wherein the transformation module, when performing inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to obtain the first coordinates of each contour point in the vehicle body physical coordinate system, is specifically used for:

[0047] The following formula is used to perform inverse perspective transformation on the outline points of each of the surrounding environment objects:

[0048]

[0049] Where (u,v) represents the coordinates of the contour point in the fisheye image; (X,Y) represents the first coordinates of the contour point in the vehicle's physical coordinate system; K is the camera parameter; R cv T cv Represent the rotation matrix and translation vector from the autonomous vehicle to the fisheye camera; [] col:1,2,4 This indicates taking the 1st, 2nd, and 4th columns of the matrix; λ is a scalar in the calculation process;

[0050] Each first coordinate corresponds to a λ; when the transformation module removes contour points above the horizon in the fisheye image based on the first coordinates corresponding to each contour point, it is specifically used for:

[0051] Based on the λ corresponding to the first coordinate of each contour point, remove the contour points corresponding to the first coordinate where λ is negative.

[0052] In conjunction with the third or fourth possible implementation of the second aspect, this application provides a fifth possible implementation of the second aspect, wherein the removal module, when used to remove non-ground-contact contour points from contour points located below the horizon in the fisheye image to obtain ground-contact contour points, is specifically used for:

[0053] For each contour point located below the horizon in the fisheye image, the first coordinate of the contour point in the vehicle body physical coordinate system is transformed into the second coordinate in the polar coordinate system.

[0054] The target polar angle is determined from all polar angles according to a preset angle interval; wherein, two adjacent target polar angles differ by the preset angle interval.

[0055] For each target polar angle, the polar angle is selected from all the second coordinates as the third coordinate of the target polar angle, and the third coordinate with the smallest polar radius is selected from all the third coordinates as the target coordinate corresponding to the target polar angle;

[0056] Among the contour points located below the horizon in the fisheye image, the contour point corresponding to the target coordinates is determined as the landing contour point.

[0057] In conjunction with the second aspect, this application provides a sixth possible implementation of the second aspect, wherein the autonomous vehicle is equipped with a plurality of fisheye cameras; different fisheye cameras are respectively arranged around the autonomous vehicle.

[0058] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in any of the possible implementations of the first aspect described above are performed.

[0059] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps in any of the possible implementations of the first aspect described above.

[0060] This application provides an autonomous driving visual perception method, device, electronic device, and readable storage medium. It captures images of the surrounding environment of an autonomous vehicle using a fisheye camera, then performs semantic segmentation on the surrounding objects within the fisheye images to obtain contour points representing the contours of each object. Contour points above the horizon are progressively removed, and non-grounded contour points are further removed to obtain grounded contour points. Based on the positions of each grounded contour point in the vehicle's physical coordinate system, the distance information between the corresponding surrounding objects and the autonomous vehicle is determined. In this embodiment, the large field of view of the fisheye camera helps reduce blind spots and missed detections. Furthermore, progressively removing contour points above the horizon and non-grounded contour points reduces interference from these contour points, thereby improving the accuracy of distance perception between surrounding objects and the autonomous vehicle.

[0061] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart of an autonomous driving visual perception method provided in an embodiment of this application is shown;

[0064] Figure 2 This illustration shows a schematic diagram of semantic segmentation of a fisheye image according to an embodiment of this application;

[0065] Figure 3 This illustration shows a schematic diagram of a non-grounding contour point provided in an embodiment of this application;

[0066] Figure 4 A schematic diagram of a vehicle body physical coordinate system provided in an embodiment of this application is shown;

[0067] Figure 5 A schematic diagram of a polar coordinate system provided in an embodiment of this application is shown;

[0068] Figure 6 This paper shows a schematic diagram of the structure of an autonomous driving visual perception device provided in an embodiment of this application;

[0069] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0071] During operation, autonomous vehicles need to perceive their surroundings to measure the distance between the vehicle and objects in the environment. Currently, autonomous driving perception often relies on LiDAR sensors, which provide distance information between the vehicle and obstacles and are often used as the primary perception sensor for autonomous driving. However, LiDAR sensors have limitations, such as limited semantic information (they can only perceive the distance to obstacles, not the type of obstacle), frequent missed detections of small objects, and potential blind spots at close range.

[0072] Furthermore, besides sensing the surrounding environment through LiDAR sensors, perception can also be achieved through visual 2D object detection (i.e., planar images). Currently, when obtaining depth information (i.e., distance information between the vehicle and surrounding objects) from planar images, depth estimation can be performed using deep learning models, or obstacle depth (distance) information can be obtained by performing inverse perspective transformation on the landing points. However, deep learning estimation methods are hardware-intensive, time-consuming, and have low accuracy. Using landing points for perspective transformation often only labels and regresses one or two landing points on the same obstacle, only obtaining distance information for approximately one or two points on the obstacle, resulting in inaccurate distance estimation.

[0073] In view of the above problems, this application provides an autonomous driving visual perception method, device, electronic device and readable storage medium to reduce missed detections and blind spots and improve the accuracy of perception. The following is a description through embodiments.

[0074] Example 1:

[0075] To facilitate understanding of this embodiment, a detailed description of an autonomous driving visual perception method disclosed in this application embodiment will be provided first. Figure 1 A flowchart of an autonomous driving visual perception method provided in an embodiment of this application is shown, as follows: Figure 1 As shown, this method is applied to autonomous vehicles, and includes the following steps S101-S104:

[0076] S101: Perform semantic segmentation on each surrounding environment object in the fisheye image containing the surrounding environment of the autonomous vehicle to obtain contour points used to represent the contours of each surrounding environment object.

[0077] In this embodiment, the autonomous vehicle is specifically a port autonomous vehicle, meaning it operates within a port environment containing objects such as docks and containers. The autonomous vehicle frequently needs to pass through narrow docks or get very close to containers. Specifically, the autonomous vehicle can be a container truck or flatbed truck used for transporting goods in the port.

[0078] In one possible implementation, the autonomous vehicle is equipped with multiple fisheye cameras; different fisheye cameras are positioned around the autonomous vehicle.

[0079] For example, an autonomous vehicle is equipped with four fisheye cameras, which are respectively positioned in the front, rear, left, and right directions of the autonomous vehicle. In this embodiment, the fisheye cameras can be wide-angle fisheye cameras, and the field of view of the captured images can reach 190 degrees. Therefore, through these four fisheye cameras, the surrounding environment of the autonomous vehicle can be captured in 360 degrees.

[0080] In this embodiment, for any fisheye camera, a fisheye image captured by the camera is acquired. The fisheye image contains the surrounding environment of the autonomous vehicle from the perspective of the fisheye camera. The fisheye image is then input into an image semantic segmentation model. Through the image semantic segmentation model, semantic segmentation is performed on each surrounding environment object in the fisheye image to obtain contour points used to represent the contours of each surrounding environment object.

[0081] The surrounding environment includes objects such as the standing position, containers, trees, pools, and the ground. Each surrounding environment object corresponds to multiple contour points, and these contour points are used to depict the contour of that object.

[0082] In the technical solution of this application, by segmenting only the contour points of each surrounding environment object, compared with extracting all the pixels of each surrounding environment object, it is beneficial to reduce the amount of subsequent data processing.

[0083] In one possible implementation, when performing step S101, the following steps may be performed: semantic segmentation of each surrounding environment object in the fisheye image containing the surrounding environment of the autonomous vehicle to obtain the area occupied by each surrounding environment object in the fisheye image; and determining contour points to characterize the contours of each surrounding environment object based on the area occupied by each surrounding environment object in the fisheye image.

[0084] In this embodiment, the fisheye camera is input into an image semantic segmentation model. The model performs semantic segmentation on each surrounding object in the fisheye image, outputting the area occupied by each object in the fisheye image and the category to which each object belongs. The category can be the name or identifier of the object.

[0085] In this embodiment, different colors can be used to mark the areas occupied by different surrounding objects in the fisheye image. For example, green can be used to mark the area occupied by the object in the fisheye image.

[0086] For example, Figure 2 This illustration shows a schematic diagram of semantic segmentation of a fisheye image according to an embodiment of this application, such as... Figure 2 As shown, different objects in the surrounding environment in the fisheye image are marked using different grayscale colors.

[0087] S102: Based on the camera parameters of the fisheye camera used to capture the fisheye image, perform inverse perspective transformation on the outline points of each surrounding object to remove outline points above the horizon in the fisheye image.

[0088] Specifically, the camera parameters can be the camera's intrinsic parameters. The horizon is the horizon in the fisheye image.

[0089] In one possible implementation, when performing step S102, specifically, the contour points of each surrounding environment object can be subjected to inverse perspective transformation based on the camera parameters of the fisheye camera used to capture the fisheye image, so as to obtain the first coordinates of each contour point in the vehicle body physical coordinate system; based on the first coordinates corresponding to each contour point, the contour points located above the horizon in the fisheye image are removed.

[0090] The vehicle body physical coordinate system is actually a coordinate system with the autonomous vehicle as the origin, the autonomous vehicle's driving direction as the x-axis, and the autonomous vehicle's left-side direction as the y-axis. The vehicle body physical coordinate system is a coordinate system viewed from above.

[0091] In this embodiment, the inverse perspective transformation of the outline points of each surrounding object can be performed using the following formula:

[0092]

[0093] Where (u,v) represents the coordinates of the contour point in the fisheye image; (X,Y) represents the first coordinate of the contour point in the vehicle's physical coordinate system; K is the camera parameter; R cv T cv Represent the rotation matrix and translation vector from the autonomous vehicle to the fisheye camera; [] col:1,2,4 This indicates taking the 1st, 2nd, and 4th columns of the matrix; λ is a scalar in the calculation process; each first coordinate corresponds to one λ.

[0094] Based on the first coordinate corresponding to each contour point and its corresponding λ, contour points corresponding to first coordinates with negative λ are removed. In this embodiment, contour points corresponding to first coordinates with negative λ (i.e., λ is less than 0) are contour points located above the horizon in the fisheye image.

[0095] S103: For contour points located below the horizon in the fisheye image, remove the non-ground contour points to obtain ground contour points.

[0096] In this embodiment, Figure 3 This application provides a schematic diagram of a non-grounding contour point according to an embodiment of the present application. Figure 3 As shown, Figure 3 The white dot represents one of the non-grounded contour points. This non-grounded contour point is actually a contour point on the column, meaning it is not actually located on the ground. Therefore, it is a non-grounded contour point. A grounded contour point is a contour point located on the ground, such as the part at the bottom of the column that connects to the ground.

[0097] In this application, such as Figure 3 As shown, if the non-ground contour point is not removed, it will affect the position of the pillar in the vehicle's physical coordinate system. Figure 4 A schematic diagram of a vehicle body physical coordinate system provided in an embodiment of this application is shown, as follows: Figure 4 As shown, the actual position of the pillar in the vehicle's physical coordinate system is position A. However, the aforementioned non-grounded contour point, after being converted to the vehicle's physical coordinate system, may be located at position B. This non-grounded contour point is actually a point on the pillar. Therefore, if this non-grounded contour point is not removed, it will affect the pillar's position in the vehicle's physical coordinate system. In this application, removing the non-grounded contour point helps improve the accuracy of the positions of various surrounding objects in the vehicle's physical coordinate system, thereby improving the accuracy of distance perception.

[0098] In one possible implementation, when performing step S103, the following steps S1031-S1034 can be specifically performed:

[0099] S1031: For each contour point located below the horizon in the fisheye image, convert the first coordinate of the contour point in the vehicle body physical coordinate system into the second coordinate in the polar coordinate system.

[0100] In this embodiment, the first coordinates (X,Y) of the contour points in the vehicle's physical coordinate system can be converted into the second coordinates (r,θ) in the polar coordinate system using the following formula:

[0101]

[0102]

[0103] The origin of the polar coordinate system is the same as the origin of the vehicle's physical coordinate system, both being any point on the autonomous vehicle. The polar axis of the polar coordinate system is parallel to the y-axis of the vehicle's physical coordinate system, but in the opposite direction.

[0104] S1032: Determine the target polar angle from all polar angles according to the preset angle interval; wherein, the two adjacent target polar angles differ by the preset angle interval.

[0105] The preset angle interval can be 1 degree, 2 degrees, 5 degrees, etc., and this application does not limit it. For example, when the preset angle interval is 2 degrees, the target polar angle can include 1 degree, 3 degrees, 5 degrees, etc.

[0106] S1033: For each target polar angle, select the polar angle from all the second coordinates as the third coordinate of the target polar angle, and select the third coordinate with the smallest polar radius from all the third coordinates as the target coordinate corresponding to the target polar angle.

[0107] Figure 5 A schematic diagram of a polar coordinate system provided in an embodiment of this application is shown, as follows: Figure 5 As shown, when the target polar angle is Figure 5 When the target polar angle is θ1, the target polar angle corresponds to two second coordinates (r1, θ1) and (r2, θ1). These two second coordinates are used as the third coordinates corresponding to the target polar angle. From these two third coordinates, the third coordinate with the smallest polar radius is selected, where r1 is less than r2. Therefore, the selected target coordinates are (r1, θ1).

[0108] S1034: Among the contour points located below the horizon in the fisheye image, the contour point corresponding to the target coordinates is determined as the landing contour point.

[0109] In this embodiment, there are multiple target coordinates, and each target coordinate corresponds to a landing contour point.

[0110] S104: Based on the position of each ground contact point in the vehicle's physical coordinate system, determine the distance information between the surrounding environment objects corresponding to each ground contact point and the autonomous vehicle.

[0111] Based on the distance between the position coordinates (i.e. target coordinates) of each landing contour point in the vehicle's physical coordinate system and the origin of the vehicle's physical coordinate system (autonomous vehicle), the distance information between the surrounding environmental objects corresponding to each landing contour point and the autonomous vehicle is determined.

[0112] In one possible implementation, when performing step S101, an image semantic segmentation model can be used to perform semantic segmentation on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, to obtain contour points that characterize the contours of each surrounding environment object and the object category to which each surrounding environment object belongs.

[0113] After executing step S104, the autonomous vehicle can be controlled to avoid obstacles based on the distance information between the surrounding environment objects corresponding to each landing contour point and the autonomous vehicle, as well as the object category to which the surrounding environment objects corresponding to each landing contour point belong.

[0114] Example 2:

[0115] Based on the same technical concept, this application also provides an autonomous driving visual perception device. Figure 6 This application provides a schematic diagram of the structure of an autonomous driving visual perception device according to an embodiment of the present application. Figure 6 As shown, the device includes:

[0116] The semantic segmentation module 601 is used to perform semantic segmentation on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, and obtain contour points to characterize the contours of each of the surrounding environment objects.

[0117] The transformation module 602 is used to perform inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, so as to remove the contour points located above the horizon in the fisheye image.

[0118] The removal module 603 is used to remove non-ground contour points from contour points located below the horizon in the fisheye image to obtain ground contour points.

[0119] The determination module 604 is used to determine the distance information between the surrounding environmental objects corresponding to each of the ground contact points and the autonomous vehicle based on the position of each of the ground contact points in the vehicle's physical coordinate system.

[0120] Optionally, when the semantic segmentation module 601 performs semantic segmentation on various surrounding environment objects in a fisheye image containing the surrounding environment of the autonomous vehicle to obtain contour points representing the contours of each of the surrounding environment objects, it is specifically used for:

[0121] Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the environment around an autonomous vehicle to obtain the area occupied by each surrounding environment object in the fisheye image.

[0122] Based on the area occupied by each of the surrounding environment objects in the fisheye image, contour points are determined to characterize the contours of each of the surrounding environment objects.

[0123] Optionally, when the semantic segmentation module 601 performs semantic segmentation on various surrounding environment objects in a fisheye image containing the surrounding environment of the autonomous vehicle to obtain contour points representing the contours of each of the surrounding environment objects, it is specifically used for:

[0124] Using an image semantic segmentation model, semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, to obtain contour points that characterize the contours of each surrounding environment object and the object category to which each surrounding environment object belongs.

[0125] Also includes:

[0126] The control module is used to control the autonomous vehicle to perform obstacle avoidance driving based on the distance information between the surrounding environmental objects corresponding to each landing contour point and the autonomous vehicle, after the determining module 604 determines the distance information between the surrounding environmental objects corresponding to each landing contour point and the autonomous vehicle, and the object category to which the surrounding environmental objects corresponding to each landing contour point belong.

[0127] Optionally, when the transformation module 602 performs inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, in order to remove contour points above the horizon in the fisheye image, it is specifically used for:

[0128] Based on the camera parameters of the fisheye camera used to capture the fisheye image, the contour points of each of the surrounding environment objects are subjected to inverse perspective transformation to obtain the first coordinates of each contour point in the vehicle body physical coordinate system.

[0129] Based on the first coordinates corresponding to each of the contour points, remove the contour points located above the horizon in the fisheye image.

[0130] Optionally, when the transformation module 602 performs inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to obtain the first coordinates of each contour point in the vehicle body physical coordinate system, it is specifically used for:

[0131] The following formula is used to perform inverse perspective transformation on the outline points of each of the surrounding environment objects:

[0132]

[0133] Where (u,v) represents the coordinates of the contour point in the fisheye image; (X,Y) represents the first coordinates of the contour point in the vehicle's physical coordinate system; K is the camera parameter; R cv T cv Represent the rotation matrix and translation vector from the autonomous vehicle to the fisheye camera; [] col:1,2,4 This indicates taking the 1st, 2nd, and 4th columns of the matrix; λ is a scalar in the calculation process;

[0134] Each first coordinate corresponds to a λ; when the transformation module 602 removes contour points above the horizon in the fisheye image based on the first coordinates corresponding to each contour point, it is specifically used for:

[0135] Based on the λ corresponding to the first coordinate of each contour point, remove the contour points corresponding to the first coordinate where λ is negative.

[0136] Optionally, when the removal module 603 removes non-ground-contact contour points from contour points located below the horizon in the fisheye image to obtain ground-contact contour points, it is specifically used for:

[0137] For each contour point located below the horizon in the fisheye image, the first coordinate of the contour point in the vehicle body physical coordinate system is transformed into the second coordinate in the polar coordinate system.

[0138] The target polar angle is determined from all polar angles according to a preset angle interval; wherein, two adjacent target polar angles differ by the preset angle interval.

[0139] For each target polar angle, the polar angle is selected from all the second coordinates as the third coordinate of the target polar angle, and the third coordinate with the smallest polar radius is selected from all the third coordinates as the target coordinate corresponding to the target polar angle;

[0140] Among the contour points located below the horizon in the fisheye image, the contour point corresponding to the target coordinates is determined as the landing contour point.

[0141] Example 3:

[0142] Figure 7 A schematic diagram of an electronic device provided in this application embodiment includes: a processor 701, a memory 702, and a bus 703. The memory 702 stores machine-readable instructions executable by the processor 701. When the electronic device runs the above-described information processing method, the processor 701 and the memory 702 communicate through the bus 703. The processor 701 executes the machine-readable instructions to perform the steps of the method described in Embodiment 1.

[0143] Example 4:

[0144] Embodiment 4 of this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps described in Embodiment 1.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, electronic devices, and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0149] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. An autonomous driving visual perception method, characterized in that, The method includes: Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle to obtain contour points that characterize the contours of each surrounding environment object. Based on the camera parameters of the fisheye camera used to capture the fisheye image, inverse perspective transformation is performed on the contour points of each of the surrounding environment objects to obtain the first coordinates of each contour point in the vehicle body physical coordinate system. Based on the first coordinates corresponding to each of the contour points, remove the contour points located above the horizon in the fisheye image; For each contour point located below the horizon in the fisheye image, the first coordinate of the contour point in the vehicle body physical coordinate system is transformed into the second coordinate in the polar coordinate system. The target polar angle is determined from all polar angles according to a preset angle interval; wherein, two adjacent target polar angles differ by the preset angle interval. For each target polar angle, the polar angle is selected from all the second coordinates as the third coordinate of the target polar angle, and the third coordinate with the smallest polar radius is selected from all the third coordinates as the target coordinate corresponding to the target polar angle; Among the contour points located below the horizon in the fisheye image, the contour point corresponding to the target coordinates is determined as the landing contour point; Based on the position of each of the ground contact points in the vehicle's physical coordinate system, the distance information between the surrounding environmental objects corresponding to each ground contact point and the autonomous vehicle is determined.

2. The method according to claim 1, characterized in that, The step of semantically segmenting the various surrounding objects in a fisheye image containing the environment around the autonomous vehicle to obtain contour points representing the contours of each of the surrounding objects includes: Semantic segmentation is performed on each surrounding environment object in a fisheye image containing the environment around an autonomous vehicle to obtain the area occupied by each surrounding environment object in the fisheye image. Based on the area occupied by each of the surrounding environment objects in the fisheye image, contour points are determined to characterize the contours of each of the surrounding environment objects.

3. The method according to claim 1, characterized in that, The step of semantically segmenting the various surrounding objects in a fisheye image containing the environment around the autonomous vehicle to obtain contour points representing the contours of each of the surrounding objects includes: Using an image semantic segmentation model, semantic segmentation is performed on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, to obtain contour points that characterize the contours of each surrounding environment object and the object category to which each surrounding environment object belongs. After determining the distance information between the surrounding environmental objects and the autonomous vehicle corresponding to each of the landing contour points based on their positions in the vehicle's physical coordinate system, the method further includes: Based on the distance information between the surrounding environment objects corresponding to each landing contour point and the autonomous vehicle, and the object category to which the surrounding environment objects corresponding to each landing contour point belong, the autonomous vehicle is controlled to perform obstacle avoidance driving.

4. The method according to claim 1, characterized in that, The step of performing inverse perspective transformation on the contour points of each of the surrounding environment objects based on the camera parameters of the fisheye camera used to capture the fisheye image, to obtain the first coordinates of each contour point in the vehicle's physical coordinate system, includes: The following formula is used to perform inverse perspective transformation on the outline points of each of the surrounding environment objects: Where (u,v) represents the coordinates of the contour point in the fisheye image; (X,Y) represents the first coordinates of the contour point in the vehicle's physical coordinate system; K is the camera parameter; Represents the rotation matrix and translation vector from the autonomous vehicle to the fisheye camera; This indicates taking the 1st, 2nd, and 4th columns of the matrix; For scalars in the calculation process; Each first coordinate corresponds to one The step of removing contour points above the horizon in the fisheye image based on the first coordinates corresponding to each contour point includes: According to the first coordinates corresponding to each of the contour points Remove The contour point corresponding to the first coordinate that is negative.

5. The method according to claim 1, characterized in that, The autonomous vehicle is equipped with multiple fisheye cameras; different fisheye cameras are respectively located around the autonomous vehicle.

6. An autonomous driving visual perception device, characterized in that, include: The semantic segmentation module is used to perform semantic segmentation on each surrounding environment object in a fisheye image containing the surrounding environment of an autonomous vehicle, and to obtain contour points that characterize the contours of each of the surrounding environment objects. The transformation module is used to perform inverse perspective transformation on the contour points of each of the surrounding environment objects according to the camera parameters of the fisheye camera used to capture the fisheye image, to obtain the first coordinates of each contour point in the vehicle body physical coordinate system; and to remove the contour points above the horizon in the fisheye image according to the first coordinates corresponding to each contour point. The removal module is used to convert the first coordinates of each contour point located below the horizon in the fisheye image into a second coordinate in the vehicle's physical coordinate system; determine the target polar angle from all polar angles according to a preset angle interval; wherein two adjacent target polar angles differ by the preset angle interval; for each target polar angle, select the polar angle from all the second coordinates as the third coordinate of the target polar angle, and select the third coordinate with the smallest polar radius from all the third coordinates as the target coordinate corresponding to the target polar angle; and determine the contour point corresponding to the target coordinate among the contour points located below the horizon in the fisheye image as the ground contact contour point; The determination module is used to determine the distance information between the surrounding environmental objects corresponding to each of the ground contact points and the autonomous vehicle based on the position of each ground contact point in the vehicle's physical coordinate system.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 5.

Images