Fusion calculation-based obstacle identification method, apparatus and device, and storage medium

Abstract

The embodiment of the invention discloses an obstacle recognition method and device based on fusion calculation, equipment and a storage medium, and relates to the technical field of equipment control. The method comprises the following steps: obtaining point cloud data through a radar detector, and obtaining image data through an image collector; performing semantic segmentation on the image data to obtain semantic information of each pixel point in the image data; and projecting the point cloud data to the image data with the semantic information, and identifying an obstacle in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point. According to the technical scheme, the point cloud data acquired by the radar detector can be projected to the image data with the semantic information to identify whether the obstacle exists in the current environment of the unmanned equipment, so that the unmanned equipment can more comprehensively and reliably perceive the obstacle, and the accuracy of obstacle identification is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of equipment control technology, and in particular to an obstacle recognition method, apparatus, device and storage medium based on fusion calculation. Background Art

[0002] The unmanned equipment market has grown rapidly in recent years, and obstacle avoidance technology, as a guarantee for the safe operation of unmanned equipment, has also been continuously upgraded and improved. During operation, unmanned equipment uses its sensors to collect information about the surrounding environment and measure distances, thereby generating corresponding action instructions to achieve the purpose of obstacle avoidance.

[0003] In the related technologies, taking radar detection technology as an example, the use of millimeter-wave radar to sense obstacles has the advantages of short wavelength and wide bandwidth. However, for the perception of slender objects such as wires, the radar is most sensitive to the part of the wire facing it, and the wires on both sides will reflect the waves emitted by the radar to the left and right. The return signal received by the radar is relatively weak and will be filtered out. Therefore, it is not easy for the radar to identify the boundary information of the wires, nor can it distinguish them from other obstacles. The planner of unmanned equipment will plan the left and right obstacle-avoiding flight routes by default, and aircraft explosion accidents often occur.

[0004] Therefore, how to make improvements based on radar detection so that unmanned equipment can identify obstacles more comprehensively and accurately is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The embodiments of the present invention provide an obstacle recognition method, device, equipment and storage medium based on fusion calculation. By projecting the point cloud data obtained by the radar detector into image data with semantic information, it is possible to identify whether there are obstacles in the current environment of the unmanned equipment, so that the unmanned equipment's perception of obstacles is more comprehensive and reliable, and the accuracy of obstacle recognition is effectively improved.

[0006] In a first aspect, an embodiment of the present invention provides an obstacle recognition method based on fusion calculation, the method comprising:

[0007] Acquire point cloud data through a radar detector, and acquire image data through an image collector;

[0008] Performing semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0009] The point cloud data is projected into image data with the semantic information, and obstacles ahead of the unmanned equipment are identified based on the semantic information corresponding to the projection points of the radar points.

[0010] In a second aspect, an embodiment of the present invention further provides an obstacle recognition device based on fusion calculation, the device comprising:

[0011] A data acquisition module, used to acquire point cloud data through a radar detector and to acquire image data through an image collector;

[0012] A semantic information acquisition module, used to perform semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0013] The obstacle determination module is used to project the point cloud data into image data with the semantic information, and identify obstacles in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point.

[0014] In a third aspect, an embodiment of the present invention further provides an unmanned device, which is configured with an image collector and a radar detector, and includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the obstacle recognition method based on fusion calculation described in an embodiment of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the obstacle recognition method based on fusion calculation described in an embodiment of the present invention.

[0016] In an embodiment of the present invention, point cloud data is acquired through a radar detector, and image data is acquired through an image collector; semantic segmentation is performed on the image data to obtain semantic information of each pixel in the image data; the point cloud data is projected into image data with the semantic information, and obstacles in front of the unmanned device are identified based on the semantic information corresponding to the projection point of the radar point. Through the above-mentioned obstacle identification method based on fusion calculation, it is possible to identify whether there are obstacles in the current environment of the unmanned device by projecting the point cloud data acquired by the radar detector into the image data with semantic information, so that the perception of obstacles by the unmanned device is more comprehensive and reliable, and the accuracy of obstacle identification is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of an obstacle recognition method based on fusion calculation provided in the first embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a flow chart of an obstacle recognition method based on fusion calculation provided in the second embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a flow chart of determining an effective obstacle extension direction provided in Embodiment 3 of the present invention;

[0020] Figure 4 is a flow chart of a normalized plane conversion method provided in Embodiment 4 of the present application;

[0021] Figure 5 : is a flowchart of a method for verifying the extension direction of a valid obstacle provided in Embodiment 5 of the present application;

[0022] Figure 6 A schematic diagram of a process for confirming effective extension points provided in Embodiment 6 of the present invention;

[0023] Figure 7 is a flow chart of an obstacle identification method based on fusion calculation provided in Embodiment 7 of the present application;

[0024] Figure 8 is a schematic diagram of a process for determining a 3D image of an obstacle provided in Embodiment 7 of the present application;

[0025] Fig. 9 is a schematic diagram of the structure of an obstacle recognition device based on fusion calculation provided in Embodiment 8 of the present application;

[0026] Fig.10 A schematic diagram of the structure of an unmanned device provided in Embodiment 9 of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The obstacle recognition method based on fusion calculation provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Embodiment 1

[0031] Figure 1 FIG. 1 is a flow chart of an obstacle recognition method based on fusion calculation provided in the first embodiment of the present application. Figure 1 As shown, the specific steps include:

[0032] S101, acquiring point cloud data through a radar detector, and acquiring image data through an image collector;

[0033] The application scenario of this solution can be a scenario where unmanned equipment identifies and avoids obstacles while moving in a specific direction or waiting for task instructions on the spot.

[0034] Based on the above usage scenarios, it can be understood that the executor of the present application can be the unmanned equipment control terminal, specifically the processor running the unmanned equipment control terminal, etc., and no excessive limitations are made here.

[0035] Unmanned equipment can be unmanned equipment that can execute tasks or instructions issued by staff, such as unmanned aerial vehicles and plant protection drones.

[0036] It can be understood that when unmanned equipment executes this solution, taking a drone as an example, its operating state can be any of the following states: forward flight, vertical upward and downward flight, turning, and hovering in the air; or, taking an unmanned vehicle as an example, it can be any of the following states: forward flight, reverse flight, turning, and standby in place, all of which can realize the operation of obstacle recognition.

[0037] A radar detector can be a device that detects obstacles and determines their spatial position by emitting electromagnetic waves and receiving their echoes. It can be installed in a position with a wide field of view on the outer shell of the unmanned equipment in front of it. Point cloud data can be a collection of points composed of obstacles in a three-dimensional coordinate system collected by the radar detector.

[0038] The image collector may be a device that converts optical image information into electrical signals, and may be arranged side by side with the radar detector at a distance of 10 cm, 15 cm, etc. on the housing of the unmanned device, or may be arranged vertically with the radar detector at a distance of 5 cm, 8 cm, etc. on the housing of the unmanned device. The image data may be the scene image in front of the unmanned device acquired by the image collector and the pixel information contained in the image.

[0039] The way to obtain point cloud data is that the radar detector emits millimeter waves and receives the echo signal reflected by the obstacle back to the radar detector. The position and distance of the obstacle can be obtained based on the time interval between the transmitted signal and the received signal and the propagation speed of the millimeter wave. Then, based on the path difference and phase difference generated when the echo signal reaches each receiving antenna in the radar detector, the angle between the obstacle and the radar detector can be calculated. Finally, based on the distance and angle of the obstacle, the specific position and three-dimensional coordinates of each point of the obstacle in the three-dimensional coordinate system can be obtained, thereby obtaining point cloud data.

[0040] The way to obtain image data is that after light passes through the lens and enters the image collector, it hits the photosensitive unit on the photosensitive element. The photons in the photosensitive unit cause the release of electrons and gather at the output end of the chip to form a voltage signal. These voltage signals are converted into digital signals by the processing chip to generate images.

[0041] S102, performing semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0042] Semantic segmentation can be to divide the pixels contained in the image into different areas according to the similarities and differences in categories. For example, in the image captured by the image collector in front of the unmanned equipment, the image can be divided into sky area, tree area, house area, vehicle area, etc. according to the different pixel semantic information. Semantic information can be to set the attribute type of each pixel in the image after semantic segmentation, such as sky, tree, etc.

[0043] The training method of the neural network model for semantic segmentation can be to pre-process the image first, and then put it into the neural network model for training. Using the training results, test the performance of the network in the validation set, observe the segmentation effect through the semantic segmentation indicators, and then save the corresponding weight value calculated according to the accuracy of the segmentation effect to perform image annotation correction. Take the weight with the best effect as the parameter of the neural network model, input the collected image data into the neural network model, and obtain the semantic information of each pixel in the image according to the output result of the neural network model. The way to obtain semantic information can be to affix semantic information labels to the areas divided in the image after semantic segmentation according to their respective classification standards.

[0044] S103: Projecting the point cloud data into image data with the semantic information, and identifying obstacles ahead of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point.

[0045] Radar points can be points in the point cloud data obtained by the radar detector. For example, the point cloud data of the wire is obtained by the radar detector, and the points representing the position of the wire and other obstacles in the radar detector coordinate system are all radar points. Projection points can be the projection points of the radar points in the image data when the point cloud data is projected into the image data. For example, the coordinate system where the point cloud data is located is converted into the image data, and the depth coordinates of the radar points are cleared to move the radar points to the plane where the image data is located. At this time, the position of the radar point can be used as the projection point of the radar point in the image data.

[0046] The method of projecting the point cloud data to the image data can be to project the three-dimensional coordinate system where the point cloud data is located into the two-dimensional coordinate system to obtain the coordinates of the point cloud data in the two-dimensional coordinate system. The image of the point cloud data projected from the three-dimensional coordinate system to the two-dimensional coordinate system is divided into rectangles of the same size (for example, rectangles with a width and height of 1 meter), and the two-dimensional real coordinates of the point cloud are mapped to the divided pixel coordinates to obtain the pixel image of the point cloud data. The obtained pixel image is scaled to the same resolution as the corresponding image data, and the scaled pixel image of the point cloud data and the corresponding image data are placed in the same coordinate system so that the image of the point cloud data overlaps with the image data.

[0047] The method of determining whether there is an obstacle in front of the unmanned equipment can be to collect point cloud data in front of the current unmanned equipment through a radar detector, and at the same time, the image collector collects image data in front of the current unmanned equipment. After semantic segmentation of the image collector, the point cloud data is projected into the image data with the semantic information. If the semantic information of the projection points of at least two radar points is identified to have obstacles, it is determined that there is an obstacle in front of the unmanned equipment.

[0048] Alternatively, the method of determining whether there is an obstacle in front of the unmanned equipment can also be to project the point cloud data onto the image data with the semantic information, and then circle the range of all the projection points with each projection point as the center and the preset range as the size data. Identify whether there are pixel points with semantic information of obstacle type in all the circled ranges. If there are at least two such pixel points, it is confirmed that there is an obstacle in front of the unmanned equipment. Among them, the preset range can be a pre-set range of pixel points in the image data to detect whether the projection point is an obstacle. Since the acquisition of point cloud data, the conversion of point cloud coordinates, and the projection of point cloud data to image data may have insufficient processor calculation accuracy, or external forces such as wind force and tension cause the obstacle to move or vibrate, the coordinates of the projection point have position deviations, so it is necessary to set a preset range to prevent missing projection points within the range of obstacle pixels. For example, with the projection point as the center, query whether there are pixel points belonging to electric wires in a field with a width and height of 64 pixels.

[0049] It should be noted that, in this solution, fusion calculation may refer to fusing point cloud data into image data with voice information, and determining the position of obstacles in the point cloud data through a series of recognition calculations, so as to accurately avoid obstacles. The fusion calculation is detailed in the above embodiments, so it will not be emphasized again.

[0050] In an embodiment of the present invention, point cloud data is acquired through a radar detector, and image data is acquired through an image collector; semantic segmentation is performed on the image data to obtain semantic information of each pixel in the image data; the point cloud data is projected into image data with the semantic information, and obstacles in front of the unmanned device are identified based on the semantic information corresponding to the projection point of the radar point. Through the above-mentioned obstacle identification method based on fusion calculation, it is possible to identify whether there are obstacles in the current environment of the unmanned device by projecting the point cloud data acquired by the radar detector into the image data with semantic information. This makes the perception of obstacles by the unmanned device more comprehensive and reliable, and effectively improves the accuracy of obstacle identification.

[0051] Embodiment 2

[0052] Figure 2 This is a flow chart of the obstacle identification method based on fusion calculation provided in Example 2 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: according to the semantic information corresponding to the projection point of the radar point, the obstacle in front of the unmanned equipment is identified, including: taking the pixel point where the obstacle exists in the semantic information corresponding to the projection point as the base point, performing regional growth clustering to obtain a set of pixel points of the same obstacle; when the number of pixel points in the pixel point set exceeds the preset number threshold, the obstacle is determined to be a valid obstacle.

[0053] like Figure 2 As shown, specifically including the following:

[0054] S201, acquiring point cloud data through a radar detector, and acquiring image data through an image collector;

[0055] S202, performing semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0056] S203, projecting the point cloud data onto the image data with the semantic information, taking the pixel points of obstacles in the semantic information corresponding to the projection points as the base points, performing region growing clustering to obtain a set of pixel points of the same obstacle;

[0057] The basic point may be a radar point corresponding to a projection point of a pixel point with a mark within the circled range that is identified after projecting the point cloud data onto the image data.

[0058] The method of performing region growing clustering can be to segment the image into similar regions according to the specified criteria, that is, the similarity of the grayscale values ​​of the image. Specifically, the point cloud data can be projected into the image data with the semantic information, with the basic point as the starting position, and the pixels with grayscale values ​​close to the basic point are extracted from the connected pixels in the basic point area and added to the segmented region, and the pixel point is used as the new basic point to repeatedly extract the pixel point with grayscale value close to the basic point. When the absolute value of the difference between the grayscale value of the pixel point to be added and the average grayscale value of all the pixels in the segmented area is less than or equal to the preset range, the pixel point is added to the segmented area until there is no new basic point. At this time, all the pixels in the segmented area are a set of pixels of the same obstacle.

[0059] This embodiment provides a region growing clustering procedure for reference:

[0060]

[0061]

[0062]

[0063] S204: when the number of pixel points in the pixel point set exceeds a preset number threshold, determining that the obstacle is a valid obstacle;

[0064] The preset number threshold may be a preset number limit of a pixel point set for determining whether the current pixel point set is a valid obstacle. For example, it is preset that when the number of elements in the pixel point set obtained by the region growing clustering algorithm is greater than 3, the pixel point set is valid.

[0065] The method of determining a valid obstacle may be to obtain a pixel point set of the same obstacle, calculate the number of all pixels in the pixel point set, and when the number of pixels exceeds a preset number threshold, determine the obstacle as a valid obstacle.

[0066] For example, in this embodiment, a method for determining effective obstacles using electric wires as obstacles is provided for reference:

[0067] The acquired wire point cloud is projected onto the captured wire image data to obtain the projection points of the wire point cloud data projected onto its image data. For each projection point, query whether there is a pixel point with semantic information of a wire in the area centered on the projection point and 64 pixels wide and 64 pixels high. If there is a pixel point of a wire, select the pixel point closest to the center point cloud projection point and use the regional growing clustering algorithm to find the pixel points of all wires on the same straight line. The center point cloud projection point at this time and the pixel points of all wires found are taken as a set W. When the number of elements in W is greater than 3, the wire is determined to be a valid obstacle, and the straight line composed of pixel points with semantic information of wires in the current image data is recorded as a 2D straight line.

[0068] The advantage of this arrangement of the present invention is that the existence of obstacles can be verified by detecting the number of pixels in the pixel set, thereby avoiding wasting time and resources due to misjudgment and improving the efficiency and accuracy of obstacle determination.

[0069] Embodiment 3

[0070] Figure 3 A schematic diagram of the process of determining the extension direction of an effective obstacle provided in the third embodiment of the present invention. This solution makes a better improvement on the above embodiment, and the specific improvement is: after determining that the obstacle is a valid obstacle, the method also includes: determining the first endpoint and the second endpoint of the effective obstacle according to the pixel point set; performing a normalized plane transformation on the first endpoint and the second endpoint to obtain the transformed normalized coordinates of the first end and the normalized coordinates of the second end; constructing a first plane based on the normalized coordinates of the first end and the normalized coordinates of the second end, and the coordinates of the image collector; constructing a second plane based on the radar points; and determining the extension direction of the effective obstacle based on the first plane and the second plane.

[0071] like Figure 3 As shown, specifically including the following:

[0072] S301, projecting the point cloud data into the image data with the semantic information, performing region growing clustering based on the pixel points with obstacles in the semantic information corresponding to the projection points, and obtaining a set of pixel points with the same obstacle.

[0073] S302: when the number of pixel points in the pixel point set exceeds a preset number threshold, determining that the obstacle is a valid obstacle;

[0074] S303, determining a first endpoint and a second endpoint of a valid obstacle according to the pixel point set;

[0075] The first endpoint may be a boundary point on the left side of the obstacle, which is closest to the left edge of the image in the pixel point set. The second endpoint may be a boundary point on the right side of the obstacle, which is closest to the right edge of the image in the pixel point set.

[0076] The first endpoint and the second endpoint of the effective obstacle may be determined by traversing all elements in the pixel point set of the same obstacle, finding the two pixel points closest to the left and right sides of the image data, and recording them as the first endpoint and the second endpoint respectively.

[0077] S304, performing normalized plane transformation on the first endpoint and the second endpoint to obtain transformed normalized coordinates of the first end and normalized coordinates of the second end;

[0078] The normalized plane transformation may be to transform the pixel coordinates in the image data into coordinates in the camera coordinate system. The first end normalized coordinates may be the coordinates of the point after the first endpoint is normalized plane transformed. The second endpoint may be the coordinates of the point after the second endpoint is normalized plane transformed.

[0079] The method for obtaining the normalized coordinates of the first end and the normalized coordinates of the second end can be to extract the pixel coordinates of the first endpoint and the second endpoint, and translate and / or rotate the pixel coordinates through the pinhole imaging principle and the similar triangle principle to obtain the normalized coordinates of the first end and the second end.

[0080] In this embodiment, a normalized plane conversion method is provided for reference:

[0081] If the pixel coordinates of a pixel point in the set are (u, v) and the focal length in the camera parameters is f, then the normalized coordinates (X, Y, Z) obtained by normalizing the pixel coordinates by plane transformation satisfy the following formula:

[0082]

[0083] By using a method similar to the above, the normalized coordinates of each pixel point can be obtained.

[0084] S305: constructing a first plane based on the normalized coordinates of the first end, the normalized coordinates of the second end, and the coordinates of the image collector;

[0085] The coordinates of the first plane can be constructed in a manner that the position of the image collector is used as the coordinate origin, the normalized coordinates of the first end and the normalized coordinates of the second end are used as known points in the first plane, and the first plane is determined by three points.

[0086] In this embodiment, the following method for determining the first plane by using the position coordinates of the image collector, the normalized coordinates of the first end, and the normalized coordinates of the second end is provided for reference:

[0087] If the image collector's position coordinates are (x 1 ,y 1 , z 1 ), the normalized coordinate of the first end is (x 2 ,y 2 , z 2 ), the normalized coordinates of the second end are (x 3 ,y 3 , z 3 ), then the first plane can be determined as:

[0088] Ax+By+Cz+D=0;

[0089] Where A=(y 3 -y 1 )(z 3 -z 1 )-(z 2 -z 1 )(y 3 -y 1 );

[0090] B=(x 3 -x 1 )(z 2 -z 1 )-(x 2 -x 1 )(z 3 -z 1 );

[0091] C=(x 2 -x 1 )(y 3 -y 1 )-(x 3 -x 1 )(y 2 -y 1 );

[0092] D=-(Ax 1 +By 1 +Cz 1 );

[0093] Using a similar method to the above, the expression of the first plane can be determined.

[0094] S306, constructing a second plane according to the radar points;

[0095] The second plane can be constructed by selecting pixel points in a set of pixel points located at the same obstacle, obtaining the coordinates of the corresponding radar point, constructing a second surface normal vector pointing to the radar point using the coordinates of the radar detector's location and the radar point coordinates, and calculating an expression for the second plane based on the information that the second plane is perpendicular to its normal vector and passes through the radar point.

[0096] For example, if the normal vector of the second surface is (a, b, c), the radar point coordinates are (x 0 ,y 0 , z 0 ), then the second plane can be:

[0097] ax+by+cz+d=0;

[0098] Where d = -ax 0 -by 0 -cz 0 ;

[0099] The expression of the second plane can be obtained by using a method similar to the above.

[0100] S307: Determine an extension direction of the effective obstacle according to the first plane and the second plane.

[0101] The extension direction of the effective obstacle can be determined by extending the first plane and the second plane until the two planes intersect, and the intersection line is the extension line of the effective obstacle. The leftmost endpoint and the rightmost endpoint of the extension line are selected and the coordinates of each endpoint are obtained, and direction vectors are constructed with the radar point respectively. The direction of the direction vector is the extension direction.

[0102] For example, in this embodiment, a method for determining the extension direction of an effective obstacle using a wire as an obstacle is provided for reference:

[0103] Traverse all elements in the set of the center point cloud projection point and its wire pixel points, and find the leftmost and rightmost pixels in the image data, which are recorded as L2d_left and L2d_right respectively. Convert L2d_left and L2d_right to normalized coordinates. A plane Plane_camera can be determined by two normalized coordinates and the origin in the camera coordinate system. Get the radar point coordinates of the center point cloud projection point in the set, and determine the plane Plane_radar with the direction vector of the radar point reaching the radar detector as the plane normal vector. In addition, the radar point corresponding to the center point cloud projection point is on the plane Plane_radar. Extend the plane Plane_camera and the plane Plane_radar respectively until the two planes intersect, and take the intersection of the two planes as the 3D straight line of the wire. The extension direction of the effective obstacle is the extension direction of the 3D straight line.

[0104] The advantage of this setting of the present scheme is that the first plane and the second plane can be constructed according to the coordinates of the endpoints of the effective obstacle and the radar point, and the extension direction of the effective obstacle can be determined by the first plane and the second plane, which is conducive to the unmanned equipment to more comprehensively perceive the size range of the obstacle and formulate a reasonable avoidance route.

[0105] Embodiment 4

[0106] Figure 4 It is a flow chart of the normalized plane conversion method provided in Example 4 of the present application. This scheme makes a better improvement on the above-mentioned embodiment, and the specific improvement is: performing normalized plane conversion on the first endpoint and the second endpoint to obtain the normalized coordinates of the first end and the normalized coordinates of the second end after conversion, including: constructing a normalized plane at a preset distance directly in front of the image collector based on the parameters of the image collector; determining the normalized coordinates of the first endpoint at the first end of the normalized plane and the normalized coordinates of the second endpoint at the second end of the normalized plane according to the pixel point coordinates of the first endpoint and the second endpoint in the image data.

[0107] like Figure 4 As shown, specifically including the following:

[0108] S4061. Constructing a normalized plane at a preset distance directly in front of the image collector based on the parameters of the image collector;

[0109] The parameters of the image collector may include the focal length, sensitivity, shooting angle and other parameters of the image collector.

[0110] The normalized plane can be constructed by taking the optical center of the image collector as the origin, the direction coinciding with the optical axis, i.e. pointing to the front of the image collector (perpendicular to the imaging plane) as the x-axis, and the direction perpendicular to the x-axis as the y-axis.

[0111] S4062: Determine the normalized coordinates of the first endpoint at a first end of the normalized plane and the normalized coordinates of the second endpoint at a second end of the normalized plane according to the pixel coordinates of the first endpoint and the second endpoint in the image data;

[0112] The method for determining the normalized coordinates of the first end and the normalized coordinates of the second end can be to use the focal length of the image collector as the coordinates of the first endpoint and the second endpoint in the image data plane and the scale of the normalized coordinates of the first end and the second end, scale the coordinates of the first endpoint and the second endpoint according to the scale, map the scaled coordinates to the normalized plane, and determine the corresponding normalized coordinates of the first end and the normalized coordinates of the second end.

[0113] The advantage of this setting of the scheme is that a normalized plane can be constructed based on the parameters of the image collector, and the normalized coordinates of the obstacle endpoints can be determined, which is conducive to more accurate determination of the obstacle position and improving the efficiency of obstacle detection.

[0114] Embodiment 5

[0115] Figure 5 This is a flow chart of the method for verifying the extension direction of an effective obstacle provided in Example 5 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: after determining the extension direction of the effective obstacle according to the first plane and the second plane, the method further includes: taking the radar point as the center, taking sampling points along the extension direction with a preset step size, and verifying the extension direction based on the sampling points and the pixel point set.

[0116] like Figure 5 As shown, specifically including the following:

[0117] S501, determining a first endpoint and a second endpoint of a valid obstacle according to the pixel point set;

[0118] S502, performing normalized plane transformation on the first endpoint and the second endpoint to obtain transformed normalized coordinates of the first end and normalized coordinates of the second end;

[0119] S503, constructing a first plane based on the normalized coordinates of the first end, the normalized coordinates of the second end, and the coordinates of the image collector;

[0120] S504, constructing a second plane according to the radar points;

[0121] S505. Determine an extension direction of the effective obstacle according to the first plane and the second plane;

[0122] S506, taking the radar point as the center, taking sampling points along the extension direction with a preset step length, and verifying the extension direction based on the sampling points and the pixel point set;

[0123] The preset step length may be a preset interval for selecting sampling points, which may be set to 1 meter, 1.5 meters or 2 meters, etc., and is not limited here. The sampling point may be a sample point used to verify whether the obstacle extension defense line is correctly collected.

[0124] The extension direction verification method may be to take a sampling point in the extension direction, obtain the projection point of the sampling point in the image data, and detect whether the projection point is within the pixel point range of the obstacle. If the projection point is within the pixel point range of the obstacle, the extension direction verification is passed.

[0125] Alternatively, the extension direction verification may also be performed by extracting a set of pixel points of the obstacle after semantic segmentation of the image data, and calculating the direction vector of the obstacle according to the coordinates of at least two pixel points. A sampling point closest to the radar point is selected from the sampling points, and the direction vector of the extension direction is obtained according to the coordinates of the sampling point and the radar point, and the direction vector of the extension direction is matched with the direction vector of the obstacle. If the direction vector of the extension direction matches the direction vector of the obstacle, the extension direction verification is passed.

[0126] For example, in this embodiment, a method for verifying the extension direction of an effective obstacle using a wire as an obstacle is provided for reference:

[0127] Find the point closest to the radar point corresponding to the projection point of the center point cloud in the 3D straight line of the wire, recorded as P3d. With this point as the center, collect sampling points at intervals of 1 meter at both ends along the 3D straight line. Obtain the projection point of each sampling point in the wire image data through the camera internal parameters, calculate the distance between each projection point and the 2D straight line, select the projection point whose distance to the 2D straight line is less than 40 pixels and is included by L2d_left and L2d_right, and calculate whether the direction vector between it and the radar point is consistent with the extension direction of the 2D straight line. If consistent, the extension direction test is passed, and the corresponding sampling point is a valid extension point.

[0128] The advantage of this setting of the scheme is that the extension direction of the obstacle can be verified by collecting sampling points, making the determination of the extension direction of the obstacle more accurate and reasonable, and avoiding damage to the normal operation of the unmanned equipment caused by incorrect judgment of the extension direction.

[0129] Embodiment 6

[0130] Figure 6 A schematic diagram of the confirmation process of the effective extension point provided in the sixth embodiment of the present invention. This solution makes a better improvement on the above embodiment, and the specific improvement is: verification based on the sampling point and the pixel point set, including: determining the sampling projection point of the sampling point in the image data based on the sampling point and the parameters of the image collector; identifying the deviation distance between the sampling projection point and the pixel point set that constitutes an effective obstacle; and determining the current sampling point as a valid extension point when the deviation distance is less than a set threshold.

[0131] like Figure 6 As shown, specifically including the following:

[0132] S6061. Determine a sampling projection point of the sampling point in the image data based on the sampling point and parameters of the image collector;

[0133] The sampling projection point can be determined by projecting the sampling point from the three-dimensional coordinate system to the two-dimensional coordinate system where the image data is located according to the projection method of the point cloud data, marking the sampling projection point projected to the image data plane, and calculating the pixel coordinates of the sampling projection point in the image data.

[0134] S6062, identifying the deviation distance between the sampling projection point and the pixel point set constituting a valid obstacle;

[0135] The deviation distance can be the distance between the sampling projection point and the set of valid obstacle pixels.

[0136] The deviation distance can be obtained by obtaining the distance between the sampling projection point and the pixel point closest to the sampling projection point in the effective obstacle pixel point collection. Specifically, the distance between the sampling projection point and all the pixel points in the effective obstacle pixel point collection can be calculated, and the shortest distance can be taken as the deviation distance. Alternatively, the distance relationship between regular geometric figures can be used to obtain the distance closest to the sampling projection point as the deviation distance. For example, if the effective obstacle is a horizontal electric wire, the pixel point collection of the effective obstacle can be regarded as a straight line. In this case, the deviation distance is the distance from the sampling projection point to the straight line.

[0137] S6063: When the deviation distance is less than a set threshold, determine the current sampling point as a valid extension point.

[0138] The set threshold value may be a range of values ​​used to determine whether the sampled projection point is a pixel point of an obstacle. For example, if the deviation distance is less than 40 pixels, it is determined that the deviation distance is less than the set threshold value, and the sampled projection point is a pixel point of an obstacle.

[0139] The method for determining that the current sampling point is a valid extension point is to compare the deviation distance with a set threshold. When the deviation distance is less than the set threshold, it is determined whether the coordinates of the sampling projection point are between the first endpoint coordinates and the second endpoint coordinates. If so, the sampling point corresponding to the current sampling projection point is determined to be a valid extension point.

[0140] The advantage of this arrangement of the present invention is that the extension direction of the obstacle can be verified by calculating the deviation distance between the sampling projection point and the effective obstacle, thereby improving the efficiency of the extension direction verification.

[0141] Embodiment 7

[0142] Figure 7This is a flow chart of the obstacle identification method based on fusion calculation provided in Example 7 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: after acquiring point cloud data through a radar detector and acquiring image data through an image collector, the method further includes: performing time registration on the point cloud data and the image data; converting the coordinate system of the point cloud data to the coordinate system of the image data.

[0143] like Figure 7 As shown, specifically including the following:

[0144] S701, acquiring point cloud data through a radar detector, and acquiring image data through an image collector;

[0145] S702, performing time registration on the point cloud data and the image data;

[0146] The method of performing time registration is that the unmanned equipment control terminal extracts the time of collecting point cloud data and collecting image data respectively, and pairs the point cloud data and image data with the same collection time, and marks them with the same time tag.

[0147] S703, converting the coordinate system of the point cloud data into the coordinate system of the image data;

[0148] The way to convert the coordinate system of the point cloud data to the coordinate system of the image data is to select a group of point cloud data and image data with the same time tag, extract the coordinates of the midpoint of the point cloud data, take the midpoint coordinates of the point cloud data (X, Y, Z) and the midpoint coordinates of the image data (u, v) as an example, and substitute them into the formula:

[0149]

[0150] in, is the camera intrinsic parameter matrix, is the camera extrinsic matrix. Using an algorithm similar to the above, the coordinates of the points in the point cloud data can be converted into coordinates in the image data coordinate system.

[0151] S704, performing semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0152] S705: Project the point cloud data into image data with the semantic information, and identify obstacles in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point.

[0153] For example, using wires as obstacles, combined with the above embodiments, Figure 8 : is a schematic diagram of a process for determining a 3D image of an obstacle provided in Embodiment 7 of the present application. Figure 8As shown in the figure, the collected camera image of the wire is matched with the radar point cloud in time and space, and registered at the same time. The registered camera image is semantically segmented to obtain the semantic information of each pixel. The radar point cloud is converted into a coordinate system and projected onto the camera image to obtain the semantic image of the wire. The projection points in the semantic image are traversed, and all the projection points projected within the range of the wire pixel points are extracted to form a 3D straight line set of the wire. The parameters of the 3D straight line are estimated according to the position of each element in the set, and the boundary of the 3D straight line is determined by the distance between the projection point corresponding to each element in the set and the boundary pixel point of the wire.

[0154] The advantage of this setting of the scheme is that the point cloud data and the image data can be registered according to the acquisition time, and the coordinate system of the point cloud data can be converted to the coordinate system of the image data, thereby avoiding mismatching of the point cloud data and the image data and providing a basis for obstacle recognition.

[0155] Embodiment 8

[0156] Fig. 9 It is a structural diagram of an obstacle recognition device based on fusion calculation provided in Example 8 of the present application.

[0157] like Fig. 9 As shown, specifically including the following:

[0158] The data acquisition module 901 is used to acquire point cloud data through a radar detector and to acquire image data through an image collector;

[0159] The semantic information acquisition module 902 is used to perform semantic segmentation on the image data to obtain semantic information of each pixel in the image data;

[0160] The obstacle determination module 903 is used to project the point cloud data into the image data with the semantic information, and identify the obstacles in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point.

[0161] In an embodiment of the present invention, a data acquisition module is used to acquire point cloud data through a radar detector and to acquire image data through an image collector; a semantic information acquisition module is used to perform semantic segmentation on the image data to obtain semantic information of each pixel in the image data; an obstacle determination module is used to project the point cloud data into image data with the semantic information, and identify obstacles in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point. Through the above-mentioned obstacle identification device based on fusion calculation, it is possible to identify whether there are obstacles in the current environment of the unmanned equipment by projecting the point cloud data acquired by the radar detector into the image data with semantic information. This makes the perception of obstacles by the unmanned equipment more comprehensive and reliable, and effectively improves the accuracy of obstacle identification.

[0162] The obstacle recognition device based on fusion calculation provided in the embodiment of the present application and the obstacle recognition method based on fusion calculation provided in the above embodiment have the same beneficial effects, and will not be described again here to avoid repetition.

[0163] Embodiment 9

[0164] Fig.10 This is a schematic diagram of the structure of an unmanned device provided in Embodiment 9 of the present invention. Fig.10 As shown, the embodiment of the present application also provides an unmanned device, which is configured with an image collector 1001 and a radar detector 1002, and includes: one or more processors 1003; a storage device 1004, which is used to store one or more programs. The one or more programs can be executed by the one or more processors 1003, and the processor 1003 can implement the various processes of the above-mentioned obstacle recognition method embodiment based on fusion calculation, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0165] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile device, mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. The obstacle recognition method based on fusion calculation is characterized by: The method is performed by an unmanned device, wherein the unmanned device is equipped with an image collector and a radar detector; the method comprises: Acquire point cloud data through a radar detector, and acquire image data through an image collector; Performing semantic segmentation on the image data to obtain semantic information of each pixel in the image data; The point cloud data is projected into image data with the semantic information, and obstacles ahead of the unmanned equipment are identified based on the semantic information corresponding to the projection points of the radar points.

2. The obstacle recognition method based on fusion calculation according to claim 1, characterized in that: According to the semantic information corresponding to the projection point of the radar point, the obstacle in front of the unmanned equipment is identified, including: Taking the pixel points with obstacles in the semantic information corresponding to the projection point as the base point, perform region growing clustering to obtain a set of pixel points with the same obstacle; When the number of pixels in the pixel point set exceeds a preset number threshold, the obstacle is determined to be a valid obstacle.

3. The obstacle recognition method based on fusion calculation according to claim 2, characterized in that: After determining that the obstacle is a valid obstacle, the method further includes: Determine a first endpoint and a second endpoint of a valid obstacle according to the pixel point set; Performing normalized plane transformation on the first endpoint and the second endpoint to obtain transformed normalized coordinates of the first end and normalized coordinates of the second end; constructing a first plane based on the normalized coordinates of the first end, the normalized coordinates of the second end, and the coordinates of the image collector; constructing a second plane according to the radar points; An extension direction of the effective obstacle is determined according to the first plane and the second plane.

4. The obstacle recognition method based on fusion calculation according to claim 3 is characterized in that: Performing normalized plane transformation on the first endpoint and the second endpoint to obtain transformed normalized coordinates of the first end and normalized coordinates of the second end, including: Constructing a normalized plane at a preset distance directly in front of the image collector based on the parameters of the image collector; According to the pixel point coordinates of the first endpoint and the second endpoint in the image data, the normalized coordinates of the first endpoint at the first end of the normalized plane and the normalized coordinates of the second endpoint at the second end of the normalized plane are determined.

5. The obstacle recognition method based on fusion calculation according to claim 3 is characterized in that: After determining the extension direction of the effective obstacle according to the first plane and the second plane, the method further includes: Taking the radar point as the center, sampling points are taken along the extension direction with a preset step size, and the extension direction is verified based on the sampling points and the pixel point set.

6. The obstacle recognition method based on fusion calculation according to claim 5, characterized in that: Verifying based on the sampling point and the pixel point set includes: Based on the sampling point and the parameters of the image collector, determining a sampling projection point of the sampling point in the image data; Identify the deviation distance between the sampling projection point and the pixel point set that constitutes a valid obstacle; When the deviation distance is less than a set threshold, the current sampling point is determined to be a valid extension point.

7. The obstacle recognition method based on fusion calculation according to claim 1, characterized in that: After acquiring point cloud data through the radar detector and acquiring image data through the image collector, the method further includes: Performing temporal registration on the point cloud data and the image data; The coordinate system of the point cloud data is converted to the coordinate system of the image data.

8. The obstacle recognition device based on fusion calculation is characterized in that: The method is performed by an unmanned device, wherein the unmanned device is equipped with an image collector and a radar detector; the device comprises: A data acquisition module, used to acquire point cloud data through a radar detector and to acquire image data through an image collector; A semantic information acquisition module, used to perform semantic segmentation on the image data to obtain semantic information of each pixel in the image data; The obstacle determination module is used to project the point cloud data into image data with the semantic information, and identify obstacles in front of the unmanned equipment according to the semantic information corresponding to the projection point of the radar point.

9. An unmanned device, characterized in that: The unmanned equipment is configured with an image collector and a radar detector, and includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the obstacle recognition method based on fusion calculation described in any one of claims 1-7.

10. A storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the obstacle recognition method based on fusion calculation according to any one of claims 1 to 7 when executed by a computer processor.

Images