Obstacle position identification method and device and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an obstacle position recognition method and device and a storage medium. Comprising the following steps: acquiring three-dimensional point cloud data and two-dimensional image data in a preset time period; performing identification processing on the three-dimensional point cloud data and the two-dimensional image data to generate three-dimensional information and two-dimensional information of at least one obstacle; for each obstacle, when the three-dimensional block diagram corresponding to the first two-dimensional block diagram of the obstacle does not exist, taking the first two-dimensional block diagram as a candidate block diagram of the obstacle; performing back projection processing on the candidate block diagram according to the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional block diagram corresponding to the first two-dimensional block diagram; and determining the position of the three-dimensional block diagram as the position of the obstacle. The embodiment of the invention is used for solving the problem of low accuracy of an obstacle position recognition result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, and storage medium for identifying an obstacle position. Background Art

[0002] To improve autonomous driving capabilities, existing vehicles are beginning to utilize various sensing devices to detect objects around the vehicle, thereby more accurately perceiving the vehicle's surrounding environment. For example, image sensors and radars are used to detect objects around the vehicle. Due to the high cost of radars themselves, currently, most vehicles only have one radar installed to reduce costs. However, due to installation location, vehicle body obstructions, and other factors, the radar's detection range is limited, making it unable to detect objects close to the vehicle. When an object passes by the vehicle, it will be out of the radar's detection range for a period of time, causing the lidar to only collect partial data on the object or fail to detect the object. However, during this period, the image sensor can still detect objects close to the vehicle, resulting in differences in the perception of the object between two-dimensional and three-dimensional data. This difference may adversely affect subsequent decision-making.

[0003] Therefore, how to clearly identify the location of an object outside the radar range when the object leaves the radar's detection range becomes an urgent problem to be solved. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides an obstacle position identification method, device and storage medium, which can improve the obstacle identification accuracy.

[0005] In a first aspect, the present application provides a method for identifying an obstacle position, comprising: acquiring three-dimensional point cloud data and two-dimensional image data within a preset time period; performing identification processing on the three-dimensional point cloud data and the two-dimensional image data respectively to obtain three-dimensional information and two-dimensional information of at least one obstacle; the three-dimensional information includes at least one three-dimensional frame diagram, and the two-dimensional information includes at least one two-dimensional frame diagram; for each obstacle, when a three-dimensional frame diagram corresponding to a first two-dimensional frame diagram of the obstacle does not exist, using the first two-dimensional frame diagram as a candidate frame diagram of the obstacle; the first two-dimensional frame diagram is a two-dimensional frame diagram in the first two-dimensional information, the first two-dimensional information being generated using two-dimensional image data of the obstacle acquired at a first time point; the first time point being any time point within the preset time period; performing back-projection processing on the candidate frame diagram based on the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram; the candidate three-dimensional information being generated using three-dimensional point cloud data of the obstacle acquired at a second time point, the second time point being a time point adjacent to the first time point; and determining the location of the three-dimensional complementary frame diagram as the location of the obstacle at the first time point.

[0006] In a second aspect, the present application provides an obstacle position identification device, comprising: an acquisition module for acquiring three-dimensional point cloud data and two-dimensional image data within a preset time period; an identification module for respectively identifying and processing the three-dimensional point cloud data and the two-dimensional image data to generate three-dimensional information and two-dimensional information of at least one obstacle; the three-dimensional information including at least one three-dimensional frame diagram, and the two-dimensional information including at least one two-dimensional frame diagram; a determination module for, for each obstacle, when a three-dimensional frame diagram corresponding to a first two-dimensional frame diagram of the obstacle does not exist, using the first two-dimensional frame diagram as a candidate frame diagram of the obstacle; the first two-dimensional frame diagram is a two-dimensional frame diagram in the first two-dimensional information, the first two-dimensional information being generated using two-dimensional image data of the obstacle acquired at a first time point; the first time point being any time point within the preset time period; a projection module for back-projecting the candidate frame diagram based on the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram; the candidate three-dimensional information being generated using three-dimensional point cloud data of the obstacle acquired at a second time point, the second time point being a time point adjacent to the first time point; and the determination module for further determining the location of the three-dimensional complementary frame diagram as the location of the obstacle at the first time point.

[0007] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the obstacle position identification method according to the first aspect is implemented.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the obstacle position identification method as in the first aspect is implemented.

[0009] In a fifth aspect, the present application provides a computer program product, comprising: when the computer program product is run on a computer, the computer is enabled to implement the obstacle position identification method as in the first aspect.

[0010] The technical solution provided by this application offers the following advantages over existing technologies: Three-dimensional point cloud data and two-dimensional image data within a preset time period are acquired, and both the three-dimensional point cloud data and the two-dimensional image data are identified and processed to obtain three-dimensional information and two-dimensional information of at least one obstacle. Subsequently, for each obstacle, if a three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, the first two-dimensional frame diagram is used as a candidate frame diagram for the obstacle. Based on the first two-dimensional information and the candidate three-dimensional information, the candidate frame diagram is back-projected to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram. The first two-dimensional frame diagram is the two-dimensional frame diagram within the first two-dimensional information, generated using the two-dimensional image data of the obstacle acquired at a first time point; the first time point is any time point within the preset time period. Finally, the location of the three-dimensional complementary frame diagram is determined as the location of the obstacle at the first time point. In this way, when identifying the position of obstacles, the three-dimensional point cloud data and the two-dimensional image data are combined for identification. The two-dimensional candidate frame diagram can be back-projected to obtain a three-dimensional supplementary frame diagram that does not exist in the three-dimensional information. That is, the two-dimensional recognition result can be used to supplement the obstacle information lost in the three-dimensional recognition result, thereby improving the accuracy of the recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 A schematic diagram of a scenario of an obstacle location identification method provided in an embodiment of the present application;

[0014] Figure 2 This is a flow chart of a method for identifying an obstacle position according to an embodiment of the present application;

[0015] Figure 3 The second flowchart of the obstacle location identification method provided in the embodiment of the present application;

[0016] Figure 4 A schematic diagram of a projection scene is provided for an embodiment of the present application;

[0017] Figure 5 The third flowchart of the obstacle location identification method provided in the embodiment of the present application;

[0018] Figure 6 Flowchart 4 of the obstacle location identification method provided in the embodiment of the present application;

[0019] Figure 7 Flowchart 5 of the obstacle location identification method provided in the embodiment of the present application;

[0020] Figure 8 This is one of the structural diagrams of the obstacle position identification device provided in an embodiment of the present application;

[0021] Figure 9 This is a second structural diagram of the obstacle position identification device provided in an embodiment of the present application;

[0022] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present application, not all of the embodiments.

[0025] Figure 1 A schematic diagram of an application scenario for the obstacle location identification method provided in an embodiment of the present application includes a vehicle 11, six two-dimensional data acquisition devices 12, and a three-dimensional data acquisition device 13. The three-dimensional data acquisition device 11 is located on top of the vehicle 10 and is used to collect three-dimensional point cloud data around the vehicle. The six two-dimensional data acquisition devices 12 are located at six peripheral viewing angles of the vehicle 11 (including the front view, left front view, right front view, left rear view, right rear view, and rear view) and are used to collect two-dimensional image data around the vehicle. Finally, the obstacle location identification device aggregates the three-dimensional point cloud data and the two-dimensional image data to implement the obstacle location identification method of the present application.

[0026] In some embodiments, the 3D data acquisition device 11 is a device capable of acquiring 3D point cloud data, such as a 360-degree laser radar, a structured light sensor, a stereo camera, etc. The 2D data acquisition device 12 is a device capable of acquiring 2D image data, such as a wide-angle camera, an infrared camera, a webcam, etc.

[0027] In some embodiments, the number of the two-dimensional data acquisition devices 12 may vary, and this application does not limit this.

[0028] The obstacle position identification device provided in the embodiment of the present application can be located inside the vehicle 11 to assist in driving the vehicle 11, or it can be located outside the vehicle 11 to provide support for other software or hardware that needs to make decisions based on the identification results. In addition, the obstacle position identification device provided in the embodiment of the present application can be hardware or software. When the obstacle position identification device is hardware, it can be various electronic devices with the function of running obstacle position identification, including but not limited to vehicle-mounted equipment, smart vehicles, mobile phones, computers, etc. When the obstacle position identification device is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or it can be implemented as a single software or software module, which is not specifically limited here.

[0029] Figure 2 A flow chart of the obstacle position identification method provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the obstacle position recognition method may include the following steps.

[0030] S21. Acquire three-dimensional point cloud data and two-dimensional image data within a preset time period.

[0031] The 3D point cloud data is collected by a 3D data acquisition device installed on the vehicle (e.g., on top of the vehicle). Examples include a 360-degree LiDAR, a structured light sensor, or a stereo camera. The 2D image data is collected by at least one 2D data acquisition device installed on the vehicle (e.g., around the vehicle body). Examples include a wide-angle camera, an infrared camera, or a webcam.

[0032] S22: Perform recognition processing on the three-dimensional point cloud data and the two-dimensional image data respectively to generate three-dimensional information and two-dimensional information of at least one obstacle.

[0033] The three-dimensional information includes a three-dimensional frame diagram and a three-dimensional track ID, and the two-dimensional information includes a two-dimensional frame diagram and a two-dimensional track ID.

[0034] In some embodiments, the 3D point cloud data may be identified and processed to obtain 3D information about at least one obstacle by first dividing the 3D point cloud data into different point cloud clusters using a clustering algorithm (such as a density clustering algorithm or a Euclidean clustering algorithm), with each point cloud cluster representing one obstacle. A target detection algorithm is then used to perform target detection on each point cloud cluster, obtaining a 3D block diagram and 3D obstacle identifier corresponding to each obstacle in each point cloud cluster. Furthermore, a multi-target tracking algorithm is used to link obstacles at different time points to achieve 3D target tracking and obtain a 3D trajectory identifier corresponding to each obstacle after association. One 3D trajectory identifier corresponds to one obstacle trajectory, and one 3D trajectory identifier corresponds to multiple 3D obstacle identifiers.

[0035] In some embodiments, the method for identifying and processing 2D image data to obtain 2D information about at least one obstacle may be to first use a target detection algorithm to detect obstacles in the 2D image data, obtaining a 2D block diagram and 2D obstacle identifier corresponding to each obstacle. Then, a multi-target tracking algorithm is used to correlate obstacles detected in different frames of image data to achieve 2D target tracking and obtain a 2D trajectory identifier corresponding to each associated obstacle. One 2D trajectory identifier corresponds to one obstacle trajectory, and one 2D trajectory identifier corresponds to multiple 2D obstacle identifiers.

[0036] S23. For each obstacle, when the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, use the first two-dimensional frame diagram as a candidate frame diagram of the obstacle.

[0037] The first two-dimensional frame diagram is a two-dimensional frame diagram in the first two-dimensional information, and the first two-dimensional information is generated by obtaining two-dimensional image data of the obstacle at a first time point; the first time point is any time point within a preset time period.

[0038] In some embodiments, as Figure 3 As shown, before step S23, the method further includes the following steps:

[0039] S231 . For each obstacle, project at least one three-dimensional frame diagram into a two-dimensional coordinate system to obtain at least one projected frame diagram.

[0040] The two-dimensional coordinate system is the coordinate system where the two-dimensional frame diagram is located.

[0041] In some embodiments, at least one three-dimensional frame diagram is projected into a two-dimensional coordinate system to obtain at least one projection frame diagram by projecting the three-dimensional frame diagram into a two-dimensional coordinate system according to the minimum bounding rectangle principle to obtain at least one projection frame diagram; the minimum bounding rectangle principle means that the minimum bounding rectangle corresponding to the target three-dimensional frame diagram is used as the projection frame diagram of the target three-dimensional frame diagram; the target three-dimensional frame diagram is any three-dimensional frame diagram. For example, in the example Figure 4 In the projection scene shown, the three-dimensional frame diagram is represented as a three-dimensional frame diagram 401 in the two-dimensional coordinate system. According to the small bounding rectangle principle, the minimum bounding rectangle 402 of the three-dimensional frame diagram 401 is taken as the three-dimensional projection frame diagram of the three-dimensional frame diagram 401 in the two-dimensional coordinate system.

[0042] S232: Within a first time period, when a first number of two-dimensional frame images match a first number of projection frame images, determine that the two-dimensional trajectory identifiers corresponding to the two-dimensional frame images are credible identifiers.

[0043] Among them, the first time period is less than the preset time period; the two-dimensional frame diagram is matched to the projection frame diagram, including: the two-dimensional trajectory identifier corresponding to the two-dimensional frame diagram and the three-dimensional trajectory identifier corresponding to the projection frame diagram are the same, and the two-dimensional frame diagram and the projection frame diagram satisfy the greedy rule or the Hungarian matching rule.

[0044] In some embodiments, the first number of two-dimensional frame diagrams are matched to the first number of projection frame diagrams in a manner that the two-dimensional frame diagrams in a continuous first number of image frames are all matched to the first number of projection frame diagrams; or the two-dimensional frame diagrams in a discontinuous first number of image frames are all matched to the first number of projection frame diagrams.

[0045] In some embodiments, when the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, the method of using the first two-dimensional frame diagram as a candidate frame diagram of the obstacle may be that when the two-dimensional trajectory corresponding to the first two-dimensional frame diagram is identified as a credible identifier and the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram does not exist, the first two-dimensional frame diagram is used as a candidate frame diagram of the obstacle.

[0046] In the above scheme, at least one three-dimensional frame diagram is first projected onto a two-dimensional coordinate system for each obstacle to obtain at least one projected frame diagram. Within a first time period, when a first number of two-dimensional frame diagrams match a first number of projected frame diagrams, the two-dimensional trajectory of the two-dimensional frame diagram is determined to be a credible identifier. Finally, when the two-dimensional trajectory corresponding to the first two-dimensional frame diagram is a credible identifier and the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram does not exist, the first two-dimensional frame diagram is used as a candidate frame diagram for the obstacle. In this way, two-dimensional frame diagrams that do not exist in the three-dimensional information can be determined while ensuring the credibility of the two-dimensional frame diagrams. This allows the missing obstacle information in the three-dimensional recognition results to be supplemented based on the two-dimensional frame diagrams that do not exist in the three-dimensional information, thereby improving the accuracy of the recognition results.

[0047] S24 , performing back-projection processing on the candidate frame diagram according to the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram.

[0048] The candidate 3D information is generated using 3D point cloud data of the obstacle acquired at a second time point, where the second time point is adjacent to the first time point. The 2D information also includes the 2D coordinates of the 2D block diagram, and the 3D information also includes the 3D coordinates of the 3D block diagram. The 2D coordinates are the coordinates of the 2D block diagram in a 2D coordinate system, and the 3D coordinates are the coordinates of the 3D block diagram in a 3D coordinate system. The 3D coordinate system is the coordinate system of the vehicle body.

[0049] In some embodiments, as Figure 5 As shown, a method of back-projecting the candidate frame diagram according to the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram may include the following steps:

[0050] S1. Determine the back-projection height according to the three-dimensional coordinates in the candidate three-dimensional information.

[0051] The three-dimensional patch frame diagram is a cuboid including six faces. The back-projection height is the distance between the first face and the second face of the three-dimensional patch frame diagram. The first face and the second face are opposite faces, that is, the first face and the second face have no contacting edges.

[0052] S2. According to the center point coordinates and back-projection height of the target edge in the candidate frame graph, the candidate frame graph is back-projected to obtain a three-dimensional filler frame subgraph.

[0053] The target edge is located on the first surface or the second surface after back projection.

[0054] Specifically, a back-projection algorithm can be used to back-project the candidate frame image based on the center coordinates of the target edge in the candidate frame image and the back-projection height to obtain a three-dimensional patch frame sub-image. For example, an inverse transformation algorithm for perspective projection or a mathematical model for perspective projection can be used.

[0055] S3. Determine the three-dimensional frame-patch subgraph as a three-dimensional frame-patch graph.

[0056] In the above scheme, after back-projection processing on the two-dimensional candidate frame diagram, a three-dimensional supplementary frame diagram that does not exist in the three-dimensional information is obtained. That is, the two-dimensional recognition result can be used to supplement the obstacle information lost in the three-dimensional recognition result, thereby improving the accuracy of the obstacle recognition result.

[0057] In some embodiments, as Figure 6 As shown, the method of back-projecting the candidate frame diagram according to the first two-dimensional information and the candidate three-dimensional information to obtain the three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram may further include the following steps:

[0058] S4. Determine the back-projection height according to the three-dimensional coordinates in the candidate three-dimensional information.

[0059] The three-dimensional patch frame diagram is a cuboid including six faces. The back projection height is the distance between the third face and the fourth face of the three-dimensional patch frame diagram. The third face and the fourth face are opposite faces, that is, the third face and the fourth face have no contacting edges.

[0060] S5. Back-project the candidate frame image according to the center point coordinates of the first side of the candidate frame image and the back-projection height to obtain a first three-dimensional supplementary frame sub-image.

[0061] Among them, the first side is located on the third surface after back projection.

[0062] S6. Back-project the candidate frame image according to the center point coordinates of the second side of the candidate frame image and the back-projection height to obtain a second three-dimensional supplementary frame sub-image.

[0063] The first side and the second side are opposite sides, and the second side is located on the fourth side after back projection.

[0064] S7. Determine a first intersection-over-union ratio and a second intersection-over-union ratio.

[0065] Among them, the first intersection-and-union ratio is the intersection-and-union ratio between the projection of the first three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame image, and the second intersection-and-union ratio is the intersection-and-union ratio between the projection of the second three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame image.

[0066] Specifically, the method for determining the first intersection-and-union ratio can be to calculate the intersection area and union area between the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the candidate frame image based on the coordinates of the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the coordinates of the candidate frame image in the two-dimensional coordinate system, and then calculate the intersection-and-union ratio between the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the candidate frame image by iou=i / u; wherein, iou is used to represent the intersection-and-union ratio between the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the candidate frame image, i is used to represent the intersection area between the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the candidate frame image, and u is used to represent the union area between the projection of the first three-dimensional patch sub-image in the two-dimensional coordinate system and the candidate frame image.

[0067] Similarly, the intersection-and-union ratio between the projection of the second three-dimensional patch frame sub-graph in the two-dimensional coordinate system and the candidate frame graph is determined.

[0068] S8. Determine the three-dimensional frame patching subgraph corresponding to the largest intersection-and-union ratio among the first intersection-and-union ratio and the second intersection-and-union ratio as the three-dimensional frame patching subgraph.

[0069] In the above scheme, two three-dimensional patch frame sub-graphs are back-projected based on the two opposite sides of the second candidate frame graph, and the three-dimensional patch frame sub-graph with the largest intersection-union ratio with the candidate frame graph in the projections of the two three-dimensional patch frame sub-graphs is determined as the final three-dimensional patch frame graph. That is, the three-dimensional patch frame sub-graph with the largest intersection-union ratio with the candidate frame graph can be determined as the final three-dimensional patch frame graph, thereby improving the accuracy of the determined three-dimensional patch frame graph.

[0070] In some embodiments, after step S3 or step S8, the obstacle position identification method further includes determining the 3D frame sub-image within a target range as a 3D frame sub-image. The target range is a default range or a range set by a person according to the size of the vehicle, for example, the target range is a blind spot range of the vehicle body, or the target range is a 5-meter range of the vehicle body.

[0071] In the above solution, the three-dimensional frame patching sub-graphs within the target range can be determined as the three-dimensional frame patching graph, which narrows the range of the three-dimensional frame patching graphs that need to be supplemented and saves computing resources.

[0072] In some embodiments, the two-dimensional information also includes the coordinates of the two-dimensional frame in the two-dimensional coordinate system. Figure 7 As shown, after step S3 or step S8, the obstacle position recognition method further includes the following steps:

[0073] S9. For each frame of two-dimensional image data, calculate the center point distance between every two candidate frames.

[0074] Specifically, the method for calculating the center point distance between each two candidate frame diagrams can be to calculate the center point distance between each two candidate frame diagrams based on the coordinates of the same position of each two candidate frame diagrams in the two-dimensional coordinate system, that is, the straight-line distance between the two coordinates is the center point distance between the two candidate frame diagrams; or the method for calculating the center point distance between each two candidate frame diagrams based on the coordinates of the center point positions in each two candidate frame diagrams in the two-dimensional coordinate system.

[0075] S10: When the center point distance is less than the distance threshold and the intersection-to-union ratios of the two candidate frame graphs are both greater than the ratio threshold, the candidate frame graph with the smaller intersection-to-union ratio is deleted.

[0076] The distance threshold is preset, for example, a default value, or a value set by relevant personnel based on actual circumstances. For another example, the distance threshold is 0.5 meters. The intersection-over-union ratio of the candidate frame diagram refers to the intersection-over-union ratio of the candidate frame diagram to its corresponding three-dimensional complement frame diagram.

[0077] In the above scheme, when the center point distance is less than the distance threshold and the intersection-and-union ratio of two candidate frames is greater than the ratio threshold, the candidate frame with the smaller intersection-and-union ratio is deleted. A center point distance of two candidate frames less than the distance threshold indicates that the two candidate frames are relatively close; an intersection-and-union ratio greater than the ratio threshold indicates that the three-dimensional patch frames corresponding to both candidate frames are relatively accurate. Because two-dimensional data acquisition equipment may have duplicate acquisition areas, the candidate frame with the smaller intersection-and-union ratio, i.e., the three-dimensional patch frames corresponding to the two candidate frames with insufficient accuracy, are deleted. This further improves the accuracy of obstacle recognition results by removing duplicate three-dimensional patch frames.

[0078] S25. Determine the location of the three-dimensional frame patching image as the location of the obstacle at the first time point.

[0079] In the above scheme, 3D point cloud data and 2D image data within a preset time period are acquired and both are identified and processed to obtain 3D and 2D information of at least one obstacle. Then, for each obstacle, if a 3D frame diagram corresponding to the first 2D frame diagram of the obstacle does not exist, the first 2D frame diagram is used as a candidate frame diagram for the obstacle. Based on the first 2D information and the candidate 3D information, the candidate frame diagram is back-projected to obtain a 3D complementary frame diagram corresponding to the first 2D frame diagram. The first 2D frame diagram is the 2D frame diagram in the first 2D information, generated using 2D image data of the obstacle acquired at a first time point; the first time point is any time point within the preset time period. Finally, the location of the 3D complementary frame diagram is determined as the location of the obstacle at the first time point. In this way, when identifying the position of obstacles, the three-dimensional point cloud data and the two-dimensional image data are combined for identification. The two-dimensional candidate frame diagram can be back-projected to obtain a three-dimensional supplementary frame diagram that does not exist in the three-dimensional information. That is, the two-dimensional recognition result can be used to supplement the obstacle information lost in the three-dimensional recognition result, thereby improving the accuracy of the recognition result.

[0080] In the embodiment of the present application, the functional modules of the obstacle position recognition device can be divided according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, other division methods may be used.

[0081] like Figure 8 , which is a schematic structural diagram of an obstacle position recognition device provided in an embodiment of the present application, the obstacle position recognition device includes an acquisition module 701 , an identification module 702 , a determination module 703 and a projection module 704 .

[0082] The acquisition module 701 is used to acquire three-dimensional point cloud data and two-dimensional image data within a preset time period; the recognition module 702 is used to respectively recognize and process the three-dimensional point cloud data and the two-dimensional image data to generate three-dimensional information and two-dimensional information of at least one obstacle; the three-dimensional information includes at least one three-dimensional frame diagram, and the two-dimensional information includes at least one two-dimensional frame diagram; the determination module 703 is used to, for each obstacle, when the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, use the first two-dimensional frame diagram as a candidate frame diagram of the obstacle; the first two-dimensional frame diagram is the two-dimensional frame diagram in the first two-dimensional information. The first two-dimensional information is generated by using the two-dimensional image data of the obstacle obtained at a first time point; the first time point is any time point within a preset time period; the projection module 704 is used to perform back-projection processing on the candidate frame diagram based on the first two-dimensional information and the candidate three-dimensional information to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram; the candidate three-dimensional information is generated by using the three-dimensional point cloud data of the obstacle obtained at a second time point, and the second time point is a time point adjacent to the first time point; the determination module 703 is further used to determine the position of the three-dimensional complementary frame diagram as the position of the obstacle at the first time point.

[0083] In some embodiments, the two-dimensional information also includes a two-dimensional trajectory identifier, and the three-dimensional information also includes a three-dimensional trajectory identifier; the determination module 703 is further used to project at least one three-dimensional frame diagram into a two-dimensional coordinate system for each obstacle to obtain at least one projection frame diagram; the two-dimensional coordinate system is the coordinate system where the two-dimensional frame diagram is located; within a first time period, when a first number of two-dimensional frame diagrams match a first number of projection frame diagrams, the two-dimensional trajectory identifier corresponding to the two-dimensional frame diagram is determined to be a credible identifier; the first time period is less than a preset time period; the two-dimensional frame diagram matches the projection frame diagram, including: the two-dimensional trajectory identifier corresponding to the two-dimensional frame diagram and the three-dimensional trajectory identifier corresponding to the projection frame diagram are the same, and the two-dimensional frame diagram and the projection frame diagram satisfy the greedy rule or the Hungarian matching rule; the determination module 703 is specifically used to use the first two-dimensional frame diagram as a candidate frame diagram for the obstacle when the two-dimensional trajectory identifier corresponding to the first two-dimensional frame diagram is a credible identifier and the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram does not exist.

[0084] In some embodiments, the determination module 703 is specifically used to project the three-dimensional frame diagram into a two-dimensional coordinate system according to the minimum bounding rectangle principle to obtain at least one projected frame diagram; the minimum bounding rectangle principle refers to using the minimum bounding rectangle corresponding to the target three-dimensional frame diagram as the projected frame diagram of the target three-dimensional frame diagram; the target three-dimensional frame diagram is any three-dimensional frame diagram.

[0085] In some embodiments, the two-dimensional information also includes the two-dimensional coordinates of the two-dimensional frame diagram, and the three-dimensional information also includes the three-dimensional coordinates of the three-dimensional frame diagram; the projection module 704 is specifically used to determine the back-projection height based on the three-dimensional coordinates in the candidate three-dimensional information; the back-projection height is the distance between the first side and the second side of the three-dimensional patch frame diagram, and the first side and the second side are opposite sides; according to the center point coordinates and the back-projection height of the target edge in the candidate frame diagram, the candidate frame diagram is back-projected to obtain a three-dimensional patch frame sub-graph; after back-projection, the target edge is located on the first side or the second side; the three-dimensional patch frame sub-graph is determined as a three-dimensional patch frame diagram.

[0086] In some embodiments, the two-dimensional information further includes the two-dimensional coordinates of the two-dimensional frame diagram, and the three-dimensional information further includes the three-dimensional coordinates of the three-dimensional frame diagram; the projection module 704 is specifically configured to determine a back-projection height based on the three-dimensional coordinates in the candidate three-dimensional information, where the back-projection height is the distance between the third and fourth sides of the three-dimensional patch frame diagram, where the third and fourth sides are opposite each other; back-project the candidate frame diagram based on the center point coordinates of the first side of the candidate frame diagram and the back-projection height to obtain a first three-dimensional patch frame sub-image; after back-projection, the first side is located on the third side; back-project the candidate frame diagram based on the center point coordinates of the second side of the candidate frame diagram and the back-projection height to obtain a second three-dimensional patch frame sub-image; the first side and the second side are opposite sides; determine a first intersection-and-union ratio (IoU) and a second IoU ratio (IoU); the first IoU ratio is the IoU ratio between the projection of the first three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame diagram, and the second IoU ratio is the IoU ratio between the projection of the second three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame diagram; and determine the three-dimensional patch frame sub-image corresponding to the largest IoU ratio between the first IoU ratio and the second IoU ratio as the three-dimensional patch frame diagram.

[0087] In some embodiments, the projection module 704 is specifically configured to determine the 3D patch frame sub-image within the target range as the 3D patch frame image.

[0088] In some embodiments, as Figure 9 As shown, the obstacle position recognition device also includes a calculation module 705 and a deletion module 706; the calculation module 705 is used to calculate the center point distance between each two candidate frame images for each frame of two-dimensional image data; the deletion module 706 is used to delete the candidate frame image with a smaller intersection-and-union ratio when the center point distance is less than a distance threshold and the intersection-and-union ratios of the two candidate frame images are both greater than a ratio threshold; the intersection-and-union ratio of the candidate frame images is the intersection-and-union ratio of the candidate frame image and its corresponding three-dimensional complement frame image.

[0089] The obstacle position identification device provided in this embodiment can execute the obstacle position identification method provided in the above method embodiment. Its implementation principle and technical effects are similar to those of the above method and will not be repeated here.

[0090] Figure 10An electronic device according to an exemplary embodiment may include a processor 802 configured to execute application code to implement the obstacle location recognition method of the present application.

[0091] The processor 802 may be a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0092] like Figure 10 As shown, the electronic device may further include a memory 803. The memory 803 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 802.

[0093] The memory 803 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 803 may exist independently and be connected to the processor 802 via the bus 804. The memory 803 may also be integrated with the processor 802.

[0094] like Figure 10 As shown, the electronic device may further include a communication interface 801, wherein the communication interface 801, the processor 802, and the memory 803 may be coupled to each other, for example, via a bus 804. The communication interface 801 is used to exchange information with other devices, for example, to support information exchange between the electronic device and other devices.

[0095] It should be pointed out that Figure 10 The device structure shown in the figure does not constitute a limitation on the electronic device, except Figure 9In addition to the components shown, the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. Furthermore, the electronic device provided in this embodiment can execute the obstacle location identification method provided in the above method embodiment. Its implementation principles and technical effects are similar to those of the above method and will not be further described here.

[0096] An embodiment of the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the various processes of the obstacle position identification method in the above-mentioned method embodiment are implemented, and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0097] The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0098] An embodiment of the present application provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the various processes of the obstacle position identification method in the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0100] In this application, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0102] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data and carrier waves.

[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0104] The foregoing description is intended only to provide specific embodiments of the present application, which will enable those skilled in the art to understand and implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments described herein, but is intended to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying an obstacle position, characterized in that: include: Acquire 3D point cloud data and 2D image data within a preset time period; Recognize and process the three-dimensional point cloud data and the two-dimensional image data respectively to obtain three-dimensional information and two-dimensional information of at least one obstacle; the three-dimensional information includes at least one three-dimensional block diagram, and the two-dimensional information includes at least one two-dimensional block diagram; For each obstacle, when a three-dimensional frame diagram corresponding to a first two-dimensional frame diagram of the obstacle does not exist, using the first two-dimensional frame diagram as a candidate frame diagram of the obstacle; The first two-dimensional frame diagram is a two-dimensional frame diagram in the first two-dimensional information, and the first two-dimensional information is generated by using the two-dimensional image data of the obstacle obtained at a first time point; the first time point is any time point within the preset time period; According to the first two-dimensional information and the candidate three-dimensional information, the candidate frame diagram is back-projected to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram; The candidate three-dimensional information is generated by using three-dimensional point cloud data of the obstacle acquired at a second time point, where the second time point is a time point adjacent to the first time point; The position of the three-dimensional frame patch image is determined as the position of the obstacle at the first time point.

2. The obstacle position recognition method according to claim 1, characterized in that: The two-dimensional information further includes a two-dimensional trajectory identifier, and the three-dimensional information further includes a three-dimensional trajectory identifier. For each obstacle, when a three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, before using the first two-dimensional frame diagram as a candidate frame diagram for the obstacle, the method further includes: For each obstacle, project at least one three-dimensional frame diagram into a two-dimensional coordinate system to obtain at least one projected frame diagram; the two-dimensional coordinate system is the coordinate system where the two-dimensional frame diagram is located; Within a first time period, when a first number of two-dimensional block diagrams match a first number of projection block diagrams, determining that the two-dimensional trajectory identifiers corresponding to the two-dimensional block diagrams are credible identifiers; the first time period is less than the preset time period; the two-dimensional block diagrams matching the projection block diagrams includes: the two-dimensional trajectory identifiers corresponding to the two-dimensional block diagrams and the three-dimensional trajectory identifiers corresponding to the projection block diagrams are the same, and the two-dimensional block diagrams and the projection block diagrams satisfy a greedy rule or a Hungarian matching rule; When a three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist, using the first two-dimensional frame diagram as a candidate frame diagram of the obstacle includes: When the two-dimensional trajectory identifier corresponding to the first two-dimensional frame diagram is a credible identifier and the three-dimensional frame diagram corresponding to the first two-dimensional frame diagram does not exist, the first two-dimensional frame diagram is used as a candidate frame diagram for the obstacle.

3. The obstacle position recognition method according to claim 2, characterized in that: The projecting of at least one three-dimensional frame diagram into a two-dimensional coordinate system to obtain at least one projected frame diagram includes: The three-dimensional frame diagram is projected into a two-dimensional coordinate system according to the minimum bounding rectangle principle to obtain at least one projection frame diagram; the minimum bounding rectangle principle refers to using the minimum bounding rectangle corresponding to the target three-dimensional frame diagram as the projection frame diagram of the target three-dimensional frame diagram; the target three-dimensional frame diagram is any three-dimensional frame diagram.

4. The obstacle position recognition method according to claim 1, characterized in that: The two-dimensional information further includes the two-dimensional coordinates of the two-dimensional frame diagram, and the three-dimensional information further includes the three-dimensional coordinates of the three-dimensional frame diagram; the back-projection processing of the candidate frame diagram based on the first two-dimensional information and the candidate three-dimensional information to obtain the three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram includes: Determine a back-projection height according to the three-dimensional coordinates in the candidate three-dimensional information; the back-projection height is the distance between the first surface and the second surface of the three-dimensional patch frame, the first surface and the second surface being opposite to each other; Back-projecting the candidate frame image according to the center point coordinates of the target edge in the candidate frame image and the back-projection height to obtain a three-dimensional patch frame sub-image; after the back-projection, the target edge is located on the first surface or the second surface; The three-dimensional frame patching sub-graph is determined as a three-dimensional frame patching graph.

5. The obstacle position recognition method according to claim 1, characterized in that: The two-dimensional information further includes the two-dimensional coordinates of the two-dimensional frame diagram, and the three-dimensional information further includes the three-dimensional coordinates of the three-dimensional frame diagram; the back-projection processing of the candidate frame diagram based on the first two-dimensional information and the candidate three-dimensional information to obtain the three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram includes: Determine a back-projection height according to the three-dimensional coordinates in the candidate three-dimensional information, wherein the back-projection height is the distance between the third surface and the fourth surface of the three-dimensional patch frame image, and the third surface and the fourth surface are opposite to each other; Back-projecting the candidate frame image according to the center point coordinates of the first side of the candidate frame image and the back-projection height to obtain a first three-dimensional patch frame sub-image; after the back-projection, the first side is located on the third surface; Back-projecting the candidate frame image according to the center point coordinates of the second side of the candidate frame image and the back-projection height to obtain a second three-dimensional filler frame sub-image; the first side and the second side are opposite sides; Determining a first intersection-and-union ratio (IOR) and a second IOR; wherein the first IOR is the IOR between the projection of the first three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame image, and the second IOR is the IOR between the projection of the second three-dimensional patch frame sub-image in the two-dimensional coordinate system and the candidate frame image; The three-dimensional frame patching subgraph corresponding to the largest intersection-and-union ratio between the first intersection-and-union ratio and the second intersection-and-union ratio is determined as the three-dimensional frame patching subgraph.

6. The obstacle position recognition method according to claim 4 or 5, characterized in that: After obtaining the three-dimensional frame-filling sub-image, the method further includes: The three-dimensional patching frame sub-image within the target range is determined as the three-dimensional patching frame image.

7. The obstacle position recognition method according to claim 4 or 5, characterized in that: For each obstacle, when a three-dimensional frame diagram corresponding to a first two-dimensional frame diagram of the obstacle does not exist, the method uses the first two-dimensional frame diagram as a candidate frame diagram of the obstacle. For each frame of two-dimensional image data, the center point distance between each two candidate frames is calculated; When the center point distance is less than the distance threshold and the intersection-and-union ratios of the two candidate frame diagrams are both greater than the ratio threshold, the candidate frame diagram with the smaller intersection-and-union ratio is deleted; the intersection-and-union of the candidate frame diagrams is the intersection-and-union ratio of the candidate frame diagram and its corresponding three-dimensional complement frame diagram.

8. An obstacle position recognition device, characterized in that: include: An acquisition module, used to acquire three-dimensional point cloud data and two-dimensional image data within a preset time period; an identification module, configured to perform identification processing on the three-dimensional point cloud data and the two-dimensional image data, respectively, to generate three-dimensional information and two-dimensional information of at least one obstacle; The three-dimensional information includes at least one three-dimensional block diagram, and the two-dimensional information includes at least one two-dimensional block diagram; a determination module configured to, for each obstacle, use the first two-dimensional frame diagram as a candidate frame diagram for the obstacle when a three-dimensional frame diagram corresponding to the first two-dimensional frame diagram of the obstacle does not exist; The first two-dimensional frame diagram is a two-dimensional frame diagram in the first two-dimensional information, and the first two-dimensional information is generated by using the two-dimensional image data of the obstacle obtained at a first time point; the first time point is any time point within the preset time period; a projection module, configured to perform back-projection processing on the candidate frame diagram according to the first two-dimensional information and the candidate three-dimensional information, to obtain a three-dimensional complementary frame diagram corresponding to the first two-dimensional frame diagram; The candidate three-dimensional information is generated by using three-dimensional point cloud data of the obstacle acquired at a second time point, where the second time point is a time point adjacent to the first time point; The determining module is further configured to determine the location of the three-dimensional frame patch image as the location of the obstacle at the first time point.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the obstacle position recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the obstacle position identification method according to any one of claims 1 to 7 is implemented.

Images