3D Data Processing Method, Apparatus, Device, and Storage Medium for High-Precision Maps

Abstract

The present disclosure provides an image processing method, apparatus, device, and computer storage medium, which relate to the field of computer technologies, and particularly to technologies such as autonomous driving. The specific implementation solution is as follows: obtaining a first image frame captured by an image capturing device; establishing an association relationship between a first element in the first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data acquisition device, and the three-dimensional data acquisition device and the image capturing device are in a set relative position; and determining the three-dimensional position of the first element according to the association relationship. The embodiments of the present disclosure can provide good assistance for map generation and processing work.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to fields such as autonomous driving technology. Background Art

[0002] With the development of computer technology, mobile terminal technology has also advanced rapidly, and mobile terminal technology is involved in all aspects of people's clothing, food, housing, and transportation. For example, in terms of travel, due to factors such as the acceleration of urbanization construction and the development of the tourism industry, the frequency of mobile terminal users using electronic maps is increasing, and the dependence of users on electronic maps is also increasing, thus continuously putting forward higher requirements for electronic maps.

[0003] With the development of electronic maps, products such as high-precision maps have emerged, and at the same time, more requirements have been put forward for the production and manufacturing of electronic maps. It is necessary to improve the accuracy of data collection and the frequency of data update for making higher-quality maps, and at the same time, it is necessary to quickly adjust the map in the case of changes in the road environment. Therefore, it is necessary to improve the generation and manufacturing steps and methods of electronic maps in order to improve the quality of the generated maps. Summary of the Invention

[0004] The present disclosure provides a three-dimensional data processing method, apparatus, device, and storage medium. Figure 3 According to one aspect of the present disclosure, a three-dimensional data processing method is provided, including:

[0005] Obtaining a first image frame captured by an image capturing device;

[0006] Establishing an association relationship between a first element in the first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data collection device, and the three-dimensional data collection device and the image capturing device are in a set relative position;

[0007] Determining the three-dimensional position of the first element according to the association relationship.

[0008] According to another aspect of the present disclosure, a three-dimensional data processing apparatus is provided, including:

[0009] A first image frame obtaining module, configured to obtain a first image frame captured by an image capturing device;

[0010] An association relationship establishing module, configured to establish an association relationship between a first element in the first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data collection device, and the three-dimensional data collection device and the image capturing device are in a set relative position;

[0011] A three-dimensional position determining module, configured to determine the three-dimensional position of the first element according to the association relationship.

[0012] A three-dimensional position determination module, configured to determine the three-dimensional position of a first element according to an association relationship.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method in any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the method in any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the method in any embodiment of the present disclosure is implemented.

[0019] According to the technology of the present disclosure, the first element in the first image frame can be associated with a three-dimensional coordinate system, thereby realizing the three-dimensional visualization of elements in a two-dimensional image frame, automatically optimizing the 3D position of point elements, and in the annotation tool for map original acquisition data, the annotation result visualization can be realized based on the three-dimensional position data, facilitating the subsequent annotation work of map operators on the map.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0022] Figure 1 is a schematic diagram of a three-dimensional data processing method according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a three-dimensional data processing method according to another embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a three-dimensional data processing method according to still another embodiment of the present disclosure;

[0025] Figure 4Schematic diagram of a three-dimensional data processing method according to another embodiment of the present disclosure;

[0026] Figure 5 Schematic diagram of a three-dimensional data processing method according to another embodiment of the present disclosure;

[0027] Figure 6 Schematic diagram of a three-dimensional data processing method according to another embodiment of the present disclosure;

[0028] Figure 7 Schematic diagram of a three-dimensional data processing method according to an example of the present disclosure;

[0029] Figure 8 Schematic diagram of a three-dimensional data processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 9 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0031] Figure 10 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0032] Figure 11 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0033] Figure 12 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0034] Figure 13 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0035] Figure 14 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0036] Figure 15 Schematic diagram of a three-dimensional data processing apparatus according to another embodiment of the present disclosure;

[0037] Figure 16 Block diagram of an electronic device for implementing the three-dimensional data processing method of the embodiments of the present disclosure. Detailed implementation manners

[0038] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted below for clarity and conciseness.

[0039] According to an embodiment of the present disclosure, a three-dimensional data processing method is provided. Figure 1 FIG. Figure 1 is a schematic flowchart of a three-dimensional data processing method according to an embodiment of the present disclosure. This method can be applied to a three-dimensional data processing device. For example, when the device is deployed on a terminal, server, or other processing device for execution, steps such as obtaining a synthetic image frame and establishing an association relationship can be performed. Among them, the terminal can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementation manners, this method can also be implemented by a processor invoking computer-readable instructions stored in a memory. As Figure 1 shown, the three-dimensional data processing method includes:

[0040] Step S11: Obtain a first image frame captured by an image capturing device.

[0041] Step S12: Establish an association relationship between a first element in the first image frame and a map with three-dimensional (3D) information. The map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data acquisition device. The three-dimensional data acquisition device and the image capturing device are in a set relative position.

[0042] Step S13: Determine the three-dimensional position of the first element according to the association relationship.

[0043] In this embodiment, the image capturing device can be a device such as a camera, scanner, camera, image signal sensor, etc. that can obtain images of the surrounding environment.

[0044] The first image frame can be an image captured by the image capturing device, or a frame in a video captured by the image capturing device.

[0045] The first element in the first image frame can be a composition, object, icon, etc. in the first image frame. For example, it can be an object such as a road surface, building, icon drawn on the road, road sign set on both sides of the road, road fence, bus stop sign, etc. that can be captured in the image frame.

[0046] In a possible implementation manner, the first element is a planar element, that is, the element can be approximately regarded as a plane in the real world. For example, an icon drawn on the road, a sign set on both sides of the road, the side of a building, the top surface of a building, the road surface, the side of a vehicle, a road barrier, etc.

[0047] A map with three-dimensional information can be a map in which all points have coordinates in the world coordinate system, or a map mainly composed of objects or elements with three-dimensional coordinate information, or a map in which all or some points have coordinates in an arbitrary three-dimensional coordinate system.

[0048] In a map with three-dimensional information, it can also contain a large number of elements with three-dimensional coordinate information, which can be point elements, line elements, surface elements, etc. Establishing an association relationship between the first element in the first image frame and the map with three-dimensional information can be establishing an association relationship between the element in the first image frame and the element with three-dimensional coordinates in the map. Since the element in the first image frame only has the two-dimensional coordinate information presented in the first image frame, that is, the pixel coordinate information within the image frame plane, but the first image frame is an image taken of the real environment, the object corresponding to the element in the first image frame has real coordinates in the three-dimensional world coordinate system. Therefore, it is possible to establish an association relationship between the element in the first image frame and the element representing the same object in the map with three-dimensional information.

[0049] The three-dimensional data acquisition device and the image capturing device are in a set relative position, which can be that the three-dimensional data acquisition device and the image capturing device are in a fixed relative position, so that according to the specific position of one of them and the set relative position, the specific position of the other can be deduced.

[0050] In a possible implementation, the position of the three-dimensional data acquisition device, the position of the image capturing device, or their relative positions can include any relative data related to position and direction, such as their relative position, relative angle, relative pose, etc. in the same three-dimensional or two-dimensional reference coordinate system.

[0051] Determining the three-dimensional position of the first element according to the association relationship can be determining the position of the first element in the three-dimensional coordinate system according to the association relationship, that is, determining the three-dimensional coordinates of the first element. Specifically, the three-dimensional equation expression of the plane where the first element is located can be determined, or the three-dimensional coordinates of a finite number of points on the first element can be determined, or the plane expression of the first element can be determined.

[0052] In a possible implementation, the operations shown can be performed for each element in the first image frame Figure 1 so that each element in the first image frame is associated with the map.

[0053] The high-precision point cloud map is a type of high-precision map that can be generated based on the point cloud data collected by radar. During the process of creating the high-precision point cloud map, it is necessary to label the data such as images and point clouds collected by the map acquisition device, and mark the three-dimensional positions of point elements such as arrows and signs. However, there are certain difficulties in labeling the three-dimensional positions of elements. For example, when labeling elements in the basement scenario, there is a problem that the point cloud is blurred and difficult to label. In this embodiment, the first element in the first image frame can be associated with the three-dimensional coordinate system, thereby realizing the three-dimensional visualization of the element in the two-dimensional image frame, automatically optimizing the 3D position of the point element, and visualizing the labeling result in the tool, which is convenient for the map operator to perform subsequent map labeling work.

[0054] In one implementation, an association relationship is established between the first element in the first image frame and the map with three-dimensional information, as Figure 2 shown, including:

[0055] Step S21: Determine the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point in the first image frame according to the three-dimensional coordinates of the three-dimensional point in the map and the set relative position;

[0056] Step S22: Determine the three-dimensional coordinates of the key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point;

[0057] Step S23: Use the three-dimensional coordinates of the key points as the association relationship.

[0058] In this embodiment, the three-dimensional point in the map can be any three-dimensional point with clear three-dimensional coordinates in the map.

[0059] The three-dimensional point can be a point with coordinate data in three dimensions, that is, a point with three-dimensional coordinates; the two-dimensional point can be a point with coordinate data in only two dimensions, that is, a point with two-dimensional coordinates. The three-dimensional coordinates of the three-dimensional point in the map can be the coordinates of the three-dimensional point in the three-dimensional space determined according to the map data. The aforementioned three-dimensional space can be the reference coordinate system space of the map or the world coordinate system space.

[0060] The first two-dimensional point corresponding to the three-dimensional point can be the two-dimensional point in the first image frame that is the same point as the three-dimensional point.

[0061] The first image frame can be regarded as an image frame composed of multiple two-dimensional points. The points in the first image frame only have coordinates in the two-dimensional plane where the image frame is located. Therefore, the points in the first image frame are all two-dimensional points with only two-dimensional coordinates in the pixel coordinate system of the image frame.

[0062] In a possible implementation, since it is impossible to mark the coordinates of all points in the three-dimensional space when creating a map, the three-dimensional points marked with three-dimensional coordinate data when creating the map are only a finite number of points in the map.

[0063] The key points included in the first element can be the points set on the first element. In the case where the first element contains multiple classifications, for each classification of the first element, the key points can be determined according to the set rules. For example, in the case where the first element is a rectangular long line segment, the key points included in the first element can be the points at the four corners of the rectangular long line segment. In the case where the first element is a planar arrow, the key points included in the first element can include the point at the tip of the arrow, the line segment endpoints of the triangular head of the arrow, the points at the four corners of the rectangular part of the arrow, etc. In the case where the first element is a planar circular element, the key points included in the first element can include the center of the circle and any point on the circumference. In the case where the first element is a cube element, the key points included in the first element can be the points at the eight corners of the cube element.

[0064] Since the finite number of three-dimensional points in the map have known three-dimensional coordinates without additional calculation, and the two-dimensional points corresponding to the three-dimensional points with known three-dimensional coordinates in the map are not exactly the key points of the elements. At the same time, in the process of creating the association relationship between the elements in the first image frame and the map, it is necessary to know the key points of each element in the real environment. Therefore, it is necessary to calculate the three-dimensional coordinate data of the key points according to the three-dimensional points with known three-dimensional coordinate data.

[0065] In this embodiment, the three-dimensional coordinates of the key points of the elements in the first image frame can be determined according to the three-dimensional points with known three-dimensional coordinates in the map, so that the two-dimensional coordinate points in the first image frame can be associated with the three-dimensional position information in the map, which is convenient for performing map element annotation and other work according to the first image frame.

[0066] In one implementation, determining the three-dimensional coordinates of the key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point includes:

[0067] In the case where the key point overlaps with the first two-dimensional point, the three-dimensional coordinates of the three-dimensional point are used as the three-dimensional coordinates of the key point;

[0068] In the case where the key point does not overlap with the first two-dimensional point, the three-dimensional coordinates of the key point are determined according to the two-dimensional coordinates of the first two-dimensional point, the relative position between the key point and the first two-dimensional point, and the three-dimensional coordinates of the three-dimensional point.

[0069] In this embodiment, according to the different overlapping situations between the key points and the first two-dimensional points, the three-dimensional coordinates corresponding to the three-dimensional points can be directly used as the three-dimensional coordinates of the key points, or the three-dimensional coordinates of the key points can be calculated based on the three-dimensional coordinates of the three-dimensional points, so as to determine the three-dimensional coordinates of the key points of each element in the first image frame.

[0070] In one implementation manner, when the first element is a planar element, according to the two-dimensional coordinates of the first two-dimensional points, the relative positions between the key points and the first two-dimensional points, and the three-dimensional coordinates of the three-dimensional points, the three-dimensional coordinates of the key points are determined, as Figure 3 shown, including:

[0071] Step S31: In the map, determine the first plane passing through the three-dimensional point and parallel to the planar element;

[0072] Step S32: Determine the straight line obtained by back-projecting the key point into the three-dimensional space;

[0073] Step S33: Use the three-dimensional coordinates of the intersection point of the first plane and the straight line as the three-dimensional coordinates of the key point.

[0074] The first plane passing through the three-dimensional point and parallel to the planar element can be a plane passing through the three-dimensional point and parallel to the plane where the first element is located in the three-dimensional coordinate system.

[0075] Since the key point has two-dimensional coordinates in the pixel coordinate system of the first image frame, and the image capturing device of the first image frame is in a relatively fixed position and pose with respect to the data acquisition device of the map, therefore, according to the key point, a straight line passing through the key point in the three-dimensional space can be determined.

[0076] In this embodiment, the three-dimensional coordinates of the intersection point of the first plane and the straight line are used as the three-dimensional coordinates of the key point, so as to be able to determine the unique three-dimensional coordinates corresponding to the key point in the map.

[0077] In one implementation manner, according to the three-dimensional coordinates of the three-dimensional points in the map and the set relative positions, the two-dimensional coordinates of the first two-dimensional points corresponding to the three-dimensional points in the first image frame are determined, including:

[0078] In the map, determine the second plane within the set range of the position of the image capturing device;

[0079] According to the pose information of the image capturing device, the equation of the second plane in the three-dimensional space, the internal parameters of the image capturing device, and the pixel coordinates of the first element in the pixel plane of the first image frame, project the three-dimensional points in the second plane onto the pixel plane of the first image frame to obtain the two-dimensional coordinates of the first two-dimensional points.

[0080] The second plane within the set range of the position of the image capturing device can be any plane that actually exists in the three-dimensional space near the image capturing device.

[0081] In this embodiment, the three-dimensional points in the second plane can be projected onto the pixel plane of the first image frame, so as to establish an association relationship between the points in the three-dimensional map and the points in the first image frame.

[0082] In one implementation, as Figure 4 shown, the three-dimensional data processing method further includes:

[0083] Step S41: Project the three-dimensional points onto the pixel plane of the image frame to obtain second two-dimensional points;

[0084] Step S42: Determine a first error according to the position difference between the second two-dimensional points and the first two-dimensional points;

[0085] Step S43: Adjust the three-dimensional position of the element according to the first error.

[0086] In this embodiment, projecting the three-dimensional points onto the pixel plane of the image frame to obtain second two-dimensional points may be to further re-project the three-dimensional points that have been projected in the previous steps to obtain second two-dimensional points. Since in the process of re-projection, the three-dimensional points to be projected and the three-dimensional points corresponding to the first two-dimensional points are the same three-dimensional points, therefore, ideally, the first two-dimensional points should coincide with the second two-dimensional points.

[0087] In the case where the first two-dimensional points and the second two-dimensional points do not coincide, based on the second two-dimensional points obtained by the secondary re-projection, the error between the two can be determined, so as to determine the possible error when associating the elements in the first image frame with the map, and then the three-dimensional position of the element can be adjusted according to the error.

[0088] In one implementation, determining the first error according to the second two-dimensional points and the first two-dimensional points includes:

[0089] Calculate the first error according to the pose of the image capturing device, the internal parameters of the image capturing device, the two-dimensional coordinates of the second two-dimensional points, and the scale data.

[0090] In this embodiment, the first error can be calculated according to the information of the first two-dimensional points and the information of the second two-dimensional points, so that the three-dimensional position of the first element can be corrected according to the first error, making the three-dimensional position of the first element more accurate.

[0091] In one implementation, when the first element is a planar element, as Figure 5 shown, the three-dimensional data processing method further includes:

[0092] Step S51: Determine the connection line between the three-dimensional coordinates of the key points included in the first element and the three-dimensional coordinates of any point in the three-dimensional plane corresponding to the first element in the map;

[0093] Step S52: Determine the coplanarity degree of the connection line and the three-dimensional plane;

[0094] Step S53: Determine the second error according to the coplanarity degree;

[0095] Step S54: Adjust the three-dimensional coordinates of the first element according to the second error.

[0096] In this embodiment, when the first element is a planar element, any three points on the first element should be coplanar points. Based on this, the three-dimensional position of the first element can be optimized and adjusted to make the three-dimensional position of the first element more accurate.

[0097] In one implementation manner, adjusting the three-dimensional position of the element according to the second error includes:

[0098] Calculate the total error according to the first error and the second error;

[0099] Adjust the three-dimensional position of the first element according to the total error.

[0100] In this embodiment, the first error and the second error can be weighted and summed, so as to be able to adjust the three-dimensional position of the first element as a whole, making the overall accuracy of the three-dimensional position of the first element higher.

[0101] In one implementation manner, determining the coplanarity degree of the connection line and the three-dimensional plane includes:

[0102] Calculate the coplanarity degree according to the three-dimensional coordinates of the key points included in the first element, the three-dimensional coordinates of any point, and the normal vector of the three-dimensional plane.

[0103] In this embodiment, the coplanarity degree of the connection line between the key points and other points on the first element and the plane parallel to the first element can be calculated according to the three-dimensional coordinates of the key points, the three-dimensional coordinates of any point, and the normal vector of the three-dimensional plane, so as to be able to determine the coplanarity error of the three-dimensional position of the first element.

[0104] In one implementation manner, the three-dimensional data processing method further includes:

[0105] Determine a second element corresponding to the first element in the second image frame, where the second image frame is an image frame sorted after the first image frame in time sequence;

[0106] Determine the three-dimensional position of the second element according to the three-dimensional position of the first element.

[0107] In this embodiment, the second image frame can be an image frame before or after the first image frame, and can be captured by the image capturing device before or after the first image frame. The second image frame can be any image frame that contains the physical object corresponding to the first element in the picture.

[0108] Determining the second element corresponding to the first element may include determining that the first element and the second element correspond to the same physical object, that is, they are actually the manifestations of the same element in different image frames.

[0109] Determining the three-dimensional position of the second element according to the three-dimensional position of the first element may be taking the three-dimensional position of the first element as the three-dimensional position of the second element.

[0110] In this embodiment, it is possible to determine the three-dimensional position of the second element according to the three-dimensional position of the first element, so that the elements in multiple image frames obtained by the image capturing device can be associated with the map, facilitating the execution of subsequent operations related to generating map data according to the image frames.

[0111] In one implementation manner, determining the second element corresponding to the first element in the second image frame, as Figure 6 shown, includes:

[0112] Step S61: Determine the second element according to the intersection over union of all elements in the second image frame and the first element;

[0113] Step S62: Determine the three-dimensional coordinates of the key points of the second element according to the preset numbers of the key points of the second element, the three-dimensional coordinates of the key points included in the first element, and the preset numbers of the key points included in the first element;

[0114] Step S63: Take the three-dimensional coordinates of the key points of the second element as the three-dimensional position of the second element.

[0115] In this embodiment, determining the second element according to the intersection over union of all elements in the second image frame and the first element may be taking the element in the second image frame with the largest overlap degree with the first element as the second element.

[0116] Determining the second element according to the intersection over union of all elements in the second image frame and the first element may also be taking the elements in the second image frame whose intersection over union meets the threshold limit as the second elements.

[0117] In this embodiment, determining the second element through the intersection over union can associate the image frames with the same element one by one without separate calculation, thus simplifying the operation and the calculation process.

[0118] In one implementation manner, determining the second element corresponding to the first element in the second image frame includes:

[0119] Project the three-dimensional points corresponding to the key points included in the first element onto the plane where the second image frame is located to obtain the projected points;

[0120] Determine the three-dimensional coordinates of the key points of the second element according to the key points of the second element closest to the projected points, the types of the key points of the second element, and the types of the key points included in the first element;

[0121] Determine the three-dimensional position of the second element according to the three-dimensional coordinates of the key points of the second element.

[0122] In this embodiment, for some elements of set types, numbers can be assigned to the key points of the elements so that specific key points can be determined through the elements and the numbers. For example, for an element of the planar arrow type, the key point at its tip can be set as the first number, and in a clockwise or counterclockwise direction, the other key points are set as the second number, the third number, the fourth number... For key points of other shapes, key points with high recognizability can also be selected as the first number, and the remaining key points are sequentially set as the second number, the third number, etc. in a clockwise or counterclockwise direction.

[0123] In this embodiment, the three-dimensional coordinates of the key points of the elements can be determined according to the correspondence relationship between the elements and the numbers of the key points among different image frames, and then the three-dimensional positions of the elements with a correspondence relationship in different image frames can be determined, thereby reducing the amount of calculation and improving the efficiency of determining the association relationship and the three-dimensional positions of the elements.

[0124] In an example of the present disclosure, the three-dimensional data processing method includes steps as Figure 7 shown:

[0125] Step S71: Initialize the 3D position of the element key points.

[0126] During the initialization process, the timestamp of the first image frame, the element, the element key points, the map with three-dimensional position information, the trajectory of the three-dimensional data acquisition device, the fixed extrinsic parameters from the image capture device to the three-dimensional data acquisition device, and the intrinsic matrix of the image capture device can be used as input data or data to be processed.

[0127] During the processing of the input data or data to be processed, the pose of the image capture device corresponding to the first image frame can be determined through data such as the timestamp of the first image frame, the trajectory of the three-dimensional data acquisition device, the fixed extrinsic parameters from the image capture device to the three-dimensional data acquisition device, and the intrinsic matrix of the image capture device.

[0128] In a possible implementation, the three-dimensional data acquisition device can be a radar, and the three-dimensional data collected by the radar can be point cloud data.

[0129] In a possible implementation, the image capturing device may be a camera, and the first image frame may be an image frame in a video captured by the camera.

[0130] The 3D data acquisition device and the image capturing device may be simultaneously located on the same moving road data acquisition device, such as a drone, a road data acquisition vehicle, etc. Since the road data acquisition device is in a moving state, the pose information of the image capturing device such as a camera relative to the world coordinate system may also be in a changing process. In this case, it is necessary to calculate the pose of the image capturing device according to the trajectory of the 3D data acquisition device or the trajectory of the road data acquisition device. Considering that both the 3D data acquisition device and the image capturing device acquire images or 3D data at certain time intervals, when the acquisition time of the first image frame is inconsistent with the acquisition time of the trajectory of the 3D data acquisition device, the pose of the image capturing device at the moment of obtaining the first image frame can be determined by means of interpolation calculation.

[0131] After determining the pose of the image capturing device, the key points in the first image frame can be associated and the key points in the first image frame can be associated to the map.

[0132] In a possible implementation, the map may be a Mesh (Wireless Mesh Network) map, which includes 3D objects composed of triangular faces. Among the triangular faces, there are a finite number of points with determined 3D coordinate data, specifically, the points at the three corners of the triangle. During the association process, the triangular faces around the image capturing device can be first determined. For example, the triangular faces within N×M meters around the image capturing device can be determined. All the 3D points with 3D coordinates in the selected triangular faces are projected onto the pixel plane where the first image frame is located. Since the 2D points in the first image frame only have 2D coordinate data, there may be a situation where multiple 3D points are projected onto the same 2D point among the projected 3D points. If so, the 3D point closest to the pose of the image capturing device is selected as the 3D point associated with the 2D point. If the 3D point with known 3D coordinates coincides with the key point included in the first element in the pixel plane of the first image frame after projection, the 3D coordinates of the 3D point that coincides with the key point included in the first element are used as the 3D coordinates of the key point included in the first element.

[0133] When the 3D coordinates of the 3D point are known, the corresponding 2D point after the 3D point is projected onto the pixel plane can be calculated by the following formula:

[0134]

[0135] u is the coordinate of the two-dimensional point projected from the three-dimensional point; s is the scale factor; K is the internal parameter of the image capture device; T is the camera pose data; P can be the three-dimensional point position in the world coordinate system.

[0136] If the three-dimensional point with known three-dimensional coordinates does not coincide with the key point included in the first element after being projected onto the pixel plane, then according to the key point A included in the first element of the associated one with a relatively short distance, the corresponding triangular surface can be found, the plane where the triangular surface is located can be parameterized, and the plane plane1 is obtained. Back-project A into the 3D space to get a straight line, parameterize the straight line, and calculate the intersection point of the straight line and plane1, which is the 3D point associated with A.

[0137] Step S72: Associate the two-dimensional points in the first image frame with the three-dimensional data in the map.

[0138] In this step, according to the initialization result, the key points included in the first element in the first image frame can be associated with the three-dimensional points in the map.

[0139] Step S73: According to the relationship between the elements in other image frames and the first element, associate the key points of the elements in other image frames with the map.

[0140] For example, the second element in the next image frame of the first image frame can be determined. The second element is associated with the first element, so that the three-dimensional position of the second element can be determined according to the three-dimensional position of the first element. The first element and the second element can be elements representing the same actual object in different image frames.

[0141] For other elements that do not have corresponding elements in the first image frame, the three-dimensional position can be determined in the same way as determined by the three-dimensional position of the first element.

[0142] For a certain keypoint in the current frame, if it is not associated with a 3D point, initialize and associate it.

[0143] At this time, all keypoints in this image frame have been associated with 3D points. Through the association information of this image frame, part of the 2D-to-3D data association of the next adjacent image frame is constructed.

[0144] For all the image frames obtained by the image capturing device, the association relationship between the first element and the second element, as well as the association relationships of the elements in all different image frames representing the same actual object, can be determined by constructing the element-level association between each image frame and the next adjacent frame image frame. Project each 3D element of the current frame onto the next adjacent frame to obtain a projected Polygon, calculate the IoU (Intersection over Union) between the projected Polygon and the candidate matching element, retain the candidate elements with IoU > 50%, and if there are multiple candidate elements, retain the element with the largest IoU.

[0145] In the case of determining the association relationship between elements, the association of keypoints within the element can be constructed. The index of the keypoints has a fixed order and a natural association. Associations with a pixel distance > 15.0 can be rejected, so that the three-dimensional coordinates of at least some keypoints of the next frame image can be determined through the association relationship between elements.

[0146] In the case of determining the association relationship between elements, the keypoints of the previous image frame can also be projected onto the next image frame to obtain a two-dimensional point B on the next image frame. Search for the keypoint C near the projected two-dimensional point B in the next image frame. In the case where C is the keypoint of the same type closest to B, it is determined as the keypoint associated with B, and thus the three-dimensional coordinates of C can be determined based on the three-dimensional coordinates of B.

[0147] Step S74: Output the association result.

[0148] For the current image frame, the three-dimensional coordinates corresponding to all the keypoints of the current frame in the map can be output. For the next adjacent image frame of the current image frame, the three-dimensional coordinates of at least some keypoints of at least some elements in the next image frame can be determined.

[0149] Step S75: Optimize the association result according to the reprojection error and the coplanarity constraint error.

[0150] In this step, when calculating the reprojection error, the point corresponding to the three-dimensional coordinates of each keypoint within the image frame can be projected onto the pixel plane of the image frame to obtain a reprojected point, and the reprojection error can be calculated based on the error between the reprojected point and the keypoint.

[0151] Specifically, given the pixel coordinates u of a certain keypoint i and the corresponding camera pose T k , and the associated 3D point coordinates P i , the camera internal parameter K, and the scale coordinates s corresponding to the keypointi , the calculation formula of the reprojection error e_proj is:

[0152]

[0153] Since the mesh map includes triangular faces, a coplanarity constraint can be calculated for the 3D points corresponding to the keypoints in the image frame. Given the 3D coordinates P of a certain keypoint i , with the normal vector n i , and the coordinates P of another point on the mesh that is closer to this point m , the calculation formula of the coplanarity constraint error e_cop is:

[0154] e_cop = ((P i - P m ) · n i ) 2 .

[0155] Assume there are m image capture devices, and n keypoints are extracted from the image frames captured by each image capture device; assume that the 3D points of l keypoints are finally reconstructed, and a weight W is set to balance the dimension problem, then the calculation formula of the total error can be:

[0156]

[0157] In the example of the present disclosure, the key point positions of the elements are detected on the image frames obtained by the image capture device, and with the help of the constructed mesh map, the initial 3D positions of the key points of the elements are obtained. Since the same physical element in the real environment may exist in different image frames captured by different cameras, and there may be a reprojection error between the 3D coordinates of the key points of the elements representing the same physical object in multiple image frames, Bundle Adjustment is used to adjust the association results. At the same time, considering the prior information that the key points of the same element are coplanar, a coplanarity constraint is constructed with the triangular patches in the mesh map. Thus, this example can realize the automatic optimization of the 3D positions of the point elements, and realize the visualization of the 3D coordinates of the key points and the 3D positions of the elements in the map data processing tool, which is convenient for the map processing operators to mark.

[0158] The embodiment of the present disclosure also provides a three-dimensional data processing device, as Figure 8 shown, including:

[0159] The first image frame acquisition module 81, which is used to acquire the first image frame captured by the image capture device;

[0160] An association relationship establishment module 82, configured to establish an association relationship between a first element in a first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data acquisition device, and the three-dimensional data acquisition device and the image capturing device are in a set relative position;

[0161] A three-dimensional position determination module 83, configured to determine the three-dimensional position of the first element according to the association relationship.

[0162] In one implementation, as Figure 9 shown, the association relationship establishment module includes:

[0163] A two-dimensional coordinate determination unit 91, configured to determine the two-dimensional coordinates of a first two-dimensional point corresponding to the three-dimensional point in the first image frame according to the three-dimensional coordinates of the three-dimensional point in the map and the set relative position;

[0164] A three-dimensional coordinate determination unit 92, configured to determine the three-dimensional coordinates of the key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point;

[0165] A three-dimensional coordinate processing unit 93, configured to use the three-dimensional coordinates of the key points as the association relationship.

[0166] In one implementation, the three-dimensional coordinate determination unit is further configured to:

[0167] When the key point overlaps with the first two-dimensional point, use the three-dimensional coordinates of the three-dimensional point as the three-dimensional coordinates of the key point;

[0168] When the key point does not overlap with the first two-dimensional point, determine the three-dimensional coordinates of the key point according to the two-dimensional coordinates of the first two-dimensional point, the relative position between the key point and the first two-dimensional point, and the three-dimensional coordinates of the three-dimensional point.

[0169] In one implementation, the three-dimensional coordinate determination unit is further configured to:

[0170] In the map, determine a first plane passing through the three-dimensional point and parallel to the planar element;

[0171] Determine the line obtained by back-projecting the key point into the three-dimensional space;

[0172] Use the three-dimensional coordinates of the intersection point of the first plane and the line as the three-dimensional coordinates of the key point.

[0173] In one implementation, the two-dimensional coordinate determination unit is further configured to:

[0174] In the map, determine a second plane within the set range of the position where the image capturing device is located;

[0175] According to the pose information of the image capturing device, the equation of the second plane in the three-dimensional space, the internal parameters of the image capturing device, and the pixel coordinates of the first element in the pixel plane of the first image frame, project the three-dimensional points in the second plane onto the pixel plane of the first image frame to obtain the two-dimensional coordinates of the first two-dimensional points.

[0176] In one implementation, as Figure 10 shown, the three-dimensional data processing device further includes:

[0177] A second two-dimensional point obtaining module 101, configured to project three-dimensional points onto the pixel plane of the image frame to obtain second two-dimensional points;

[0178] A first error determination module 102, configured to determine a first error according to the position difference between the second two-dimensional points and the first two-dimensional points;

[0179] A first adjustment module 103, configured to adjust the three-dimensional position of the element according to the first error.

[0180] In one implementation, as Figure 11 shown, the first error determination module includes:

[0181] A first calculation unit 111, configured to calculate a first error according to the pose of the image capturing device, the internal parameters of the image capturing device, the two-dimensional coordinates of the second two-dimensional points, and the scale data.

[0182] In one implementation, when the first element is a planar element, as Figure 12 shown, the three-dimensional data processing device further includes:

[0183] A connection determination module 121, configured to determine the connection line between the three-dimensional coordinates of the key points included in the first element and the three-dimensional coordinates of any point in the three-dimensional plane corresponding to the first element in the map;

[0184] A coplanarity determination module 122, configured to determine the coplanarity between the connection line and the three-dimensional plane;

[0185] A second error determination module 123, configured to determine a second error according to the coplanarity;

[0186] A second adjustment module 124, configured to adjust the three-dimensional coordinates of the first element according to the second error.

[0187] In one implementation, as Figure 13 shown, the second adjustment module includes:

[0188] A total error unit 131, configured to calculate a total error according to the first error and the second error;

[0189] The total error processing unit 132 is configured to adjust the three-dimensional position of the first element according to the total error.

[0190] In one embodiment, as Figure 14 shown, the coplanarity determination module includes:

[0191] The second calculation unit 141 is configured to calculate the coplanarity according to the three-dimensional coordinates of the key points included in the first element, the three-dimensional coordinates of any point, and the normal vector of the three-dimensional plane.

[0192] In one embodiment, as Figure 15 shown, the three-dimensional data processing device further includes:

[0193] The second image frame determination unit 151 is configured to determine a second element corresponding to the first element in the second image frame, where the second image frame is an image frame sorted after the first image frame in time sequence;

[0194] The second element determination unit 152 is configured to determine the three-dimensional position of the second element according to the three-dimensional position of the first element.

[0195] In one embodiment, the second image frame determination unit is further configured to:

[0196] Determine the second element according to the intersection over union of all elements in the second image frame and the first element;

[0197] Determine the three-dimensional coordinates of the key points of the second element according to the preset numbers of the key points of the second element, the three-dimensional coordinates of the key points included in the first element, and the preset numbers of the key points included in the first element;

[0198] Use the three-dimensional coordinates of the key points of the second element as the three-dimensional position of the second element.

[0199] In one embodiment, the second image frame determination unit is further configured to:

[0200] Project the three-dimensional points corresponding to the key points included in the first element onto the plane where the second image frame is located to obtain projection points;

[0201] Determine the three-dimensional coordinates of the key points of the second element according to the key points of the second element closest to the projection points, the types of the key points of the second element, and the types of the key points included in the first element;

[0202] Determine the three-dimensional position of the second element according to the three-dimensional coordinates of the key points of the second element.

[0203] The embodiments of the present disclosure can be applied to the field of computer technology, and particularly to the field of intelligent transportation technology.

[0204] In the technical solutions of the present disclosure, the acquisition, storage, and application of the user's personal information comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0205] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0206] Figure 16 FIG. shows a schematic block diagram of an exemplary electronic device 1600 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0207] As Figure 16 shown, the device 1600 includes a computing unit 1601 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 16016 into a random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the device 1600 can also be stored. The computing unit 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0208] A plurality of components in the device 1600 are connected to the I / O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1607, such as various types of displays, speakers, etc.; a storage unit 16016, such as a magnetic disk, an optical disk, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows the device 1600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0209] The computing unit 1601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 executes the various methods and processes described above, such as the three-dimensional data processing method. For example, in some embodiments, the three-dimensional data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 16016. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1600 via the ROM 1602 and / or the communication unit 1609. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, one or more steps of the three-dimensional data processing method described above can be executed. Alternatively, in other embodiments, the computing unit 1601 can be configured to execute the three-dimensional data processing method in any other suitable manner (e.g., by means of firmware).

[0210] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0211] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0212] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0213] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0214] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0215] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0216] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0217] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A three-dimensional data processing method, comprising: Obtaining a first image frame captured by an image capturing device; Establishing an association relationship between a first element in the first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data acquisition device, and the three-dimensional data acquisition device and the image capturing device are in a set relative position; Determining the three-dimensional position of the first element according to the association relationship; The step of establishing an association relationship between the first element in the first image frame and the map with three-dimensional information includes: determining the two-dimensional coordinates of a first two-dimensional point corresponding to the three-dimensional point in the first image frame according to the three-dimensional coordinates of the three-dimensional points in the map and the set relative position; Determining the three-dimensional coordinates of key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point; Taking the three-dimensional coordinates of the key points as the association relationship.

2. The method according to claim 1, wherein, The step of determining the three-dimensional coordinates of key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point includes: When the key point overlaps with the first two-dimensional point, taking the three-dimensional coordinates of the three-dimensional point as the three-dimensional coordinates of the key point; When the key point does not overlap with the first two-dimensional point, determining the three-dimensional coordinates of the key point according to the two-dimensional coordinates of the first two-dimensional point, the relative position between the key point and the first two-dimensional point, and the three-dimensional coordinates of the three-dimensional point.

3. The method according to claim 2, wherein, When the first element is a planar element, the step of determining the three-dimensional coordinates of the key point according to the two-dimensional coordinates of the first two-dimensional point, the relative position between the key point and the first two-dimensional point, and the three-dimensional coordinates of the three-dimensional point includes: In the map, determining a first plane passing through the three-dimensional point and parallel to the planar element; Determining a straight line obtained by back-projecting the key point into three-dimensional space; Taking the three-dimensional coordinates of the intersection point of the first plane and the straight line as the three-dimensional coordinates of the key point.

4. The method according to any one of claims 1 to 3, wherein, The step of determining the two-dimensional coordinates of a first two-dimensional point corresponding to the three-dimensional point in the first image frame according to the three-dimensional coordinates of the three-dimensional points in the map and the set relative position includes: In the map, determining a second plane within a set range of the position where the image capturing device is located; Projecting the three-dimensional points in the second plane onto the pixel plane of the first image frame according to the pose information of the image capturing device, the equation of the second plane in three-dimensional space, the internal parameters of the image capturing device, and the pixel coordinates of the first element in the pixel plane of the first image frame, to obtain the two-dimensional coordinates of the first two-dimensional point.

5. The method according to any one of claims 1-3, the method further comprising: Projecting the three-dimensional point onto the pixel plane of the image frame to obtain a second two-dimensional point; Determining a first error according to the position difference between the second two-dimensional point and the first two-dimensional point; Adjusting the three-dimensional position of the first element according to the first error.

6. The method according to claim 5, wherein, Determining a first error based on the second two-dimensional point and the first two-dimensional point includes: Calculating the first error according to the pose of the image capturing device, the internal parameters of the image capturing device, the two-dimensional coordinates of the second two-dimensional point, and the scale data.

7. The method according to claim 6, wherein when the first element is a planar element, the method further includes: Determining a connection line between the three-dimensional coordinates of the key points included in the first element and the three-dimensional coordinates of any point in the corresponding three-dimensional plane of the first element in the map; Determining the coplanarity degree between the connection line and the three-dimensional plane; Determining a second error according to the coplanarity degree; Adjusting the three-dimensional coordinates of the first element according to the second error.

8. The method according to claim 7, wherein The adjusting the three-dimensional coordinates of the first element according to the second error includes: Calculating a total error according to the first error and the second error; Adjusting the three-dimensional position of the first element according to the total error.

9. The method according to claim 7, wherein, The determining the coplanarity degree between the connection line and the three-dimensional plane includes: Calculating the coplanarity degree according to the three-dimensional coordinates of the key points included in the first element, the three-dimensional coordinates of the any point, and the normal vector of the three-dimensional plane.

10. The method according to any one of claims 6-9, the method further includes: Determining a second element corresponding to the first element in a second image frame, where the second image frame is an image frame sorted after the first image frame in time sequence; Determining the three-dimensional position of the second element according to the three-dimensional position of the first element.

11. The method according to claim 10, wherein, The determining a second element corresponding to the first element in the second image frame includes: Determining the second element according to the intersection over union ratio between all elements in the second image frame and the first element; Determining the three-dimensional coordinates of the key points of the second element according to the preset numbers of the key points of the second element, the three-dimensional coordinates of the key points included in the first element, and the preset numbers of the key points included in the first element; Taking the three-dimensional coordinates of the key points of the second element as the three-dimensional position of the second element.

12. The method according to claim 10, wherein The determining a second element corresponding to the first element in the second image frame includes: Projecting the three-dimensional points corresponding to the key points included in the first element onto the plane where the second image frame is located to obtain projection points; Determining the three-dimensional coordinates of the key points of the second element according to the key points of the second element closest to the projection points, the types of the key points of the second element, and the types of the key points included in the first element; Determining the three-dimensional position of the second element according to the three-dimensional coordinates of the key points of the second element.

13. A three-dimensional data processing device, including: A first image frame acquisition module, configured to acquire a first image frame captured by an image capturing device; An association relationship establishment module, configured to establish an association relationship between a first element in the first image frame and a map with three-dimensional information, where the map with three-dimensional information is generated based on three-dimensional data collected by a three-dimensional data acquisition device, and the three-dimensional data acquisition device and the image capturing device are in a set relative position; A three-dimensional position determination module, configured to determine the three-dimensional position of the first element according to the association relationship; The association relationship establishment module includes: a two-dimensional coordinate determination unit, configured to determine the two-dimensional coordinates of a first two-dimensional point corresponding to the three-dimensional point in the first image frame according to the three-dimensional coordinates of the three-dimensional point in the map and the set relative position; A three-dimensional coordinate determination unit, configured to determine the three-dimensional coordinates of key points included in the first element according to the two-dimensional coordinates of the first two-dimensional point corresponding to the three-dimensional point and the three-dimensional coordinates of the three-dimensional point; A three-dimensional coordinate processing unit, configured to use the three-dimensional coordinates of the key points as the association relationship.

14. The apparatus according to claim 13, wherein The three-dimensional coordinate determination unit is further configured to: In the case where the key point overlaps with the first two-dimensional point, use the three-dimensional coordinates of the three-dimensional point as the three-dimensional coordinates of the key point; In the case where the key point does not overlap with the first two-dimensional point, determine the three-dimensional coordinates of the key point according to the two-dimensional coordinates of the first two-dimensional point, the relative position between the key point and the first two-dimensional point, and the three-dimensional coordinates of the three-dimensional point.

15. The apparatus according to claim 14, wherein, In the case where the first element is a planar element, the three-dimensional coordinate determination unit is further configured to: In the map, determine a first plane passing through the three-dimensional point and parallel to the planar element; Determine the line obtained by back-projecting the key point into three-dimensional space; Use the three-dimensional coordinates of the intersection point of the first plane and the line as the three-dimensional coordinates of the key point.

16. The device according to any one of claims 13 - 15, wherein, The two-dimensional coordinate determination unit is further configured to: In the map, determine a second plane within a set range of the position where the image capturing device is located; According to the pose information of the image capturing device, the equation of the second plane in three-dimensional space, the internal parameters of the image capturing device, and the pixel coordinates of the first element in the pixel plane of the first image frame, project the three-dimensional points in the second plane onto the pixel plane of the first image frame to obtain the position of the first two-dimensional point.

17. The apparatus according to any one of claims 13-15, the apparatus further includes: A second two-dimensional point obtaining module, configured to project the three-dimensional point onto the pixel plane of the image frame to obtain a second two-dimensional point; A first error determination module, configured to determine a first error according to the position difference between the second two-dimensional point and the first two-dimensional point; A first adjustment module, configured to adjust the three-dimensional position of the first element according to the first error.

18. The device according to claim 17, wherein, The first error determination module includes: A first calculation unit, configured to calculate the first error according to the pose of the image capturing device, the internal parameters of the image capturing device, the two-dimensional coordinates of the second two-dimensional point, and scale data.

19. The apparatus according to claim 18, in the case where the first element is a planar element, the apparatus further includes: A connection determination module, configured to determine a connection line between the three-dimensional coordinates of the key points included in the first element and the three-dimensional coordinates of any point in the three-dimensional plane corresponding to the first element in the map; A coplanarity determination module, configured to determine the coplanarity between the connection line and the three-dimensional plane; A second error determination module, configured to determine a second error according to the coplanarity; A second adjustment module, configured to adjust the three-dimensional coordinates of the first element according to the second error.

20. The apparatus according to claim 19, wherein, The second adjustment module includes: A total error unit, configured to calculate a total error according to the first error and the second error; A total error processing unit, configured to adjust the three-dimensional position of the first element according to the total error.

21. The apparatus according to claim 19, wherein, The coplanarity determination module includes: A second calculation unit, configured to calculate the coplanarity according to the three-dimensional coordinates of the key points included in the first element, the three-dimensional coordinates of any point, and the normal vector of the three-dimensional plane.

22. The apparatus according to any one of claims 18-21, the apparatus further comprising: A second image frame determination unit, configured to determine a second element corresponding to the first element in a second image frame, where the second image frame is an image frame sorted after the first image frame in time sequence; A second element determination unit, configured to determine the three-dimensional position of the second element according to the three-dimensional position of the first element.

23. The device according to claim 22, wherein The second image frame determination unit is further configured to: Determine the second element according to the intersection-over-union ratio between all elements in the second image frame and the first element; Determine the three-dimensional coordinates of the key points of the second element according to the preset numbers of the key points of the second element, the three-dimensional coordinates of the key points included in the first element, and the preset numbers of the key points included in the first element; Use the three-dimensional coordinates of the key points of the second element as the three-dimensional position of the second element.

24. The apparatus according to claim 22, wherein, The second image frame determination unit is further configured to: Project the three-dimensional points corresponding to the key points included in the first element onto the plane where the second image frame is located to obtain projection points; Determine the three-dimensional coordinates of the key points of the second element according to the key points of the second element closest to the projection points, the types of the key points of the second element, and the types of the key points included in the first element; Determine the three-dimensional position of the second element according to the three-dimensional coordinates of the key points of the second element.

25. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images