Vehicle end camera 3D marking method based on roadside perception and related device

Abstract

The invention provides a vehicle end camera 3D marking method based on roadside perception and a related device. In the scheme, the sensing result of the roadside equipment is adopted to label the data of the vehicle-side camera. A fixed target on the ground serves as a known and stable reference point, the roadside equipment converts an obtained initial 3D sensing result into a world coordinate system based on latitude and longitude information of related equipment, and the 3D sensing result under the world coordinate system is obtained and sent to a target vehicle. And the vehicle converts the 3D perception result under the world coordinate system into a local coordinate system of the vehicle, the 3D perception result under the local coordinate system of the vehicle is obtained and then is projected into an image acquired by the vehicle, and marking of the perception result of the roadside equipment on the pure visual image of the vehicle end is completed. Through the method, the problems of high cost, long time and limited quantity of the existing vehicle end pure vision 3D detection data acquisition technology are solved, and the quality and acquisition efficiency of the 3D training data are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent connected transportation technology, and in particular to a vehicle-side camera 3D labeling method based on roadside perception and related devices. Background Art

[0002] The global automotive competition landscape is constantly evolving, and intelligent competitiveness led by artificial intelligence has become a new development highland and strategic fulcrum for industries and enterprises. Mass-produced passenger cars are extremely sensitive to cost, so low-cost pure vision solutions are usually adopted. Traditional two-dimensional (2D) detection and semantic segmentation technologies have limitations when dealing with complex autonomous driving scenarios, and 3D detection technology has become the key to solving this problem.

[0003] Training a visual three-dimensional (3D) detection model requires a large amount of high-quality data, and quickly obtaining this data has become one of the main constraints. Traditionally, the collection of 3D pure visual training data usually relies on professional data collection vehicles equipped with sensors such as lidar, multi-camera, high-precision global positioning system (GPS) and millimeter-wave radar to collect raw data, and project the 3D detection frame onto the image through external parameter calibration to form the true value. However, this process not only relies on expensive equipment, but also requires manual adjustment of inaccurate 3D frames, making data collection and processing very time-consuming.

[0004] In summary, how to solve the problems of high cost, long time and limited quantity of existing technology for collecting pure visual 3D detection data on the vehicle side and provide an efficient and high-quality data acquisition method is an urgent problem to be solved. Summary of the invention

[0005] The present application provides a vehicle-side camera 3D labeling method and related devices based on roadside perception, which are used to solve the problems of high cost, long time and limited quantity of existing technology for collecting vehicle-side pure visual 3D detection data, and provide a high-efficiency and high-quality data acquisition method.

[0006] In a first aspect, the present application provides a vehicle-side camera 3D annotation method based on roadside perception, which is applied to roadside equipment, including:

[0007] Based on the latitude and longitude information of the fixed target, the obtained initial three-dimensional 3D perception result is converted into a world coordinate system to obtain a first 3D perception result;

[0008] The first 3D perception result is sent to a target vehicle, so that the vehicle obtains a 3D annotation result according to the first 3D perception result.

[0009] In a specific implementation, the method of converting the acquired initial three-dimensional 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target to obtain a first 3D perception result includes:

[0010] Converting the initial 3D perception result into a local coordinate system of the fixed target within a preset range of the perception device to obtain an intermediate 3D perception result;

[0011] Based on the latitude and longitude information of the fixed target, the intermediate 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

[0012] In a specific implementation, before converting the acquired initial 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target, the method further includes:

[0013] Acquire perception results collected by multiple perception devices respectively, where the multiple perception devices include cameras, laser radars, and millimeter wave radars arranged on the roadside;

[0014] The perception results of the multiple perception devices are fused to obtain the initial 3D perception result.

[0015] In a specific implementation, fusing the perception results of the multiple perception devices to obtain the initial 3D perception result includes:

[0016] Performing 2D detection and semantic segmentation on the image captured by the camera to obtain a first processing result;

[0017] Performing 3D target detection and point cloud semantic segmentation on the point cloud collected by the laser radar to obtain a second processing result;

[0018] According to the first processing result, the second processing result and the structured data collected by the millimeter wave radar, a post-fusion algorithm is used to perform fusion processing to obtain the initial 3D perception result.

[0019] In a specific implementation, the converting the intermediate 3D perception result into the world coordinate system based on the latitude and longitude information of the fixed target to obtain the first 3D perception result includes:

[0020] Convert the latitude and longitude information of the fixed target into the Transverse Mercator Grid System (UTM) to obtain the position of the fixed target in the UTM coordinate system;

[0021] Determining a transformation matrix based on the position of the fixed object in the UTM coordinate system;

[0022] According to the conversion matrix, the intermediate 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

[0023] In a specific implementation, the fixed target includes a traffic sign pole, a street light pole or an information signboard arranged within a preset range of the sensing device.

[0024] In the second aspect, the present application provides a vehicle-side camera 3D annotation method based on roadside perception, which is applied to a vehicle, including:

[0025] Receiving a first 3D perception result sent by a roadside device;

[0026] According to the latitude and longitude information of the vehicle, the first 3D perception result is converted into a local coordinate system of the vehicle to obtain a second 3D perception result;

[0027] The second 3D perception result is projected onto the image captured by the vehicle camera to obtain a 3D annotation result.

[0028] In a specific implementation, converting the first 3D perception result into a local coordinate system of the vehicle according to the latitude and longitude information of the vehicle to obtain a second 3D perception result includes:

[0029] Convert the latitude and longitude information of the vehicle to a UTM coordinate system to obtain the position of the vehicle in the UTM coordinate system;

[0030] Determining a transformation matrix based on the position of the vehicle in the UTM coordinate system;

[0031] According to the transformation matrix, the first 3D perception result is transformed into the local coordinate system of the vehicle to obtain a second 3D perception result.

[0032] In a specific implementation, determining a transformation matrix based on the position of the vehicle in the UTM coordinate system includes:

[0033] If the local coordinate system of the vehicle is aligned with the UTM coordinate system, the identity matrix is ​​determined as the rotation matrix;

[0034] If there is a deviation between the local coordinate system of the vehicle and the UTM coordinate system, determining a rotation matrix based on the deviation angle;

[0035] Determining a translation matrix based on the position of the vehicle in the UTM coordinate system;

[0036] Wherein, the transformation matrix includes the rotation matrix and the translation matrix.

[0037] In a third aspect, the present application provides a vehicle-side camera 3D annotation device based on roadside perception, which is applied to roadside equipment, including:

[0038] A first processing module is used to convert the acquired initial three-dimensional 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target to obtain a first 3D perception result;

[0039] The second processing module is used to send the first 3D perception result to the target vehicle, so that the vehicle obtains the 3D annotation result according to the first 3D perception result.

[0040] In a fourth aspect, the present application provides a vehicle-side camera 3D annotation device based on roadside perception, which is applied to a vehicle, including:

[0041] A first processing module, configured to receive a first 3D perception result sent by a roadside device;

[0042] A second processing module, configured to convert the first 3D perception result into a local coordinate system of the vehicle according to the latitude and longitude information of the vehicle to obtain a second 3D perception result;

[0043] The third processing module is used to project the second 3D perception result onto the image captured by the vehicle camera to obtain a 3D annotation result.

[0044] In a fifth aspect, the present application provides an electronic device, comprising:

[0045] Processor, memory and communication interface;

[0046] The memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the vehicle-side camera 3D labeling method based on roadside perception as described in the first aspect or the second aspect.

[0048] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the vehicle-side camera 3D labeling method based on roadside perception as described in the first aspect or the second aspect.

[0049] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the vehicle-side camera 3D labeling method based on roadside perception as described in the first aspect or the second aspect.

[0050] The present application provides a 3D labeling method for vehicle-side cameras based on roadside perception and related devices. In this solution, the perception results of the roadside equipment are used to label the vehicle-side camera data. Based on the longitude and latitude information of the fixed target on the ground, the roadside equipment converts the initial 3D perception results obtained into the world coordinate system, obtains the 3D perception results in the world coordinate system and sends them to the target vehicle. The vehicle converts the 3D perception results in the world coordinate system into its own local coordinate system based on its own longitude and latitude information, obtains the 3D perception results in the local coordinate system of the vehicle and projects them into the image collected by the vehicle, completing the labeling of the pure visual image of the vehicle with the perception results of the roadside equipment. Through the above method, the problems of high cost, long time and limited quantity of the existing technology for collecting pure visual 3D detection data on the vehicle side are solved, and the quality and acquisition efficiency of 3D training data are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 A schematic diagram of an application scenario of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a process flow of a first embodiment of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a flow chart of a second embodiment of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a specific implementation of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application;

[0056] Figure 5 Schematic diagram of the overall process of the vehicle-side camera 3D annotation method based on roadside perception provided in the embodiment of the present application

[0057] Figure 6 A schematic diagram of the structure of a first embodiment of a vehicle-side camera 3D annotation device based on roadside perception provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a second embodiment of a vehicle-side camera 3D annotation device based on roadside perception provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0060] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0062] Before introducing the embodiments of the present application, the application background of the embodiments of the present application is first explained:

[0063] The global automotive competition landscape is constantly evolving, and intelligent competitiveness led by artificial intelligence has become a new development highland and strategic fulcrum for industries and enterprises. Intelligence is not only a means to improve vehicle performance and user experience, but also a key factor for enterprises to stand out in the fierce market competition.

[0064] In the pursuit of intelligence, mass-produced passenger cars are extremely sensitive to costs. Finding a balance between technological advancement and cost-effectiveness is a realistic challenge that mass-produced passenger car manufacturers must face. In order to reduce production costs, many manufacturers choose to adopt pure visual solutions as the basis for autonomous driving and driver assistance systems. Traditional 2D detection and semantic segmentation technologies have limitations in dealing with complex autonomous driving scenarios, such as scenes that require depth perception and spatial understanding, such as occlusion by obstacles, changes in lighting, and complex backgrounds. 3D detection technology has become the key to solving this problem. By introducing depth information, 3D detection can obtain its precise position and size in space, more accurately identify and locate objects in three-dimensional space, and provide more comprehensive environmental perception capabilities. The artificial intelligence (AI) model of visual 3D detection requires a large amount of training data, and the rapid acquisition of training data has become one of the key constraints. Traditionally, the collection of 3D pure visual training data usually relies on professional data collection vehicles equipped with sensors such as LiDAR, multi-camera, high-precision GPS positioning equipment, and millimeter-wave radar to collect raw data, perform three-dimensional target detection on the LiDAR point cloud data, and project the 3D detection frame onto the image through the external parameters between the LiDAR and the camera to complete the preliminary annotation of the vehicle-mounted image. At this time, the 3D frame generated by the LiDAR is used as the true value for training the visual model. However, this process not only relies on expensive equipment, but also requires manual adjustment of inaccurate 3D frames, making data collection and processing very time-consuming.

[0065] In summary, the existing data collection method using dedicated data collection vehicles is time-consuming, costly, and has limited scenarios and data quantity. In the context of rapid model iteration, the speed of data collection and processing cannot meet the needs of algorithm development. Therefore, providing an efficient and high-quality data acquisition method is an urgent problem to be solved.

[0066] Based on the above technical problems, the inventors studied how to improve efficiency, reduce manual intervention, and ensure high data quality in the process of 3D detection model data collection to meet the needs of rapid iteration and algorithm development. They found that with a fixed target on the ground as a known and stable reference point, and based on the longitude and latitude information of related equipment, the roadside equipment can accurately convert the 3D perception results obtained by the perception device into the local coordinate system of the target vehicle, and further complete the annotation of the image data collected by the vehicle camera, thereby improving the quality and acquisition efficiency of 3D training data. Based on this, the present application provides a 3D annotation method for a vehicle-side camera based on roadside perception and related devices, which uses a fixed target on the ground as a known and stable reference point, and based on the longitude and latitude information of related equipment, converts the initial 3D perception results obtained by the roadside equipment into the local coordinate system of the target vehicle, obtains the 3D perception results in the local coordinate system of the vehicle, and projects them into the image collected by the vehicle, completing the annotation of the pure visual image of the vehicle-side with the perception results of the roadside equipment. Through the above method, the perception results of roadside equipment are effectively utilized, the dependence on vehicle-side equipment when acquiring data is reduced, the cost and time of data collection are significantly reduced, and the quality and acquisition efficiency of 3D training data are improved, providing richer and more reliable data support for the development of autonomous driving technology.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 1 A schematic diagram of an application scenario of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application, such as Figure 1 As shown, the scene at least includes: roadside equipment, several fixed targets on the ground, and vehicles. When the target vehicle drives into the perception area of ​​the roadside equipment, such as an intersection, the vehicle sends its own longitude and latitude information to the roadside equipment. The roadside equipment fuses the 3D perception results detected in the perception area, converts them into the world coordinate system with the fixed targets on the ground as reference points, and sends them to the vehicle. The vehicle converts the 3D perception results into the local coordinate system of the vehicle based on its own longitude and latitude information, and projects the 3D perception results in the local coordinate system of the vehicle into the image collected by the vehicle, completing the annotation of the pure visual image of the vehicle end with the 3D perception results of the roadside equipment.

[0069] The roadside equipment includes roadside sensing equipment and roadside edge computing equipment, which have data processing capabilities such as sensing, communication and computing. The roadside sensing equipment includes cameras, laser radars and millimeter wave radars installed on the roadside. The specific deployment location, quantity, preset range and model of the roadside equipment are set according to actual needs, and this plan does not limit it. The vehicle is equipped with a high-precision positioning system. The several ground fixed targets can be traffic sign poles, street light poles, information signs, etc. The specific equipment form is not limited by the plan.

[0070] The physical devices involved in the above description are exemplary in the figures and do not represent the only ones. This application does not specifically limit the specific form and type of the specific devices involved.

[0071] Figure 2 A schematic diagram of a process flow of a vehicle-side camera 3D annotation method embodiment 1 based on roadside perception provided in an embodiment of the present application, such as Figure 2 As shown, the vehicle-side camera 3D annotation method based on roadside perception provided in this embodiment specifically includes:

[0072] S201: Based on the latitude and longitude information of the fixed target, the acquired initial three-dimensional 3D perception result is converted into a world coordinate system to obtain a first 3D perception result.

[0073] In a specific implementation, before this step, the vehicle-side camera 3D labeling method based on roadside perception further includes: acquiring perception results respectively collected by multiple perception devices, the multiple perception devices including a camera, a laser radar, and a millimeter-wave radar set on the roadside;

[0074] In this step, the roadside sensing equipment is divided and set on both sides of the road according to the preset distance, and the specific division distance is set according to actual needs. The sensing equipment is used to collect road information. The camera set on the roadside obtains road-related image data, and the image data provides rich color and texture information; the laser radar obtains road-related point cloud data, and the point cloud data reflects the spatial position and shape characteristics of obstacles in the road environment; the millimeter wave radar obtains road-related structured data, which reflects the distance, speed and relative motion information of objects in the road environment. Compared with laser radar and camera, millimeter wave radar has stronger penetration ability in various weather conditions and provides reliable data in harsh environments such as low visibility. Lidar, camera and millimeter wave radar cover the same field of view.

[0075] The perception results of multiple perception devices are fused to obtain the initial 3D perception result.

[0076] In this step, the roadside edge computing device can detect and fuse the data obtained by the roadside cameras, lidars, and millimeter-wave radars. The roadside sensing device can sense all obstacles in the device area and obtain the initial 3D sensing results. By fusing multi-source data, the accuracy and reliability of the sensing results are significantly improved, and the possible false detection and missed detection of a single sensor are reduced, ensuring that the roadside equipment continues to provide high-quality sensing results in complex and dynamic traffic environments.

[0077] Specifically, 2D detection and semantic segmentation are performed on the image captured by the camera to obtain a first processing result; 3D target detection and point cloud semantic segmentation are performed on the point cloud captured by the lidar to obtain a second processing result; based on the first processing result, the second processing result and the structured data collected by the millimeter wave radar, a post-fusion algorithm is used to perform fusion processing to obtain an initial 3D perception result.

[0078] In a specific implementation, the roadside edge computing device performs 2D target detection and semantic segmentation on the image collected by the camera to obtain a first processing result, which includes a 2D target detection frame and target category; performs 3D target detection and semantic segmentation on the point cloud collected by the laser radar to obtain a second processing result, which includes a 3D target detection frame and target category. Then, based on the first processing result, the second processing result and the structured data collected by the millimeter wave radar, a post-fusion algorithm is used for fusion processing to obtain an initial 3D perception result.

[0079] Specifically, the 2D target detection frame and the 3D target detection frame are associated, the 3D target detection frame is projected onto the 2D target detection frame, and the overlap between the 3D target detection frame and the 2D target detection frame is calculated. If the overlap exceeds the preset threshold, it indicates that the 3D target detection frame and the 2D target detection frame are successfully associated and belong to the same target, otherwise they do not belong to the same target. Then the 3D target detection frame is associated with the structured data collected by the millimeter wave radar, that is, the Euclidean distance between the center point of the 3D target detection frame and the millimeter wave radar data point is calculated, and the association is performed through the Hungarian matching algorithm. The Hungarian algorithm is a combinatorial optimization algorithm for solving task allocation problems in polynomial time. In the task of associating lidar data with millimeter wave radar data, the 3D target detection frame and the millimeter wave radar data point are regarded as two vertex sets of a bipartite graph, and the Euclidean distance between the two is regarded as the weight of the edge between the vertices. The number of matches is gradually increased by finding an augmenting path, and pairs of points with closer distances are given priority for matching until no new augmenting path can be found. Through this step, the precise association between the 3D target detection frame and the structured data collected by the millimeter-wave radar is achieved, providing accurate target position and motion information for subsequent autonomous driving decisions and planning. So far, the 3D perception result obtained, that is, the 3D detection frame of the obstacle provides an estimate of the position and size of the target object in three-dimensional space, including the target's center point coordinates, size (length, width and height), heading angle, category, and speed information provided by the millimeter-wave radar. After that, the Kalman filter is used to continuously track and update the state of the obstacle. First, the position, speed and acceleration of the obstacle are initialized, and then the predicted value is updated according to the actual 3D target detection frame and the structured data of the millimeter-wave radar. This process includes Kalman gain calculation, state update and covariance update. By continuously repeating the prediction and update process, the Kalman filter can recursively estimate the state of the system, realize real-time tracking and updating of the obstacle position and speed, and complete the post-fusion of the detection data of each sensor, so that the pure visual data on the vehicle side can be annotated with the fused data with higher accuracy in the future, thereby improving the quality of subsequent data annotation.

[0080] In the process of converting the acquired initial three-dimensional 3D perception results into the world coordinate system based on the latitude and longitude information of the fixed target to obtain the first 3D perception results, the initial 3D perception results integrate the laser radar point cloud data, two-dimensional image data and structured data of the millimeter wave radar. Due to the different physical positions, orientations and internal processing mechanisms of each sensor, these data are all in the local coordinate system of each sensor. In order to ensure the effective fusion and analysis of multi-source data, the 3D perception results need to be converted into a unified laser radar coordinate system. Then, the 3D perception results in the laser radar coordinate system are converted into the world coordinate system with the ground fixed target as the reference point, including coordinate translation, that is, moving the origin of the laser radar coordinate system to the origin of the world coordinate system, and coordinate rotation, that is, aligning the axis of the laser radar coordinate system with the axis of the world coordinate system. The coordinate translation and coordinate rotation are represented by the homogeneous transformation matrix. Through the transformation matrix, the initial 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

[0081] S202: Send the first 3D perception result to the target vehicle, so that the vehicle obtains the 3D annotation result according to the first 3D perception result.

[0082] For the vehicle side, the first 3D perception result sent by the roadside equipment is received.

[0083] In this step, when the target vehicle travels into the perception area of ​​the roadside device, such as an intersection, the vehicle sends its own latitude and longitude information to the roadside device, that is, a communication connection is established between the roadside device and the vehicle. After the roadside device obtains the first 3D perception result, it sends it to the target vehicle. The target vehicle receives the first 3D perception result so that the 3D perception result can be used on the vehicle side to annotate the pure visual data on the vehicle side, obtain the 3D annotation result, and obtain the training data.

[0084] S203: According to the latitude and longitude information of the vehicle, the first 3D perception result is converted into the local coordinate system of the vehicle to obtain a second 3D perception result.

[0085] In this step, after the target vehicle receives the first 3D perception result, the latitude and longitude information of the vehicle is first converted to the Universal Transverse Mercator Grid System (UTM) to obtain the position of the vehicle in the UTM coordinate system, and the conversion matrix is ​​determined based on the position of the vehicle in the UTM coordinate system. The longitude and latitude information is a representation in the geographic coordinate system and is not directly applicable to subsequent mathematical transformations and calculations. The UTM coordinate system is a plane rectangular coordinate system based on the transverse Mercator projection. The coordinate system divides the earth's surface into 60 longitudinal regions, each of which has a transverse projection surface, and the earth's surface is mapped onto the projection surface.

[0086] Specifically, convert the latitude and longitude information of the vehicle into the UTM coordinate system. First, convert the latitude and longitude values ​​into radians, that is, multiply the longitude and latitude by π / 180 respectively. Then calculate the UTM zone number zoneNumber. The UTM coordinate system uses 6-degree zones, starting from 180 degrees east longitude or 180 degrees west longitude, and counting from west to east. The calculation formula of the UTM zone number is usually expressed as:

[0087] zoneNumber=floor((longTemp+180) / 6)+1

[0088] In the formula, floor means rounding down, and longTemp means normalizing the longitude to a value between -180 and 180 degrees.

[0089] Then, the relevant parameters of the UTM longitudinal zone are calculated according to the dimension value, including the radius of curvature N of the meridian circle, the tangent value T of the tangent azimuth of the curvature plane, the radius of curvature C of the meridian circle, etc. Finally, the UTM zone number, the relevant parameters of the UTM longitudinal zone and the arc value are used to calculate the easting coordinate UTM Easting and the northing coordinate UTM Northing of the UTM coordinate system, that is, the position of the vehicle in the UTM coordinate system is obtained, which is convenient for the subsequent calculation of the coordinate system.

[0090] The local coordinate system of the vehicle takes the center of mass of the vehicle as the origin, the driving direction of the vehicle as the X-axis, the Y-axis perpendicular to the driving direction and pointing to the left side of the vehicle as the Y-axis, and the Z-axis perpendicular to the ground and pointing to the sky as the Z-axis. The transformation matrix includes the rotation matrix and the translation matrix. If the local coordinate system of the target vehicle is aligned with the UTM coordinate system, such as the vehicle's forward direction is aligned with the north direction of the UTM coordinate system, the unit matrix is ​​determined as the rotation matrix; if there is a deviation between the local coordinate system of the target vehicle and the UTM coordinate system, the sine and cosine values ​​of the heading angle are calculated through the vehicle's heading angle, that is, the angle between the front of the vehicle and the north direction of the UTM coordinate system to construct a rotation matrix. Based on the position of the vehicle in the UTM coordinate system, the translation matrix is ​​determined.

[0091] In a specific implementation, the first 3D perception result is transformed into a local coordinate system of the vehicle according to a transformation matrix to obtain a second 3D perception result.

[0092] The coordinates of the first 3D perception result are multiplied by the inverse matrix of the transformation matrix determined by the rotation matrix and the translation matrix, thereby obtaining the second 3D perception result in the local coordinate system of the vehicle.

[0093] S204: Project the second 3D perception result onto the image captured by the vehicle camera to obtain a 3D annotation result.

[0094] In this step, the second 3D perception result contains the category of obstacles, the length, width, height, heading angle, speed and other information of the 3D target detection box. The second 3D perception result is projected into the image captured by the vehicle camera using the intrinsic parameter matrix of the vehicle camera. The intrinsic parameter matrix contains the focal length of the camera, the coordinates of the intersection of the principal point, i.e., the optical axis and the image plane, and possible radial and tangential distortion parameters. The moving targets such as vehicles, pedestrians, and cyclists in the camera images collected by the vehicle have 3D annotation results, which are used as the true value of the subsequent training data training model.

[0095] The vehicle-side camera 3D annotation method based on roadside perception provided in the embodiment of the present application integrates the data of the camera, lidar and millimeter-wave radar set on the roadside. The roadside edge computing device uses the advantages of each sensor to fuse the multi-source data and generate an initial 3D perception result. The initial 3D perception result is then converted to the world coordinate system and sent to the target vehicle through communication. The 3D perception results received by the vehicle are further converted to the local coordinate system of the vehicle for comparison and annotation with the real-time data of the vehicle. Finally, by projecting these data into the image captured by the vehicle camera, high-precision 3D annotation results are generated. These annotation results can be used to train the autonomous driving model to improve the accuracy and robustness of the model. Through the above method, high-quality perception and annotation in complex traffic environments are ensured, providing a solid foundation for the further development of autonomous driving technology.

[0096] Figure 3 A schematic diagram of a second embodiment of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, based on the above implementation S201, based on the latitude and longitude information of the fixed target, the obtained initial three-dimensional 3D perception result is converted into the world coordinate system to obtain the first 3D perception result, which specifically includes:

[0097] S301: Convert the initial 3D perception result into the local coordinate system of the fixed target within a preset range of the perception device to obtain an intermediate 3D perception result.

[0098] In this step, in order to improve the accuracy of the roadside 3D perception results, the recognition of fixed targets on the road surface will be added in the target detection of the roadside lidar. Fixed targets include traffic sign poles, street light poles or information signs set within the preset range of the perception device. Identifying and utilizing fixed targets on the road surface is a key strategy to improve perception accuracy. By identifying fixed targets such as road signs, light poles, guardrails, etc., a stable reference coordinate system can be established. Since the constancy of their positions provides reliable reference points, the perception system can more accurately calibrate and align the data of different sensors to improve the spatial consistency of the perception results and enhance the robustness and reliability of the system in complex traffic environments.

[0099] Figure 4 This is a schematic diagram of a specific implementation of a vehicle-side camera 3D annotation method based on roadside perception provided in an embodiment of the present application. Specifically, in the target detection of the roadside laser radar, the first rod-shaped object O2, the second rod-shaped object P, and the center point Q of the obstacle are detected. The distribution and position relationship of the first rod-shaped object, the second rod-shaped object, and the obstacle are as follows: Figure 4 As shown, coordinate system O1 represents the laser radar coordinate system, and coordinate system O2 represents the local coordinate system of the first rod-shaped object, which uses the three directions of northeast and sky as coordinate axes. The coordinates of the second rod-shaped object P in coordinate system O2 are (x1, y1), the coordinates of the center Q of the obstacle in coordinate system O2 are (x2, y2), and the heading angle of the center Q of the obstacle and the X-axis of coordinate system O2 are θ. In coordinate system O1, the length between the first rod-shaped object O2 and the second rod-shaped object P is r1, the length between the second rod-shaped object P and the center point Q of the obstacle is r2, and the length between the center point Q of the obstacle and the first rod-shaped object O2 is r3.

[0100] Calculate the coordinate value and heading angle of the center point Q of the obstacle in the coordinate system O2.

[0101] In the coordinate system O1, calculate the angle between the line connecting the first rod-shaped object O2 and the second rod-shaped object P and the heading angle of the obstacle Q, and define the clockwise direction as the positive direction. Among them, x1 = r1, y1 = 0, then only the values ​​of x2 and y2 need to be calculated. According to the distance formula between two points in plane geometry:

[0102]

[0103] Calculation yields:

[0104]

[0105] It should be noted that if the angle between the heading angle of the center Q of the obstacle and the X-axis of the coordinate system O2 is θ greater than 180 degrees, y2 takes a negative value; if the angle between the heading angle of the center Q of the obstacle and the X-axis of the coordinate system O2 is θ less than or equal to 180 degrees, y2 takes a positive value.

[0106] Based on the local coordinate system O2 of the first rod-shaped object, the center point position and heading angle of the initial 3D perception result in the laser radar coordinate system are calculated to obtain an intermediate 3D perception result in the local coordinate system O2 of the first rod-shaped object.

[0107] S302: Based on the latitude and longitude information of the fixed target, the intermediate 3D perception result is converted into a world coordinate system to obtain a first 3D perception result.

[0108] In this step, the latitude and longitude information of the fixed target can be measured according to the real-time kinematic (RTK) device, which can obtain the centimeter-level positioning accuracy of the target in real time outdoors through real-time dynamic positioning of carrier phase observations. The RTK device consists of a fixed base station and one or more mobile stations. The base station is located at a known geographical location, receives signals from GNSS satellites, and calculates differential correction data. The mobile station receives differential signals and global navigation satellite signals at the target location, so that the precise longitude and latitude coordinates of the fixed target can be calculated in real time.

[0109] Specifically, the latitude and longitude information of the first rod-shaped object O2 and the second rod-shaped object P are measured by the RTK marking device. The angle between the line connecting the first rod-shaped object O2 and the second rod-shaped object P and the X-axis of the world coordinate system is a fixed value, and its rotation matrix is ​​recorded as R y , the position of the first rod-shaped object O2 converted from its longitude and latitude information to the UTM coordinate system is recorded as T. The implementation principle and technical effect of obtaining the position of the first rod-shaped object O2 in the UTM coordinate system are similar to the implementation principle and technical effect of obtaining the position of the target vehicle in the UTM coordinate system, which will not be repeated here. Determine the conversion matrix as [R y |T]. Based on the transformation matrix, the intermediate 3D perception result is transformed from the local coordinate system O2 of the first rod-shaped object to the world coordinate system to obtain a first 3D perception result.

[0110] It should be noted that the first rod-shaped object O2 and the second rod-shaped object P are only used as examples. In a specific implementation, at least two ground fixed targets can be selected as reference points. In determining the transformation matrix, two fixed targets are selected at random, and the transformation matrix is ​​determined by determining the angle between the line between the two fixed targets and the X-axis of the world coordinate system. This solution does not limit the number and form of specific fixed targets.

[0111] The vehicle-side camera 3D annotation method based on roadside perception provided in the embodiment of the present application improves perception accuracy by identifying and utilizing fixed targets such as poles, road signs, etc. to establish a stable reference coordinate system. In the target detection of the roadside lidar, fixed targets such as the first rod-shaped object and the second rod-shaped object are identified, and based on this, the center point of the obstacle and its heading angle in the local coordinate system are calculated. Through the real-time dynamic dotting equipment, the longitude and latitude information of the fixed target is accurately measured and converted to the UTM coordinate system, and a conversion matrix is ​​constructed to convert the intermediate 3D perception result from the local coordinate system to the world coordinate system to obtain the first 3D perception result. Through the above method, the spatial consistency and accuracy of the perception results are significantly improved, the error accumulation is reduced, the reliability and applicability of the data are improved, the robustness and reliability of the system in complex traffic environments are enhanced, and a more accurate environmental information basis is provided for autonomous driving decisions.

[0112] Figure 5 The overall process diagram of the vehicle-side camera 3D annotation method based on roadside perception provided in the embodiment of the present application is as follows: Figure 5 As shown, the overall logic of this solution is:

[0113] When the target vehicle drives into the perception area of ​​the roadside device, such as an intersection, the target vehicle sends its longitude and latitude information to the roadside edge computing device, which converts the 3D perception results into the world coordinate system and sends them to the target vehicle. The target vehicle converts the 3D perception results into the local coordinate system of the vehicle, and projects the 3D perception results in the local coordinate system of the vehicle into the image coordinate system, completing the annotation of the pure visual image of the vehicle end by the 3D perception results of the roadside device.

[0114] Figure 6 This is a structural diagram of a first embodiment of a vehicle-side camera 3D annotation device based on roadside perception provided in an embodiment of the present application, such as Figure 6 As shown, the vehicle-side camera 3D annotation device 600 based on roadside perception is applied to roadside equipment, including:

[0115] The first processing module 601 is used to convert the acquired initial three-dimensional 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target to obtain a first 3D perception result;

[0116] The second processing module 602 is used to send the first 3D perception result to the target vehicle, so that the vehicle obtains the 3D annotation result according to the first 3D perception result.

[0117] The vehicle-side camera 3D annotation device based on roadside perception provided in this embodiment is used to execute the technical solution in the aforementioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0118] On the basis of the above-mentioned embodiment, the vehicle-side camera 3D labeling device 600 based on roadside perception further includes: a third processing module 603, which is used to:

[0119] Acquire perception results collected by multiple perception devices, including cameras, laser radars, and millimeter-wave radars installed on the roadside;

[0120] The perception results of multiple perception devices are fused to obtain the initial 3D perception result.

[0121] In a specific implementation, the first processing module 601 is specifically used to:

[0122] Convert the initial 3D perception result into the local coordinate system of the fixed target within the preset range of the perception device to obtain an intermediate 3D perception result;

[0123] Based on the latitude and longitude information of the fixed target, the intermediate 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

[0124] In a specific implementation, the first processing module 601 is further configured to:

[0125] Convert the latitude and longitude information of the fixed target to the Transverse Mercator Grid System UTM to obtain the position of the fixed target in the UTM coordinate system;

[0126] Determine the transformation matrix based on the position of the fixed target in the UTM coordinate system;

[0127] According to the transformation matrix, the intermediate 3D perception result is transformed into the world coordinate system to obtain the first 3D perception result.

[0128] Fixed targets include traffic sign poles, street light poles or information signs set within the preset range of the sensing device.

[0129] In a specific implementation, the third processing module 603 is specifically used to:

[0130] Perform 2D detection and semantic segmentation on the image captured by the camera to obtain a first processing result;

[0131] Perform 3D target detection and point cloud semantic segmentation on the point cloud collected by the laser radar to obtain a second processing result;

[0132] According to the first processing result, the second processing result and the structured data collected by the millimeter wave radar, a post-fusion algorithm is used to perform fusion processing to obtain an initial 3D perception result.

[0133] The vehicle-side camera 3D annotation device based on roadside perception provided in any of the above embodiments is used to execute the technical solution in the above-mentioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0134] Figure 7 This is a structural diagram of a second embodiment of a vehicle-side camera 3D annotation device based on roadside perception provided in an embodiment of the present application, such as Figure 7 As shown, the vehicle-side camera 3D annotation device 700 based on roadside perception is applied to a vehicle, including:

[0135] The first processing module 701 is used to receive a first 3D perception result sent by a roadside device;

[0136] The second processing module 702 is used to convert the first 3D perception result into a local coordinate system of the vehicle according to the latitude and longitude information of the vehicle to obtain a second 3D perception result;

[0137] The third processing module 703 is used to project the second 3D perception result onto the image captured by the vehicle camera to obtain a 3D annotation result.

[0138] The vehicle-side camera 3D annotation device based on roadside perception provided in this embodiment is used to execute the technical solution in the aforementioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0139] In a specific implementation, the second processing module 702 is specifically used to:

[0140] Convert the latitude and longitude information of the vehicle to the UTM coordinate system to obtain the position of the vehicle in the UTM coordinate system;

[0141] Based on the position of the vehicle in the UTM coordinate system, a transformation matrix is ​​determined;

[0142] According to the transformation matrix, the first 3D perception result is transformed into the local coordinate system of the vehicle to obtain a second 3D perception result.

[0143] In a specific implementation, the second processing module 702 is further configured to:

[0144] If the vehicle's local coordinate system is aligned with the UTM coordinate system, the identity matrix is ​​determined as the rotation matrix;

[0145] If there is a deviation between the vehicle's local coordinate system and the UTM coordinate system, the rotation matrix is ​​determined based on the deviation angle;

[0146] Based on the position of the vehicle in the UTM coordinate system, a translation matrix is ​​determined;

[0147] The transformation matrix includes a rotation matrix and a translation matrix.

[0148] The vehicle-side camera 3D annotation device based on roadside perception provided in any of the above embodiments is used to execute the technical solution in the above-mentioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0149] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes:

[0150] Processor, memory and communication interface;

[0151] Memory stores computer-executable instructions;

[0152] The processor executes the computer execution instructions stored in the memory, so that the processor executes the technical solution provided in the aforementioned method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0153] An embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the technical solution in the aforementioned method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0154] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the technical solution in the aforementioned method embodiment is implemented. The implementation principle and technical effect are similar and will not be repeated here.

[0155] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0156] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0158] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A 3D annotation method for vehicle-side cameras based on roadside perception, characterized in that: Applied to roadside equipment, including: Based on the latitude and longitude information of the fixed target, the obtained initial three-dimensional 3D perception result is converted into a world coordinate system to obtain a first 3D perception result; The first 3D perception result is sent to a target vehicle, so that the vehicle obtains a 3D annotation result according to the first 3D perception result.

2. The method according to claim 1, characterized in that The method converts the acquired initial three-dimensional 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target to obtain a first 3D perception result, including: Converting the initial 3D perception result into a local coordinate system of the fixed target within a preset range of the perception device to obtain an intermediate 3D perception result; Based on the latitude and longitude information of the fixed target, the intermediate 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

3. The method according to claim 1 or 2, characterized in that: Before converting the acquired initial 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target, the method further includes: Acquire perception results collected by multiple perception devices respectively, where the multiple perception devices include cameras, laser radars, and millimeter wave radars arranged on the roadside; The perception results of the multiple perception devices are fused to obtain the initial 3D perception result.

4. The method according to claim 3, characterized in that The fusing the perception results of the multiple perception devices to obtain the initial 3D perception result includes: Performing 2D detection and semantic segmentation on the image captured by the camera to obtain a first processing result; Performing 3D target detection and point cloud semantic segmentation on the point cloud collected by the laser radar to obtain a second processing result; According to the first processing result, the second processing result and the structured data collected by the millimeter wave radar, a post-fusion algorithm is used to perform fusion processing to obtain the initial 3D perception result.

5. The method according to claim 2, characterized in that: The converting the intermediate 3D perception result into the world coordinate system based on the latitude and longitude information of the fixed target to obtain the first 3D perception result includes: Convert the latitude and longitude information of the fixed target into the Transverse Mercator Grid System (UTM) to obtain the position of the fixed target in the UTM coordinate system; Determining a transformation matrix based on the position of the fixed object in the UTM coordinate system; According to the conversion matrix, the intermediate 3D perception result is converted into the world coordinate system to obtain the first 3D perception result.

6. The method according to claim 1 or 2, characterized in that: The fixed target includes a traffic sign pole, a street light pole or an information signboard arranged within a preset range of the sensing device.

7. A 3D annotation method for vehicle-side cameras based on roadside perception, characterized in that: Applied to vehicles, including: Receiving a first 3D perception result sent by a roadside device; According to the latitude and longitude information of the vehicle, the first 3D perception result is converted into a local coordinate system of the vehicle to obtain a second 3D perception result; The second 3D perception result is projected onto the image captured by the vehicle camera to obtain a 3D annotation result.

8. The method according to claim 7, characterized in that The converting the first 3D perception result into a local coordinate system of the vehicle according to the latitude and longitude information of the vehicle to obtain a second 3D perception result includes: Convert the latitude and longitude information of the vehicle to a UTM coordinate system to obtain the position of the vehicle in the UTM coordinate system; Determining a transformation matrix based on the position of the vehicle in the UTM coordinate system; According to the transformation matrix, the first 3D perception result is transformed into the local coordinate system of the vehicle to obtain a second 3D perception result.

9. The method according to claim 8, characterized in that The determining of the transformation matrix based on the position of the vehicle in the UTM coordinate system includes: If the local coordinate system of the vehicle is aligned with the UTM coordinate system, the identity matrix is ​​determined as the rotation matrix; If there is a deviation between the local coordinate system of the vehicle and the UTM coordinate system, determining a rotation matrix based on the deviation angle; Determining a translation matrix based on the position of the vehicle in the UTM coordinate system; The transformation matrix includes the rotation matrix and the translation matrix.

10. A vehicle-side camera 3D annotation device based on roadside perception, characterized in that: Applied to roadside equipment, including: A first processing module is used to convert the acquired initial three-dimensional 3D perception result into a world coordinate system based on the latitude and longitude information of the fixed target to obtain a first 3D perception result; The second processing module is used to send the first 3D perception result to the target vehicle, so that the vehicle obtains the 3D annotation result according to the first 3D perception result.

11. A vehicle-side camera 3D annotation device based on roadside perception, characterized in that: Applied to vehicles, including: A first processing module, configured to receive a first 3D perception result sent by a roadside device; A second processing module, configured to convert the first 3D perception result into a local coordinate system of the vehicle according to the latitude and longitude information of the vehicle to obtain a second 3D perception result; The third processing module is used to project the second 3D perception result onto the image captured by the vehicle camera to obtain a 3D annotation result.

12. An electronic device, characterized in that: include: Processor, memory and communication interface; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the vehicle-side camera 3D labeling method based on roadside perception as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the vehicle-side camera 3D labeling method based on roadside perception as described in any one of claims 1 to 9.

14. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the vehicle-side camera 3D labeling method based on roadside perception as described in any one of claims 1 to 9.

Images