A vehicle key point marking method and device, a vehicle assisted driving method and system, an electronic device, a computer readable storage medium and a vehicle

Abstract

The embodiments of the present invention disclose a method, device and electronic device for marking key points of a vehicle, which relate to the field of computer vision technology and can improve the automation and consistency of marking. The marking method preprocesses the full picture of the vehicle to be marked to obtain a picture block and a binary mask thereof, inputs each picture block into a vehicle model retrieval algorithm module to be marked, and the retrieval algorithm interacts with the vehicle 3D model library to detect the model of each vehicle to be marked; inputs the full picture and the picture block into the vehicle pose solution and optimization algorithm module to be marked to obtain the optimized pose of the vehicle to be marked; inputs each model, pose and binary mask into a 2D key point generation algorithm module to generate a 2D semantic key point marking result. The present invention is suitable for scenarios such as vehicle 3D target perception, cloud-based pure perspective 3D marking and autonomous driving.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a vehicle key point labeling method and device, a vehicle assisted driving method and system, an electronic device, a computer-readable storage medium and a vehicle. Background Art

[0002] Vehicle 2D key points have a wide range of applications, especially in pure visual vehicle 3D annotation. In order to obtain better annotation results, it is generally necessary to use large-scale vehicle 2D semantic key point annotation data to train the key point detection model. However, it is difficult to achieve large-scale and high-quality vehicle 2D semantic key point data generation with the commonly used manual annotation. The main reasons are: 1) Many aspects of annotation require manual participation, which is slow and costly; 2) Affected by factors such as the experience of the annotation personnel, the annotation consistency is relatively poor and the data noise is large; 3) When the definition of 2D semantic key points changes, there is a lot of manual intervention when adjusting the annotation, and the flexibility is poor. Therefore, how to automatically annotate vehicle 2D semantic key points is an urgent problem to be solved.

[0003] The Chinese invention application "Vehicle Key Point Information Detection Method, Vehicle Control Method and Device, Vehicle" with publication number CN112489126A discloses that firstly, three-dimensional scanning data for the target vehicle and the image to be annotated containing the target vehicle are obtained; secondly, based on the three-dimensional scanning data, the three-dimensional coordinate information of multiple key points corresponding to the target vehicle is determined; finally, according to the coordinate conversion relationship between the three-dimensional coordinate system of the target vehicle and the two-dimensional coordinate system of the image to be annotated, as well as the three-dimensional coordinate information of multiple key points corresponding to the target vehicle, the two-dimensional coordinate information of the multiple key points in the image to be annotated is determined to complete the annotation. Although this solution uses 3D model projection to replace manual direct annotation on the image, in order to achieve better results, manual annotation may be required in the pose solution link of the vehicle to be annotated, manual selection may be required in the vehicle model retrieval link, and visibility also requires manual annotation. The degree of automation is low, which is not conducive to annotating large-scale data.

[0004] The Chinese invention application "A method, system, device and medium for marking vehicle key points" with publication number CN113869215A discloses that the vehicle model to be marked is first determined, and the corresponding three-dimensional annotation model is obtained according to the determined vehicle model, and the three-dimensional key points of the vehicle to be marked are annotated; then the three-dimensional key point annotation results are projected in two dimensions to obtain the two-dimensional projection results of all key points; at the same time, the two-dimensional key points of the vehicle to be marked are annotated, and the final key point annotation results of the vehicle to be annotated are determined based on the two-dimensional key point annotation results and the two-dimensional projection results. This solution associates the two-dimensional projection results with the two-dimensional key point annotation results, and uses the annotation results of the 2D annotation model and the two-dimensional projection results after the projection of the 3D annotation model as the key point annotation results of the vehicle to be annotated, thereby solving the problem of annotation errors when using only 2D annotation models or 3D annotation models. Although this solution also uses 3D model projection, it still requires manual annotation directly on the picture for verification, which has poor consistency and low automation; At the same time, when the definition of semantic key points changes, the solution needs to re-perform the entire process of annotation, which has poor flexibility.

[0005] The Chinese invention application "Key Point Annotation Method and Device" with publication number CN114677734A discloses that, first, N images of a target object are obtained by photographing the target object with a camera array, and the semantic key points of the target object are pre-annotated on the N images to obtain the first annotation position of the semantic key point on each image; the first spatial position corresponding to the semantic key point is determined according to the N first annotation positions corresponding to the semantic key point; the first spatial position is projected onto the N images according to the N groups of camera parameters corresponding to the camera array to obtain the second annotation position of the semantic key point on each image; the target annotation position that meets the preset adjustment conditions is determined in the N first annotation positions according to the positional relationship between the first annotation position and the corresponding second annotation position on the N images, and the target annotation position is adjusted. This solution still requires manual annotation directly on the image, which is inefficient. Summary of the invention

[0006] In view of this, embodiments of the present invention provide a vehicle key point labeling method and device, a vehicle assisted driving method and system, an electronic device, a computer-readable storage medium and a vehicle, which can improve the degree of automation of vehicle key point labeling.

[0007] In a first aspect, an embodiment of the present invention provides a method for labeling vehicle key points, including: pre-building a vehicle model library: the vehicle model library includes a vehicle 3D model and a semantic key point library, in which the 3D model, 3D semantic key points and 3D point cloud of each vehicle model are stored; data preprocessing: data preprocessing is performed on the full picture of each vehicle to be labeled to obtain the picture block and the binary mask of the picture block of each vehicle to be labeled; vehicle model retrieval for the vehicle to be labeled: generating a vehicle point cloud from the picture block, comparing the vehicle point cloud with each of the 3D point clouds in the vehicle model library, and retrieving the model to which each vehicle to be labeled belongs; solving and optimizing the posture of the vehicle to be labeled: inputting the full picture of the vehicle to be labeled into the 3D point cloud library D target detection model, obtain the initial value of the pose of each vehicle to be labeled; invert the initial value of the pose to obtain the initial value of the view angle of each vehicle to be labeled; input the initial value of the view angle and the picture block of the vehicle to be labeled into the Vehicle_nerf algorithm module at the same time to obtain each optimized view angle; invert the optimized view angle to obtain each optimized pose; 2D key point generation: according to the retrieved vehicle model, obtain the 3D semantic key point of each vehicle to be labeled in the vehicle coordinate system from the vehicle model library; according to the optimized pose, convert the 3D semantic key point to the camera coordinate system; project the 3D semantic key point in the camera coordinate system to the picture coordinate system to obtain the 2D semantic key point labeling result of each vehicle to be labeled.

[0008] According to a specific implementation method of an embodiment of the present invention, the data preprocessing step includes: using a 2D detection model to extract 2D detection frames of the vehicles to be labeled in the entire image; cropping the image according to each detection frame to obtain image blocks of each vehicle to be labeled; inputting the image blocks of each vehicle to be labeled into a 2D segmentation model to obtain a binary mask of each image block.

[0009] According to a specific implementation method of an embodiment of the present invention, the data preprocessing step includes: inputting the full image of the picture into a 2D instance segmentation model to obtain a binary mask full image of each vehicle to be labeled; using an image cropping algorithm to crop the full image and the binary mask full image respectively to obtain image blocks of each vehicle to be labeled and binary masks of the image blocks.

[0010] According to a specific implementation method of an embodiment of the present invention, the 3D semantic key points of each vehicle model in the vehicle model library have multiple versions; when changing the version to mark the key points, the binary mask of the image block of the vehicle to be marked, the vehicle model retrieval result and the pose solution result are all kept unchanged for reuse, and only the 3D semantic key point projection needs to be re-executed.

[0011] According to a specific implementation method of an embodiment of the present invention, the step of retrieving the vehicle model to be labeled includes: using Vehicle_nerf to generate a rendering point cloud of each vehicle to be labeled; calculating the similarity between the rendering point cloud of a vehicle to be labeled and each 3D point cloud in the vehicle model library, and taking the vehicle model in the vehicle model library corresponding to the maximum similarity as the vehicle model retrieval result.

[0012] According to a specific implementation of an embodiment of the present invention, the step of calculating the vehicle rendering point cloud includes: selecting a group of K viewing angles Covering 360° of the vehicle, including Represents the view from the kth camera to the vehicle; each view generates a depth map , forming a set of K depth maps ,in represents the image of the i-th vehicle, represents the kth depth map;

[0013] Will Convert to rendered point cloud .

[0014] According to a specific implementation of an embodiment of the present invention, the similarity calculation step includes: (21) taking out the 3D model of vehicle model j from the vehicle model library ;(22) Assume the rendered point cloud There are Y points in total, calculate the distance from each point to The minimum distance ,get ; (23) Calculate the score of each point distance and get , where each score is calculated by , is a pre-set distance threshold greater than 0, Limited to [0.0, 1.0] Between; (24) Calculate the overall score of vehicle model j ; (25) Repeat the above steps (21)-(24) until all models are traversed and the scores of all models are obtained. ; (26) Sort the scores and determine the model with the highest score as the search result .

[0015] According to a specific implementation method of an embodiment of the present invention, assume that the initial values ​​of the N vehicle poses to be labeled are , the initial viewing angle of N vehicles to be labeled is The process of solving the optimized perspective is as follows: for each vehicle to be annotated, the perspective is iteratively optimized vehicle by vehicle using Vehicle_nerf, wherein the iterative steps for the i-th vehicle include: (a) obtaining a rendering image of the i-th vehicle at the initial perspective value: ,in Represents the image of the i-th vehicle; (b) Calculate the photometric error: ; (c) Calculate the photometric error relative to the viewing angle The gradient is: ; (d) with an update function based on the gradient renew ,make ; (e) Repeat steps (a)-(d) until convergence. Optimize viewing angle .

[0016] According to a specific implementation of the embodiment of the present invention, the update function update is: ,in is an artificially set learning rate.

[0017] According to a specific implementation of an embodiment of the present invention, the step of generating 2D key points includes: (1) obtaining the 3D key points of each vehicle to be annotated in the vehicle coordinate system from the vehicle model library according to the retrieved vehicle model and the semantic key point version to be annotated , = indicates that the i-th vehicle to be labeled uses the X-th version of the 3D semantic key points of the j-th vehicle model; (2) the 3D semantic key points are converted from the vehicle coordinate system to the camera coordinate system using the pose of the vehicle to be labeled, and the 3D key points of each vehicle to be labeled in the camera coordinate system are obtained. ,in , is the optimized vehicle posture The rotating part, is the optimized vehicle posture The translation part of represents matrix multiplication; (3) the 3D key points of each vehicle to be labeled in the camera coordinate system Project to the image coordinate system to obtain the 2D key points of each vehicle to be labeled in the image coordinate system , each key point ,u, v are the pixel coordinates on the image, and the calculation method of u, v is: , is the coordinate of a 3D key point in the camera coordinate system, is the camera internal parameter, , and is the focal length in pixels, and are the pixel coordinates of the principal point.

[0018] According to a specific implementation of an embodiment of the present invention, the following steps are further included after step (2): self-occlusion judgment: Assuming that the X-axis of the camera coordinate system faces right, the Z-axis faces forward, and the Y-axis faces downward, the judgment process observed in the top view is: (a) Scan all key points clockwise with the origin of the camera coordinate system as the center to find the leftmost point B and the rightmost point A, and these two points are visible; (b) For other points P, calculate ,symbol Represents vector cross product, symbol represents vector dot product; (c) if If This means the point is not visible.

[0019] According to a specific implementation of the embodiment of the present invention, the following steps are further included after step (3): (a) Other occlusion judgment: Each of , query the pixel coordinate value corresponding to the binary mask of the corresponding vehicle image block. If it is 0, it means there is other occlusion, and if it is 1, it means there is no other occlusion; (b) Visibility generation: those without self-occlusion and other occlusion are visible, and others are invisible; then the 2D key point annotation result is:

[0020] , are the pixel coordinates of the 2D semantic keypoints, is the visibility of the 2D semantic keypoint.

[0021] According to a specific implementation method of an embodiment of the present invention, the 3D target detection model is a monocular 3D target detection model or a multimodal 3D target detection model; the input data of the multimodal 3D target detection model is multimodal data, and the multimodal data is a combination of at least two of the full image, lidar point cloud data and millimeter wave data.

[0022] According to a specific implementation method of an embodiment of the present invention, the method also includes: 3D key point labeling: using a key point detection training model obtained by training with each of the 2D semantic key point labeling result data to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled, to obtain the key points and vehicle ID of each frame, and thereby solve the 3D posture of the vehicle frame by frame; placing a 3D frame corresponding to the vehicle model at the posture to form a 3D semantic key point labeling result for the vehicle.

[0023] According to a specific implementation of an embodiment of the present invention, the 3D pose is solved by a PnP algorithm.

[0024] According to a specific implementation method of an embodiment of the present invention, the 3D posture is optimized for multiple frames, including optimizing the reprojection errors of multiple frames to obtain an optimized posture; a 3D frame corresponding to the vehicle model is placed at the optimized posture to form a 3D semantic key point labeling result of the vehicle.

[0025] In a second aspect, an embodiment of the present invention provides a vehicle key point labeling device, including: a vehicle model library module, used to pre-store 3D models, 3D semantic key points and 3D point clouds of various vehicle models; a data preprocessing module, used to perform data preprocessing on the full picture of each vehicle to be labeled, and obtain the picture block of each vehicle to be labeled and the binary mask of the picture block; a vehicle model retrieval module to be labeled, used to generate a vehicle point cloud from the picture block, and compare the vehicle point cloud with each of the 3D point clouds in the vehicle model library, and retrieve the model to which each vehicle to be labeled belongs; a vehicle posture solving and optimization module to be labeled, used to input the full picture of the vehicle to be labeled into a 3D target detection model, The initial pose value of each vehicle to be labeled is obtained; the initial pose value is inverted to obtain the initial view value of each vehicle to be labeled; the initial view value and the image block of the vehicle to be labeled are simultaneously input into the Vehicle_nerf algorithm module to obtain each optimized view; the optimized view is inverted to obtain each optimized pose; a 2D key point generation module is used to obtain the 3D semantic key points of each vehicle to be labeled in the vehicle coordinate system from the vehicle model library according to the retrieved vehicle model; according to the optimized pose, the 3D semantic key points are converted to the camera coordinate system; the 3D semantic key points in the camera coordinate system are projected to the image coordinate system to obtain the 2D semantic key point labeling results of each vehicle to be labeled.

[0026] According to a specific implementation method of an embodiment of the present invention, the device also includes a 3D key point labeling module, which is used to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled using a key point detection training model trained with each of the 2D semantic key point labeling result data, to obtain the key points and vehicle ID of each frame, and thereby solve the 3D pose of the vehicle frame by frame; a 3D frame corresponding to the vehicle model is placed at the pose to form a 3D semantic key point labeling result for the vehicle.

[0027] In the third aspect, an embodiment of the present invention provides a vehicle assisted driving method, including: the vehicle collects images and / or videos of surrounding vehicles from multiple angles in real time and continuously; based on the images and / or videos, the vehicle marks key points of surrounding vehicles by a method as described in any embodiment of the first aspect; the vehicle forms dynamic driving trajectories of surrounding vehicles based on the key points; and the vehicle assists in planning its own driving path based on the dynamic driving trajectories.

[0028] In a fourth aspect, an embodiment of the present invention provides a vehicle assisted driving system, comprising: multiple image and / or video acquisition units, for real-time and continuous acquisition of images and / or videos of vehicles around the vehicle from multiple angles; a vehicle key point marking device as described in any embodiment of the second aspect, for marking key points of vehicles around the vehicle based on the images and / or videos; a driving trajectory calculation unit, for forming a dynamic driving trajectory of the vehicles around the vehicle based on the key points; and a driving path planning unit, for assisting in planning the driving path of the vehicle based on the dynamic driving trajectory.

[0029] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any implementation method of the first aspect.

[0030] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in any embodiment of the first aspect of the present invention.

[0031] In a seventh aspect, an embodiment of the present invention further provides a vehicle, comprising a vehicle assisted driving system as described in the embodiment of the fourth aspect.

[0032] In an eighth aspect, an embodiment of the present invention further provides a vehicle, comprising the electronic device as described in the embodiment of the fifth aspect.

[0033] The embodiments of the present invention provide a vehicle key point labeling method and device, a vehicle assisted driving method and system, an electronic device, a computer-readable storage medium and a vehicle. By giving a vehicle picture, a point cloud of the vehicle can be generated using Vehicle_nerf, and the specific vehicle model can be automatically retrieved by comparing it in a pre-built 3D model library; the initial value of the vehicle posture to be labeled can be provided through monocular 3D target detection, and Vehicle_nerf can optimize the vehicle posture to be labeled by returning the photometric error, thereby realizing the automation of solving and optimizing the vehicle posture to be labeled; combined with the vehicle model, posture and binary mask of the image block, 2D semantic key point labeling results can be automatically generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 It is a principle block diagram of a first embodiment of a vehicle key point labeling method of the present invention;

[0036] Figure 2 It is a principle block diagram of a second embodiment of a vehicle key point labeling method of the present invention;

[0037] Figure 3 It is a principle block diagram of a third embodiment of a vehicle key point labeling method of the present invention;

[0038] Figure 4 A principle block diagram of a data preprocessing step in any embodiment of the vehicle key point labeling method of the present invention;

[0039] Figure 5 A principle block diagram of another data preprocessing step in any embodiment of the vehicle key point labeling method of the present invention;

[0040] Figure 6 It is a principle block diagram of a step of retrieving a vehicle model to be annotated in any embodiment of the vehicle key point annotating method of the present invention;

[0041] Figure 7 A principle block diagram of a step of solving and optimizing the posture of a vehicle to be labeled in any embodiment of the vehicle key point labeling method of the present invention;

[0042] Figure 8 A principle block diagram of another step of solving and optimizing the posture of a vehicle to be labeled in any embodiment of the vehicle key point labeling method of the present invention;

[0043] Fig. 9 It is a principle block diagram of the Vehicle_nerf algorithm in the step of retrieving the vehicle model to be labeled and the step of solving and optimizing the posture in any embodiment of the vehicle key point labeling method of the present invention;

[0044] Fig.10 It is a principle block diagram of a 2D key point generation step in any embodiment of the vehicle key point labeling method of the present invention;

[0045] Fig.11 It is a principle block diagram of a fourth embodiment of a vehicle key point labeling method of the present invention;

[0046] Fig.12It is a principle block diagram of the 3D key point marking step in the fourth embodiment of the vehicle key point marking method of the present invention;

[0047] Fig.13 It is a principle block diagram of a fifth embodiment of a vehicle key point labeling method of the present invention;

[0048] Fig.14 It is a schematic diagram of the principle of self-occlusion judgment in one embodiment of the vehicle key point labeling method of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] This embodiment provides a vehicle key point labeling method to improve the degree of automation of vehicle key point labeling.

[0051] Figure 1 FIG. 1 is a principle block diagram of a vehicle key point labeling method according to a first embodiment of the present invention. Figure 1 As shown, the application scenarios of this embodiment are vehicle 3D target perception, cloud-based pure perspective 3D labeling and autonomous driving.

[0052] The method of this embodiment may include: pre-building a vehicle model library: the vehicle model library includes a vehicle 3D model and a semantic key point library, in which the 3D model, 3D semantic key point and 3D point cloud of each vehicle model are stored; data pre-processing: data pre-processing is performed on the full picture of each vehicle to be labeled to obtain the picture block of each vehicle to be labeled and the binary mask of the picture block; vehicle model retrieval for the vehicle to be labeled: generating a vehicle point cloud from the picture block, comparing the vehicle point cloud with each of the 3D point clouds in the vehicle model library, and retrieving the model to which each vehicle to be labeled belongs; solving and optimizing the posture of the vehicle to be labeled: inputting the full picture of the vehicle to be labeled into the 3D target detection model, Obtain the initial pose value of each vehicle to be labeled; invert the initial pose value to obtain the initial view value of each vehicle to be labeled; input the initial view value and the image block of the vehicle to be labeled into the Vehicle_nerf algorithm module at the same time to obtain each optimized view; invert the optimized view to obtain each optimized pose; 2D key point generation: obtain the 3D semantic key points of each vehicle to be labeled in the vehicle coordinate system from the vehicle model library according to the retrieved vehicle model; according to the optimized pose, convert the 3D semantic key points to the camera coordinate system; project the 3D semantic key points in the camera coordinate system to the image coordinate system to obtain the 2D semantic key point labeling results of each vehicle to be labeled.

[0053] In this embodiment, the vehicle model library stores various data indexed by vehicle model, such as Figure 1 As shown, the vehicle 3D model and semantic key point library stores the 3D model, 3D semantic key points and 3D point cloud data of each vehicle model, such as the 3D model, 3D semantic key points and 3D point cloud data of a certain car.

[0054] Assuming there are J models in total, the entire library can be expressed as: .

[0055] in Represents a certain model j ( ) and its corresponding 3D vehicle model , and multiple versions of 3D semantic keypoint definitions .

[0056] A 3D vehicle model The 3D shape of the vehicle is defined by a set of geometric elements (such as spatial triangles), and its coordinate system is defined as the vehicle coordinate system, which is generally a point on the vehicle, such as the midpoint of the rear axle. The 3D vehicle model can be constructed once when a new model is available using the industry's common methods.

[0057] A version of 3D semantic keypoints It is a set of 3D points artificially defined on the vehicle 3D model. Generally, this set of points is defined on the surface of the vehicle 3D model, and the coordinate system of this set of points is the vehicle coordinate system. When the task requirements change, the number and position of these points will be changed to form a new version of 3D semantic key points.

[0058] In this embodiment, Figure 4 or Figure 5 As shown in the figure, data preprocessing is to obtain the image blocks of each vehicle to be labeled and the binary mask of the image block (Note: Binary mask is often called mask) from the full image of the vehicle. The places with 0 on the mask are background or other vehicles, and the places with 1 on the mask are vehicles to be labeled. There may be multiple vehicles of different models in the full image of the vehicle. Each vehicle needs to be identified separately and divided into its own image block and the binary mask of the image block, that is, there is only one vehicle in each image block, so as to facilitate the subsequent vehicle model recognition and posture solution operations.

[0059] In this embodiment, the vehicle model retrieval to be annotated is to identify the vehicle model in the image block, and the method adopted is to convert the vehicle image block into vehicle point cloud data, and then compare it with the 3D point cloud data of each vehicle model in the library one by one, until the 3D point cloud data of the same or most similar vehicle model is found, and then the vehicle model is determined as the vehicle model in the image block. Figure 6As shown, the Vehicle_nerf (Vehicle_Neural Radiance Fields) algorithm module (using the forward reasoning process therein) is used to generate the rendering point cloud of each vehicle to be labeled. The similarity between the rendering point cloud of a vehicle to be labeled and each 3D point cloud in the vehicle model library is calculated respectively, and the vehicle model in the vehicle model library corresponding to the maximum similarity is taken as the vehicle model retrieval result.

[0060] In this embodiment, the calculation steps of the vehicle rendering point cloud are as follows:

[0061] (11) Select a set of K perspectives Covering 360° of the vehicle, including represents the k-th camera-to-vehicle perspective; c2v is the abbreviation of camera to vehicle, which means camera to vehicle;

[0062] (12) Generate a depth map for each view , forming a set of K depth maps ,in represents the image of the i-th vehicle, represents the kth depth map;

[0063] (13) Convert to rendered point cloud .

[0064] In this embodiment, the similarity calculation steps are as follows:

[0065] (21) Take out the 3D model of model j from the model library ;

[0066] (22) Set the rendering point cloud There are Y points in total, calculate the distance from each point to The minimum distance ,get ;

[0067] (23) Calculate the score of the distance of each point and get , where each score is calculated by , is a pre-set distance threshold greater than 0, Limited to [0.0, 1.0] between;

[0068] (24) Calculate the overall score of vehicle model j ;

[0069] (25) Repeat the above steps (21)-(24) until all models are traversed and the scores of all models are obtained. ;

[0070] (26) Sort the scores and select the model with the highest score as the search result .

[0071] In this embodiment, the pose solving and optimization of the vehicle to be labeled is divided into two steps: pose initial value solving and pose initial value optimization. The pose initial value is obtained by solving the 3D target detection model, such as Figure 7 As shown in , the 3D target detection model can select a monocular 3D target detection model, and the input data is the full image; Figure 8 As shown, the 3D target detection model can select a multimodal 3D target detection model, whose input data is multimodal data, and the multimodal data is a combination of at least two of the full image, lidar point cloud data and millimeter wave data. The multimodal data helps to improve the accuracy of solving the initial value of the vehicle posture to be labeled and improve the overall labeling quality.

[0072] The optimization of the initial pose value is further divided into the following three steps: 1. Convert the initial pose value into the initial view value by inverting it; 2. Input the initial view value and the vehicle image block into the Vehicle_nerf algorithm module at the same time (using the back propagation process therein) to obtain the optimized view; 3. Invert the optimized view to obtain the optimized pose.

[0073] It should be emphasized here that the Vehicle_nerf algorithm module is a differentiable neural network that generates new views of vehicles, which is used in the above-mentioned solution and optimization of the pose of the vehicle to be labeled and the retrieval of the vehicle model to be labeled.

[0074] The forward reasoning process of Vehicle_nerf is:

[0075] , that is, input a picture of a car and a new perspective , which can generate a rendered color image corresponding to the new perspective and the rendered depth map ;if and input image If the perspective is consistent, .

[0076] The back propagation process of Vehicle_nerf is:

[0077] The generated new perspective color image is compared with the true value and produces an error. The error can be fed back and used to update and optimize the model parameters and new perspective of Vehicle_nerf.

[0078] like Fig. 9 As shown, Vehicle_nerf generally consists of four parts: encoder, decoder, sampling position and orientation generation module, and rendering module. Each part is differentiable, as detailed below:

[0079] First, the vehicle picture A set of features is extracted by the encoder , the number of features is at least 1, and each feature is a vector with a certain length. The encoder is composed of a common neural network, which is not limited here. Generally, CNN (Convolutional Neural Network) or ViT (Vision Transformer) can be used.

[0080] Simultaneously, the sampling position and orientation generation module is based on the new view , generate multiple sampling points , each point contains spatial position and orientation information The generation method is not limited here, and can be simple uniform sampling, or interacting with the decoder to implement importance sampling. The camera internal parameters generally include the focal length and principal point of the camera. The internal parameters of the same camera are constant.

[0081] Third, the decoder uses the feature set And the point information, process point by point, and get the color and density of each sampling point ,in, , The decoder is composed of a common neural network, which is not limited here. Generally, MLP (Multi Layer Perceptron, multi-layer fully connected neural network) can be used.

[0082] Finally, a rendering method is used to generate a color image and a depth map under a new perspective. The rendering method is not limited here and can be any differentiable rendering method.

[0083] In this embodiment, the pose of the vehicle to be annotated refers to the rotation and translation of the vehicle to be annotated relative to the camera in the camera coordinate system, which is generally represented by a 4×4 matrix in the following form: , in is a 3×3 rotation matrix, is a 3×1 translation vector.

[0084] The vehicle perspective to be annotated refers to the rotation and translation of the camera relative to the vehicle to be annotated in the vehicle coordinate system. When expressed as a 4×4 matrix, it is the inverse matrix of the vehicle pose to be annotated: .

[0085] In this embodiment, the process of solving and optimizing the position and posture of the vehicle to be labeled is as follows:

[0086] (1) Input the entire image into the monocular 3D target detection algorithm and obtain the initial pose values ​​of N vehicles to be labeled. , the pose accuracy is not high at this time, but it can be used as the initial value for further optimization;

[0087] (2) Invert the initial pose value to obtain the initial view value of N vehicles to be labeled: ;

[0088] (3) Optimize the initial value of the viewing angle: For each vehicle to be annotated, use Vehicle_nerf to iteratively optimize the viewing angle of each vehicle. The iterative steps for the i-th vehicle are as follows:

[0089] (a) Get the rendering image of the i-th car at the initial viewing angle: ,in represents the image of the i-th vehicle;

[0090] (b) Calculate the photometric error: ;

[0091] (c) Calculate the photometric error relative to the viewing angle The gradient is: ;

[0092] (d) With an update function based on the gradient renew ,make ;

[0093] (e) Repeat steps (a)-(d) until convergence. Optimize viewing angle ;

[0094] (4) Invert the optimized perspective to obtain the optimized pose In the above formula, represents the initial value, Indicates the optimized value.

[0095] In this embodiment, the above calculation process is fully automated, and the posture of the vehicle to be annotated can be solved and optimized without relying on the definition of semantic key points.

[0096] In this embodiment, Fig.10 As shown in the figure, the key to the 2D key point generation process is the coordinate position transformation of the 3D semantic key points. From the driver's perspective, from far to near, first transform from the vehicle coordinate system to the camera coordinate system, and then transform from the camera coordinate system to the image coordinate system. It should be noted that when projecting the 3D semantic key points of each vehicle to be annotated in the camera coordinate system to the image coordinate system, if the camera is distorted, it needs to be processed according to the general de-distortion solution.

[0097] In this embodiment, the steps of generating 2D key points are as follows:

[0098] (1) Based on the retrieved vehicle models and the semantic key point versions to be annotated, obtain the 3D key points of each vehicle to be annotated in the vehicle coordinate system from the vehicle model library. , Indicates that the i-th vehicle to be annotated uses the X-th version of the 3D semantic key points of the j-th model;

[0099] (2) Use the position and posture of the vehicle to be annotated to transform the 3D semantic key points from the vehicle coordinate system to the camera coordinate system, and obtain the 3D key points of each vehicle to be annotated in the camera coordinate system. ,in , is the optimized vehicle posture The rotating part, is the optimized vehicle posture The translation part of Represents matrix multiplication;

[0100] (3) The 3D key points of each vehicle to be annotated in the camera coordinate system Project to the image coordinate system to obtain the 2D key points of each vehicle to be labeled in the image coordinate system , each key point ,u, v are the pixel coordinates on the image, and the calculation method of u, v is: is the coordinate of a 3D key point in the camera coordinate system, is the camera internal parameter, , and is the focal length in pixels, and are the pixel coordinates of the principal point.

[0101] In a preferred embodiment, the step (2) further includes the following steps:

[0102] Self-occlusion judgment: Fig.14 As shown, assuming that the X-axis of the camera coordinate system is facing right, the Z-axis is facing forward, and the Y-axis is facing downward, the judgment process observed in the top view is:

[0103] (a) Scan all key points clockwise with the origin of the camera coordinate system as the center, find the leftmost point B and the rightmost point A, and these two points are visible;

[0104] (b) For other points P, calculate ,symbol Represents vector cross product, symbol Represents vector dot product;

[0105] (c) If It means that the point is visible, such as point D; if It means that the point is not visible, such as point C.

[0106] In a more preferred embodiment, the step (3) further includes the following steps:

[0107] (a) Other occlusion judgments: Each of , query the pixel coordinate value corresponding to the binary mask of the corresponding vehicle image block. If it is 0, it means there is other occlusion, and if it is 1, it means there is no other occlusion;

[0108] (b) Visibility generation: those without self-occlusion and other occlusion are visible, and the others are invisible;

[0109] The 2D key point annotation result is:

[0110] , are the pixel coordinates of the 2D semantic keypoints, is the visibility of the 2D semantic keypoint.

[0111] In this embodiment, the above process is fully automated. When the definition of semantic key points changes, there is no need to re-solve and optimize the annotated vehicle posture and retrieve the vehicle model. The previously obtained results can be reused, and the whole process is very efficient.

[0112] This embodiment can automatically realize vehicle model retrieval and vehicle posture solution optimization through the forward reasoning and back propagation process based on the Vehicle_nerf algorithm module, thereby realizing automatic labeling of vehicle key points according to vehicle model and posture, without manual intervention throughout the process, greatly improving the labeling efficiency and consistency of labeling quality. It does not require the input image to have overlapping continuous frames, which has better applicability.

[0113] Figure 2 FIG. 1 is a principle block diagram of a vehicle key point labeling method according to a second embodiment of the present invention. Figure 2 As shown, compared to Figure 1 In the first embodiment shown, laser radar point cloud data is added to the input end of the step of solving and optimizing the posture of the vehicle to be labeled, and the other parts remain unchanged.

[0114] In this embodiment, the laser radar point cloud data is the contour data of the vehicle measured by the laser radar, combined with the full picture collected by the camera to form bimodal data, which can improve the accuracy of the initial value solution of the vehicle pose to be labeled and the overall annotation quality. Input an original picture to be labeled, first through the data preprocessing algorithm module, obtain the picture blocks of each vehicle to be labeled and the mask of each picture block of the vehicle to be labeled. Secondly, input the picture blocks of each vehicle to be labeled into the vehicle model retrieval algorithm module to be labeled, and the retrieval algorithm interacts with the vehicle 3D model library to retrieve the model of each vehicle to be labeled. Synchronously, the full picture, the laser radar point cloud and each vehicle picture block to be labeled are input into the vehicle pose solution and optimization module to obtain the optimized pose of the vehicle to be labeled. Finally, the model, pose and mask of each vehicle to be labeled are input into the 2D key point generation module to generate 2D semantic key point annotation results.

[0115] Figure 3 FIG. 1 is a principle block diagram of a vehicle key point labeling method according to a third embodiment of the present invention. Figure 3 As shown, compared to Figure 2 In the second embodiment shown, millimeter-wave radar data is added to the input end of the step of solving and optimizing the posture of the vehicle to be labeled, and the other parts remain unchanged.

[0116] In this embodiment, the millimeter-wave radar data is the contour data of the vehicle measured by the millimeter-wave radar, which is combined with the full image and the lidar point cloud data to form tri-modal data, which can further improve the accuracy of the initial value solution of the vehicle posture to be labeled and the overall labeling quality.

[0117] Figure 4 FIG. 1 is a principle block diagram of a data preprocessing step in any embodiment of the vehicle key point labeling method of the present invention. Figure 4 As shown in FIG, the steps of data preprocessing are as follows: (1) using the 2D detection model to extract the 2D detection frames of the vehicles to be labeled in the entire image; (2) cropping the image according to each detection frame to obtain the image blocks of each vehicle to be labeled ; (3) Input the image blocks of each vehicle to be labeled into the 2D segmentation model to obtain the binary mask of each image block In this embodiment, the 2D detection model can be a 2D detection model of any known structure, and the 2D segmentation model can be a 2D segmentation model of any known structure, which can conveniently and quickly implement image segmentation of a single vehicle and provide accurate basic data for subsequent vehicle model retrieval and posture solution.

[0118] Figure 5 FIG. 1 is a schematic diagram of another data preprocessing step in any embodiment of the vehicle key point labeling method of the present invention. Figure 5As shown, the steps of data preprocessing are as follows: (1) input the full image of the image into the 2D instance segmentation model to obtain the binary mask full image of each vehicle to be labeled; (2) use the image cropping algorithm to crop the full image and the binary mask full image respectively to obtain the image block and the binary mask of each vehicle to be labeled. In this embodiment, a 2D instance segmentation model is used instead of Figure 4 Both the 2D detection model and the 2D segmentation model shown in the figure have a more compact structure and faster preprocessing speed.

[0119] Fig.11 is a principle block diagram of a vehicle key point labeling method according to a fourth embodiment of the present invention, Fig.12 FIG. 4 is a principle block diagram of the 3D key point labeling step in the vehicle key point labeling method of the fourth embodiment of the present invention. Fig.11 and Fig.12 As shown, the method of this embodiment is based on the first embodiment, and adds a 3D key point labeling step. The key point detection training model obtained by training with the 2D semantic key point labeling result data is used to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled, and the key points and vehicle ID of each frame are obtained, and the 3D posture of the vehicle is solved frame by frame; a 3D frame corresponding to the vehicle model is placed at the posture to form a 3D semantic key point labeling result of the vehicle.

[0120] Preferably, the 3D pose can be solved by a PnP (Perspective n-points, a method for finding the spatial pose after knowing the association between 2D points and 3D points in computer vision) algorithm.

[0121] More preferably, the 3D pose is optimized for multiple frames, including optimizing the reprojection errors of multiple frames to obtain an optimized pose; a 3D frame corresponding to the vehicle model is placed at the optimized pose to form a 3D semantic key point labeling result of the vehicle.

[0122] In this embodiment, since the 2D key point annotation scheme has a high degree of automation and good consistency in annotation quality, the generated 2D semantic key point annotation result data is large in scale and high in quality, and the trained key point detection training model has higher accuracy and stronger generalization ability. The model can be any key point detection training model. There is no restriction on the detection and tracking scheme; since better and larger data is used than other schemes, the pure visual vehicle 3D annotation results of this scheme are more accurate.

[0123] Fig.13 FIG. 5 is a principle block diagram of a vehicle key point labeling method according to a fifth embodiment of the present invention. Fig.13As shown, the 3D semantic key points of each vehicle model in the vehicle model library have multiple versions; when changing the version to mark the key points, the image block binary mask of the vehicle to be marked, the vehicle model retrieval result and the pose solution result are all kept unchanged for reuse, and only the 3D semantic key point projection needs to be re-executed. In this embodiment, the vehicle model, pose, and mask of each vehicle to be marked that have been used for other versions of 3D semantic key points are reused, and the three are input into the 2D key point generation module, and then the new key point definition is interacted with the vehicle model library to obtain a new version of the 3D semantic key points, and then the new version of the 3D semantic key points is used for projection to obtain a new 2D semantic key point marking result. The vehicle pose solution and vehicle model retrieval are decoupled from the semantic key point definition. The new definition of the semantic key point does not affect the obtained vehicle pose and vehicle model to be marked. The semantic key point change is convenient and flexible, and does not require manual intervention.

[0124] A vehicle key point labeling device according to an embodiment of the present invention includes:

[0125] The vehicle model library module is used to pre-store 3D models, 3D semantic key points and 3D point clouds of various vehicle models;

[0126] The data preprocessing module is used to perform data preprocessing on the entire image of each vehicle to be labeled, and obtain the image block of each vehicle to be labeled and the binary mask of the image block;

[0127] The vehicle model retrieval module to be labeled is used to generate a vehicle point cloud from the image block, compare the vehicle point cloud with each of the 3D point clouds in the model library, and retrieve the model of each vehicle to be labeled; the vehicle posture solution and optimization module to be labeled is used to input the full image of the vehicle to be labeled into the 3D target detection model to obtain the initial posture value of each vehicle to be labeled; invert the initial posture value to obtain the initial perspective value of each vehicle to be labeled; input the initial perspective value and the image block of the vehicle to be labeled into the Vehicle_nerf algorithm module at the same time to obtain each optimized perspective; invert the optimized perspective to obtain each optimized posture;

[0128] The 2D key point generation module is used to obtain the 3D semantic key points of each vehicle to be labeled in the vehicle coordinate system from the vehicle model library according to the retrieved vehicle model; according to the optimized posture, the 3D semantic key points are converted to the camera coordinate system; the 3D semantic key points in the camera coordinate system are projected to the image coordinate system to obtain the 2D semantic key point labeling results of each vehicle to be labeled.

[0129] The advantages of this embodiment are similar to those of the aforementioned related embodiments and will not be described in detail here.

[0130] In a preferred embodiment, the device further includes a 3D key point labeling module, which is used to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled using a key point detection training model trained with the 2D semantic key point labeling result data, obtain key points and vehicle IDs for each frame, and solve the 3D position and posture of the vehicle frame by frame; place a 3D frame corresponding to the vehicle model at the position and posture to form a 3D semantic key point labeling result for the vehicle. The advantages of this embodiment are similar to those of the aforementioned related embodiments, and will not be repeated here.

[0131] A vehicle assisted driving method according to an embodiment of the present invention includes: the vehicle collects images and / or videos of surrounding vehicles in real time and continuously from multiple angles; based on the images and / or videos, the vehicle marks key points of surrounding vehicles by the method described in any of the above embodiments; the vehicle forms dynamic driving trajectories of surrounding vehicles according to the key points; the vehicle assists in planning its own driving path according to the dynamic driving trajectories. The advantages of this embodiment are similar to those of the above embodiments and will not be repeated here.

[0132] A vehicle assisted driving system according to an embodiment of the present invention comprises: a plurality of image and / or video acquisition units, which are used to acquire images and / or videos of vehicles around the vehicle in real time and continuously from multiple angles; a vehicle key point labeling device as described in any embodiment of the second aspect, which is used to label key points of vehicles around the vehicle based on the images and / or videos; a driving trajectory calculation unit, which is used to form a dynamic driving trajectory of each vehicle around the vehicle according to the key points; and a driving path planning unit, which is used to assist in planning the driving path of the vehicle according to the dynamic driving trajectory. The advantages of this embodiment are similar to those of the aforementioned embodiments and will not be repeated here.

[0133] An electronic device according to an embodiment of the present invention comprises: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to each circuit or device of the electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any of the above embodiments. The advantages of this embodiment are similar to those of the above embodiments, and will not be repeated here.

[0134] A computer-readable storage medium according to an embodiment of the present invention stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in any embodiment of the present invention. The advantages of this embodiment are similar to those of the previous embodiment and are not repeated here.

[0135] A vehicle according to an embodiment of the present invention comprises the vehicle assisted driving system as described in the above embodiment. The advantages of this embodiment are similar to those of the above embodiment and will not be described in detail here.

[0136] Another vehicle according to an embodiment of the present invention comprises the electronic device as described in the above embodiment. The advantages of this embodiment are similar to those of the above embodiment, and will not be described in detail here.

[0137] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0139] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] For the convenience of description, the above device is described by dividing the functions into various units / modules. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A vehicle key point labeling method, characterized in that: include: Pre-build a vehicle model library: the vehicle model library includes a vehicle 3D model and a semantic key point library, which stores the 3D model, 3D semantic key points and 3D point cloud of each vehicle model; Data preprocessing: Perform data preprocessing on the entire image of each vehicle to be labeled to obtain the image block of each vehicle to be labeled and the binary mask of the image block; Retrieval of vehicle models to be annotated: generating a vehicle point cloud from the image block, comparing the vehicle point cloud with each of the 3D point clouds in the vehicle model library, and retrieving the model of each vehicle to be annotated; Solving and optimizing the pose of vehicles to be labeled: input the full picture of the vehicles to be labeled into the 3D target detection model to obtain the initial pose value of each vehicle to be labeled; invert the initial pose value to obtain the initial view value of each vehicle to be labeled; input the initial view value and the picture block of the vehicle to be labeled into the Vehicle_nerf algorithm module at the same time to obtain each optimized view; invert the optimized view to obtain each optimized pose; 2D key point generation: according to the retrieved vehicle model, the 3D semantic key points of each vehicle to be labeled in the vehicle coordinate system are obtained from the vehicle model library; according to the optimized posture, the 3D semantic key points are converted to the camera coordinate system; the 3D semantic key points in the camera coordinate system are projected to the image coordinate system to obtain the 2D semantic key point labeling results of each vehicle to be labeled.

2. The vehicle key point labeling method according to claim 1, characterized in that: The data preprocessing step includes: Extracting a 2D detection frame of the vehicle to be annotated in the entire image using a 2D detection model; Crop the image according to each detection frame to obtain the image block of each vehicle to be labeled; The image blocks of each vehicle to be labeled are input into the 2D segmentation model to obtain the binary mask of each image block.

3. The vehicle key point labeling method according to claim 1, characterized in that: The data preprocessing step includes: Input the full picture of the image into the 2D instance segmentation model to obtain a binary mask full picture of each vehicle to be labeled; The image cropping algorithm is used to crop the entire image and the binary mask image respectively to obtain the image blocks of each vehicle to be labeled and the binary mask of the image blocks.

4. The vehicle key point labeling method according to claim 1, characterized in that: The 3D semantic key points of each vehicle model in the vehicle model library have multiple versions; when changing the version for key point annotation, the binary mask of the image block of the vehicle to be annotated, the vehicle model retrieval result and the pose solution result are all kept unchanged for reuse, and only the 3D semantic key point projection needs to be re-executed.

5. The vehicle key point labeling method according to claim 1, characterized in that: The step of retrieving the vehicle model to be marked includes: Use Vehicle_nerf to generate the rendered point cloud of each vehicle to be annotated; The similarities between the rendered point cloud of a vehicle to be annotated and each of the 3D point clouds in the vehicle model library are calculated respectively, and the vehicle model in the vehicle model library corresponding to the maximum similarity value is taken as the vehicle model retrieval result.

6. The vehicle key point labeling method according to claim 5, characterized in that: The calculation steps of the vehicle rendering point cloud include: Select a set of K perspectives Covering 360° of the vehicle, including represents the perspective from the kth camera to the vehicle; Generate a depth map for each view , forming a set of K depth maps ,in represents the image of the i-th vehicle, represents the kth depth map; Will Convert to rendered point cloud .

7. The vehicle key point labeling method according to claim 5, characterized in that: The similarity calculation step includes: (21) Take out the 3D model of model j from the model library ; (22) Set the rendering point cloud There are Y points in total, calculate the distance from each point to The minimum distance ,get ; (23) Calculate the score of the distance of each point and get , where each score is calculated by , is a pre-set distance threshold greater than 0, Limited to between; (24) Calculate the overall score of vehicle model j ; (25) Repeat the above steps (21)-(24) until all models are traversed and the scores of all models are obtained. ; (26) Sort the scores and select the model with the highest score as the search result .

8. The vehicle key point labeling method according to claim 1, characterized in that: Assume that the initial values ​​of the N vehicle poses to be labeled are , the initial viewing angle of N vehicles to be labeled is The process of solving the optimized view angle is as follows: for each vehicle to be annotated, the view angle is iteratively optimized vehicle by vehicle using Vehicle_nerf, wherein the iterative steps for the i-th vehicle include: (a) Get the rendering image of the i-th car at the initial viewing angle: ,in represents the image of the i-th vehicle; (b) Calculate the photometric error: ; (c) Calculate the photometric error relative to the viewing angle The gradient is: ; (d) With an update function based on the gradient renew ,make ; (e) Repeat steps (a)-(d) until convergence. Optimize viewing angle .

9. The vehicle key point labeling method according to claim 8, characterized in that: The update function update is ,in is an artificially set learning rate.

10. The vehicle key point labeling method according to claim 4, characterized in that: The steps of generating 2D key points include: (1) Based on the retrieved vehicle models and the semantic key point versions to be annotated, obtain the 3D key points of each vehicle to be annotated in the vehicle coordinate system from the vehicle model library. , Indicates that the i-th vehicle to be annotated uses the X-th version of the 3D semantic key points of the j-th model; (2) Use the position and posture of the vehicle to be annotated to transform the 3D semantic key points from the vehicle coordinate system to the camera coordinate system, and obtain the 3D key points of each vehicle to be annotated in the camera coordinate system. ,in , is the optimized vehicle posture The rotating part, is the optimized vehicle posture The translation part of Represents matrix multiplication; (3) The 3D key points of each vehicle to be annotated in the camera coordinate system Project to the image coordinate system to obtain the 2D key points of each vehicle to be labeled in the image coordinate system , each key point , u, v are the pixel coordinates on the image, and the calculation method of u, v is: is the coordinate of a 3D key point in the camera coordinate system, is the camera internal parameter, , and is the focal length in pixels, and are the pixel coordinates of the principal point.

11. The vehicle key point labeling method according to claim 10, characterized in that: The step (2) further includes the following steps: Self-occlusion judgment: Assume that the X-axis of the camera coordinate system is facing right, the Z-axis is facing forward, and the Y-axis is facing downward. The judgment process observed in the top view is as follows: (a) Scan all key points clockwise with the origin of the camera coordinate system as the center, find the leftmost point B and the rightmost point A, and these two points are visible; (b) For other points P, calculate ,symbol Represents vector cross product, symbol Represents vector dot product; (c) If If This means the point is not visible.

12. The vehicle key point labeling method according to claim 11, characterized in that: The step (3) further includes the following steps: (a) Other occlusion judgments: Each of , query the pixel coordinate value corresponding to the binary mask of the corresponding vehicle image block. If it is 0, it means there is other occlusion, and if it is 1, it means there is no other occlusion; (b) Visibility generation: those without self-occlusion and other occlusion are visible, and the others are invisible; The 2D key point annotation result is: , are the pixel coordinates of the 2D semantic keypoints, is the visibility of the 2D semantic keypoint.

13. The vehicle key point labeling method according to claim 1, characterized in that: The 3D target detection model is a monocular 3D target detection model or a multimodal 3D target detection model; the input data of the multimodal 3D target detection model is multimodal data, and the multimodal data is a combination of at least two of the full image, lidar point cloud data and millimeter wave data.

14. The vehicle key point labeling method according to claim 1, characterized in that: The method further comprises: 3D key point labeling: Use the key point detection training model trained with the 2D semantic key point labeling result data to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled, obtain the key points and vehicle ID of each frame, and solve the 3D position and posture of the vehicle frame by frame; place a 3D frame corresponding to the vehicle model at the position and posture to form the 3D semantic key point labeling result of the vehicle.

15. The vehicle key point labeling method according to claim 14, characterized in that: The 3D pose is optimized in multiple frames, including optimizing the reprojection errors of the multiple frames, to obtain an optimized pose; a 3D frame corresponding to the vehicle model is placed at the optimized pose to form a 3D semantic key point labeling result of the vehicle.

16. A vehicle key point marking device, characterized in that: include: The vehicle model library module is used to pre-store 3D models, 3D semantic key points and 3D point clouds of various vehicle models; The data preprocessing module is used to perform data preprocessing on the entire image of each vehicle to be labeled, and obtain the image block of each vehicle to be labeled and the binary mask of the image block; A vehicle model retrieval module for vehicles to be annotated is used to generate a vehicle point cloud from the image block, compare the vehicle point cloud with each of the 3D point clouds in the vehicle model library, and retrieve the model of each vehicle to be annotated; The module for solving and optimizing the pose of vehicles to be labeled is used to input the full picture of the vehicles to be labeled into the 3D target detection model to obtain the initial pose value of each vehicle to be labeled; invert the initial pose value to obtain the initial view value of each vehicle to be labeled; input the initial view value and the picture block of the vehicle to be labeled into the Vehicle_nerf algorithm module at the same time to obtain each optimized view; invert the optimized view to obtain each optimized pose; The 2D key point generation module is used to obtain the 3D semantic key points of each vehicle to be labeled in the vehicle coordinate system from the vehicle model library according to the retrieved vehicle model; according to the optimized posture, the 3D semantic key points are converted to the camera coordinate system; the 3D semantic key points in the camera coordinate system are projected to the image coordinate system to obtain the 2D semantic key point labeling results of each vehicle to be labeled.

17. The vehicle key point marking device according to claim 16, characterized in that: The device also includes a 3D key point labeling module, which is used to perform frame-by-frame key point detection and tracking on the vehicle video to be labeled using a key point detection training model trained with the 2D semantic key point labeling result data, obtain key points and vehicle IDs for each frame, and solve the 3D posture of the vehicle frame by frame based on this; place a 3D frame corresponding to the vehicle model at the posture to form a 3D semantic key point labeling result for the vehicle.

18. A vehicle assisted driving method, characterized in that: include: The vehicle continuously collects images and / or videos of surrounding vehicles in real time from multiple angles; Based on the image and / or video, the vehicle marks key points of surrounding vehicles by the method according to any one of claims 1 to 15; The vehicle forms a dynamic driving trajectory of surrounding vehicles based on the key points; The vehicle assists in planning its own driving path according to the dynamic driving trajectories.

19. A vehicle assisted driving system, characterized in that: include: Multiple image and / or video acquisition units, used to acquire images and / or videos of vehicles around the vehicle in real time and continuously from multiple angles; The vehicle key point marking device according to claim 16 or 17, used to mark the key points of each vehicle around the vehicle based on the image and / or video; A driving trajectory calculation unit, used for forming a dynamic driving trajectory of each vehicle around the vehicle according to the key points; The driving path planning unit is used to assist in planning the driving path of the vehicle according to the dynamic driving trajectory.

20. An electronic device, characterized in that: The electronic device comprises: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of claims 1 to 15.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1-15.

22. A vehicle, characterized in that: Includes the vehicle assisted driving system as described in claim 19, or includes the electronic device as described in claim 20.

Images