Method and device for labeling dynamic object in image, medium and equipment
Two-dimensional and three-dimensional object predictions are performed through the image sequence collected by the camera, combined with vehicle status information, automatic annotation of dynamic objects is realized, solving the problems of low labeling efficiency and high cost in intelligent driving scenarios, and providing high-quality perceptual model training data.
Patent Information
- Application Number
- CN202411022477.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, the labeling of dynamic objects in intelligent driving scenarios is inefficient and costly, especially in the absence of lidar installation, the acquisition of point cloud data becomes difficult.
Image sequences are collected by the camera, and the motion state sequence of dynamic objects is determined using the two-dimensional and three-dimensional object prediction results. The automatic annotation of dynamic objects is realized in combination with vehicle status information, avoiding the need for manual annotation and lidar acquisition point clouds.
It improves the efficiency of dynamic object labeling, reduces costs, realizes automated labeling based on pure vision, and enhances the training data quality of the intelligent driving perception model.
Smart Images

Figure CN118968450A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to computer vision technology, and in particular to a method, device, medium and equipment for marking dynamic objects in an image. Background Art
[0002] In intelligent driving scenarios, the driving of vehicles often depends on the perception results of the vehicle's surrounding environment, and the perception results need to be obtained based on various perception models. For dynamic objects around the vehicle, it is necessary to use a large number of sample images to train the model to obtain a perception model that can perceive the state of dynamic objects in the vehicle's local coordinate system. The training of the model depends on the true state value of the dynamic object in the sample image. In the related art, point clouds are usually collected by lidar, and images are synchronously collected by cameras as sample images, and then the true state value (or state label) of the dynamic object in the sample image is obtained through manual annotation based on the radar point cloud synchronized with the sample image. On the one hand, the efficiency of manual annotation is low. On the other hand, due to the high cost of lidar and the fact that there is no lidar installed on the vehicle, the collection of point cloud data becomes difficult. Summary of the invention
[0003] The embodiments of the present disclosure provide a method, apparatus, medium and device for annotating dynamic objects in an image, which can improve annotation efficiency and reduce annotation costs.
[0004] A first aspect of the present disclosure provides a method for labeling dynamic objects in an image, comprising: determining vehicle status information corresponding to each frame of a first image sequence acquired by a camera on a vehicle; the first image sequence comprising multiple frames of images; based on the first image sequence, determining a two-dimensional object prediction result and a three-dimensional object prediction result corresponding to each frame of the first image sequence; based on the three-dimensional object prediction result corresponding to each frame of the first image sequence, determining a first motion state sequence of the dynamic objects in the first image sequence; based on the vehicle status information, the two-dimensional object prediction result and the first motion state sequence corresponding to each frame of the first image sequence, determining a dynamic object labeling result corresponding to each frame of the first image sequence.
[0005] According to a second aspect of the present disclosure, a device for labeling dynamic objects in images is provided, comprising: a first processing module, for determining vehicle status information corresponding to each frame of a first image sequence acquired by a camera on a vehicle; the first image sequence comprising multiple frames of images; a second processing module, for determining, based on the first image sequence, a two-dimensional object prediction result and a three-dimensional object prediction result corresponding to each frame of the first image sequence; a third processing module, for determining a first motion state sequence of dynamic objects in the first image sequence based on the three-dimensional object prediction result corresponding to each frame of the first image sequence; and a fourth processing module, for determining, based on the vehicle status information, the two-dimensional object prediction result and the first motion state sequence corresponding to each frame of the first image sequence, a dynamic object labeling result corresponding to each frame of the first image sequence.
[0006] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method for marking dynamic objects in an image as described in any of the above embodiments of the present disclosure.
[0007] The fourth aspect of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the method for marking dynamic objects in an image described in any of the above embodiments of the present disclosure.
[0008] A fifth aspect of the present disclosure provides a computer program product. When instructions in the computer program product are executed by a processor, the method for marking dynamic objects in an image provided by any of the above embodiments of the present disclosure is performed.
[0009] Based on the method, device, medium and equipment for marking dynamic objects in images provided by the above embodiments of the present disclosure, since the three-dimensional object prediction result can characterize the position and posture information of the dynamic object in the three-dimensional space, the first motion state sequence of the dynamic object determined based on the three-dimensional object prediction result can characterize the position and posture changes of the dynamic object in the three-dimensional space, and in combination with the vehicle state information, an association relationship between the states in the first motion state sequence is established, so that the position and posture, speed, acceleration, angular velocity and other state information corresponding to the dynamic object in each frame of the image can be obtained according to the first motion state sequence and the vehicle state information, and then the annotation result of the dynamic object in each frame of the image is obtained by combining the association relationship between the two-dimensional object prediction result and the three-dimensional object prediction result of the dynamic object, thereby achieving the purpose of realizing automatic annotation of dynamic objects in the image based on a pure visual solution, thereby effectively improving the annotation efficiency, and at the same time avoiding the use of laser radar to collect point clouds and manual annotation, greatly reducing the annotation cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 This is an exemplary application scenario of the method for marking dynamic objects in an image provided by the present disclosure;
[0011] Figure 2 is a flowchart of a method for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure;
[0012] Figure 3 is a flow chart of a method for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure;
[0013] Figure 4 is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure;
[0014] Figure 5 is a flowchart of a method for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure;
[0015] Figure 6 is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure;
[0016] Figure 7 is a flowchart of a method for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure;
[0017] Figure 8 is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure;
[0018] Fig. 9 is a flowchart of a method for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure;
[0019] Fig.10 is a flowchart of target tracking provided by an exemplary embodiment of the present disclosure;
[0020] Fig.11 It is a schematic diagram of the principle of unifying the coordinate system of the motion state sequence of an object provided by an exemplary embodiment of the present disclosure;
[0021] Fig.12 is a schematic diagram of trajectory optimization principle provided by an exemplary embodiment of the present disclosure;
[0022] Fig.13 is a flowchart of two-dimensional and three-dimensional cross-validation provided by an exemplary embodiment of the present disclosure;
[0023] Fig.14 is a structural schematic diagram of a device for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure;
[0024] Fig.15 is a structural schematic diagram of a device for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure;
[0025] Fig.16 is a structural schematic diagram of a device for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure;
[0026] Fig.17 is a structural schematic diagram of a device for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure;
[0027] Fig.18 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0028] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.
[0029] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0030] SUMMARY OF THE DISCLOSURE
[0031] In the process of realizing the present disclosure, the inventors found that in the related art, point clouds are usually collected by laser radar, and images are synchronously collected by camera as sample images, and then the state true value (or state label) of the dynamic object in the sample image is obtained by manual annotation based on the radar point cloud synchronized with the sample image. The state true value includes the position true value and size true value of the dynamic object in the local coordinate system of the vehicle. On the one hand, the efficiency of manual annotation is low, which prolongs the training time of the perception model; on the other hand, due to the high cost of laser radar and the fact that there is no laser radar installed on the vehicle, the collection of point cloud data becomes difficult.
[0032] Exemplary Overview
[0033] Figure 1 This is an exemplary application scenario of the method for marking dynamic objects in an image provided by the present disclosure. Figure 1As shown, the vehicle (i.e., the vehicle) 11 may be provided with one or more cameras 12 of different view angles. When the vehicle 11 is traveling on the road, the camera 12 may be used to collect environmental images around the vehicle 11. Multiple frames of images form an image sequence (referred to as a first image sequence), and each frame of image may include images collected by cameras of one or more view angles at the same time. And the vehicle state information at the time of collection of each frame of image may be recorded. In the case where it is necessary to annotate the dynamic objects in the image, the annotation method provided by the present disclosure may be used to determine the vehicle state information corresponding to each frame of image in the first image sequence collected by the camera on the vehicle 11; and then based on the first image sequence, determine the two-dimensional object prediction result and the three-dimensional object prediction result corresponding to each frame of image in the first image sequence; based on the three-dimensional object prediction result corresponding to each frame of image, determine the first motion state sequence of the dynamic object in the first image sequence; based on the vehicle state information, the two-dimensional object prediction result and the first motion state sequence of the dynamic object corresponding to each frame of image, determine the dynamic object annotation result corresponding to each frame of image. Dynamic objects may include, for example, other vehicles 13 around the vehicle 11 in the figure, and may also include pedestrians, cyclists (not shown), etc. around the vehicle 11. The road where the vehicle is located is divided into multiple lanes by lane lines 14. The coordinate system xoy is an exemplary vehicle local coordinate system with the center of the rear axle of the vehicle 11 as the origin. In practical applications, the origin of the vehicle local coordinate system can also be any other position on the vehicle 11. The method provided by the present disclosure, since the three-dimensional object prediction result can characterize the position and posture information of the dynamic object in the three-dimensional space, the first motion state sequence of the dynamic object determined based on the three-dimensional object prediction result can characterize the position and posture change of the dynamic object in the three-dimensional space, and the association relationship between the states in the first motion state sequence is established in combination with the vehicle state information, so that the position and posture, speed, acceleration, angular velocity and other state information corresponding to the dynamic object in each frame image can be obtained according to the first motion state sequence and the vehicle state information, and then the association relationship between the two-dimensional object prediction result and the three-dimensional object prediction result of the dynamic object can be combined to obtain the annotation result of the dynamic object of each frame image, so as to achieve the purpose of realizing the automatic annotation of the image dynamic object based on the pure vision solution, thereby effectively improving the annotation efficiency, and at the same time avoiding the use of laser radar to collect point clouds and manual annotation, greatly reducing the annotation cost.
[0034] It should be noted that the method disclosed herein can be executed offline in electronic devices such as servers and terminal devices that can meet the needs of computing resources, storage resources, etc., and can also be executed on the on-board computing platform (or on-board terminal) on the vehicle 11 when the resources are sufficient, without specific limitation. When executed in a server, terminal device, etc., the first image sequence captured by the camera and the vehicle status information corresponding to each frame of the first image sequence can be obtained from the vehicle 11. Alternatively, after completing data collection for a certain period of time, the vehicle 11 transmits the collected first image sequence and the vehicle status information corresponding to each frame of the first image sequence to electronic devices such as servers and terminal devices. The transmission method can be network transmission, U disk transmission, etc., without specific limitation.
[0035] It should also be noted that the labeling method provided by the present invention is not limited to image labeling in intelligent driving scenarios on the road, but can also be applied to any other scenarios where dynamic image object labeling can be performed, such as drone driving scenarios. The camera that collects the image is the camera on the drone, and the vehicle status information corresponding to the image is the status information of the drone.
[0036] Exemplary Methods
[0037] Figure 2 FIG. 1 is a flow chart of a method for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers, terminal devices, etc. Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:
[0038] Step 201 : determining vehicle state information corresponding to each frame of an image in a first image sequence captured by a camera on the vehicle.
[0039] The first image sequence includes multiple frames of images. The vehicle can be any vehicle equipped with a camera. For example, the vehicle can be a data collection vehicle, a user vehicle authorized by a user, etc. The camera on the vehicle can collect images of the vehicle's surrounding environment, and the vehicle status information corresponding to the image can be obtained through other sensors on the vehicle. Each frame of image can include images collected by one or more cameras. The images collected by each camera in each frame of image can be images that are time-synchronized by hardware or software. For example, a plurality of cameras are provided on the vehicle, and the plurality of cameras correspond to different viewing angles, and the plurality of cameras can include, for example, a front-view camera, a left-view camera, a right-view camera, a rear-view camera, etc. Through a corresponding synchronization mechanism, multiple frames of images collected by each camera with time synchronization are obtained to form a first image sequence. The vehicle status information corresponding to each frame of image is the vehicle status information at the time point (or moment) when the frame of image is collected. The vehicle status information can include one or more of the vehicle's position, posture (i.e., orientation), speed, acceleration, angular velocity, and other information in a global coordinate system. The global coordinate system can be a world coordinate system, a vehicle coordinate system at the start time, and the vehicle status information can be obtained by observing sensors on the vehicle. The sensor used to determine the vehicle status information may include, for example, one or more of an odometer (odometry, or wheel speed meter), an inertial measurement unit (IMU), a global navigation satellite system (GNSS), etc. The vehicle status information corresponding to each frame of image is the vehicle status information synchronized with the time of the frame of image.
[0040] In some optional embodiments, if the time of the vehicle sensor used to determine the vehicle status information is not synchronized with the time of capturing the image, the vehicle status information synchronized with the image time may be obtained by interpolation or other methods.
[0041] Step 202: Based on the first image sequence, determine the two-dimensional object prediction result and the three-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0042] Among them, the two-dimensional object prediction result corresponding to each frame image may include the two-dimensional detection box (or two-dimensional pixel coordinates) of the object in the frame image in the image. Optionally, the two-dimensional object prediction result may also include the prediction results of attributes such as the type, occlusion degree, and subdivision category (or secondary type, subtype) of the object. The type of the object may include other vehicles, pedestrians and cyclists, lane lines, curbs, buildings, signs, etc. The subdivision category may include subdivision attributes corresponding to the type. For example, the subdivision category of the vehicle type may include large vehicles, medium-sized vehicles, small vehicles, vehicle colors, etc., the subdivision category of the pedestrian type may include clothing colors, etc., and the subdivision category corresponding to the cyclist type may include the category of two-wheeled vehicles, etc. The three-dimensional object prediction result corresponding to each frame image may include the three-dimensional detection box of the object in the frame image. The three-dimensional detection box may be a detection box in a local coordinate system. The local coordinate system may be a vehicle coordinate system (Vehicle Coordinate System, referred to as: VCS), or a preconfigured coordinate system with an arbitrary position on the vehicle as the origin, which is not specifically limited. Optionally, the three-dimensional object prediction result may further include a tracking identifier of the object, where the tracking identifier is used to indicate an association relationship between three-dimensional detection boxes of the object corresponding to each frame image in the first image sequence.
[0043] In some optional embodiments, the objects in the image may include other vehicles, pedestrians, cyclists and other movable dynamic objects around the vehicle, and optionally, the objects may also include immovable static objects such as trees, traffic signs, buildings, etc. The number of objects may be one or more.
[0044] In some optional embodiments, the two-dimensional object prediction result and the three-dimensional object prediction result may be prediction results of dynamic objects. For example, the two-dimensional object prediction result includes at least one of the two-dimensional detection frame, type, occlusion degree, etc. of the dynamic object in the image, and the three-dimensional object prediction result includes at least one of the three-dimensional detection frame, tracking mark, etc. of the dynamic object in the image. This can reduce unnecessary processing of static objects, avoid subsequent separation processing of dynamic and static objects, and improve labeling efficiency.
[0045] Step 203: determining a first motion state sequence of the dynamic object in the first image sequence based on the three-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0046] The first motion state sequence of the dynamic object may include states of the dynamic object corresponding to one or more frames of images. The state of the dynamic object may include the position, posture, size and other states of the dynamic object.
[0047] In some optional embodiments, the number of the dynamic objects may be one or more. In the case of multiple dynamic objects, each dynamic object has a corresponding first motion state sequence.
[0048] In some optional embodiments, the three-dimensional object prediction result corresponding to each frame image includes a three-dimensional detection frame of the dynamic object, which represents the position, posture, size and other states of the dynamic object at the time point corresponding to the frame image. Combined with the states of the dynamic object at the time points corresponding to multiple frames of images, the first motion state sequence of the dynamic object can be determined.
[0049] In some optional embodiments, when the three-dimensional object prediction result includes the tracking mark of the object, the first motion state sequence of each dynamic object can be determined according to the tracking mark of the dynamic object and the three-dimensional detection frame in the three-dimensional object prediction result corresponding to each frame image. When the three-dimensional object prediction result does not include the tracking mark of the object, target tracking can be performed on the three-dimensional object prediction results corresponding to each frame image, and the first motion state sequence of each dynamic object can be determined according to the target tracking result.
[0050] Step 204 , based on the vehicle state information corresponding to each frame image in the first image sequence, the two-dimensional object prediction result and the first motion state sequence, determine the dynamic object labeling result corresponding to each frame image in the first image sequence.
[0051] Among them, the annotation results of the dynamic objects corresponding to each frame image may include one or more information such as the position, posture, speed, acceleration, angular velocity, type, degree of occlusion, etc. of the dynamic object in the local coordinate system of the vehicle corresponding to the frame image.
[0052] In some optional embodiments, for each dynamic object, the two-dimensional object prediction result may include the two-dimensional position of the dynamic object in the image, and the first motion state sequence may include the state (including position, posture, size, etc.) of the dynamic object in the local vehicle coordinate system corresponding to each frame image, and the vehicle state information corresponding to each frame image can characterize the posture of the local vehicle coordinate system corresponding to the frame image in the global coordinate system, so that based on the vehicle state information corresponding to each frame image, each state in the first motion state sequence can be unified into the same coordinate system, and then based on each state in the same coordinate system, the speed, acceleration, angular velocity and other states of the dynamic object are determined, and the annotation result of the dynamic object is determined by combining the two-dimensional object prediction result corresponding to the dynamic object and the obtained position, posture, speed, acceleration, angular velocity, size and other states of the dynamic object.
[0053] The method for labeling dynamic objects in images provided by the present embodiment, since the three-dimensional object prediction result can characterize the position and posture information of the dynamic object in the three-dimensional space, the first motion state sequence of the dynamic object determined based on the three-dimensional object prediction result can characterize the position and posture change of the dynamic object in the three-dimensional space, and in combination with the vehicle state information, the association relationship between the states in the first motion state sequence is established, so that the position and posture, speed, acceleration, angular velocity and other state information corresponding to the dynamic object in each frame of the image can be obtained according to the first motion state sequence and the vehicle state information, and then the labeling result of the dynamic object in each frame of the image can be obtained by combining the association relationship between the two-dimensional object prediction result and the three-dimensional object prediction result of the dynamic object, thereby achieving the purpose of realizing automatic labeling of dynamic objects in the image based on a pure vision solution, thereby effectively improving the labeling efficiency, and at the same time avoiding the use of laser radar to collect point clouds and manual labeling, greatly reducing the labeling cost.
[0054] Figure 3 It is a flowchart of a method for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure.
[0055] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, step 202 of determining the two-dimensional object prediction result and the three-dimensional object prediction result corresponding to each frame image in the first image sequence based on the first image sequence may include:
[0056] Step 2021, based on the pre-trained two-dimensional detection model, process each frame of the first image sequence to obtain a two-dimensional object prediction result corresponding to each frame of the first image sequence.
[0057] The two-dimensional detection model may adopt any visual two-dimensional detection model in the related art, which is not limited in the embodiments of the present disclosure. The two-dimensional object prediction result may refer to the above content.
[0058] In some optional embodiments, in order to further ensure the accuracy and precision of the two-dimensional detection results, the two-dimensional detection model may adopt a large visual two-dimensional detection model with higher performance.
[0059] In some optional embodiments, the two-dimensional detection model may include at least one of a first two-dimensional detection model and a second two-dimensional detection model. The first two-dimensional detection model is used to predict the two-dimensional detection frame of the object, or to predict the two-dimensional detection frame and object type of the object. The second two-dimensional detection model is used to predict the occlusion degree, subdivision category and other attributes of the object. The occlusion degree may include different levels, for example, the occlusion degree may include full occlusion, severe occlusion, slight occlusion, no occlusion, etc. The number of levels and the grading method of the specific occlusion degree can be set according to actual needs, and the embodiments of the present disclosure are not limited.
[0060] Step 2022: Based on the pre-trained three-dimensional detection model, each frame of the first image sequence is processed to obtain a three-dimensional object prediction result corresponding to each frame of the first image sequence.
[0061] Among them, the three-dimensional detection model can adopt any visual three-dimensional detection model in the relevant technology, and the embodiments of the present disclosure are not limited thereto.
[0062] In some optional embodiments, in order to further ensure the accuracy and precision of the three-dimensional object prediction results, the three-dimensional detection model can use a high-performance visual three-dimensional large model. For example, the three-dimensional detection model can use a visual BEV (Bird's-Eye-View) 3D (Three Dimensions) large model.
[0063] In some optional embodiments, the three-dimensional detection model can be a single-frame model or a time series model. The single-frame model refers to inference on one frame of image each time to obtain the three-dimensional object prediction result corresponding to the frame of image. In this case, the three-dimensional object prediction result includes the three-dimensional detection box corresponding to the object in the frame of image. The time series model means that inference can be performed on an image sequence (i.e., multiple frames of images) at a time to obtain the three-dimensional object prediction result corresponding to each frame of image in the image sequence. In this case, the three-dimensional object prediction result includes the three-dimensional detection box and tracking mark corresponding to the object in the frame of image. That is, the time series model can realize the prediction and tracking of three-dimensional objects at the same time.
[0064] It should be noted that step 2021 and step 2022 may be performed in any order.
[0065] This embodiment predicts the two-dimensional prediction and three-dimensional prediction of the object in each frame image in the first image sequence through the two-dimensional detection model and the three-dimensional detection model obtained by pre-training, respectively, which can ensure the accuracy and effectiveness of the two-dimensional object prediction results and the three-dimensional object prediction results, and thus provide accurate and effective two-dimensional object prediction results and three-dimensional object prediction results for the labeling of dynamic objects.
[0066] Figure 4 It is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0067] In some optional embodiments, such as Figure 4 As shown, step 2021 processes each frame of the first image sequence based on the pre-trained two-dimensional detection model to obtain a two-dimensional object prediction result corresponding to each frame of the first image sequence, which may include:
[0068] Step 20211: Perform object detection on each image frame in the first image sequence based on the first two-dimensional detection model to obtain a two-dimensional detection frame corresponding to each image frame in the first image sequence.
[0069] The first two-dimensional detection model is a pre-trained target detection model (or object detection model), which is used to detect targets in images, obtain a two-dimensional detection frame of an object in the image, and optionally, obtain an object type to which the two-dimensional detection frame of the object in the image belongs. Each frame of the first image sequence can be input into the first two-dimensional detection model to obtain a two-dimensional detection frame and an object type corresponding to the frame.
[0070] Step 20212: Perform object attribute detection on each image frame in the first image sequence based on the second two-dimensional detection model to obtain the object occlusion degree and object secondary type corresponding to each image frame in the first image sequence.
[0071] Among them, the second two-dimensional detection model is a pre-trained object attribute detection model, which is used to perform attribute detection on the image to obtain the occlusion degree and subdivision category of the object in the image. The object secondary type is the subdivision category (or subdivision type) of the object. The occlusion degree and subdivision category can be referred to the above content.
[0072] In some optional embodiments, the first two-dimensional detection model and the second two-dimensional detection model may adopt any model in the related art, which is not limited in the embodiments of the present disclosure.
[0073] It should be noted that step 20211 and step 20212 may be executed in any order.
[0074] Step 20213, based on the two-dimensional detection frame, object occlusion degree and object secondary type corresponding to each frame image in the first image sequence, determine the two-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0075] Among them, the two-dimensional detection box can represent the two-dimensional position of the object in the image (i.e., pixel coordinates), the object occlusion degree can represent the degree to which the object is occluded in the image, and the object secondary type can represent the subdivision attributes of the object. The two-dimensional detection box, object occlusion degree and object secondary type corresponding to each frame image can be used as the two-dimensional object prediction result corresponding to the frame image, so as to determine the partial labeling result of the dynamic object in the image.
[0076] In addition to being able to detect a two-dimensional detection frame of an object in an image, this embodiment can also detect the degree of occlusion and subdivision category of the object, and use the detection results for dynamic object labeling of the image, thereby greatly improving the richness of the dynamic object labeling result state.
[0077] Figure 5It is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0078] In some optional embodiments, such as Figure 5 As shown, step 2022 of processing each frame of the first image sequence based on the pre-trained three-dimensional detection model to obtain a three-dimensional object prediction result corresponding to each frame of the first image sequence may include:
[0079] Step 20221: Based on the pre-trained first three-dimensional detection model, each frame image in the first image sequence is processed respectively to obtain a three-dimensional detection frame in the local coordinate system corresponding to each frame image in the first image sequence.
[0080] The first 3D detection model is a single-frame detection model obtained by pre-training, that is, each time a frame of an image in the first image sequence is detected, a 3D detection frame in a local coordinate system corresponding to the frame of the image is obtained. By processing each frame of the image in the first image sequence respectively, a 3D detection frame in a local coordinate system corresponding to each frame of the image is obtained. Optionally, the first 3D detection model can also detect the object type to which the 3D detection frame of the object in the image belongs.
[0081] In some optional embodiments, the first three-dimensional detection model may adopt any three-dimensional detection model in the related art, which is not limited in the embodiments of the present disclosure.
[0082] Step 20222: Determine a three-dimensional object prediction result corresponding to each frame in the first image sequence based on the three-dimensional detection frame corresponding to each frame in the first image sequence.
[0083] Among them, the three-dimensional detection frame corresponding to each frame image can be determined as the three-dimensional object prediction result corresponding to the frame image.
[0084] This embodiment performs three-dimensional object detection on each frame in the first image sequence based on a pre-trained first three-dimensional detection model of a single frame, which can ensure the accuracy of the three-dimensional object prediction result corresponding to each frame, and further provide accurate and effective three-dimensional object prediction results for the annotation of dynamic objects.
[0085] In some optional embodiments, such as Figure 5 As shown, step 203 of determining a first motion state sequence of a dynamic object in the first image sequence based on a three-dimensional object prediction result corresponding to each frame image in the first image sequence may include:
[0086] Step 2031 , performing target tracking on the three-dimensional object prediction result corresponding to each frame image in the first image sequence to obtain a target tracking result.
[0087] The target tracking result may include a tracking identifier of a 3D detection frame corresponding to each frame image, and the tracking identifier is used to indicate that the 3D detection frames of the previous and next frames belong to the same object. That is, the tracking identifiers of the 3D detection frames of the same object in the previous and next frames are the same.
[0088] In some optional embodiments, a preset target tracking method may be used to target track the three-dimensional object prediction result corresponding to each frame image in the first image sequence to obtain a target tracking result. Exemplarily, the target tracking method is, for example, taking each frame image as the current frame image in turn, and for all three-dimensional detection frames corresponding to the current frame image, based on the target tracking result of the previous frame, using a Kalman filter algorithm or other algorithms, establishing an association relationship between the three-dimensional detection frame corresponding to the current frame image and the target tracking result of the previous frame, and obtaining the target tracking result of the current frame. Specifically, the predicted position of the three-dimensional detection frame of the previous frame in the current frame may be predicted by a Kalman filter or other algorithm, and the predicted position may be matched with the three-dimensional detection frame of the current frame. For example, the similarity may be determined by one or more methods such as GIoU (Generalized Intersection over Union), Euclidean distance, etc., to find the best match between the three-dimensional detection frame of the current frame and the predicted position of the previous frame, thereby establishing a matching relationship between the three-dimensional detection frame of the current frame and the three-dimensional detection frame of the previous frame, and combining the target tracking result of the previous frame to obtain the target tracking result of the current frame, which is used to maintain the three-dimensional detection frame trajectory (i.e., three-dimensional detection frame sequence) of each object. In the case of matching through multiple methods such as GIoU and Euclidean distance, multiple similarities can be weightedly fused as the final similarity.
[0089] In some optional embodiments, the three-dimensional object prediction results corresponding to each frame of the image may be preprocessed before target tracking. Preprocessing may include filtering low-confidence three-dimensional detection frames, filtering overlapping three-dimensional detection frames, etc. Among them, the confidence of the three-dimensional detection frame may be output while the model outputs the three-dimensional detection frame, and the three-dimensional detection frame with low confidence may be filtered according to the preconfigured confidence threshold. That is, the three-dimensional detection frames with confidence lower than the confidence threshold are filtered out, and the three-dimensional detection frames with confidence greater than or equal to the confidence threshold are retained for target tracking. Overlapping three-dimensional detection frames may be filtered by non-maximum suppression (NMS).
[0090] Step 2032: Determine a first motion state sequence of the dynamic object in the first image sequence based on the target tracking result.
[0091] Among them, the target tracking result can characterize the correlation between the three-dimensional detection frames corresponding to each frame image in the first image sequence. Therefore, according to the target tracking result, a sequence of three-dimensional detection frames belonging to the same object can be obtained, and the motion state sequence of the object can be determined based on the sequence of three-dimensional detection frames of each object. The length of the motion state sequence of each object can be determined according to the life cycle of the object being observed. The life cycle refers to the period from the first observation of the object to the disappearance of the object from the image.
[0092] In some optional embodiments, when the three-dimensional object prediction results include prediction results of static objects and dynamic objects, the dynamic objects in the object can be determined by separating the dynamic and static objects, thereby determining the first motion state sequence of the dynamic objects from the motion state sequence of the object. For example, the dynamic and static objects can be separated according to the speed of the object, the speed distribution of the object in the motion state sequence (or trajectory), the acceleration distribution, etc., to determine the dynamic objects in the object and obtain the first motion state sequence of the dynamic objects.
[0093] In some optional embodiments, during the target tracking process, the life cycle management of each dynamic object can be performed in real time, and the four states of trajectory start (birth), trajectory active (alive), no matching trajectory (no asso), and trajectory end (dead) of each dynamic object can be maintained. In practical applications, the life cycle management of static objects can also be performed on the basis of the life cycle management of dynamic objects, which is not limited in the embodiments of the present disclosure.
[0094] Through target tracking, this embodiment can establish an association relationship between the three-dimensional detection frames corresponding to multiple frames of images. According to the association relationship, the trajectory motion state sequence of the dynamic object can be effectively obtained to ensure the accuracy of the motion state sequence of the dynamic object, thereby providing an effective motion state of the dynamic object for the annotation of the dynamic object in each frame of the first image sequence.
[0095] Figure 6 It is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0096] In some optional embodiments, such as Figure 6 As shown, step 2032 of determining a first motion state sequence of the dynamic object in the first image sequence based on the target tracking result may include:
[0097] Step 20321: Based on the three-dimensional object prediction result and the target tracking result corresponding to each frame image in the first image sequence, determine the prediction state sequence corresponding to the object in the first image sequence.
[0098] The predicted state sequence corresponding to the object may include a 3D detection frame sequence corresponding to each object in the first image sequence or a sequence of states such as position and size of the object obtained by further calculating the 3D detection frame sequence. The 3D detection frames belonging to the same object in the 3D object prediction results of each frame may be established based on the target tracking result, and the predicted state sequence of the object may be determined based on the 3D detection frames belonging to the same object.
[0099] Step 20322: Determine the dynamic or static type corresponding to the object according to the predicted state sequence corresponding to the object in the first image sequence.
[0100] The dynamic and static types corresponding to the object may include dynamic and static types. The dynamic and static types corresponding to the object may be determined by dynamic and static separation. For example, at least one of the velocity distribution and acceleration distribution of the object may be determined according to the predicted state sequence corresponding to the object, and the dynamic and static types of the object may be determined according to at least one of the velocity distribution and the acceleration distribution.
[0101] Step 20323: Determine a first motion state sequence of the dynamic object in the first image sequence according to the dynamic or static type and the predicted state sequence corresponding to the object.
[0102] Among them, after determining the dynamic and static types of the object, the predicted state sequence of the dynamic object can be determined from the predicted state sequence of the object, and then the first motion state sequence of the dynamic object can be determined according to the predicted state sequence of the dynamic object. For example, the predicted state sequence of the dynamic object can be used as the first motion state sequence, or the predicted state sequence can be further processed to obtain the first motion state sequence. In the case where the predicted state sequence includes the position and posture of one or more frames of the dynamic object, further processing may include calculating the speed, acceleration, angular velocity, etc. of the dynamic object according to the predicted state sequence of the dynamic object, and the first motion state sequence may include the position, posture, speed, acceleration, angular velocity, size, etc. of the dynamic object corresponding to one or more frames of images.
[0103] In this embodiment, since the predicted state sequence of the object represents the position change of the object in the local coordinate system corresponding to multiple frames of images, the velocity and acceleration distribution of the object can be calculated based on the predicted state sequence, so that the dynamic and static types of the object can be accurately determined, thereby providing an accurate and effective dynamic and static type reference for obtaining the first motion state sequence of the dynamic object.
[0104] Figure 7 It is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0105] In some optional embodiments, such as Figure 7As shown, step 2022 of processing each frame of the first image sequence based on the pre-trained three-dimensional detection model to obtain a three-dimensional object prediction result corresponding to each frame of the first image sequence may include:
[0106] Step 2022a: Based on the pre-trained second three-dimensional detection model, the first image sequence is processed to obtain a three-dimensional detection frame and a tracking mark corresponding to each frame image in the first image sequence.
[0107] The second 3D detection model is a temporal model, that is, the first image sequence as a whole can be used as a single input to the second 3D detection model, which infers the first image sequence and outputs a 3D detection frame and tracking mark corresponding to each frame of the image.
[0108] In some optional embodiments, when the first image sequence is processed through a timing model, the vehicle status information corresponding to each frame image in the first image sequence can be input into the timing model together with the first image sequence, and the vehicle status information is used to establish an association relationship between the local coordinate systems of different frames to improve the accuracy of the three-dimensional detection box and the tracking mark.
[0109] Step 2022b: Determine a three-dimensional object prediction result corresponding to each frame in the first image sequence based on the three-dimensional detection frame and tracking mark corresponding to each frame in the first image sequence.
[0110] The three-dimensional detection frame and tracking mark corresponding to each frame image may be determined as the three-dimensional object prediction result corresponding to the frame image.
[0111] This embodiment can simultaneously obtain the three-dimensional detection frame and tracking mark corresponding to each frame image through the timing model, that is, it realizes three-dimensional detection and target tracking at the same time, which can reduce the subsequent target tracking process, improve the processing efficiency of determining the first motion state sequence of dynamic objects, and thus improve the overall processing efficiency of image annotation.
[0112] In some optional embodiments, such as Figure 7 As shown, step 203 of determining a first motion state sequence of a dynamic object in the first image sequence based on a three-dimensional object prediction result corresponding to each frame image in the first image sequence may include:
[0113] Step 203a: determining a predicted state sequence corresponding to the object in the first image sequence based on the three-dimensional detection frame and tracking mark corresponding to each frame image in the first image sequence.
[0114] The tracking mark is equivalent to the target tracking result of the above embodiment. The specific operation of step 203a can refer to the above step 20321, which will not be described in detail here.
[0115] Step 203b: Determine the dynamic or static type corresponding to the object according to the predicted state sequence corresponding to the object in the first image sequence.
[0116] Step 203c: determining a first motion state sequence of the dynamic object in the first image sequence according to the dynamic or static type and the predicted state sequence corresponding to the object.
[0117] The specific operations of step 203b and step 203c can refer to the aforementioned step 20322 and step 20323, which will not be repeated here.
[0118] In this embodiment, since the predicted state sequence of the object represents the position change of the object in the local coordinate system corresponding to multiple frames of images, the velocity and acceleration distribution of the object can be calculated based on the predicted state sequence, so that the dynamic and static types of the object can be accurately determined, thereby providing an accurate and effective dynamic and static type reference for obtaining the first motion state sequence of the dynamic object.
[0119] In some optional embodiments, after determining the dynamic and static types corresponding to the object, the size, angle, etc. of the object may be post-processed and corrected, and the first motion state sequence of the dynamic object may be determined based on the post-processing correction results to improve the accuracy and effectiveness of the first motion state sequence. The post-processing correction may include, for example: setting the speed and acceleration of the static object to zero, correcting position jumps, unifying the length, width, and height dimensions (for example, taking the mode of the length, width, and height of multiple frames), etc.
[0120] Figure 8 It is a flowchart of a method for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0121] In some optional embodiments, based on any of the above embodiments, Figure 8 As shown, step 204 of determining the dynamic object labeling result corresponding to each frame image in the first image sequence based on the vehicle state information corresponding to each frame image in the first image sequence, the two-dimensional object prediction result and the first motion state sequence may include:
[0122] Step 2041: Based on the vehicle state information corresponding to each frame of the first image sequence, the first motion state sequence is corrected to obtain a corrected second motion state sequence.
[0123] Among them, the second motion state sequence may include the optimized (corrected) state of one or more frames of the dynamic object. The state of each frame may include the optimized position, posture, speed, acceleration, angular velocity, size and other states of the dynamic object. Correcting the first motion state sequence refers to optimizing the position, posture, speed, acceleration, angular velocity and other states of each frame in the first motion state sequence, that is, optimizing the trajectory of the dynamic object, so that the trajectory of the dynamic object better meets the pre-configured constraints, so as to further improve the accuracy of the dynamic object annotation results.
[0124] In some optional embodiments, the first motion state sequence may be corrected by using a graph optimization method or any other possible optimization method to obtain a corrected second motion state sequence.
[0125] Step 2042: Determine the dynamic object labeling result corresponding to each frame in the first image sequence based on the two-dimensional object prediction result corresponding to each frame in the first image sequence and the second motion state sequence.
[0126] Among them, the second motion state sequence of each dynamic object includes the position, posture, speed, acceleration, angular velocity, size and other states of the dynamic object in the local coordinate system corresponding to the image of the corresponding frame. According to the correspondence between the state in the local coordinate system of each frame and the two-dimensional detection box in the two-dimensional object prediction result, as well as the position relationship between the two-dimensional detection box and the image, automatic labeling of the dynamic object in each frame image can be achieved to obtain the labeling result of the dynamic object in each frame image. The labeling result can include two-dimensional labeling results such as the two-dimensional detection box, object type, subdivision category, degree of occlusion of the dynamic object, as well as three-dimensional labeling results such as the position, posture, speed, acceleration, angular velocity, size and other of the dynamic object in the local coordinate system.
[0127] In some optional embodiments, each state in the first motion state sequence may also include speed, acceleration, angular velocity, etc. calculated based on the position and posture of the dynamic object. The speed and angular velocity can be obtained based on the relationship between the position and posture changes of adjacent frames and time, and the acceleration can be obtained based on the relationship between speed and time. By correcting and optimizing the first motion state sequence, the optimized position, posture, speed, acceleration, angular velocity, size, etc. of the dynamic object in each frame are obtained.
[0128] This embodiment optimizes the trajectory of the dynamic object by correcting the first motion state sequence, so that the trajectory of the dynamic object is more consistent with the actual motion trajectory of the dynamic object, thereby improving the accuracy and effectiveness of the motion trajectory of the dynamic object, and further improving the accuracy of the image dynamic object annotation results.
[0129] In some optional embodiments, the first motion state sequence is corrected based on the vehicle state information corresponding to each frame in the first image sequence to obtain a corrected second motion state sequence in step 2041, which may include: based on the vehicle state information corresponding to each frame in the first image sequence, each motion state in the first motion state sequence is converted to the local coordinate system corresponding to the vehicle in the target frame to obtain a third motion state sequence; the target frame is any frame in the multiple frames; the first motion state sequence includes the motion state corresponding to each frame in the first image sequence. Based on the relationship between position, speed, acceleration and preset constraints, the third motion state sequence is corrected to obtain a corrected second motion state sequence.
[0130] Among them, the relationship between position, speed, and speed can be expressed as a uniform acceleration motion relationship, a variable acceleration motion relationship, etc. The preset constraints may include adjacent frame acceleration constraints, adjacent frame speed constraints, speed and acceleration constraints, posture and speed, acceleration constraints, etc. The adjacent frame acceleration constraints may be that the acceleration of adjacent frames (front and back frames) is the same or close, for example, the change in the acceleration of any frame relative to the acceleration of the previous frame is less than the acceleration change threshold. The adjacent frame speed constraints may be that the speed of adjacent frames is the same or close, for example, the change in the speed of any frame relative to the speed of the previous frame is less than the speed change threshold. The speed and acceleration constraints may be that the error between the speed calculated based on the acceleration and the observed speed satisfies the speed error condition. The speed error condition may refer to that the error is less than the speed error threshold. The posture and speed, acceleration constraints may be that the relationship between the posture, speed, and acceleration conforms to the constraints of a preconfigured motion model (e.g., a uniform acceleration motion model). For example, the error between the position calculated based on the speed and acceleration and the observed position satisfies the error position error condition, and the position error condition may refer to that the error is less than the position error threshold.
[0131] In some optional embodiments, as the vehicle moves, the conversion relationship between the local coordinate system and the global coordinate system changes continuously. The vehicle state information corresponding to each frame image can characterize the position of the local coordinate system relative to the global coordinate system at the time of acquisition of the frame image. Therefore, the conversion relationship between the local coordinate system of each frame and the global coordinate system can be established based on the vehicle state information corresponding to each frame image. For each motion state in the first motion state sequence, the motion state can be converted to the global coordinate system based on the conversion relationship between the local coordinate system corresponding to the motion state and the global coordinate system, and then converted to the local coordinate system corresponding to the target frame according to the conversion relationship between the global coordinate system and the local coordinate system corresponding to the target frame, and the motion state of the motion state in the local coordinate system of the target frame is obtained, and the third motion state sequence is obtained based on the motion state of each motion state in the first motion state sequence in the local coordinate system of the target frame.
[0132] In some optional embodiments, the third motion state sequence can be corrected by graph optimization or other means based on the relationship between position, velocity, acceleration and preset constraints to obtain a corrected motion state sequence, so that the corrected motion state sequence satisfies the preset constraints. The corrected motion state sequence can be used as the second motion state sequence, or each motion state in the corrected motion state sequence can be converted from the local coordinate system of the target frame back to the local coordinate system of the frame corresponding to the motion state to obtain the second motion state sequence. That is, the second motion state sequence can be a sequence in the same local coordinate system, or it can be a sequence in each local coordinate system of each frame image. The coordinate system of the motion state can be converted as needed, and the embodiments of the present disclosure are not limited thereto.
[0133] This embodiment can more intuitively represent the changes in the trajectory of the dynamic object by uniformly converting each motion state in the first motion state sequence into a local coordinate system with the same timestamp, and facilitates trajectory optimization of the dynamic object through the relationship between position, velocity, acceleration and preset constraints, so that the optimized trajectory is more consistent with the actual motion trajectory of the dynamic object, thereby improving the accuracy and effectiveness of the dynamic object labeling results.
[0134] In some optional embodiments, determining the dynamic object labeling result corresponding to each frame image in the first image sequence based on the two-dimensional object prediction result corresponding to each frame image in the first image sequence and the second motion state sequence in step 2042 may include:
[0135] Determine the three-dimensional boxes corresponding to each motion state in the second motion state sequence; project each three-dimensional box to the image pixel coordinate system corresponding to each motion state to obtain the two-dimensional projection boxes corresponding to each three-dimensional box; match the two-dimensional detection box in the two-dimensional object prediction result corresponding to each frame image in the first image sequence with the two-dimensional projection box corresponding to the frame image to obtain the matching result corresponding to each frame image; determine the dynamic object labeling result corresponding to each frame image according to the matching result.
[0136] Among them, the three-dimensional frame corresponding to each motion state in the second motion state sequence can be determined according to the position, posture and size of the dynamic object in the motion state, that is, the three-dimensional frame is a three-dimensional frame after trajectory optimization. For example, the center point position of the three-dimensional frame can be determined according to the position of the dynamic object, the orientation of the three-dimensional frame can be determined according to the posture of the dynamic object, and the length, width and height of the three-dimensional frame can be determined according to the size of the dynamic object, so as to determine the three-dimensional frame. The three-dimensional frame can be a three-dimensional frame in the local coordinate system corresponding to the motion state. According to the conversion relationship between the local coordinate system and the image pixel coordinate system, the three-dimensional frame is projected to the image pixel coordinate system of the corresponding frame (called the target frame) to obtain the two-dimensional projection frame corresponding to the three-dimensional frame. The two-dimensional projection frame is matched with the two-dimensional detection frame in the two-dimensional object prediction result corresponding to the target frame image to obtain the matching result corresponding to the target frame image. The matching result may include the state of whether the two-dimensional projection frame matches each two-dimensional detection frame. For any two-dimensional detection frame in the two-dimensional detection frame corresponding to the target frame image, if the two-dimensional detection frame matches the two-dimensional projection frame, it is determined that the two-dimensional projection frame matches successfully, and the corresponding relationship between the two-dimensional projection frame and the matching two-dimensional detection frame can be recorded. Then, according to the corresponding relationship, the motion state corresponding to the two-dimensional projection frame and the two-dimensional detection frame, the occlusion degree and subdivision category corresponding to the two-dimensional detection frame are determined as the annotation result of the dynamic object in the target frame image. Based on the above process, the annotation result of the dynamic object in the second motion state sequence in each corresponding frame image can be obtained. In the case of multiple dynamic objects, the annotation result of each dynamic object in each frame image (each frame image corresponding to the second motion state sequence) can be obtained according to the above process based on the second motion state sequence of each dynamic object.
[0137] In some optional embodiments, the matching of the two-dimensional projection frame and the two-dimensional detection frame may be achieved by a greedy algorithm based on intersection-over-union ratio.
[0138] In some optional embodiments, for each two-dimensional projection frame, if the two-dimensional detection frame that matches the two-dimensional projection frame does not exist in the two-dimensional detection frame of the target frame image corresponding to the two-dimensional projection frame, it means that the three-dimensional frame corresponding to the two-dimensional projection frame may be blocked or invisible on the target frame image, and the three-dimensional frame can be marked as invalid (ignore), which can effectively suppress the problem of false detection.
[0139] This embodiment reprojects the three-dimensional frame of the dynamic object onto the image and matches it with the two-dimensional detection frame of the image. This can achieve cross-verification of the three-dimensional state and the two-dimensional state, ensure the visibility of the dynamic object corresponding to the three-dimensional state on the image, avoid annotating dynamic objects that are invisible on the image, and further improve the accuracy of the annotation results.
[0140] In some optional embodiments, the dynamic object annotation results can also be quality checked according to the matching results to obtain quality inspection results, and quality prompt information can be output according to the quality inspection results. For example, the similarity between the two-dimensional projection frame corresponding to each three-dimensional frame and the matching two-dimensional detection frame can be calculated, and the size relationship between each similarity and the similarity threshold can be calculated. According to the size relationship between each similarity and the similarity threshold, the proportion of two-dimensional projection frames with similarity less than the similarity threshold is calculated. If the proportion exceeds the proportion threshold, it means that there are more low-similarity matches, and it can be determined that the quality of the annotation results is low, and then the quality prompt information is output to give a quality warning to the user, so that the user can quickly analyze or locate problems such as vehicle sensor calibration parameters or time synchronization.
[0141] In some optional embodiments, during the optimization of the third motion state sequence of the dynamic object, the state observation results of the dynamic object can also be combined with other sensors to perform trajectory optimization. That is, during the optimization process, the motion state observed by the image is combined with the motion state observed by other sensors to determine the initial state of the optimization process, and preset constraints are imposed on the initial state based on graph optimization, and the optimized posture, velocity, acceleration and other information are output.
[0142] In some optional embodiments, Fig. 9 FIG. 1 is a flowchart of a method for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure. Fig. 9 As shown, the process may include:
[0143] Step 301, vehicle state processing. That is, the positioning information is collected through IMU 31, Odometry 32 and GNSS 33, and the vehicle state processing is performed based on the positioning information to obtain the vehicle state information (i.e., vehicle state information). By parsing the positioning information, the local coordinate system of the vehicle in the first frame can be converted to the global coordinate system, and the position, posture, speed, acceleration and other states of the vehicle in the global coordinate system at each subsequent time point can be obtained to obtain the vehicle state information at each time point.
[0144] In practical applications, Odometry 32 can be used alone to determine the vehicle state information, and IMU 31 and GNSS 33 are optional.
[0145] In step 302, the cloud-based visual three-dimensional detection model performs three-dimensional detection. That is, multi-view synchronized images are collected through one or more cameras 35, and multiple frames of images constitute a first image sequence. The first image sequence is subjected to three-dimensional detection by the cloud-based visual three-dimensional detection model 36, and a three-dimensional object prediction result corresponding to each frame of the first image sequence is obtained. And the vehicle status information can be time-synchronized with each frame of the first image sequence (for example, time synchronization can be achieved through interpolation) to obtain the vehicle status information corresponding to each frame of the image. For example, if the timestamp of a frame image is 150 milliseconds, and the timestamp of the vehicle status information includes 100 milliseconds, 200 milliseconds, etc., then the vehicle status information at 150 milliseconds is obtained by interpolation based on the vehicle status information at 100 milliseconds and the vehicle status information at 200 milliseconds, as the vehicle status information corresponding to the image at 150 milliseconds.
[0146] In some optional embodiments, the original image sequence captured by the camera may be extracted at preset time intervals to obtain a first image sequence, thereby further improving the effectiveness of the first image sequence.
[0147] In some optional embodiments, for the case of multiple cameras, each group of multi-view images synchronized by multiple cameras can be used as a frame of image according to the timestamp nearest neighbor method. The accuracy of the three-dimensional object prediction results can be improved by performing three-dimensional detection through multi-view images. If it is a single-frame three-dimensional detection model, one frame of multi-view images is used as the model input each time to obtain the three-dimensional object prediction result corresponding to the frame image. If it is a time-series three-dimensional detection model, the first image sequence and the vehicle status information corresponding to each frame image can be used as the model input, and the three-dimensional object prediction result and target tracking result corresponding to each frame image are output at the same time.
[0148] Step 303: uploading the three-dimensional object prediction result to a database. That is, uploading the three-dimensional object prediction result corresponding to each frame of image to a database for storage for subsequent processing.
[0149] Step 304: The cloud-based visual two-dimensional detection model performs two-dimensional detection. That is, the cloud-based visual two-dimensional detection model performs two-dimensional detection on each frame of the first image sequence to obtain a two-dimensional object prediction result corresponding to each frame of the image. The cloud-based visual two-dimensional detection model may include a first two-dimensional detection model and a second two-dimensional detection model.
[0150] Step 305, uploading the two-dimensional object prediction result to a database, that is, uploading the two-dimensional object prediction result corresponding to each frame image in the first image sequence to the database for storage and use in subsequent processing.
[0151] Step 306, target tracking. That is, target tracking is performed on the three-dimensional object prediction results corresponding to each frame image in step 303 to obtain a three-dimensional detection frame sequence or a prediction state sequence of each object. For example, the three-dimensional object prediction results in the local coordinate system of multiple frames can be filtered based on a Kalman filter algorithm to output the trajectory of the object with a tracking mark (i.e., a three-dimensional detection frame sequence or a prediction state sequence).
[0152] Step 307, target tracking post-processing. Target tracking post-processing may include separation of dynamic and static objects, object size correction, object angle correction, and other post-processing. For example, the post-processing may specifically include setting the velocity and acceleration of the static object to zero, position jump correction, length, width, and height unification (for example, taking the mode of the length, width, and height of multiple frames), etc. The first motion state sequence of each dynamic object is obtained through target tracking post-processing.
[0153] Step 308, trajectory optimization. That is, the first motion state sequence is corrected to obtain a corrected second motion state sequence. In the process of trajectory optimization, the trajectory of the first motion state sequence can be optimized in combination with the observation results of other sensors such as Lidar and Radar to obtain an optimized second motion state sequence. The observation of other sensors is optional and can be used or not according to the actual configuration of the vehicle sensors. For example, for the data returned by a vehicle equipped with Lidar and Radar, the trajectory of the first motion state sequence can be optimized based on multiple observation sources such as Lidar ranging, Radar speed measurement, and visual two-dimensional detection model internal reference ranging.
[0154] Step 309, synchronizing 2D and 3D timestamps, that is, synchronizing the motion state of each frame of the second motion state sequence with the time of each frame image, obtaining the motion state of the dynamic object corresponding to the capture moment of each frame image, and ensuring the temporal consistency of the motion state and the 2D detection frame of the image.
[0155] Step 310, 2D and 3D matching and cross-validation. That is, the 3D frame of each motion state in the second motion state sequence is reprojected onto the image of the corresponding frame, and matched with the 2D detection frame to obtain a matching result. The visibility of the dynamic object in the image is verified according to the matching result. The validity of the motion state for the annotation result (i.e., whether to use the motion state as the annotation result of the image) is determined according to the visibility, and the cross-validation of 2D and 3D is realized, which can improve the accuracy of the annotation result and effectively avoid or reduce the false detection rate.
[0156] Step 311, true value quality check. That is, according to the dynamic object annotation result, the three-dimensional frame is projected onto the corresponding image, and the similarity (such as intersection-over-union ratio, Euclidean distance, etc.) between the two-dimensional projection frame and the corresponding two-dimensional detection frame is calculated. If a large number of low-similarity matches appear, it indicates that there may be problems with the calibration parameters or time synchronization of the vehicle sensor. The user can be warned of the low true value quality, so that the user can take corresponding measures for the problem, and re-annotate the dynamic object in the image according to the method of the embodiment of the present disclosure after the problem is solved.
[0157] Step 312, obtaining the true value. That is, if the true value quality check meets the quality standard, the dynamic object labeling result corresponding to each frame image in the first image sequence can be obtained.
[0158] In some optional embodiments, a retrieval interface can be provided for users, so that users can obtain true value information (three-dimensional coordinates of dynamic objects, subdivision categories, speed, acceleration, etc.) from one or more perspectives in a specified dynamic object, a specified timestamp, and a specified coordinate system according to actual needs, so as to meet the different true value needs of different users and further enhance the user experience.
[0159] The method of the disclosed embodiment can realize the automatic annotation of multi-view dynamic objects based on a purely visual annotation method, so that the data returned by the test vehicle equipped with only a camera can also be used to generate high-precision, complete-element true values of dynamic objects, and can be used in the data closed loop of perception, testing, regulation and control simulation, that is, the true value of the dynamic object obtained by automatic annotation is used for the training of the perception model, and the trained model is used to perceive the environment to obtain the perception result, and the perception result is used for testing, regulation and control simulation, etc. The problem that images cannot be annotated and cannot be used in the absence of point clouds is solved, so that a large amount of bad case data can reflect its value. It has been verified that the method of the disclosed embodiment has become an important part of data closure capabilities. The improvement in the training performance of the three-dimensional detection task of the vehicle-side model is comparable to the training effect of the true value data generated based on the lidar point cloud.
[0160] In some optional embodiments, Fig.10 FIG. 1 is a flowchart of target tracking provided by an exemplary embodiment of the present disclosure. Fig.10 As shown, the target tracking process may include the following steps:
[0161] Step 401, obtain the three-dimensional detection frame observations in the VCS coordinate system at all times, that is, the three-dimensional object prediction results in the vehicle coordinate system corresponding to each frame image in the first image sequence.
[0162] Step 402, preprocessing. That is, preprocessing the three-dimensional object prediction results corresponding to each frame of the image. Preprocessing may include filtering low-confidence three-dimensional detection frames, filtering overlapping three-dimensional detection frames, etc. Among them, when the three-dimensional detection model outputs the three-dimensional detection frame, the confidence of the three-dimensional detection frame can be output, and the three-dimensional detection frame with low confidence is filtered according to the preconfigured confidence threshold. That is, the three-dimensional detection frames with confidence lower than the confidence threshold are filtered out, and the three-dimensional detection frames with confidence greater than or equal to the confidence threshold are retained for target tracking. Overlapping three-dimensional detection frames can be filtered by non-maximum suppression (NMS). The accuracy and effectiveness of the three-dimensional object prediction results can be improved by preprocessing.
[0163] Step 403, predict the pose at time k (called predicted pose) based on the pose at time k-1. The moment corresponding to each frame is sequentially taken as the current moment, and the current moment is represented as time k. The predicted pose at time k is predicted based on the pose of each 3D detection frame at time k-1. The pose of each 3D detection frame at time k-1 can be predicted based on Kalman filtering or other methods to obtain the predicted pose at time k.
[0164] Step 404, the predicted pose is matched with the 3D detection frame at time k. That is, each predicted pose is matched with each 3D detection frame observed at time k, similarity is calculated, and the association relationship between each predicted pose and the 3D detection frame at time k is determined based on the similarity. The similarity can be calculated based on one or more of GIoU, Euclidean distance, etc.
[0165] Step 405, object lifecycle management, maintain the lifecycle state of each object's trajectory. The lifecycle state can include four states: trajectory birth, trajectory alive, no matching trajectory, and trajectory dead. By maintaining the lifecycle state of the object, the tracking sequence of each object can be obtained, and then the predicted state sequence of each object can be obtained based on the tracking sequence.
[0166] This embodiment implements tracking of multi-frame three-dimensional object prediction results of dynamic objects through methods such as Kalman filtering, which can improve the accuracy of the first motion state sequence of the dynamic object.
[0167] In some optional embodiments, Fig.11 FIG. 1 is a schematic diagram showing the principle of unifying the coordinate system of the motion state sequence of an object provided by an exemplary embodiment of the present disclosure. Fig.11As shown, in the original observation in the global coordinate system, the ego vehicle trajectory 51 and the object trajectory 52 and the object trajectory 53 all include the postures of three time stamps t0, t1, and t2. By unifying the ego vehicle posture, the object trajectories are unified to the t0 time stamp, and the postures of the object trajectories 52 and 53 at each time stamp relative to the ego vehicle at the t0 time stamp can be obtained, and then the postures of the object estimation 52 and the object trajectory 53 in the ego vehicle VCS coordinate system at the t0 time stamp can be obtained, so that each motion state of the object is converted to the local coordinate system corresponding to the vehicle in the target frame (the frame corresponding to the time t0), and the third motion state sequence is obtained.
[0168] In some optional embodiments, Fig.12 FIG. 1 is a schematic diagram of the trajectory optimization principle provided by an exemplary embodiment of the present disclosure. Fig.12 As shown, position P represents the position in each motion state in the third motion state sequence. The position in the third motion state sequence is used as the initial position of the position optimization variable 71 for trajectory optimization. The initial speed of the speed optimization variable 72 is obtained by fitting the speed of the position of b1. The initial value of the acceleration optimization variable 73 is further fitted by the fitted speed. 74 represents the speed of the last optimization, and 75 represents the acceleration of the last optimization. The preset constraints include: the constraints between the position of b2 and the speed and acceleration of the last optimization (corresponding to the above-mentioned posture and speed and acceleration constraints), the front and rear frame speed change constraints of b3 (corresponding to the above-mentioned adjacent frame speed constraints), the speed and acceleration constraints of b4 (corresponding to the above-mentioned speed and acceleration constraints), and the front and rear frame acceleration change constraints of b5 (corresponding to the above-mentioned adjacent frame acceleration constraints). The origins of different grayscales in the figure represent the nodes of the position optimization variable 71, the speed optimization variable 72 and the acceleration optimization variable 73 in the graph optimization process, and each constraint represents the edge between the nodes. Through the graph optimization method, the motion state sequence of the dynamic object is continuously optimized to obtain the optimized motion state sequence, so that the optimized motion state sequence gradually satisfies the preset constraints. The figure only shows the nodes and edges of the motion state in the graph optimization within a part of the time window (such as the time window in the figure includes the time points t0, t1, t2, and t3). The optimization of the entire motion state sequence can be achieved by continuously moving the window, and the position, speed, and acceleration information of each frame after optimization can be obtained.
[0169] In some optional examples, the uniformly accelerated motion equation can be used to Establish kinematic constraints for the position, velocity, and acceleration optimization nodes (i.e., constraints b2 in the figure). Where t represents the time corresponding to any frame (as the current frame), t-1 represents the time before time t (i.e., the time corresponding to the frame before the current frame), and x t -x t-1 Represents the position x at time t-1t-1 The position x at time t t The displacement between t-1 represents the speed at time t-1, Δt represents the duration of movement from time t-1 to time t, and a represents acceleration.
[0170] In some optional examples, the uniformly accelerated motion equation v can be used to calculate the uniformly accelerated motion equation v. t =v t-1 +aΔt, establish the kinematic constraints of the speed and acceleration optimization variable nodes (i.e. the constraints of b4 in the figure). t-1 represents the velocity at time t-1, v t represents the speed at time t.
[0171] In some optional examples, the speed change constraint of the preceding and following frames of b3 may be that the speed change between the preceding and following frames is less than a speed change threshold, that is, the speed of adjacent frames is not allowed to change suddenly.
[0172] In some optional examples, the acceleration change constraint of b5 between the previous and next frames may be that the acceleration between the previous and next frames is the same.
[0173] In some optional embodiments, Fig.13 FIG. 1 is a flowchart of a two-dimensional and three-dimensional cross-validation process provided by an exemplary embodiment of the present disclosure. Fig.13 As shown, the cross-validation process can include the following steps:
[0174] Step 601, the three-dimensional frame is projected to the pixel coordinate system to obtain the two-dimensional projection frame corresponding to the three-dimensional frame, that is, the three-dimensional frame of each motion state in the second motion state sequence is reprojected to the image pixel coordinate system of the corresponding frame.
[0175] In some optional embodiments, the eight vertices of the three-dimensional box may be projected into a pixel coordinate system to obtain a two-dimensional projection box.
[0176] In some optional embodiments, a set of sampling points may be sampled on each side of the three-dimensional frame, and the sampling points on each side may be projected to the image pixel coordinate system of the corresponding frame to obtain the projection points corresponding to each sampling point, and the minimum circumscribed rectangle may be determined according to each projection point as the two-dimensional projection frame corresponding to the three-dimensional frame. Compared with the above-mentioned eight-vertex projection, this projection method may avoid or reduce the influence of distortion and truncation, and improve the accuracy of the two-dimensional projection frame. The specific projection method of the three-dimensional frame to the pixel coordinate system is not limited.
[0177] Step 602: Greedy IoU matching to obtain the matching result between the two-dimensional projection frame and the two-dimensional detection frame.
[0178] Specifically, the IoU greedy matching process may include: according to the object type of the dynamic object, the depth of the dynamic object in the camera coordinate system, and the coordinates of the dynamic object in the pixel coordinate system, the two-dimensional projection frame in the pixel coordinate system is customized and scaled. For example, different dynamic objects and different depths are set to different scaling values to scale the two-dimensional projection frame. The IoU value (i.e., intersection over union) is calculated for the scaled two-dimensional projection frame and each two-dimensional detection frame on the corresponding frame image, and the two-dimensional detection frame that best matches the scaled two-dimensional projection frame is selected according to the IoU value and the IoU threshold corresponding to the dynamic object type and depth. Multiple groups of parameter values may also be set, each group of parameter values including a scaling value and an IoU threshold. The matching results under each group of parameters are evaluated through a preconfigured evaluation function to obtain the evaluation score of the matching results under each group of parameters, and the matching result with the highest evaluation score is used as the matching result of the two-dimensional projection frame and the two-dimensional detection frame. The matching relationship between the three-dimensional frame and the two-dimensional detection frame corresponding to each two-dimensional projection frame is recorded.
[0179] Step 603, determine whether the two-dimensional projection frame and the two-dimensional detection frame are matched successfully. If the match is successful, go to step 604, if the match fails, go to step 605.
[0180] Step 604: Set the 3D box to be valid, and maintain the matching relationship between the 3D box and the 2D detection box. If the match is successful, it means that the dynamic object is visible on both the visual 2D detection model and the 3D detection model. Setting the 3D box to be valid means that the annotation result corresponding to the 3D box can be used as a positive sample for training the perception model.
[0181] Step 605: Set the 3D box to be invalid. If the matching fails, the 3D box is set to be invalid and can be used as a negative sample and not used for model training.
[0182] During the matching process, there may be some 2D detection frames that do not have matching 2D projection frames, that is, the dynamic object is visible in the 2D detection model but not in the 3D detection model. In this case, it can be considered that the 2D detection frame matching fails and is set as a negative sample, which does not need to be used in the training of the model.
[0183] This embodiment can effectively avoid or reduce the adverse effects of model misdetection on image annotation results through two-dimensional and three-dimensional cross-validation, and further improve the accuracy of image annotation results.
[0184] In some optional embodiments, the embodiments of the present disclosure may be executed by a server or terminal device other than a vehicle. In this case, the number of vehicles used to collect data (including the first image sequence and the vehicle status information corresponding to each frame of the first image sequence) may be one or more, that is, data may be collected by any number of vehicles to obtain one or more first image sequences and the vehicle status information corresponding to each frame of each first image sequence. In the case of collecting data by multiple vehicles, the first image sequence collected by the camera on each vehicle and the vehicle status information corresponding to each frame of the first image sequence may be determined. For each first image sequence, image annotation may be achieved according to the method of any of the above embodiments of the present disclosure.
[0185] In some optional embodiments, when data is collected by multiple vehicles, for each vehicle, one or more first image sequences collected by the camera on the vehicle and the vehicle status information corresponding to each frame image in each first image sequence can be determined. For each first image sequence, image annotation can be achieved according to the method of any of the above embodiments of the present disclosure. For example, the first image sequences collected by the camera on the vehicle at different time periods and the vehicle status information corresponding to each frame image in each first image sequence can be determined. No specific limitation is made.
[0186] In some optional embodiments, when data is collected by multiple vehicles, for each vehicle, the vehicle can collect and store multiple frames of images and vehicle status information corresponding to each frame of image while driving on the road, and send the stored multiple frames of images and vehicle status information corresponding to each frame of image to the server or terminal device in real time or regularly, or after the driving is completed, all collected images (or images of a specified time period) and vehicle status information corresponding to each frame of image are sent to the server or terminal device. Alternatively, multiple frames of images of a specified time period and vehicle status information corresponding to each frame of image are transmitted to the server or terminal device through an intermediate medium such as a USB flash drive. The server or terminal device can determine the first image sequence and the vehicle status information corresponding to each frame of image in the first image sequence based on the obtained multiple frames of images from the vehicle and the vehicle status information corresponding to each frame of image.
[0187] By collecting image data by multiple vehicles, the road scene richness of the image data and the efficiency of image data collection can be greatly improved. Furthermore, the dynamic object labeling results of each image can be obtained by the method of any embodiment of the present disclosure. The images labeled with dynamic objects are used as sample images of the dynamic object prediction model for training the dynamic object prediction model, which can greatly improve the convergence speed and model performance of the model.
[0188] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.
[0189] Any method for marking dynamic objects in an image provided by the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for marking dynamic objects in an image provided by the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for marking dynamic objects in an image mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0190] Exemplary Devices
[0191] Fig.14 FIG. 1 is a schematic diagram of a structure of a device for marking dynamic objects in an image provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, such as Fig.14 The device shown may include: a first processing module 81 , a second processing module 82 , a third processing module 83 and a fourth processing module 84 .
[0192] The first processing module 81 is used to determine the vehicle state information corresponding to each frame of an image in a first image sequence captured by a camera on the vehicle; the first image sequence includes multiple frames of images.
[0193] The second processing module 82 is used to determine the two-dimensional object prediction result and the three-dimensional object prediction result corresponding to each frame image in the first image sequence based on the first image sequence.
[0194] The third processing module 83 is used to determine a first motion state sequence of the dynamic object in the first image sequence based on the three-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0195] The fourth processing module 84 is used to determine the dynamic object labeling result corresponding to each frame image in the first image sequence based on the vehicle state information corresponding to each frame image in the first image sequence, the two-dimensional object prediction result and the first motion state sequence.
[0196] Fig.15 It is a structural schematic diagram of a device for marking dynamic objects in an image provided by another exemplary embodiment of the present disclosure.
[0197] In some optional embodiments, in the above Fig.14 Based on the embodiment shown, Fig.15 As shown, the second processing module 82 may include: a first processing unit 821 and a second processing unit 822 .
[0198] The first processing unit 821 can be used to process each frame of the first image sequence based on a pre-trained two-dimensional detection model to obtain a two-dimensional object prediction result corresponding to each frame of the first image sequence.
[0199] The second processing unit 822 can be used to process each frame of the first image sequence based on the pre-trained three-dimensional detection model to obtain a three-dimensional object prediction result corresponding to each frame of the first image sequence.
[0200] In some optional embodiments, the first processing unit 821 can be specifically used to: perform object detection on each frame image in the first image sequence based on the first two-dimensional detection model to obtain a two-dimensional detection box corresponding to each frame image in the first image sequence; perform object attribute detection on each frame image in the first image sequence based on the second two-dimensional detection model to obtain the object occlusion degree and the object secondary type corresponding to each frame image in the first image sequence; determine the two-dimensional object prediction result corresponding to each frame image in the first image sequence based on the two-dimensional detection box, object occlusion degree and object secondary type corresponding to each frame image in the first image sequence.
[0201] In some optional embodiments, the second processing unit 822 can be specifically used to: based on the first three-dimensional detection model obtained by pre-training, process each frame image in the first image sequence separately to obtain a three-dimensional detection box in the local coordinate system corresponding to each frame image in the first image sequence; based on the three-dimensional detection box corresponding to each frame image in the first image sequence, determine the three-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0202] In some optional embodiments, such as Fig.15 As shown, the third processing module 83 may include: a target tracking unit 831 and a third processing unit 832 .
[0203] The target tracking unit 831 may be used to perform target tracking on the three-dimensional object prediction result corresponding to each frame image in the first image sequence to obtain a target tracking result.
[0204] The third processing unit 832 may be configured to determine a first motion state sequence of the dynamic object in the first image sequence based on the target tracking result.
[0205] In some optional embodiments, the third processing unit 832 can be specifically used to: determine the predicted state sequence corresponding to the object in the first image sequence based on the three-dimensional object prediction results and target tracking results corresponding to each frame image in the first image sequence; determine the dynamic and static types corresponding to the object according to the predicted state sequence corresponding to the object in the first image sequence; determine the first motion state sequence of the dynamic object in the first image sequence according to the dynamic and static types corresponding to the object and the predicted state sequence.
[0206] In some optional embodiments, the second processing unit 822 can be specifically used to: process the first image sequence based on a second three-dimensional detection model obtained in advance, to obtain a three-dimensional detection box and a tracking mark corresponding to each frame image in the first image sequence; the second three-dimensional detection model is a time series model; based on the three-dimensional detection box and the tracking mark corresponding to each frame image in the first image sequence, determine the three-dimensional object prediction result corresponding to each frame image in the first image sequence.
[0207] Fig.16 It is a structural schematic diagram of a device for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0208] In some optional embodiments, such as Fig.16 As shown, the third processing module 83 may include: a first determining unit 83a, a second determining unit 83b and a third determining unit 83c.
[0209] The first determining unit 83a may be configured to determine a predicted state sequence corresponding to the object in the first image sequence based on a three-dimensional detection frame and a tracking marker corresponding to each frame image in the first image sequence.
[0210] The second determining unit 83b may be configured to determine the dynamic or static type corresponding to the object according to the predicted state sequence corresponding to the object in the first image sequence.
[0211] The third determining unit 83c may be configured to determine a first motion state sequence of the dynamic object in the first image sequence according to the dynamic or static type and the predicted state sequence corresponding to the object.
[0212] In some optional embodiments, the third processing module 83 may include the above-mentioned target tracking unit 831, the third processing unit 832, the first determination unit 83a, the second determination unit 83b and the third determination unit 83c, so that the third processing module 83 can support the determination of the first motion state sequence in both the non-temporal first three-dimensional detection model case and the temporal second three-dimensional detection model case.
[0213] Fig.17 It is a structural schematic diagram of a device for marking dynamic objects in an image provided by yet another exemplary embodiment of the present disclosure.
[0214] In some optional embodiments, based on any of the above embodiments, Fig.17 As shown, the fourth processing module 84 may include: a trajectory optimization unit 841 and a third processing unit 842 .
[0215] The trajectory optimization unit 841 may be used to correct the first motion state sequence based on the vehicle state information corresponding to each frame image in the first image sequence to obtain a corrected second motion state sequence.
[0216] The third processing unit 842 may be configured to determine a dynamic object labeling result corresponding to each frame image in the first image sequence based on a two-dimensional object prediction result corresponding to each frame image in the first image sequence and the second motion state sequence.
[0217] In some optional embodiments, the trajectory optimization unit 841 can be specifically used to: based on the vehicle state information corresponding to each frame in the first image sequence, convert each motion state in the first motion state sequence to the local coordinate system corresponding to the vehicle in the target frame to obtain a third motion state sequence; the target frame is any frame in the multiple frames; the first motion state sequence includes the motion state corresponding to each frame in the first image sequence. Based on the relationship between position, speed, acceleration and preset constraints, the third motion state sequence is corrected to obtain a corrected second motion state sequence.
[0218] In some optional embodiments, the third processing unit 842 can be specifically used to: determine the three-dimensional boxes corresponding to each motion state in the second motion state sequence; project each three-dimensional box to the image pixel coordinate system corresponding to each motion state to obtain a two-dimensional projection box corresponding to each three-dimensional box; match the two-dimensional detection box in the two-dimensional object prediction result corresponding to each frame image in the first image sequence with the two-dimensional projection box corresponding to the frame image to obtain a matching result corresponding to each frame image; and determine the dynamic object labeling result corresponding to each frame image based on the matching result.
[0219] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0220] Exemplary Electronic Devices
[0221] Fig.18 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.
[0222] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.
[0223] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.
[0224] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0225] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (such as cameras, cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to sense the surrounding environment, that is, to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to realize the positioning of movable devices (such as self-vehicles, robots, etc.) where electronic devices are located. Positioning sensors may include, for example, inertial measurement units (Inertial Measurement Unit, referred to as: IMU), global positioning systems (Global Positioning System, referred to as: GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.
[0226] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.
[0227] Of course, to simplify, Fig.18Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.
[0228] Exemplary computer program products and computer-readable storage media
[0229] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the methods of various embodiments of the present disclosure described in the above “Exemplary Methods” section.
[0230] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0231] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.
[0232] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0233] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0234] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A method for marking dynamic objects in an image, comprising: Determine vehicle status information corresponding to each frame of an image in a first image sequence captured by a camera on the vehicle; The first image sequence includes a plurality of frames of images; Based on the first image sequence, determining a two-dimensional object prediction result and a three-dimensional object prediction result corresponding to each frame image in the first image sequence; Determining a first motion state sequence of a dynamic object in the first image sequence based on the three-dimensional object prediction result corresponding to each frame image in the first image sequence; Based on the vehicle state information corresponding to each frame of the first image sequence, the two-dimensional object prediction result and the first motion state sequence, a dynamic object labeling result corresponding to each frame of the first image sequence is determined.
2. The method according to claim 1, wherein: The determining, based on the vehicle state information corresponding to each frame of the first image sequence, the two-dimensional object prediction result and the first motion state sequence, a dynamic object labeling result corresponding to each frame of the first image sequence includes: Based on the vehicle state information corresponding to each frame of the first image sequence, correcting the first motion state sequence to obtain a corrected second motion state sequence; Based on the two-dimensional object prediction result corresponding to each frame image in the first image sequence and the second motion state sequence, a dynamic object labeling result corresponding to each frame image in the first image sequence is determined.
3. The method according to claim 2, wherein: The correcting the first motion state sequence based on the vehicle state information corresponding to each frame image in the first image sequence to obtain a corrected second motion state sequence includes: Based on the vehicle state information corresponding to each frame in the first image sequence, each motion state in the first motion state sequence is converted to a local coordinate system corresponding to a target frame of the vehicle to obtain a third motion state sequence; the target frame is any frame in a plurality of frames; the first motion state sequence includes the motion state corresponding to each frame in the first image sequence; Based on the relationship among position, velocity, and acceleration and preset constraints, the third motion state sequence is corrected to obtain a corrected second motion state sequence.
4. The method according to claim 2, wherein: The determining, based on the two-dimensional object prediction result corresponding to each frame image in the first image sequence and the second motion state sequence, a dynamic object labeling result corresponding to each frame image in the first image sequence includes: Determine a three-dimensional frame corresponding to each motion state in the second motion state sequence; Projecting each of the three-dimensional frames to the image pixel coordinate system corresponding to each motion state to obtain a two-dimensional projection frame corresponding to each of the three-dimensional frames; Matching the two-dimensional detection frame in the two-dimensional object prediction result corresponding to each frame image in the first image sequence with the two-dimensional projection frame corresponding to the frame image to obtain a matching result corresponding to each frame image; According to the matching result, the dynamic object labeling result corresponding to each frame of image is determined.
5. The method according to claim 1, wherein: The determining, based on the first image sequence, a two-dimensional object prediction result and a three-dimensional object prediction result corresponding to each frame image in the first image sequence includes: Based on the pre-trained two-dimensional detection model, each frame of the first image sequence is processed to obtain the two-dimensional object prediction result corresponding to each frame of the first image sequence; Based on the pre-trained three-dimensional detection model, each frame of the first image sequence is processed to obtain the three-dimensional object prediction result corresponding to each frame of the first image sequence.
6. The method according to claim 5, wherein: The step of processing each frame of the first image sequence based on the pre-trained two-dimensional detection model to obtain the two-dimensional object prediction result corresponding to each frame of the first image sequence includes: Performing object detection on each frame of the first image sequence based on a first two-dimensional detection model to obtain a two-dimensional detection frame corresponding to each frame of the first image sequence; Performing object attribute detection on each frame of the first image sequence based on the second two-dimensional detection model to obtain an object occlusion degree and an object secondary type corresponding to each frame of the first image sequence; Based on the two-dimensional detection frame, object occlusion degree and object secondary type corresponding to each frame image in the first image sequence, the two-dimensional object prediction result corresponding to each frame image in the first image sequence is determined.
7. The method according to claim 5, wherein: The method of processing each frame of the first image sequence based on the pre-trained three-dimensional detection model to obtain the three-dimensional object prediction result corresponding to each frame of the first image sequence includes: Based on the first 3D detection model obtained by pre-training, each frame of the first image sequence is processed respectively to obtain a 3D detection frame in a local coordinate system corresponding to each frame of the first image sequence; Based on the three-dimensional detection box corresponding to each frame image in the first image sequence, the three-dimensional object prediction result corresponding to each frame image in the first image sequence is determined.
8. The method according to claim 5, wherein: The method of processing each frame of the first image sequence based on the pre-trained three-dimensional detection model to obtain the three-dimensional object prediction result corresponding to each frame of the first image sequence includes: Based on a pre-trained second 3D detection model, the first image sequence is processed to obtain a 3D detection frame and a tracking mark corresponding to each frame of the first image sequence; the second 3D detection model is a time series model; Based on the three-dimensional detection box and the tracking mark corresponding to each frame image in the first image sequence, the three-dimensional object prediction result corresponding to each frame image in the first image sequence is determined.
9. A device for marking dynamic objects in an image, comprising: A first processing module, used to determine vehicle state information corresponding to each frame of an image in a first image sequence captured by a camera on the vehicle; The first image sequence includes a plurality of frames of images; A second processing module, configured to determine, based on the first image sequence, a two-dimensional object prediction result and a three-dimensional object prediction result corresponding to each frame of the first image sequence; A third processing module, configured to determine a first motion state sequence of the dynamic object in the first image sequence based on the three-dimensional object prediction result corresponding to each frame image in the first image sequence; The fourth processing module is used to determine the dynamic object labeling result corresponding to each frame image in the first image sequence based on the vehicle state information corresponding to each frame image in the first image sequence, the two-dimensional object prediction result and the first motion state sequence.
10. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 8.
11. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.
Citation Information
Patent Citations
Image-based vehicle position and attitude determination system and corresponding method
CN110060298A
Vehicle-mounted visual real-time multi-target multi-task joint sensing method and device
CN111310574A
Image processing method, device and equipment and computer readable storage medium
CN114723955A
Object labeling method, object labeling device and computer readable storage medium
CN117197775A
Vehicle positioning method, device, equipment and medium
CN117351079A
