A data processing method, apparatus, electronic device, and storage medium

Abstract

The present invention discloses a data processing method, apparatus, electronic device and storage medium. The method includes traversing each frame image of an object motion video, obtaining first current pose information of a detected object in the current frame image and first object point cloud data of a non-detected object in the current frame image; obtaining historical pose information of the non-detected object in the previous frame image and second object point cloud data from current object trajectory data; performing matching based on the first object point cloud data and the second object point cloud data, and obtaining second current pose information of the non-detected object based on the historical pose information and the matching result; and updating the current object trajectory data based on the first current pose information and the second current pose information. By using the disclosed embodiments of the present invention, the problem that the trajectory of a non-detected object is difficult to track due to loss can be effectively solved, and the comprehensiveness, accuracy and robustness of object trajectory tracking can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving, and particularly relates to a data processing method, apparatus, electronic device, and storage medium. Background Art

[0002] An autonomous vehicle uses on-vehicle sensors to sense the surrounding environment of the vehicle, and controls the steering and speed of the vehicle according to the road, vehicle position, and obstacle information obtained by the sensing, so that the vehicle can drive safely and reliably on the road. Autonomous vehicles require a certain recognition ability for objects beyond a certain distance to ensure sufficient time for emergency braking and other safety measures. Therefore, the recognition and tracking of dynamic objects are key issues in autonomous driving technology.

[0003] In the prior art, object recognition and tracking mainly detect objects based on a deep learning model, and then track the detected objects through tracking technologies such as Hungarian matching and Kalman filtering to form trajectories. However, deep learning models often have limitations. For object types that appear less frequently in the training samples, there is a phenomenon of missed detection. Tracking object trajectories only based on the detection results of the deep learning model will inevitably result in problems of trajectory loss. Summary of the Invention

[0004] In view of the above problems in the prior art, the present invention discloses a data processing method, apparatus, electronic device, and storage medium, which can effectively solve the problem of difficult tracking of the trajectories of undetected objects, and improve the comprehensiveness, accuracy, and robustness of object trajectory tracking. The technical solutions disclosed by the present invention are as follows:

[0005] According to one aspect of the embodiments disclosed by the present invention, a data processing method is provided, including:

[0006] Obtain each frame of image in the object motion video;

[0007] When any frame of image is traversed, obtain the first current pose information of the detected object in the current frame of image and the first object point cloud data of the undetected object in the current frame of image;

[0008] Obtain the historical pose information of the undetected object in the previous frame of image of the current frame of image corresponding to the object motion video and the second object point cloud data of the undetected object in the previous frame of image;

[0009] Match the first object point cloud data of the undetected object with the second object point cloud data of the undetected object to obtain the target offset of the undetected object;

[0010] Based on the historical pose information of the non-detection object and the target offset, obtain the second current pose information of the non-detection object in the current frame image;

[0011] Based on the first current pose information of the detection object and the second current pose information of the non-detection object, update the current object trajectory data;

[0012] When the traversal ends, use the current object trajectory data as the target object trajectory data.

[0013] Optionally, the obtaining the target offset of the non-detection object by matching the first object point cloud data of the non-detection object and the second object point cloud data of the non-detection object includes:

[0014] Obtain a preset offset;

[0015] Based on the second object point cloud data and the preset offset, determine the predicted point cloud data of the non-detection object in the current frame image;

[0016] Generate distance loss information based on the predicted point cloud data and the first object point cloud data;

[0017] When the distance loss information meets the preset conditions, obtain the target offset based on the distance loss information.

[0018] Optionally, the obtaining the first current pose information of the detection object in the current frame image and the first object point cloud data of the non-detection object in the current frame image includes:

[0019] Obtain the current point cloud data of the current frame image;

[0020] Perform object detection on the current point cloud data to determine the first object position information and the first current pose information of the detection object in the current frame image;

[0021] Perform clustering processing on the current point cloud data to obtain the second object position information of at least one clustering object in the current frame image;

[0022] Based on the first object position information and the second object position information, associate the detection object and the at least one clustering object to obtain object association information;

[0023] According to the object association information, determine the non-detection object from the at least one clustering object;

[0024] Based on the second object position information, obtain the first object point cloud data from the current frame image.

[0025] Optionally, the method further includes:

[0026] Obtain first historical position information of the detection object in the previous frame image and second historical position information of the non-detection object in the previous frame image;

[0027] Based on the first historical position information, make a prediction to obtain first predicted position information of the detection object in the current frame image;

[0028] Based on the second historical position information, make a prediction to obtain second predicted position information of the non-detection object in the current frame image;

[0029] Correspondingly, the updating of the current object trajectory data based on the first current pose information of the detection object and the second current pose information of the non-detection object includes:

[0030] When the first predicted position information matches the first object position information and the second predicted position information matches the second object position information, update the current object trajectory data based on the first current pose information and the second current pose information.

[0031] Optionally, the method further includes:

[0032] When the first predicted position information does not match the first object position information and the second predicted position information matches the second object position information, create new trajectory data of the detection object, and update the current object trajectory data based on the second current pose information;

[0033] Add the newly created object trajectory data to the current object trajectory data.

[0034] Optionally, the method further includes:

[0035] When the first predicted position information matches the first object position information and the second predicted position information does not match the second object position information, create new object trajectory data of the non-detection object, and update the current object trajectory data based on the first current pose information;

[0036] Add the newly created object trajectory data to the current object trajectory data.

[0037] Optionally, before matching the first object point cloud data of the non-detection object and the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object, the method further includes:

[0038] Perform key point detection on the first object point cloud data to obtain the first object key point cloud data;

[0039] Perform key point detection on the second object point cloud data to obtain the second object key point cloud data;

[0040] Correspondingly, the matching of the first object point cloud data of the non-detection object and the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object includes:

[0041] Match the first object key point cloud data and the second object key point cloud data to obtain the target offset.

[0042] According to another aspect of the disclosed embodiments of the present invention, there is provided a data processing device, including:

[0043] An image acquisition module, configured to acquire each frame of image in the object motion video;

[0044] A first information acquisition module, configured to, when traversing to any frame of image, acquire the first current pose information of the detection object in the current frame of image and the first object point cloud data of the non-detection object in the current frame of image;

[0045] A second information acquisition module, configured to acquire the historical pose information of the non-detection object in the previous frame of image of the current frame of image corresponding to the object motion video and the second object point cloud data of the non-detection object in the previous frame of image from the current object trajectory data corresponding to the object motion video;

[0046] A target offset determination module, configured to match the first object point cloud data of the non-detection object and the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object;

[0047] A second current pose information determination module, configured to obtain the second current pose information of the non-detection object in the current frame of image based on the historical pose information of the non-detection object and the target offset;

[0048] A first trajectory data update module, configured to update the current object trajectory data based on the first current pose information of the detection object and the second current pose information of the non-detection object;

[0049] A target object trajectory data determination module, configured to, when the traversal ends, use the current object trajectory data as the target object trajectory data.

[0050] According to another aspect of the disclosed embodiments of the present invention, there is provided an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to execute the instructions to implement the data processing method as described in any one of the above.

[0051] According to another aspect of the disclosed embodiments of the present invention, there is provided a computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the data processing method as described in any one of the disclosed embodiments of the present invention.

[0052] According to another aspect of the disclosed embodiments of the present invention, there is provided a computer program product containing instructions, when it runs on a computer, enabling the computer to execute the data processing method as described in any one of the disclosed embodiments of the present invention.

[0053] The technical solutions provided by the disclosed embodiments of the present invention at least bring the following beneficial effects:

[0054] When the data processing method provided by the present invention traverses any frame image in the object motion video, it matches the two object point cloud data of the non-detected object in the current frame image and the previous frame image respectively to determine the target offset, and obtains the current pose information of the non-detected object based on its historical pose information and the target offset. Furthermore, it updates the object trajectory data based on the obtained current pose information of the detected object and the current pose information of the non-detected object, which can effectively solve the problem that the trajectory of the non-detected object is difficult to track due to loss, and improve the comprehensiveness, accuracy and robustness of object trajectory tracking.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0056] The drawings here are incorporated into the specification and constitute a part of this specification, showing the embodiments that conform to the disclosed embodiments of the present invention, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation to the disclosed embodiments of the present invention.

[0057] Figure 1 is a flowchart of a data processing method shown according to an exemplary embodiment;

[0058] Figure 2 is a flowchart of a method for obtaining first object point cloud data shown according to an exemplary embodiment;

[0059] Figure 3 is a block diagram of a data processing device shown according to an exemplary embodiment;

[0060] Figure 4A block diagram of a terminal electronic device for data processing shown according to an exemplary embodiment;

[0061] Figure 5 A block diagram of a server electronic device for data processing shown according to an exemplary embodiment. Detailed implementation manners

[0062] In order to enable those of ordinary skill in the art to better understand the technical solutions disclosed in the present invention, the technical solutions in the disclosed embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention disclosed herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0064] A data processing method provided by the present invention, which can be applied to the generation and update of object trajectory data.

[0065] Figure 1 A flowchart of a data processing method shown according to an exemplary embodiment, as Figure 1 shown, the data processing method includes the following steps.

[0066] S101: Obtain each frame of image in the object motion video.

[0067] In a specific embodiment, the object motion video may be a set of several frames of images in at least one object motion arranged in chronological order. Specifically, the object may include a vehicle, a pedestrian, etc. Each frame of image may be a three-dimensional point cloud image collected by a lidar. Each frame of image may include the trajectory data of any object in the frame of image and the three-dimensional point cloud data corresponding to any object; the point cloud data may include point cloud position information. Specifically, the point cloud position information may be the three-dimensional coordinate information of the point cloud in the earth coordinate system.

[0068] S103: In the case of traversing any frame of image, obtain the first current pose information of the detection object in the current frame of image and the first object point cloud data of the non-detection object in the current frame of image.

[0069] In a specific embodiment, each frame of image is traversed based on the temporal order of the object motion video.

[0070] In an alternative embodiment, Figure 2 is a flowchart of a method for obtaining first object point cloud data shown according to an exemplary embodiment, as Figure 2 shown, the obtaining of the first current pose information of the detection object in the current frame of image and the first object point cloud data of the non-detection object in the current frame of image includes:

[0071] S201: Obtain the current point cloud data of the current frame of image.

[0072] S203: Perform object detection on the current point cloud data to determine the first object position information and the first current pose information of the detection object in the current frame of image.

[0073] In a specific embodiment, the detection object may be an object determined by performing object detection on point cloud data through a target detection model. Specifically, the above target detection model may be obtained by pre-training a preset deep learning model based on training data; perform object detection on the point cloud data in combination with the target detection model. Specifically, the above training data may include the point cloud data of each frame of image in the sample video, the sample object position information and the sample pose information of each frame of image in the sample video.

[0074] In a specific embodiment, the output information of the target detection model may include the pose information, metric information, category information, detection confidence, etc. of the object (detection object) in the current point cloud data. Specifically, the pose information may include position information and orientation information. The position information may be the three-dimensional coordinate information of the detection object in the earth coordinate system. The metric information may include the length information, width information and height information of the detection object. The category information may indicate the category corresponding to the above detection object. Specifically, the category may include vehicles, pedestrians, etc. The detection confidence may indicate the probability that the above detection object is included in the current frame of image.

[0075] S205: Perform clustering processing on the current point cloud data to obtain the second object position information of at least one clustering object in the current frame of image.

[0076] In a specific embodiment, the clustering object can be an object determined by clustering point cloud data through a clustering method. The output information of the clustering process can include the position information of the point cloud data corresponding to the object (clustering object) in the current point cloud data and the convex hull of the point cloud data. Specifically, the clustering method can include clustering methods based on voxel filtering, Range Image (depth image), etc. The convex hull can be the smallest circumscribed polygon containing the point cloud data; the position information of the clustering object can be the position information of the point cloud data corresponding to the clustering object.

[0077] S207: Based on the first object position information and the second object position information, associate the detected object and the at least one clustering object to obtain object association information.

[0078] In a specific embodiment, the object association information can indicate whether the detected object is associated with the clustering object. The association between the detected object and the clustering object can mean that the detected object and the clustering object are the same object. Specifically, determine the two-dimensional projection of the detected object based on the first object position information and the measurement information of the detected object, and determine the convex hull of the point cloud data corresponding to the clustering object based on the second object position information; in the case where there is an overlapping area between the two-dimensional projection of the detected object and the convex hull of the point cloud data of the clustering object, the detected object and the clustering object are the same object, otherwise the detected object and the clustering object are not the same object.

[0079] S209: According to the object association information, determine the non-detected object from the at least one clustering object.

[0080] In a specific embodiment, the non-detected object can be a clustering object that is not associated with any detected object.

[0081] S211: Based on the second object position information, obtain the first object point cloud data from the current frame image.

[0082] In a specific embodiment, use the point cloud data in the corresponding area of the second object position information in the current frame image as the first object point cloud data.

[0083] In the above embodiment, by associating the detected object and the clustering object, and then determining the non-detected object from the clustering object, it is possible to effectively identify the objects not detected by the detection model, solve the problem that the trajectory of the non-detected object is difficult to track due to loss, and thus improve the comprehensiveness of object trajectory tracking.

[0084] S105: Obtain the historical pose information of the non-detected object in the previous frame image of the current frame image corresponding to the object motion video and the second object point cloud data of the non-detected object in the previous frame image from the current object trajectory data corresponding to the object motion video.

[0085] In a specific embodiment, the current object trajectory data may carry the second object point cloud data of the non-detected object in the previous frame image.

[0086] S107: Match the first object point cloud data of the non-detected object with the second object point cloud data of the non-detected object to obtain the target offset of the non-detected object.

[0087] In an alternative embodiment, the matching of the first object point cloud data with the second object point cloud data to obtain the target offset may include:

[0088] Obtain a preset offset;

[0089] Based on the second object point cloud data and the preset offset, determine the predicted point cloud data of the non-detected object in the current frame image;

[0090] Generate distance loss information based on the predicted point cloud data and the first object point cloud data;

[0091] When the distance loss information meets a preset condition, obtain the target offset based on the distance loss information.

[0092] In a specific embodiment, the preset offset may characterize the offset degree of the first object point cloud data and the second object point cloud data in the horizontal direction. The preset offset may be the horizontal offset between two adjacent frames of point clouds. Specifically, the offset may include the lateral offset between two adjacent frames of point clouds in the x-axis direction of the earth coordinate system, the longitudinal offset between two adjacent frames of point clouds in the y-axis direction of the earth coordinate system, and the rotation angle between two adjacent frames of point clouds. The initial value of the preset offset may be the offset between the above historical pose information and the predicted pose information of the non-detected object in the current frame image.

[0093] In a specific embodiment, the predicted pose information may be obtained by performing prediction processing on the above historical pose information. Specifically, the above predicted pose information may be obtained through the following prediction formula:

[0094] x′(k) = A * x(k - 1) + B * u(k)

[0095] P′(k) = A * P(k - 1) * A T + Q

[0096] Among them, x′(k) represents the predicted state information of any object at time k, x(k - 1) represents the historical state information of any object at time k - 1, P′(k) represents the predicted state covariance matrix at time k, P(k - 1) represents the historical state covariance matrix at time k - 1, u(k) represents the fluctuation at time k, B represents the influence degree of the fluctuation on the prediction, A represents the state transition matrix, and Q represents the process noise matrix.

[0097] In a specific embodiment, the predicted state information may include predicted pose information, predicted speed information, and predicted metric information. Correspondingly, the historical state information may include historical pose information, historical speed information, and historical metric information. Specifically, the predicted metric information of any object may retain the historical metric information of the object, such as the length information, width information, height information, etc. of the object.

[0098] In a specific embodiment, the coordinates of any predicted point in the predicted point cloud data can be expressed as

[0099] (x pre , y pre ) = (x′ + Δx, y′ + Δy),

[0100] Among them, x′ = (x1 - x o ) * cos(Δθ) - (y1 - y o ) * sin(Δθ), y′ = (x1 - x o ) * sin(Δθ) + (y1 - y0) * cos(Δθ).

[0101] Among them, Δx represents the offset between two adjacent frames of point clouds in the x-axis direction of the earth coordinate system, Δy represents the offset between two adjacent frames of point clouds in the y-axis direction of the earth coordinate system, Δθ represents the rotation angle between two adjacent frames of point clouds, (x0, y0) represents the center point coordinates of the second object's point cloud data, and (x1, y1) represents the coordinates of any point in the second object's point cloud data.

[0102] In a specific embodiment, the distance loss information may be the shortest distance between the first object's point cloud data and the predicted point cloud data. Specifically, the distance loss information can be expressed as

[0103]

[0104] Among them, lost represents the distance loss information, and (x2, y2) represents the coordinates of any point in the first object's point cloud data.

[0105] In a specific embodiment, when the distance loss information meets a preset condition, obtaining the target offset based on the distance loss information may include: adjusting a preset offset based on the preset offset and the distance loss information corresponding to the preset offset; taking the adjusted preset offset as a new preset offset, repeating the above steps until the distance loss information meets the preset condition, and taking the preset offset corresponding to the distance loss information as the target offset. The above preset condition can be set according to the actual application situation.

[0106] In a specific embodiment, the distance loss information can be designed as a residual module. When the point cloud data has excessive noise, other residual modules can also be added as a regularization term.

[0107] In the above embodiment, the offset obtained by matching the point cloud data of two object points of the non-detection object in the current frame image and the previous frame image respectively, and then combining the historical pose information of the non-detection object to determine its current pose information can ensure the accuracy of the current pose information of the non-detection object, thereby improving the accuracy of the trajectory tracking of the non-detection object.

[0108] In an alternative embodiment, before matching the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object, the above method may further include:

[0109] Performing key point detection on the first object point cloud data to obtain first object key point cloud data;

[0110] Performing key point detection on the second object point cloud data to obtain second object key point cloud data.

[0111] Correspondingly, the matching of the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object may further include:

[0112] Matching the first object key point cloud data with the second object key point cloud data to obtain the target offset.

[0113] In a specific embodiment, the first object key point cloud data may be the key point cloud data in the first object point cloud data. Specifically, the first object key point cloud data may be the point cloud data on the boundary curve near the lidar center in the first object point cloud data; correspondingly, the second object key point cloud data may be the point cloud data on the boundary curve near the lidar center in the second object point cloud data.

[0114] In the above embodiments, based on the key point cloud data of the first object and the key point cloud data of the second object, fast matching of the point cloud data of the two objects can be achieved, thereby improving the efficiency of object trajectory data tracking.

[0115] S109: Based on the historical pose information of the non-detected object and the target offset, obtain the second current pose information of the non-detected object in the current frame image.

[0116] In a specific embodiment, based on the target offset, perform a pose transformation on the historical pose information to obtain the second current pose information.

[0117] In a specific embodiment, the target offset may include a target position offset and a target rotation angle. Specifically, the target position offset may include a lateral target offset in the x-axis direction of the earth coordinate system and a longitudinal target offset in the y-axis direction of the earth coordinate system. The historical pose information may include historical position information and historical orientation information. Specifically, the historical position information may be the historical coordinate information of the non-detected object in the earth coordinate system.

[0118] In a specific embodiment, performing a pose transformation on the historical pose information based on the target offset to obtain the second current pose information may include: performing a position transformation on the historical coordinate information of the non-detected object based on the above-mentioned lateral target offset and longitudinal target offset to obtain the current position information; performing a rotation transformation on the historical orientation information based on the above-mentioned target rotation angle to obtain the current orientation information; and using the current position information and the current orientation information as the second current pose information.

[0119] In the above embodiments, based on the matching of the point cloud data of the non-detected object in the current frame image and the previous frame image, determine the target offset, and obtain the current pose information of the non-detected object based on its historical pose information and the target offset. Furthermore, based on the current pose information of the detected object and the current pose information of the non-detected object obtained, the object trajectory data can be updated, effectively solving the problem that the trajectory of the non-detected object is lost and difficult to track, and improving the comprehensiveness, accuracy, and robustness of object trajectory tracking.

[0120] S111: Based on the first current pose information of the detected object and the second current pose information of the non-detected object, update the current object trajectory data.

[0121] In a specific embodiment, the updating of the current object trajectory data based on the first current pose information and the second current pose information may include: using the first current pose information and the second current pose information as new observations, and updating the Kalman filter based on the new observations.

[0122] In a specific embodiment, the trajectory update formula can be:

[0123] x(k) = x′(k) + K(k) * (z(k) - H * x′(k))

[0124] P(k) = (I - K(k) * H) * P′(k)

[0125] K(k) = P′(k) * H T * (H * P′(k) * H T + R) -1

[0126] Wherein, x(k) represents the current state information of any object at time k, x′(k) represents the predicted state information of any object at time k, K(k) represents the Kalman gain at time k, z(k) represents the observed value at time k, P(k) represents the current state covariance matrix at time k, P′(k) represents the predicted state covariance matrix at time k, H represents the observation matrix, R represents the observation noise matrix, and I represents the identity matrix.

[0127] In an alternative embodiment, the above method may further include:

[0128] Obtain the first historical position information of the detected object in the previous frame image and the second historical position information of the non-detected object in the previous frame image;

[0129] Perform prediction based on the first historical position information to obtain the first predicted position information of the detected object in the current frame image;

[0130] Perform prediction based on the second historical position information to obtain the second predicted position information of the non-detected object in the current frame image.

[0131] In a specific embodiment, the predicted position information can be obtained through the historical position information and the above prediction formula.

[0132] Correspondingly, the updating of the current object trajectory data based on the first current pose information of the detected object and the second current pose information of the non-detected object may further include:

[0133] When the first predicted position information matches the first object position information and the second predicted position information matches the second object position information, update the current object trajectory data based on the first current pose information and the second current pose information.

[0134] In a specific embodiment, the matching of the first predicted position information and the first object position information may indicate that there is object trajectory data corresponding to the above-mentioned detected object in the current object trajectory data, and the matching of the second predicted position information and the second object position information may indicate that there is object trajectory data corresponding to the above-mentioned non-detected object in the current object trajectory data; correspondingly, the non-matching of the first predicted position information and the first object position information may indicate that there is no object trajectory data corresponding to the above-mentioned detected object in the current object trajectory data, and the non-matching of the second predicted position information and the second object position information may indicate that there is no object trajectory data corresponding to the above-mentioned non-detected object in the current object trajectory data.

[0135] In a specific embodiment, determining whether the first predicted position information and the first object position information match may include: based on a preset matching algorithm, determining the position matching information between the first predicted position information and the first object position information, where the position matching information indicates whether the first predicted position information and the first object position information match; if the position matching information meets the matching conditions corresponding to the preset matching algorithm, then the first predicted position information and the first object position information match, otherwise they do not match. Specifically, the preset matching algorithm may include the IOU matching algorithm, the KM (Kuhn-Munkras) matching algorithm, etc.

[0136] Correspondingly, for determining whether the second predicted position information and the second object position information match, the above-mentioned detailed steps can be referred to and will not be elaborated here.

[0137] In an alternative embodiment, the above method may further include:

[0138] In the case where the first predicted position information does not match the first object position information and the second predicted position information matches the second object position information, create new trajectory data for the detected object, and update the current object trajectory data based on the second current pose information;

[0139] Add the newly created object trajectory data to the current object trajectory data.

[0140] In an alternative embodiment, the above method may further include:

[0141] In the case where the first predicted position information matches the first object position information and the second predicted position information does not match the second object position information, create new object trajectory data for the non-detected object, and update the current object trajectory data based on the first current pose information;

[0142] Add the newly created object trajectory data to the current object trajectory data.

[0143] In an optional embodiment, the above method may further include:

[0144] In the case where the first predicted position information does not match the first object position information and the second predicted position information does not match the second object position information, create new object trajectory data for the detected object and the non-detected object;

[0145] Add the newly created object trajectory data to the current object trajectory data.

[0146] In the above embodiment, based on whether the predicted position information of any object matches the current object position information, it is determined whether there is object trajectory data corresponding to the object in the current object trajectory data. In the case where there is object trajectory data corresponding to the object in the current object trajectory data, update the corresponding object trajectory data based on the current pose information of the object. In the case where there is no object trajectory data corresponding to the object in the current object trajectory data, create new object trajectory data for the object and perform trajectory update later, improving the comprehensiveness and accuracy of object trajectory tracking.

[0147] In an optional embodiment, the above method may further include:

[0148] In the case where within a preset number of frames, the first predicted position information does not match the first object position information, or the second predicted position information does not match the second object position information, perform a deletion process on the current object trajectory data of the detected object or the non-detected object.

[0149] In a specific embodiment, the preset number of frames can be set according to the actual application situation. For example, the preset number of frames can be 3 frames.

[0150] As can be seen from the technical solutions provided in the embodiments of this specification above, in this specification, when traversing any frame image in the object motion video, based on the two object point cloud data of the non-detection object in the current frame image and the previous frame image respectively, the target offset is determined, and the current pose information of the non-detection object is obtained based on its historical pose information and the target offset, ensuring the accuracy of the current pose information of the non-detection object. Furthermore, based on the obtained current pose information of the detection object and the current pose information of the non-detection object, the object trajectory data is updated, effectively solving the problem that the trajectory of the non-detection object is lost and difficult to track, and improving the comprehensiveness, accuracy, and robustness of object trajectory tracking. In addition, by associating the detection object and the clustered objects, and then determining the non-detection object from the clustered objects, objects that are not detected by the detection model can be effectively identified, thereby improving the comprehensiveness of object trajectory tracking. At the same time, based on the first object key point cloud data and the second object key point cloud data, rapid matching of the two object point cloud data can be achieved, thereby improving the efficiency of object trajectory data tracking.

[0151] Figure 3 is a block diagram of a data processing device shown according to an exemplary embodiment. Referring to Figure 3 , the device includes:

[0152] An image acquisition module 310, configured to acquire each frame image in the object motion video;

[0153] A first information acquisition module 320, configured to, when traversing any frame image, acquire the first current pose information of the detection object in the current frame image and the first object point cloud data of the non-detection object in the current frame image;

[0154] A second information acquisition module 330, configured to acquire the historical pose information of the non-detection object in the previous frame image of the current frame image and the second object point cloud data of the non-detection object in the previous frame image from the current object trajectory data corresponding to the object motion video;

[0155] A target offset determination module 340, configured to match the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object to obtain the target offset of the non-detection object;

[0156] A second current pose information determination module 350, configured to obtain the second current pose information of the non-detection object in the current frame image based on the historical pose information of the non-detection object and the target offset;

[0157] The first trajectory data update module 360 is used to update the current object trajectory data based on the first current pose information of the detected object and the second current pose information of the undetected object;

[0158] The target object trajectory data determination module 370 is used to, when the traversal ends, use the current object trajectory data as the target object trajectory data.

[0159] Optionally, the target offset determination module 340 may include:

[0160] The preset offset acquisition unit is used to acquire a preset offset;

[0161] The predicted point cloud data determination unit is used to determine the predicted point cloud data of the undetected object in the current frame image based on the second object point cloud data and the preset offset;

[0162] The distance loss information generation unit is used to generate distance loss information based on the predicted point cloud data and the first object point cloud data;

[0163] The target offset determination unit is used to, when the distance loss information meets a preset condition, obtain the target offset based on the distance loss information.

[0164] Optionally, the first information acquisition module 320 may include:

[0165] The current point cloud data acquisition unit is used to acquire the current point cloud data of the current frame image;

[0166] The object detection unit is used to perform object detection on the current point cloud data to determine the first object position information and the first current pose information of the detected object in the current frame image;

[0167] The clustering processing unit is used to perform clustering processing on the current point cloud data to obtain the second object position information of at least one clustering object in the current frame image;

[0168] The association unit is used to associate the detected object and the at least one clustering object based on the first object position information and the second object position information to obtain object association information;

[0169] The undetected object determination unit is used to determine the undetected object from the at least one clustering object according to the object association information;

[0170] The first object point cloud data acquisition unit is used to acquire the first object point cloud data from the current frame image based on the second object position information.

[0171] Optionally, the device may further include:

[0172] A historical position information acquisition module, configured to acquire first historical position information of the detection object in the previous frame image and second historical position information of the non-detection object in the previous frame image;

[0173] A first prediction module, configured to perform prediction based on the first historical position information to obtain first predicted position information of the detection object in the current frame image;

[0174] A second prediction module, configured to perform prediction based on the second historical position information to obtain second predicted position information of the non-detection object in the current frame image;

[0175] Correspondingly, the first trajectory data update module 360 includes:

[0176] A first trajectory data update unit, configured to update the current object trajectory data based on the first current pose information and the second current pose information when the first predicted position information matches the first object position information and the second predicted position information matches the second object position information.

[0177] Optionally, the device may further include:

[0178] A second trajectory data update module, configured to create new trajectory data of the detection object and update the current object trajectory data based on the second current pose information when the first predicted position information does not match the first object position information and the second predicted position information matches the second object position information;

[0179] A first trajectory data addition module, configured to add the newly created object trajectory data to the current object trajectory data.

[0180] Optionally, the device may further include:

[0181] A third trajectory data update module, configured to create new object trajectory data of the non-detection object and update the current object trajectory data based on the first current pose information when the first predicted position information matches the first object position information and the second predicted position information does not match the second object position information;

[0182] A second trajectory data addition module, configured to add the newly created object trajectory data to the current object trajectory data.

[0183] Optionally, the device may further include:

[0184] The first key point detection module is used to detect key points from the first object point cloud data to obtain the first object key point cloud data;

[0185] The second key point detection module is used to detect key points from the second object point cloud data to obtain the second object key point cloud data;

[0186] Correspondingly, the target offset determination module 340 includes:

[0187] The target offset determination unit is used to match the first object key point cloud data with the second object key point cloud data to obtain the target offset.

[0188] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0189] Figure 4 is a block diagram of an electronic device for data processing shown according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as Figure 4 shown. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a data processing method. The display screen of the electronic device may be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0190] Figure 5 is a block diagram of an electronic device for data processing shown according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as Figure 5As shown. The electronic device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0191] Those skilled in the art can understand that Figure 4 or Figure 5 the structure shown in is only a block diagram of some structures related to the disclosed solution of the present invention, and does not constitute a limitation on the electronic device to which the disclosed solution of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component arrangement.

[0192] In an exemplary embodiment, there is also provided an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to execute the instructions to implement the data processing method in the disclosed embodiment of the present invention.

[0193] In an exemplary embodiment, there is also provided a computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the data processing method in the disclosed embodiment of the present invention.

[0194] In an exemplary embodiment, there is also provided a computer program product containing instructions, when it runs on a computer, the computer executes the data processing method in the disclosed embodiment of the present invention.

[0195] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. This computer program can be stored in a non-volatile computer-readable storage medium. When this computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0196] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the invention. The present invention is intended to cover any variations, uses, or adaptations of the invention disclosed, which follow the general principles of the invention disclosed and include known common general knowledge or conventional technical means in the technical field not disclosed by the present invention. The specification and embodiments are to be considered illustrative only, and the true scope and spirit of the invention disclosed are pointed out by the following claims.

[0197] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A data processing method, characterized in that, Including: Obtain each frame image in the object motion video; When traversing to any frame image, obtain the first current pose information of the detected object in the current frame image and the first object point cloud data of the undetected object in the current frame image; Obtain the historical pose information of the undetected object in the previous frame image of the current frame image from the current object trajectory data corresponding to the object motion video and the second object point cloud data of the undetected object in the previous frame image; Match the first object point cloud data of the undetected object with the second object point cloud data of the undetected object to obtain the target offset of the undetected object; Based on the historical pose information of the undetected object and the target offset, obtain the second current pose information of the undetected object in the current frame image; Update the current object trajectory data based on the first current pose information of the detected object and the second current pose information of the undetected object; When the traversal ends, use the current object trajectory data as the target object trajectory data.

2. The data processing method according to claim 1, characterized in that The matching the first object point cloud data of the undetected object with the second object point cloud data of the undetected object to obtain the target offset of the undetected object includes: Obtain a preset offset; Based on the second object point cloud data and the preset offset, determine the predicted point cloud data of the undetected object in the current frame image; Generate distance loss information based on the predicted point cloud data and the first object point cloud data; When the distance loss information meets the preset conditions, obtain the target offset based on the distance loss information.

3. The data processing method according to claim 1, wherein The obtaining the first current pose information of the detected object in the current frame image and the first object point cloud data of the undetected object in the current frame image includes: Obtain the current point cloud data of the current frame image; Perform object detection on the current point cloud data to determine the first object position information and the first current pose information of the detected object in the current frame image; Perform clustering processing on the current point cloud data to obtain the second object position information of at least one clustering object in the current frame image; Based on the first object position information and the second object position information, associate the detected object and the at least one clustering object to obtain object association information; According to the object association information, determine the undetected object from the at least one clustering object; Based on the second object position information, obtain the first object point cloud data from the current frame image.

4. A data processing method according to claim 1, characterized in that, The method further includes: Obtain the first historical position information of the detected object in the previous frame image and the second historical position information of the undetected object in the previous frame image; Perform prediction based on the first historical position information to obtain the first predicted position information of the detected object in the current frame image; Perform prediction based on the second historical position information to obtain the second predicted position information of the undetected object in the current frame image; Updating the current object trajectory data based on the first current pose information of the detection object and the second current pose information of the non-detection object includes: When the first predicted position information matches the first object position information and the second predicted position information matches the second object position information, updating the current object trajectory data based on the first current pose information and the second current pose information.

5. A data processing method according to claim 4, characterized in that, The method further includes: When the first predicted position information does not match the first object position information and the second predicted position information matches the second object position information, creating new trajectory data for the detection object and updating the current object trajectory data based on the second current pose information; Adding the newly created object trajectory data to the current object trajectory data.

6. The data processing method according to claim 4, wherein The method further includes: When the first predicted position information matches the first object position information and the second predicted position information does not match the second object position information, creating new object trajectory data for the non-detection object and updating the current object trajectory data based on the first current pose information; Adding the newly created object trajectory data to the current object trajectory data.

7. A data processing method according to any one of claims 1-6, characterized in that Before obtaining the target offset of the non-detection object by matching the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object, the method further includes: Performing key point detection on the first object point cloud data to obtain first object key point cloud data; Performing key point detection on the second object point cloud data to obtain second object key point cloud data; The obtaining of the target offset of the non-detection object by matching the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object includes: Obtaining the target offset by matching the first object key point cloud data with the second object key point cloud data.

8. A data processing device, characterized in that, Includes: An image acquisition module for acquiring each frame of image in the object motion video; A first information acquisition module for, when traversing to any frame of image, acquiring the first current pose information of the detection object in the current frame of image and the first object point cloud data of the non-detection object in the current frame of image; A second information acquisition module for acquiring the historical pose information of the non-detection object in the previous frame of the current frame of image corresponding to the object motion video and the second object point cloud data of the non-detection object in the previous frame of image from the current object trajectory data; A target offset determination module for obtaining the target offset of the non-detection object by matching the first object point cloud data of the non-detection object with the second object point cloud data of the non-detection object; A second current pose information determination module for obtaining the second current pose information of the non-detection object in the current frame of image based on the historical pose information of the non-detection object and the target offset; The first trajectory data update module is configured to update the current object trajectory data based on the first current pose information of the detection object and the second current pose information of the non-detection object; The target object trajectory data determination module is configured to use the current object trajectory data as the target object trajectory data when the traversal ends.

9. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the data processing method according to any one of claims 1 to 7.

Images