Method, device, mobile device and storage medium for motion or static estimation of an object

Abstract

Embodiments of the present application provide a kind of dynamic and static estimation method of object, equipment, mobile device and storage medium.The method comprises: with camera image as benchmark, time-synchronized laser point cloud is projected into camera image, form depth image, and extract the dense feature point of object perceived in depth image;Predictive positioning information of object in world coordinate system at current time is estimated using the dense feature point of object at last time;The dense feature point of object at last time is tracked by optical flow, and the observed positioning information of object at current time is determined;Dynamic and static estimation is carried out to object based on predictive positioning information and observed positioning information.The embodiments of the present application realize the algorithm of visual and laser fusion and carry out dynamic and static detection in world coordinate system.In the process of dynamic and static detection, it is not necessary to carry out additional labeling and training work, and it is simple and easy, and it has more obvious effect for large target object such as vehicle.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and more particularly to a method, apparatus, mobile device, and storage medium for estimating the dynamic and static states of an object. Background Technology

[0002] In the field of autonomous driving, vehicle detection on roads is one of the most common and important tasks. The main function of vehicle detection is to determine whether there are other objects (e.g., other vehicles on the road) in camera images or radar point clouds. If so, it provides a two-dimensional or three-dimensional bounding box as the vehicle's location information. Existing technologies also provide other information, such as vehicle ID and color, to identify vehicle differences. However, traditional object detection methods are limited to extracting target objects from a single image and do not consider the object's position, pose, or dynamic / static properties.

[0003] In autonomous driving, determining whether objects on the road are static or dynamic is essential information for many subsequent decisions. Here, "static" or "dynamic" refers to a coordinate system relative to the inertial coordinate system, not relative to the autonomous vehicle itself. For example, vehicles parked on either side of the road are moving relative to the vehicle, but are static relative to the world coordinate system; therefore, the autonomous driving algorithm doesn't need to consider lane-changing issues for these vehicles. Conversely, vehicles moving on the road may be stationary relative to the vehicle, but are moving relative to the world coordinate system. Autonomous driving decisions must consider the different behaviors of various vehicles. For instance, at an intersection, when the light is red, a vehicle waits at the red light, thus being stationary; when the light turns green, the vehicle begins to move, thus being dynamic. This necessitates that the static / dynamic determination method must operate in real-time, rather than being calculated retrospectively.

[0004] In general, the dynamic and static information of a vehicle is a time-dependent physical quantity, and it is very difficult to determine its dynamic and static information using a single image or a single laser point cloud. Therefore, sensor data at multiple time points are required to determine whether an object is moving. Depending on the type of sensor, the existing technology includes the following methods for determining dynamic and static information:

[0005] 1. Forcefully use deep learning to reconstruct the position and velocity of objects from a single image or a single point cloud. Intuitively, humans can distinguish between vehicles parked on the side of the road and vehicles driving in the middle of the road, so neural networks should also have this recognition ability. Therefore, dynamic and static information can be directly labeled on vehicle data in images, and then the neural network can be trained to learn the ability to distinguish between dynamic and static states.

[0006] 2. First, detect and cluster the image or point cloud, then compare the positions of similar objects at different times to determine whether each object is moving. This type of method can determine dynamic and static information at the entire object level.

[0007] 3. By matching feature points in point clouds and images, the dynamic and static information of each feature point is obtained. Then, the dynamic and static status of the object itself is determined by the dynamic and static status of the feature points. This type of method can use individual image or point cloud data, or it can fuse the two together.

[0008] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0009] 1. Forcibly using neural networks to regress object position and pose completely ignores information in the time dimension. Although it can achieve some results, it suffers from most of the drawbacks of neural network methods. First, the trained network is highly correlated with the labeled data, achieving good results on well-labeled data, but its general applicability is often poor.

[0010] In practice, changes to the radar type, camera angle, and installation location can all affect the algorithm's performance. Secondly, neural networks currently lack good interpretability, making it difficult to analyze and debug when errors occur, and it's also hard to know what the network has actually learned. Finally, deep learning methods require a large amount of labeled data, and detailed velocity information of objects is difficult for labelers to directly annotate, which also presents certain difficulties in actual operation.

[0011] 2. While the procedures for detection and clustering methods are relatively clear, the conditions for their practical application are quite stringent. To detect whether an object is moving or static, it must be assumed that the same object can be accurately detected at all times and correctly associated with the same object. However, in reality, the accuracy of object detection is not 100%, and the probability of correctly detecting an object across multiple frames decreases multiplicatively.

[0012] On the other hand, when a vehicle is moving at high speed, the position of an object in the image may change significantly, making it more difficult to correctly correlate objects across multiple images. Therefore, this type of method is feasible in low-speed scenes, but encounters some difficulties in high-speed scenes.

[0013] 3. Utilizing the dynamics of feature points in point clouds and images to determine the dynamics of an object is a relatively simple and feasible method. In practical applications, the main difficulty lies in the sparsity of laser point clouds. An object in a laser point cloud is typically represented by several scattered spatial points. Due to the imaging mechanism of lidar, these points themselves exhibit a certain degree of slight fluctuation, and the number of points representing distant vehicles is significantly less than that of nearby vehicles. To determine whether this fluctuation is caused by normal imaging mechanisms or by object motion, careful selection of various thresholds in the algorithm is necessary to obtain satisfactory results. Summary of the Invention

[0014] To address at least the problems in existing technologies, such as neural network methods for regressing object position and pose completely ignoring temporal information, low accuracy of detection and clustering methods in high-speed scenes, and difficulty in obtaining accurate results due to the sparsity of laser point clouds, this invention provides a method for estimating the dynamic and static states of objects, applied to a mobile device equipped with a camera sensor and a laser sensor, comprising:

[0015] Using the camera image as a reference, a time-synchronized laser point cloud is projected onto the camera image to form a depth image, and dense feature points of objects perceived in the depth image are extracted.

[0016] The dense feature points of the object at the previous moment are used to estimate the predicted positioning information of the object in the world coordinate system at the current moment;

[0017] Optical flow tracing is performed on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment;

[0018] Based on the predicted positioning information and the observed positioning information, the object is subjected to dynamic and static estimation.

[0019] In a second aspect, embodiments of the present invention provide a device for performing dynamic and static estimation of an object, comprising:

[0020] The dense feature point extraction module is used to project a time-synchronized laser point cloud onto the camera image as a reference to form a depth image, and extract the dense feature points of the objects perceived in the depth image.

[0021] The prediction and positioning information determination module is used to estimate the predicted positioning information of the object in the world coordinate system at the current moment by using the dense feature points of the object at the previous moment.

[0022] The observation and positioning information determination module is used to perform optical flow tracing on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment.

[0023] The dynamic and static estimation module is used to perform dynamic and static estimation of the object based on the predicted positioning information and the observed positioning information.

[0024] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the dynamic and static estimation method for an object according to any embodiment of the present invention.

[0025] Fourthly, embodiments of the present invention provide a mobile device, including a main body and an electronic device according to any embodiment of the present invention mounted on the main body.

[0026] Fifthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the dynamic and static estimation method for an object according to any embodiment of the present invention.

[0027] Sixthly, embodiments of the present invention also provide a computer program product that, when run on a computer, causes the computer to execute the dynamic and static estimation method for an object as described in any one of the embodiments of the present invention.

[0028] The beneficial effects of this invention are as follows: It provides a basis for autonomous driving decisions by real-time detecting the dynamic and static information of vehicles in the world coordinate system, and realizes an algorithm for dynamic and static detection in the world coordinate system through the fusion of vision and laser. Simultaneously, it utilizes optical flow to trace feature points and perform back projection, judging physical dynamics based on the inconsistency between observation and prediction. During the dynamic and static detection process, no additional annotation and training work is required, making it simple and easy to implement, and it has a more significant effect on large target objects such as vehicles. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a method for estimating the dynamic and static states of an object according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram comparing dense and sparse features of an object's dynamic and static estimation method according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the back projection geometry of a dynamic and static estimation method for an object provided in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of two-dimensional and three-dimensional dynamic and static detection results of an object dynamic and static estimation method provided in an embodiment of the present invention;

[0034] Figure 5 This is a flowchart of the dynamic and static determination algorithm of an object dynamic and static estimation method provided in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of the structure of a dynamic and static estimation execution device for an object according to an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of an embodiment of an electronic device for dynamic and static estimation of an object, provided as an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Those skilled in the art will recognize that embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] For ease of understanding, the technical terms used in this application are explained below:

[0040] The term "mobile device" as used in this application includes, but is not limited to, vehicles with the six levels of autonomous driving technology (L0-L5) as defined by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Classification of Driving Automation for Automobiles".

[0041] In some embodiments, the mobile device may be a vehicle or robot having various functions, such as:

[0042] (1) Passenger transport function, such as family cars, buses, etc.;

[0043] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailers, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.

[0044] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, hoists, excavators, bulldozers, loaders, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.

[0045] (4) Entertainment functions, such as recreational vehicles, amusement park automatic driving devices, balance bikes, etc.;

[0046] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, and engineering emergency rescue vehicles.

[0047] like Figure 1 The diagram shows a flowchart of a method for estimating the dynamic and static states of an object according to an embodiment of the present invention, which includes the following steps:

[0048] S11: Based on the camera image, project the time-synchronized laser point cloud onto the camera image to form a depth image, and extract the dense feature points of the object perceived in the depth image.

[0049] S12: Utilize the dense feature points of the object from the previous moment to estimate the predicted positioning information of the object in the world coordinate system at the current moment;

[0050] S13: Perform optical flow tracing on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment;

[0051] S14: Perform dynamic and static estimation of the object based on the predicted positioning information and the observed positioning information.

[0052] In this embodiment of the invention, S12 can assume that the object is stationary and use the dense feature points of the object at the previous moment to estimate the predicted positioning information of the object in the world coordinate system at the current moment.

[0053] In this embodiment, during the movement of the mobile device, the laser sensor on which it is mounted can collect laser point cloud data in real time, and the camera sensor can collect camera images in real time. The collected laser point cloud data and camera images are used to determine the motion or static state of objects (mainly vehicles on the road).

[0054] For step S11, in order to determine the motion or static state of objects within the mobile device's field of view, the laser point cloud and camera image are used as inputs to the algorithm for determining the motion or static state of objects within the mobile device's field of view. It is important to note that the laser point cloud and camera image are time-synchronized. After the mobile device's sensors acquire the laser point cloud and camera image, they need to undergo time synchronization processing. After this processing, the camera image and laser point cloud can be considered to have been acquired at the same time and are used to calculate subsequent steps.

[0055] By extracting dense image features from camera images and simultaneously projecting the laser point cloud onto the forward-looking camera image, the depth of each point is calculated to form a depth image. The main purpose of extracting dense feature points is to use these points to determine the motion or static state of objects. Extracting traditional sparse features might result in too few features for distant objects, leading to unstable judgment results. Examples of sparse and dense feature points are shown below. Figure 2 As shown (dense on the left, sparse on the right), it can be seen that dense features are significantly more numerous than sparse features, making the judgment more accurate.

[0056] Specifically, using the camera image as a reference, projecting the time-synchronized laser point cloud onto the camera image to form a depth image specifically includes:

[0057] Based on the intrinsic and extrinsic parameters of the camera sensor and laser sensor mounted on the mobile device, the projection position of the laser point cloud in the camera image is calculated to obtain the depth information of each laser point in the laser point cloud in the camera image.

[0058] The depth information is fused into the camera image to form a depth image suitable for autonomous driving.

[0059] In this embodiment, the positions of the camera sensor and laser sensor mounted on the mobile device are known, allowing the determination of their respective intrinsic and extrinsic parameters. Using these parameters, the projection positions of the laser point cloud onto the camera image are calculated, yielding the depth information of each laser point in the laser point cloud within the camera image. Fusing the camera image with this depth information results in a depth image suitable for autonomous driving (a depth image contains depth information, while a regular camera image only contains color information. A regular image provides information on a two-dimensional plane, while a depth image provides information in three-dimensional space, adding an extra dimension). It's important to note that current technologies generally lack depth image sensors (depth cameras), so depth is derived solely from the projection of laser point clouds. Currently, some indoor robots and mobile phones carry depth cameras to capture 3D information for better obstacle or face recognition. However, due to limitations in sensor principles, these cameras typically operate only indoors, with a measurement range generally within 10 meters, making them unsuitable for autonomous driving environments. Through the steps described in this method, a depth image suitable for autonomous driving can be obtained even without a depth image sensor (depth camera).

[0060] For step S12, the mobile device can perceive different objects from the depth image (for example, it can perceive other vehicles on the road while driving on the road). Let's assume the mobile device is a smart car A and the perceived object is vehicle B.

[0061] As one implementation method, it includes: performing back-projection processing on the two-dimensional positions of the dense feature points of the object in the depth image at the previous moment to determine the three-dimensional positioning information of the object in the world coordinate system at the previous moment;

[0062] The three-dimensional positioning information of the object in the world coordinate system at the previous moment is determined as the predicted positioning information of the object in the world coordinate system.

[0063] In this embodiment, at the previous moment, intelligent vehicle A can perceive the current position information of vehicle B in the world coordinate system through depth images. Assuming vehicle B is stationary, the change in vehicle B's position information in the world coordinate system as intelligent vehicle A moves is inferable. (For example, conventionally speaking, assuming vehicle B is in front of intelligent vehicle A, if vehicle B is stationary, as intelligent vehicle A moves forward, the area occupied by vehicle B in the camera image will increase. However, regardless of how large the area in the camera image increases, if vehicle B is stationary, its positioning information in the world coordinate system remains unchanged.)

[0064] Specifically, the dense feature points of vehicle B are back-projected onto the two-dimensional positions of the depth image at the previous moment (converting the two-dimensional coordinates into three-dimensional coordinates) to determine the three-dimensional positioning information of vehicle B in the world coordinate system at the previous moment. Assuming vehicle B is stationary, this three-dimensional positioning information in the world coordinate system at the previous moment is used as the predicted positioning information of the object in the world coordinate system. This predicted positioning information is only used to determine whether vehicle B is stationary and does not represent the actual position of vehicle B at the current moment.

[0065] For step S13, in the previous step, this method estimated the predicted location information of vehicle B at the current moment if vehicle B were stationary. This step aims to determine the correct location information of vehicle B as much as possible.

[0066] At the previous time step, optical flow tracing was performed on the dense feature points of vehicle B to determine its current location. It's important to note that while optical flow tracing can directly determine whether an object is moving, this method does not directly use it. This is because optical flow tracing determines the object's motion relative to itself (for example, if intelligent car A and vehicle B are moving at the same constant speed, optical flow tracing would indicate that vehicle B is stationary relative to intelligent car A. However, vehicle B is actually moving in the real environment, meaning that directly using optical flow tracing to determine static or dynamic status is inaccurate in practical applications. This method aims to determine vehicle B's motion in the world coordinate system, not its motion relative to its own vehicle. There is a fundamental difference between these two, and determining the stationary or moving status of other vehicles relative to the world coordinate system has many practical implications. For example, vehicles parked on the roadside are stationary relative to the world coordinate system, and their trajectories do not need to be predicted. Vehicles stationary relative to their own vehicle are moving, and their trajectories need to be calculated). Therefore, optical flow tracing is performed on the dense feature points of vehicle B at the previous time step to determine its current location. Optical flow tracing can be implemented using DIS (Direct Inverse Search) or other dense optical flow algorithms. Since DIS optical flow is computationally fast and does not require additional hardware resources, it can be primarily used as the basis for dynamic and static determination in the implementation.

[0067] As one implementation method, optical flow tracing is performed on the dense feature points of the object at the previous moment to determine the object's observation and positioning information at the current moment, specifically including:

[0068] When the dense feature points of the object at the current moment only have camera image data, the two-dimensional image position of the dense feature points in the camera image coordinate system at the current moment is determined as the observation and positioning information of the object at the current moment.

[0069] When the dense feature points of the object at the current moment have camera image data and laser point cloud data, the three-dimensional position of the dense feature points in the depth image at the current moment is back-projected to obtain the observation and positioning information of the object in the world coordinate system at the current moment.

[0070] Because intelligent car A is constantly moving, it calculates its current position at the previous moment, and then calculates its position for the next moment, making continuous judgments in real time. While the vehicle is moving, due to vibrations, turns, or other situations, the point cloud may be incomplete, resulting in only a two-dimensional image position. The position of dense feature points in the two-dimensional image coordinate system of the camera at the current moment is determined as the observation and positioning information of vehicle B at that moment.

[0071] Specifically, based on the predicted positioning information, the predicted two-dimensional image position of the object in the camera image coordinates at the current moment is determined;

[0072] Based on epipolar geometric constraints or two-dimensional projection constraints, determine the amount of change in the predicted two-dimensional image position compared to the observed positioning information of the object at the current moment;

[0073] Based on the magnitude of the change, the object is subjected to dynamic and static estimation.

[0074] The predicted positioning information of vehicle B has been determined in the above steps. This information can be mapped to the predicted 2D image position in image coordinates. Simultaneously, optical flow tracing is used to determine the observed positioning information of vehicle B at the current moment from the 2D image position of dense feature points in the camera image coordinate system. By applying epipolar geometric constraints or 2D projection constraints to the predicted 2D image position and observed positioning information of vehicle B, the change in the observed positioning information at the current moment can be determined. For example, if the change exceeds a preset threshold, vehicle B can be determined to be dynamic relative to the world coordinate system; if the change does not exceed the threshold, vehicle B can be determined to be static relative to the world coordinate system.

[0075] Under normal circumstances, dense feature points have both camera image data and laser point cloud data. The three-dimensional positions of the dense feature points in the depth image at the current moment are back-projected to obtain the observation and positioning information of vehicle B in the world coordinate system at the current moment.

[0076] In the field of 3D vision, reprojection refers to projecting an image's location onto its spatial location. Similarly, the process of projecting a spatial location onto an image is called orthographic projection, or simply projection.

[0077] Since the camera image captured by the camera sensor is a two-dimensional image, its 3D position can be obtained by combining it with the depth information from the LiDAR. This process is also a back-projection process, which requires the camera's intrinsic and extrinsic parameters. The specific formula is:

[0078]

[0079] Among them, ZP uv These are 2D image coordinates, where Z is the depth (u is the coordinate value in the x-direction, v is the coordinate value in the y-direction, 1 is a pixel), K is the camera intrinsic parameter matrix, RPw+t is the camera extrinsic parameter, and KTP is the extrinsic parameter. w These are 3D coordinates. Projection from right to left is called projection, and back projection from left to right is called back projection.

[0080] Similarly, in the world coordinate system, based on the magnitude of the change in vehicle B's predicted positioning information compared to the observed positioning information at the current moment, dynamic and static estimations are performed on the object. The comparison process will not be elaborated here.

[0081] Specifically, the schematic diagram of the above back projection calculation is as follows: Figure 3 As shown in the attached diagram, the steps described above are explained in detail:

[0082] 1. At the first moment (the position indicated by the dashed line in the attached diagram), intelligent car A (the vehicle in the diagram) extracts dense feature points through images and point clouds. This allows the determination of the initial position of vehicle B (i.e., the moving vehicle and the stationary vehicle in the attached diagram).

[0083] 2. As intelligent car A moves, reaching the second moment (the position indicated by the solid line in the attached diagram), its own positioning module can determine its own vehicle positioning information at each moment. The optical flow tracing algorithm provides the image positions of vehicle B's feature points from the first moment at the second moment; this is the observed positioning information.

[0084] 3. If we assume vehicle B is stationary, then based on the positions of the dense feature points at the first moment, we can predict where these dense feature points should fall in the image at the second moment. This is the predictive localization information.

[0085] 4. Compare the predicted positioning information with the observed positioning information. If the prediction matches the observation (judged by the change in magnitude), it indicates that vehicle B is indeed stationary; otherwise, the assumption of stationary status is invalid, and vehicle B should be moving. (After obtaining the change in magnitude, a classifier determines whether it is dynamic. The classifier simply detects whether its movement exceeds a preset threshold. This threshold can be adaptively adjusted according to different needs.)

[0086] The above steps describe how to determine motion / static status at two time points. This two-time-point algorithm has an initialization process, meaning it cannot determine the motion / static information of the first frame. Extending it to a queue model, it continuously extracts new feature points and tracks old ones, making the algorithm output smoother. The algorithm can ultimately output the motion / static information of image feature points and point clouds, such as... Figure 4 As shown.

[0087] As can be seen in this intersection scene, the vehicle on the left is still waiting for the traffic light and is stationary. The vehicle on the right has already started moving and is in motion. The 3D structure on the right also shows the same result. This state determination is independent of whether the vehicle itself is moving; it can determine the information of other objects (vehicles) in the world coordinate system. In summary, the dynamic / static determination algorithm flow of this method is as follows: Figure 5 As shown.

[0088] This implementation demonstrates that by real-time detecting the vehicle's dynamic and static information in the world coordinate system, a basis for autonomous driving decisions is provided, achieving an algorithm for dynamic and static detection in the world coordinate system through the fusion of vision and laser technologies. Simultaneously, feature points are traced using DIS optical flow for back-projection, and physical dynamic and static status is determined based on the inconsistency between observation and prediction. No additional annotation and training are required during the dynamic and static detection process, making it simple and easy to implement, and it shows more significant effects on large target objects such as vehicles.

[0089] like Figure 6 The diagram shown is a schematic diagram of a dynamic and static estimation execution device for an object according to an embodiment of the present invention. The system can execute the dynamic and static estimation method of the object described in any of the above embodiments and is configured in a terminal.

[0090] This embodiment provides an object dynamic and static estimation execution device 10, which includes: a dense feature point extraction module 11, a prediction positioning information determination module 12, an observation positioning information determination module 13, and a dynamic and static estimation module 14.

[0091] The dense feature point extraction module 11 is used to project a time-synchronized laser point cloud onto the camera image to form a depth image, and extract the dense feature points of the object perceived in the depth image; the prediction positioning information determination module 12 is used to estimate the predicted positioning information of the object in the world coordinate system at the current moment using the dense feature points of the object at the previous moment; the observation positioning information determination module 13 is used to perform optical flow tracing on the dense feature points of the object at the previous moment to determine the observation positioning information of the object at the current moment; and the dynamic and static estimation module 14 is used to perform dynamic and static estimation of the object based on the predicted positioning information and the observation positioning information.

[0092] Furthermore, the observation and positioning information determination module is used for:

[0093] When the dense feature points of the object at the current moment only have camera image data, the two-dimensional image position of the dense feature points in the camera image coordinate system at the current moment is determined as the observation and positioning information of the object at the current moment.

[0094] Furthermore, the dynamic and static estimation module is used for:

[0095] Based on the predicted positioning information, the predicted two-dimensional image position of the object in the camera image coordinates at the current moment is determined;

[0096] Based on epipolar geometric constraints or two-dimensional projection constraints, determine the amount of change in the predicted two-dimensional image position compared to the observed positioning information of the object at the current moment;

[0097] Based on the magnitude of the change, the object is subjected to dynamic and static estimation.

[0098] Furthermore, the observation and positioning information determination module is used for:

[0099] When the dense feature points of the object at the current moment have camera image data and laser point cloud data, the three-dimensional position of the dense feature points in the depth image at the current moment is back-projected to obtain the observation and positioning information of the object in the world coordinate system at the current moment.

[0100] Furthermore, the dynamic and static estimation module is used for:

[0101] In the world coordinate system, the object is dynamically and statically estimated based on the magnitude of the change in the predicted positioning information of the object at the current moment compared to the observed positioning information.

[0102] Furthermore, the predicted positioning information determination module is used for:

[0103] The dense feature points of the object are back-projected onto the two-dimensional positions of the depth image at the previous moment to determine the three-dimensional positioning information of the object in the world coordinate system at the previous moment.

[0104] The three-dimensional positioning information of the object in the world coordinate system at the previous moment is determined as the predicted positioning information of the object in the world coordinate system.

[0105] Furthermore, the dense feature point extraction module is used for:

[0106] Based on the intrinsic and extrinsic parameters of the camera sensor and laser sensor mounted on the mobile device, the projection position of the laser point cloud in the camera image is calculated to obtain the depth information of each laser point in the laser point cloud in the camera image.

[0107] The depth information is fused into the camera image to form a depth image suitable for autonomous driving.

[0108] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the object dynamic and static estimation method in any of the above method embodiments.

[0109] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0110] Using the camera image as a reference, a time-synchronized laser point cloud is projected onto the camera image to form a depth image, and dense feature points of objects perceived in the depth image are extracted.

[0111] The dense feature points of the object at the previous moment are used to estimate the predicted positioning information of the object in the world coordinate system at the current moment;

[0112] Optical flow tracing is performed on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment;

[0113] Based on the predicted positioning information and the observed positioning information, the object is subjected to dynamic and static estimation.

[0114] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the dynamic and static estimation method of the object in any of the above method embodiments is performed.

[0115] This invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a dynamic and static estimation method for an object.

[0116] In some embodiments, the present invention also provides a mobile device, including a body and an electronic device according to any of the foregoing embodiments mounted on the body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, unmanned floor scrubber, unmanned logistics vehicle, unmanned passenger car, unmanned sanitation vehicle, unmanned minibus / bus, truck, mining truck, etc., or it may be a robot, etc.

[0117] In some embodiments, the present invention also provides a computer program product that, when run on a computer, causes the computer to execute the dynamic and static estimation method for an object as described in any one of the embodiments of the present invention.

[0118] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for a method of estimating the dynamic and static states of an object according to another embodiment of this application, as shown below. Figure 7 As shown, the device includes:

[0119] One or more processors 710 and memory 720, Figure 7 Taking a processor 710 as an example, the device for the dynamic and static estimation method of an object may also include an input device 730 and an output device 740.

[0120] The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0121] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the object dynamic and static estimation method in the embodiments of this application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the object dynamic and static estimation method of the above-described method embodiments.

[0122] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0123] Input device 730 can receive input numerical or character information. Output device 740 may include display devices such as a display screen.

[0124] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, they perform the dynamic and static estimation method of the object in any of the above method embodiments.

[0125] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0126] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the object dynamic and static estimation method of any embodiment of this invention.

[0128] The electronic devices described in this application exist in various forms, including but not limited to:

[0129] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0130] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0131] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0132] (4) Other electronic devices with data processing functions.

[0133] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the dynamic and static states of an object, applied to a mobile device equipped with a camera sensor and a laser sensor, comprising: Using the camera image as a reference, a time-synchronized laser point cloud is projected onto the camera image to form a depth image, and dense feature points of objects perceived in the depth image are extracted. The dense feature points of the object are back-projected into the two-dimensional position of the depth image at the previous moment to determine the three-dimensional positioning information of the object in the world coordinate system at the previous moment. The three-dimensional positioning information of the object in the world coordinate system at the previous moment is determined as the predicted positioning information of the object in the world coordinate system. Optical flow tracing is performed on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment. When the dense feature points of the object at the current moment only have camera image data, the two-dimensional image position of the dense feature points in the camera image coordinate system at the current moment is determined as the observation and positioning information of the object at the current moment. The method for performing dynamic and static estimation of the object based on the predicted positioning information and the observed positioning information includes: determining the predicted two-dimensional image position of the object in the camera image coordinates at the current moment based on the predicted positioning information; determining the amount of change of the predicted two-dimensional image position compared with the observed positioning information of the object at the current moment according to epipolar geometric constraints or two-dimensional projection constraints; and performing dynamic and static estimation of the object according to the magnitude of the change.

2. The method according to claim 1, characterized in that, Optical flow tracing is performed on the dense feature points of the object at the previous time step to determine the object's observation and positioning information at the current time step, specifically including: When the dense feature points of the object at the current moment have camera image data and laser point cloud data, the three-dimensional position of the dense feature points in the depth image at the current moment is back-projected to obtain the observation and positioning information of the object in the world coordinate system at the current moment.

3. The method according to claim 2, characterized in that, Based on the predicted positioning information and the observed positioning information, dynamic and static estimation of the object is performed, specifically including: In the world coordinate system, the object is dynamically and statically estimated based on the magnitude of the change in the predicted positioning information of the object at the current moment compared to the observed positioning information.

4. The method according to claim 1, characterized in that, Using the camera image as a reference, the time-synchronized laser point cloud is projected onto the camera image to form a depth image, specifically including: Based on the intrinsic and extrinsic parameters of the camera sensor and laser sensor mounted on the mobile device, the projection position of the laser point cloud in the camera image is calculated to obtain the depth information of each laser point in the laser point cloud in the camera image. The depth information is fused into the camera image to form a depth image suitable for autonomous driving.

5. A device for performing dynamic and static estimation of an object, comprising: The dense feature point extraction module is used to project a time-synchronized laser point cloud onto the camera image as a reference to form a depth image, and extract the dense feature points of the objects perceived in the depth image. The prediction and positioning information determination module is used to back-project the two-dimensional positions of the dense feature points of the object in the depth image of the previous moment to determine the three-dimensional positioning information of the object in the world coordinate system at the previous moment, and to determine the three-dimensional positioning information of the object in the world coordinate system at the previous moment as the prediction and positioning information of the object in the world coordinate system. The observation and positioning information determination module is used to perform optical flow tracing on the dense feature points of the object at the previous moment to determine the observation and positioning information of the object at the current moment. When the dense feature points of the object at the current moment only have camera image data, the two-dimensional image position of the dense feature points in the camera image coordinate system at the current moment is determined as the observation and positioning information of the object at the current moment. The dynamic and static estimation module is used to perform dynamic and static estimation of the object based on the predicted positioning information and the observed positioning information, including: determining the predicted two-dimensional image position of the object in the camera image coordinates at the current moment based on the predicted positioning information; determining the amount of change of the predicted two-dimensional image position compared with the observed positioning information of the object at the current moment according to epipolar geometric constraints or two-dimensional projection constraints; and performing dynamic and static estimation of the object according to the magnitude of the change.

6. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.

7. A mobile device comprising a body and an electronic device according to claim 6 mounted on the body.

8. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.

9. A computer program product, characterized in that, When the computer program product is run on a computer, the computer performs the dynamic and static estimation method for an object as described in any one of claims 1-4.

Images