An object state estimation method, apparatus, computing device, and vehicle

Abstract

This disclosure provides an object state estimation apparatus, computing device, and vehicle to address the problems of low robustness and the need for target detection in each frame of point cloud in existing state estimation methods. The method includes: calculating the estimated state of the object in the next point cloud frame based on its optimized state in the current point cloud frame; determining the estimated point cloud of the object in the next point cloud frame based on the estimated state; and obtaining the optimized state of the object in the next point cloud frame by optimizing the estimated point cloud. The estimated state and the optimized state both include at least one of the object's position and orientation parameters. This disclosure eliminates the reliance on target detection, and by estimating and optimizing the relevant point cloud of the object in the point cloud frame, a high-precision state of the object can be determined. The algorithm exhibits high robustness and wide applicability.

Description

Technical Field

[0001] This disclosure relates to the field of point cloud data processing, and in particular to an object state estimation method, apparatus, computing device, and vehicle. Background Technology

[0002] Currently, with the development of autonomous driving and mobile robot technologies, LiDAR equipment has been widely used in autonomous vehicles and mobile robots. To ensure the normal operation of autonomous vehicles and mobile robots, it is generally necessary to collect point cloud data of the surrounding environment using LiDAR. This helps the vehicles and robots perceive their surroundings, especially the status of other vehicles, including their position and orientation. Therefore, accurately estimating the status of surrounding vehicles within a point cloud sequence is crucial for the efficiency and safety of autonomous driving.

[0003] Existing state estimation methods are mainly based on the multi-object tracking paradigm of detection and tracking. For the input multi-frame point cloud, the detection method is first used to extract detection boxes in each single frame point cloud, where the position and orientation of the detection box represent the state of the vehicle; then the detection boxes in different frames are associated according to whether they belong to the same vehicle, so as to obtain the state of the vehicle in the multi-frame point cloud.

[0004] This method first requires object detection in each single-frame point cloud, and its performance heavily depends on the quality of each frame. The widespread sparsity of point cloud data means that a large number of low-quality data points exist, leading to insufficient robustness of existing methods. Furthermore, this approach relies on an object detection model that requires deep learning training to develop a strong ability to recognize the shape of objects, thus limiting its ability to extend to object shape categories not covered in the training set. Additionally, due to the inherent sparsity of point clouds, object detection often struggles to achieve centimeter-level accuracy, affecting the accuracy of state estimation and consequently impacting some downstream tasks in autonomous driving.

[0005] Therefore, there is a need for a more efficient and accurate method to determine the vehicle status. Summary of the Invention

[0006] Embodiments of this disclosure provide an object state estimation method, apparatus, computing device, and vehicle to improve the accuracy and efficiency of object state calculation.

[0007] To achieve the above objectives, the embodiments of this disclosure adopt the following technical solutions:

[0008] A first aspect of this disclosure provides an object state estimation method, comprising: calculating an estimated state of an object in a next point cloud frame based on an optimized state of the object in a current point cloud frame; determining an estimated point cloud of the object in the next point cloud frame based on the estimated state; and obtaining an optimized state of the object in the next point cloud frame by optimizing the estimated point cloud; wherein both the estimated state and the optimized state include at least one of the object's position parameters and orientation parameters.

[0009] A second aspect of this disclosure provides an object state estimation apparatus, comprising: a state prediction module, adapted to calculate an estimated state of an object in a next point cloud frame based on an optimized state of the object in a current point cloud frame; a point cloud prediction module, adapted to determine an estimated point cloud of the object in the next point cloud frame based on the predicted state; and a state optimization module, adapted to obtain an optimized state of the object in the next point cloud frame by optimizing the predicted point cloud; wherein the predicted state and the optimized state both include at least one of the object's position and orientation.

[0010] A third aspect of this disclosure provides a computing device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor; wherein, when the processor runs the computer program, it executes the object state estimation method as described above.

[0011] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the object state estimation method as described above.

[0012] A fifth aspect of this disclosure provides a vehicle including the computing device described above.

[0013] This disclosure provides an object state estimation scheme. For an acquired point cloud frame sequence, based on the object's optimized state in the current point cloud frame, the estimated state of the object in the next point cloud frame is calculated. Based on the estimated state, the estimated point cloud of the object in the next point cloud frame is determined. The optimized state of the object in the next point cloud frame is obtained by optimizing the estimated point cloud. This disclosure directly processes point cloud data to obtain higher-precision state estimation, eliminating the dependence on target detection for each frame of point cloud data. It can be extended to other categories of object state detection without introducing additional prior models for the object's shape. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A structural diagram of a vehicle 100 provided in an embodiment of this disclosure;

[0016] Figure 2 A flowchart of an object state estimation method 200 provided in an embodiment of this disclosure;

[0017] Figure 3 A flowchart of another object state estimation method 300 provided in this embodiment of the disclosure;

[0018] Figure 4 This is a schematic diagram illustrating the vehicle state estimation and 3D reconstruction in an embodiment of this disclosure;

[0019] Figure 5 A structural diagram of an object state estimation device 500 provided in an embodiment of this disclosure;

[0020] Figure 6 A structural diagram of another object state estimation device 600 provided in an embodiment of this disclosure;

[0021] Figure 7 This is a structural diagram of a computing device 700 provided in an embodiment of the present disclosure. Detailed Implementation

[0022] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] To enable those skilled in the art to better understand this disclosure, some technical terms appearing in the embodiments of this disclosure are explained below:

[0025] Object: The target object in each frame of data, which can include static and dynamic objects, such as pedestrians, vehicles, animals, obstacles, traffic lights, road signs, etc.

[0026] State: The positional information of an object in the three-dimensional world, including its position coordinates and orientation.

[0027] State estimation: Calculating the state of an object in the three-dimensional world using known information.

[0028] Point cloud: Data about the surrounding environment collected by radar (such as lidar) is represented by a set of sparse three-dimensional spatial points.

[0029] Point cloud sequence: Multiple consecutive point cloud images within a certain period of time, or point clouds collected by lidar within a continuous time period.

[0030] Point cloud registration: By aligning two point clouds of the same object, the motion of the object between the two point clouds can be calculated.

[0031] ICP: A point cloud registration algorithm that associates point cloud points according to nearest neighbors based on predicted object motion and minimizes the distance between associated point pairs.

[0032] Frame: The measurement data received by a sensor after completing one observation. For example, a frame of data from a camera is an image, and a frame of data from a lidar is a set of laser point clouds.

[0033] Target detection: using algorithms to find the location of target objects in raw sensor data, typically represented by rectangles or cuboids to indicate the position of the object in 2D or 3D space.

[0034] Target tracking: Given sensor input data and a given target object over a period of time, calculate the state of the given target object at each moment.

[0035] 3D reconstruction: obtaining a 3D model of an object, usually expressed as a dense point cloud or CAD model.

[0036] Figure 1 This is a schematic diagram of a vehicle 100 in which the various technologies disclosed herein can be implemented. Vehicle 100 can be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawnmower, excavator, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment, tram, golf cart, train, trolleybus, or other vehicle. Vehicle 100 can operate fully or partially in an autonomous driving mode. In autonomous driving mode, vehicle 100 can control itself; for example, vehicle 100 can determine the current state of the vehicle and the current state of the environment in which the vehicle is located, determine the predicted behavior of at least one other vehicle in the environment, determine the trust level corresponding to the probability that the at least one other vehicle will perform the predicted behavior, and control vehicle 100 itself based on the determined information. In autonomous driving mode, vehicle 100 can operate without human interaction.

[0037] Vehicle 100 may include various vehicle systems, such as drive system 142, sensor system 144, control system 146, user interface system 148, control computer system 150, and communication system 152. Vehicle 100 may include more or fewer systems, and each system may include multiple units. Furthermore, each system and unit of vehicle 100 may be interconnected. For example, control computer system 150 is capable of data communication with one or more of vehicle systems 142-148 and 152. Thus, one or more of the described functions of vehicle 100 may be divided into additional functional components or physical components, or combined into a smaller number of functional components or physical components. In a further example, additional functional components or physical components may be increased to, for example... Figure 1 In the example shown.

[0038] The drive system 142 may include a plurality of operable components (or units) that provide kinetic energy to the vehicle 100. In one embodiment, the drive system 142 may include an engine or electric motor, wheels, a transmission, electronic systems, and a power source (or power source). The engine or electric motor may be any combination of internal combustion engines, electric motors, steam engines, fuel cell engines, propane engines, or other forms of engines or electric motors. In some embodiments, the engine may convert a power source into mechanical energy. In some embodiments, the drive system 142 may include multiple engines or electric motors. For example, a hybrid vehicle may include a gasoline engine and an electric motor, or other configurations may be included.

[0039] The wheels of vehicle 100 can be standard wheels. The wheels of vehicle 100 can be of various forms, including single-wheel, two-wheel, three-wheel, or four-wheeled, such as the four wheels on a car or truck. Other numbers of wheels are also possible, such as six or more wheels. One or more wheels of vehicle 100 can be operated to rotate in a different direction than the other wheels. A wheel can be at least one wheel fixedly connected to a transmission. The wheel can include a combination of metal and rubber, or other materials. The transmission can include units operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission can include a gearbox, clutch, differential gears, and driveshaft. The transmission can also include other units. The driveshaft can include one or more axles that match the wheels. The electronic system can include units for transmitting or controlling electronic signals of vehicle 100. These electronic signals can be used to activate multiple lights, multiple servo mechanisms, multiple electric motors, and other electronic drives or controls in vehicle 100. The power source can be an energy source that provides power to the engine or electric motor, either wholly or partially. That is, the engine or electric motor is capable of converting the power source into mechanical energy. For example, the power source may include gasoline, petroleum, petroleum-based fuels, propane, other compressed gaseous fuels, ethanol, fuel cells, solar panels, batteries, and other electrical energy sources. The power source may optionally include any combination of a fuel tank, battery, capacitor, or flywheel. The power source may also provide energy to other systems of vehicle 100.

[0040] Sensor system 144 may include multiple sensors for sensing information about the environment and conditions of vehicle 100. For example, sensor system 144 may include an inertial measurement unit (IMU), a global positioning system (GPS) transceiver, a radar (RADAR) unit, a laser rangefinder / LIDAR unit (or other distance measurement device), acoustic sensors, and cameras or image capture devices. Sensor system 144 may include multiple sensors for monitoring vehicle 100 (e.g., oxygen (O2) monitor, fuel gauge sensor, engine oil pressure sensor, etc.). Other sensors may also be configured. One or more sensors included in sensor system 144 may be driven individually or collectively to update the position, orientation, or both of the sensors.

[0041] The IMU may include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing changes in the position and orientation of vehicle 100 based on inertial acceleration. The GPS transceiver may be any sensor used to estimate the geographic location of vehicle 100. For this purpose, the GPS transceiver may include a receiver / transmitter to provide position information of vehicle 100 relative to the Earth. It should be noted that GPS is an example of a Global Navigation Satellite System; therefore, in some embodiments, the GPS transceiver may be replaced with a BeiDou Navigation Satellite System transceiver or a Galileo Navigation Satellite System transceiver. The radar unit may use radio signals to sense objects in the environment in which vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit may also be used to sense the speed and direction of travel of objects approaching vehicle 100. The laser rangefinder or LIDAR unit (or other distance measurement device) may be any sensor that uses lasers to sense objects in the environment in which vehicle 100 is located. In one embodiment, the laser rangefinder / LIDAR unit may include a laser source, a laser scanner, and a detector. The laser rangefinder / LIDAR unit is used to operate in continuous (e.g., using heterodyne detection) or discontinuous detection modes. The camera may include means for capturing multiple images of the environment in which the vehicle 100 is located. The camera may be a still image camera or a video camera.

[0042] The control system 146 is used to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 may include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0043] The steering unit may be a combination of mechanisms for adjusting the forward direction of vehicle 100. A power control unit (e.g., a throttle) may be used to control the engine speed, thereby controlling the speed of vehicle 100. The braking unit may include a combination of mechanisms for decelerating vehicle 100. The braking unit may utilize friction to decelerate the vehicle in a standard manner. In other embodiments, the braking unit may convert the kinetic energy of the wheels into electrical current. The braking unit may also take other forms. The navigation unit may be any system that determines a driving path or route for vehicle 100. The navigation unit may also dynamically update the driving path as vehicle 100 travels. The control system 146 may also additionally or optionally include other components (or units) not shown or described.

[0044] User interface system 148 can be used to allow vehicle 100 to interact with external sensors, other vehicles, other computer systems, and / or the user of vehicle 100. For example, user interface system 148 may include standard visual display devices (e.g., plasma displays, liquid crystal displays (LCDs), touchscreen displays, head-mounted displays, or other similar displays), speakers or other audio output devices, microphones or other audio input devices. For example, user interface system 148 may also include navigation interfaces and interfaces for controlling the internal environment of vehicle 100 (e.g., temperature, fan, etc.).

[0045] Communication system 152 can provide vehicle 100 with a means of communicating with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, communication system 152 can communicate with one or more devices directly or through a communication network. Communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (e.g., CDMA, EVDO, GSM / GPRS) or 4G cellular communication (e.g., WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (e.g., using...). In some embodiments, the communication system 152 can communicate directly with one or more devices or other vehicles in the vicinity, for example, using infrared light. Or ZigBee. Other wireless protocols, such as various vehicular communication systems, are also within the scope of this application. For example, the communication system may include one or more Dedicated Short Range Communication (DSRC) devices, V2V devices, or V2X devices that conduct public or private data communication with vehicles and / or roadside stations.

[0046] The control computer system 150 can control some or all of the functions of the vehicle 100. The autonomous driving control unit in the control computer system 150 can be used to identify, assess, and avoid or traverse potential obstacles in the environment in which the vehicle 100 is located. Typically, the autonomous driving control unit can be used to control the vehicle 100 without a driver or to assist a driver in controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from a GPS transceiver, radar data, LiDAR data, camera data, and data from other vehicle systems to determine the driving path or trajectory of the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in autonomous driving mode.

[0047] The control computer system 150 may include at least one processor (which may include at least one microprocessor), which executes processing instructions (i.e., machine-executable instructions) stored in a non-volatile computer-readable medium (e.g., a data storage device or memory). The memory stores at least one machine-executable instruction, and the processor executes this instruction to implement functions including a map engine, a positioning module, a perception module, a navigation or path module, and an automatic control module. The map engine and positioning module provide map and positioning information. The perception module perceives objects in the vehicle's environment based on information acquired by the sensor system and map information provided by the map engine. The navigation or path module plans a driving path for the vehicle based on the processing results of the map engine, positioning module, and perception module. The automatic control module parses and converts the decision information input from modules such as the navigation or path module into control commands for the vehicle control system, and sends these commands to corresponding components in the vehicle control system via an in-vehicle network (e.g., an in-vehicle electronic network system implemented via CAN bus, local area network, multimedia orientation system transmission, etc.) to achieve automatic vehicle control; the automatic control module can also obtain information about various components in the vehicle via the in-vehicle network.

[0048] The control computer system 150 may also be multiple computing devices that distribute and control components or systems of the vehicle 100. In some embodiments, the memory may contain processing instructions (e.g., program logic) that are executed by a processor to implement various functions of the vehicle 100. In one embodiment, the control computer system 150 is capable of data communication with systems 142, 144, 146, 148, and / or 152. Interfaces within the control computer system facilitate data communication between the control computer system 150 and systems 142, 144, 146, 148, and 152.

[0049] The memory may also include other instructions, including instructions for sending data, instructions for receiving data, instructions for interaction, or instructions for controlling the drive system 140, sensor system 144, control system 146, or user interface system 148.

[0050] In addition to storing processing instructions, the memory can store various types of information or data, such as image processing parameters, road maps, and route information. This information can be used by the vehicle 100 and the control computer system 150 while the vehicle 100 is operating in automatic, semi-automatic, and / or manual mode.

[0051] Although the autonomous driving control unit is shown as separate from the processor and memory, it should be understood that in some embodiments, some or all of the functions of the autonomous driving control unit may be implemented using program code instructions residing in one or more memories (or data storage devices) and executed by one or more processors, and in some cases, the autonomous driving control unit may be implemented using the same processor and / or memory (or data storage device). In some embodiments, the autonomous driving control unit may be implemented at least in part using various special-purpose circuit logics, various processors, various field-programmable gate arrays (“FPGAs”), various application-specific integrated circuits (“ASICs”), various real-time controllers, and hardware.

[0052] The control computer system 150 can control the functions of the vehicle 100 based on inputs received from various vehicle systems (e.g., drive system 142, sensor system 144, and control system 146) or from the user interface system 148. For example, the control computer system 150 can use inputs from the control system 146 to control the steering unit to avoid obstacles detected by the sensor system 144. In one embodiment, the control computer system 150 can be used to control multiple aspects of the vehicle 100 and its systems.

[0053] Although Figure 1 The diagram shows various components (or units) integrated into vehicle 100, one or more of which may be mounted on or separately associated with vehicle 100. For example, a control computer system may exist partially or entirely independent of vehicle 100. Thus, vehicle 100 can exist as separate or integrated device units. The device units constituting vehicle 105 can communicate with each other via wired or wireless communication. In some embodiments, additional components or units may be added to or removed from various systems (e.g., ...). Figure 1 (LiDAR or radar shown).

[0054] As mentioned earlier, existing state estimation methods are mainly based on the multi-object tracking paradigm of detection and tracking. This requires first extracting bounding boxes in each single-frame point cloud using object detection methods, and then matching these bounding boxes across multiple frames. This approach heavily relies on the quality of the single-frame point cloud and the object detection model, lacking robustness when point cloud data is sparse. Furthermore, it requires deep learning training to develop a strong cognitive ability regarding the shape of target objects, and cannot be extended to object shape categories not covered in the training set. Therefore, this disclosure aims to propose a state estimation scheme that is more robust and has a wider range of applications.

[0055] like Figure 2As shown, this disclosure provides an object state estimation method 200, including:

[0056] Step S201: Calculate the estimated state of the object in the next point cloud frame based on the optimized state of the object in the current point cloud frame.

[0057] Step S202: Determine the estimated point cloud of the object in the next point cloud frame based on the estimated state of the object in the next point cloud frame.

[0058] Step S203: Optimize the predicted point cloud of the object in the next point cloud frame to obtain the optimized state of the object in the next point cloud frame.

[0059] The predicted state and the optimized state both include at least one of the object's position parameters and orientation parameters. The position parameters include at least one of the object's key points in a spatial coordinate system: a first coordinate, a second coordinate, and a third coordinate. The first, second, and third coordinates can correspond to the x-axis, y-axis, and z-axis coordinates in the spatial coordinate system. The orientation parameters include at least one of pitch angle, yaw angle, and roll angle. There can be one or more key points, such as the center point of the object, the center of a specific part of the object, or a set of these points. For example, for a vehicle, its key points could be the vehicle's center point, the center point of the front of the vehicle, the center points of the two rear wheels, or a set of multiple body points. This invention does not limit the number or location of these key points.

[0060] To enable those skilled in the art to better understand this disclosure, the embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings and examples. It is worth noting that object state estimation in the embodiments of this disclosure can refer to the online perception and detection of the surrounding environment by autonomous vehicles, intelligent robots, drones, etc., equipped with radar, and the identification of the position and orientation of objects of interest (such as vehicles, pedestrians, obstacles, etc., but not limited to these). It can also refer to the estimation of each frame in a point cloud sequence using computing devices in offline scenarios. Of course, there are many other scenarios that apply to object state estimation. It should be understood that any scenario where object state estimation can be applied may be applicable to the embodiments of this disclosure, and these embodiments will not be listed one by one here.

[0061] In one embodiment of this disclosure, such as Figure 3 As shown, an object state estimation method 300 is provided, including:

[0062] Step S301: Determine the initial frame of the point cloud sequence and the annotation state of the object in the initial frame. Set the initial frame as the current frame and use the annotation state as the optimized state of the object in the initial frame. The annotation state can be obtained through manual annotation or object detection results.

[0063] Here, point cloud data of the actual road conditions are collected using lidar, in the form of a point cloud sequence T1 to T of length L. L For the target object, a high-quality initial frame is first determined in the point cloud sequence. This initial frame can be a manually selected frame, the first frame of the point cloud sequence, or a frame where the target detection result reaches a predetermined confidence threshold. The target detection result and annotation status of the initial frame can be obtained through manual annotation or through a pre-trained target detection network; this disclosure does not impose any restrictions on this. Generally, in online scenarios, target detection algorithms can be used to filter detection results with higher confidence, while in offline application scenarios, manual annotation can be used.

[0064] The object detection results, or the annotation results obtained after annotation, can include the bounding box of the object in the initial frame and the initial state. The point cloud associated with the object is used as the initial point cloud, and the shape of the object is used as the initial shape. The bounding box can be a two-dimensional box (annotated with length and width) or a three-dimensional box (annotated with length, width, and height). Considering the object as a rigid object, its bounding box maintains a fixed size in the point cloud sequence. One objective of this application is to obtain the object's state (i.e., optimized state), the point cloud associated with the object (i.e., optimized point cloud), and the object's shape (i.e., optimized shape) in each frame of data after optimization. For consistent terminology, this application uses the annotated state in the initial frame as the optimized state of the object in the initial frame, the initial point cloud in the initial frame as the optimized point cloud of the object in the initial frame, and the initial shape in the initial frame as the optimized shape of the object in the initial frame. The initial shape is the initial model of the object in the point cloud sequence, which can be a two-dimensional model, preferably a three-dimensional model.

[0065] In fact, this disclosure only needs to obtain the target detection results or annotation results of the initial frame. Target detection is not required for other frames. The optimized state of the objects in each frame can be obtained solely based on the calculation of the point cloud data.

[0066] Step S302: Calculate the estimated state of the object in the next point cloud frame based on the optimized state of the object in the current point cloud frame.

[0067] In one implementation, the optimization state S of the object in the current point cloud frame is used as the basis. k and inter-frame displacement value ΔS prior The estimated state S of the calculated object in the next point cloud frame. k+1Let 'k' represent the point cloud in frame k. Specifically, the estimated state of the next point cloud frame = the optimized state in the current point cloud frame + inter-frame displacement value. Corresponding to the state parameters, the inter-frame displacement value also includes at least one of the position parameter change value and the orientation parameter change value. The position parameter change value includes at least one of the first coordinate change value, second coordinate change value, and third coordinate change value of the object's key points in the spatial coordinate system; the orientation parameter includes at least one of the pitch angle change value, yaw angle change value, and roll angle change value. The first coordinate change value, second coordinate change value, and third coordinate change value can correspond to the x-axis coordinate change value, y-axis coordinate change value, and z-axis coordinate change value of the object in the spatial coordinate system. Assuming the state parameters are represented as x, y, z, θ, the corresponding position parameter change values ​​are Δx, Δy, Δz, and Δθ.

[0068] If the current frame is the initial frame, there is limited reference information. Therefore, the inter-frame displacement value ΔS of the next frame relative to the initial frame is set. prior =0; If the current frame is not the initial frame, the optimized state of some frames has been obtained. Based on the optimized state of these known frames, the displacement value between each two frames can be obtained. Then, by weighted averaging, the inter-frame displacement value applied to the current point cloud frame can be obtained, which is the inter-frame displacement value of the next frame relative to the current frame. That is, the inter-frame displacement value is at least the average displacement value between the previous frames. The specific calculation process of this inter-frame displacement value will be detailed later.

[0069] Step S303: Determine the predicted point cloud of the object in the next point cloud frame based on the predicted state of the object in the next point cloud frame.

[0070] In one implementation, determining the estimated point cloud of an object in the next point cloud frame based on its estimated state can be achieved as follows: A first detection box is generated based on the estimated state of the object in the next point cloud frame, and point clouds related to the object are filtered in the next point cloud frame based on this first detection box as the estimated point cloud. Optionally, point cloud filtering is performed based on the object state and the length, width, and height information of the detection box. The filtering rule is: only points within the cuboid specified by the object state and the detection box are determined to be related to the object and retained. The object state can be an estimated state or an optimized state (described later). If it is an estimated state, a predicted point cloud is obtained; if it is an optimized state, an optimized point cloud is obtained. Generally, if the current frame is the initial frame, the first detection box is a third multiple of the bounding box, that is, the bounding box obtained in the initial frame is enlarged by a third factor to obtain the first detection box. In this case, there is relatively little reference information when estimating the second frame, so it is necessary to filter point clouds related to the object over a larger range. Optionally, the third multiple is set to 2-4 times, preferably 3 times, but it can also be set to other values, and this disclosure does not limit it.

[0071] Correspondingly, if the current frame is not the initial frame, the first detection box is the second multiple of the bounding box. Since there is already some reference information from the previous frame, it can be magnified by a slightly smaller factor, such as 1.5 times, but it is not limited to this.

[0072] Step S304: By optimizing the predicted point cloud in the next point cloud frame, the optimized state of the object in the next point cloud frame is obtained.

[0073] In one implementation, the optimized state of the object in the next point cloud frame is obtained by optimizing the estimated point cloud. This can be achieved by: determining an optimization frame set, which includes the next point cloud frame and at least one of the previous frames of the next point cloud frame; and then, optimizing the point cloud within the optimization frame set to obtain the optimized state of the object in the next point cloud frame.

[0074] It should be understood that before predicting the optimization state of the point cloud in the next frame, the optimization states of the object's point cloud in each previous frame are already known. Therefore, in the embodiments of this application, when optimizing the point cloud within the optimization frame set, the optimization state of the object in at least one previous frame is also updated to obtain the corresponding second or third optimization state, etc.

[0075] Here, the optimization frame set can be a sliding window set with a preset frame length, such as four frames. When calculating the optimization state of each next point cloud frame, the point cloud optimization is performed in conjunction with the three preceding frames. Therefore, while optimizing the next point cloud frame, the point clouds and states of the three preceding frames are also updated. Of course, if the current optimization frame set does not meet the preset frame length, such as when optimizing the third frame but there are only two preceding frames, then all existing preceding frames are included in the optimization frame set.

[0076] Generally, after obtaining the optimized state of the point cloud each time, the point cloud related to the object in that frame of data can be determined based on the optimized state, and this optimized point cloud is then used. Accordingly, the first optimized state has a corresponding first optimized point cloud, the second optimized state has a corresponding second optimized point cloud, and so on. This may involve multiple filtering iterations. When the maximum number of iterations is reached, the current optimized point cloud and optimized state are determined.

[0077] As shown in the table below, when the current frame is the initial frame, the optimized state is the labeled state. At this point, the optimization frame set only contains the initial frame and the frame following it. By optimizing the initial point cloud in the initial frame and the estimated point cloud in the next frame, the optimized state of the object in the next point cloud frame is obtained. Then, based on this initial state... and the current inter-frame displacement value ΔS priorCalculate the predicted state S2' of the second frame. By optimizing the point clouds of the first two frames, that is, optimizing the optimized point cloud of the first frame and the predicted point cloud of the second frame, we obtain the first optimized state S' of the object in the second frame. 21 This leads to the first optimized point cloud of the object in the second frame point cloud.

[0078]

[0079] Then, based on the object's first optimized state S in the second frame... 21 The predicted state S3' of the object in the third frame point cloud is calculated by optimizing the point clouds of the first three frames, i.e., optimizing the optimized point clouds of the first two frames and the predicted point cloud of the third frame, to obtain the first optimized state S of the object in the third frame. 31 At the same time, the second optimized state S of the object in the second frame will be obtained. 22 .

[0080] Similarly, based on the object's first optimized state S in the third frame... 31 The predicted state S4' of the object in the fourth frame is calculated. By optimizing the point cloud of the first four frames, the first optimized state S of the object in the fourth frame is obtained. 41 At the same time, the second optimized state S of the object in the third frame will be obtained. 32 and the object's third optimized state S in the second frame. 23 .

[0081] Subsequently, based on the object's first optimized state S in the fourth frame... 41 The predicted state S5' of the object in the fifth frame is calculated. By optimizing the point cloud in frames 2-5, the first optimized state S of the object in the fifth frame is obtained. 51 At the same time, the second optimized state S of the object in the fourth frame will be obtained. 42 and the object's third optimization state S in the third frame. 33 and the object's fourth optimization state S in the second frame. 24 .

[0082] This process continues until the last frame of the point cloud sequence is optimized. Assuming a preset set of n optimization frames, each frame (excluding the initial and last few frames) will participate in n point cloud optimizations, resulting in n optimized states. After each optimization, the latest optimized point cloud for each frame is obtained, which can then be used to participate in point cloud optimization within subsequent frames. Generally, online scenarios have higher real-time requirements for data; therefore, online scenarios can output the first optimization state of each frame, while offline scenarios can output the last updated optimization state and optimized point cloud for each frame.

[0083] Furthermore, considering the relatively high confidence level of the initial frame data, it can be included in each set of optimized frames to optimize other point clouds within the set. When an initial frame is present in the set of optimized frames, its annotation state and initial point cloud can be updated. However, considering the high confidence level of the initial frame data, its annotation state and initial point cloud may not need to be updated.

[0084] According to one embodiment, optimizing the point cloud within the optimization frame set can specifically involve: optimizing the point cloud related to the object within the optimization frame set, calculating the optimization loss function, and determining the object's optimization state in the next point cloud frame based on the loss function. The optimization of the object state comes from minimizing the loss function, which includes, but is not limited to, the following constraints:

[0085] 1) Constraint on the change of inter-frame displacement values ​​between multiple consecutive point cloud frames: that is, the change of inter-frame displacement values ​​between multiple consecutive frames cannot exceed a predetermined threshold.

[0086] 2) Constraints on the height difference between the object and the ground in the initial frame and the next point cloud frame: This means the height difference between the object and the ground in the initial frame and the next point cloud frame cannot exceed a predetermined threshold. Specifically, the ground point cloud is first determined, and the ground height is determined based on this ground point cloud. The height of the object above the ground is then determined based on the point cloud associated with the object. The point cloud associated with the object can be either a predicted point cloud or an optimized point cloud, depending on whether the frame has been optimized.

[0087] 3) At least one registration distance constraint for the optimized point cloud between previous frames: such that the registration distance between any two previous frames cannot exceed a predetermined threshold.

[0088] 4) Consistency constraint of object orientation and direction of movement

[0089] 5) At least one registration distance constraint between the optimized point cloud in the previous frame and the predicted point cloud in the next frame: such as satisfying the consistency constraint after registration between the previous frame and the next frame, which may include:

[0090] a) First registration distance constraint between the optimized point cloud of the current point cloud frame and the predicted point cloud of the next point cloud frame.

[0091] b) Second registration distance constraint between the point cloud within the optimized shape of the initial point cloud frame and the predicted point cloud of the next point cloud frame.

[0092] c) Third registration distance constraint between the point cloud within the optimized shape of the current point cloud frame and the predicted point cloud of the next point cloud frame.

[0093] Each of the above constraints has its corresponding weight. By weighting these constraints, the loss function can be obtained, and this loss function can be used to optimize the point cloud within the set of optimization frames. The calculation formulas for some of the constraints are as follows:

[0094]

[0095]

[0096]

[0097]

[0098] Among them, O k-1 The optimized point cloud representing the point cloud in frame k-1. This represents the predicted point cloud of the k-th frame. Representing O k-1 and The registration point set between them representative point set The number of points in (ΔS) k ,S k-1 )⊙p represents moving point p from state S in frame k-1. k-1 Move ΔS k The distance is q, where q is the point registered by point p in the point cloud of the kth frame, and ||||2 represents the L2 norm.

[0099] M1 represents the optimized shape of the point cloud in the initial frame. Representing M1 and The registration point set between them This represents moving point p from the initial state of the point cloud in the initial frame. move The distance S k This represents the optimized state of the point cloud in frame k. M k-1 This represents the optimized shape of the point cloud in frame k-1. Representing M k-1 and The registration point set. v is the inter-frame displacement value of the point cloud from frame k-1 to frame k, Δ xk Let θ be one of the coordinate displacement values. k-1 and θ k These are the rotation angles of the point cloud in frame (k-1) and frame (k), respectively.

[0100] Step S305: Determine the optimized point cloud of the object in the next point cloud frame based on the optimized state of the object in the next point cloud frame, and determine the optimized point cloud of the object in at least one previous frame based on the optimized state of the object after being updated in at least one previous frame.

[0101] In one implementation, the optimized point cloud of the object in the next point cloud frame, based on the object's optimized state, can be achieved as follows: A second detection box is generated based on the object's optimized state in the next point cloud frame; point clouds related to the object are selected in the next point cloud frame based on the second detection box, and these are used as the object's optimized point cloud. The first detection box is greater than or equal to the second detection box, and the second detection box is a first multiple of the object's bounding box (referring to the previous statement, the third multiple > the second multiple > the first multiple ≥ 1). Optionally, the first multiple is set to 1-1.5 times, preferably 1.1 times, but it is not limited to this.

[0102] Step S306: Based on the optimized state of the object in the current point cloud frame and the next point cloud frame, calculate the displacement value of the object between the two point cloud frames, and update the inter-frame displacement value based on the calculated displacement value, so as to calculate the estimated state of the object in other point cloud frames based on the updated inter-frame displacement value.

[0103] This mainly involves updating the motion model of the object, specifically the inter-frame displacement values. This embodiment utilizes the vehicle's motion model to maintain a motion observation value, representing a prediction of the motion in the next frame. In actual computation, a moving average method is used to maintain this value; whenever a new inter-frame motion is predicted, the inter-frame displacement value is automatically updated. Specifically, the inter-frame displacement value can be updated based on the next point cloud frame and at least one of its optimized states in a previous frame. This at least one previous frame can be only a frame within the optimized frame set, or it can include frames outside the optimized frame set, or even all previous frames.

[0104] In one implementation, the optimized state of the previous frame is the first optimized state, and it is assumed that the inter-frame displacement value of the point cloud in the (k-1)th frame is ΔS. prior The optimized states of the (k-1)th frame and the kth frame are S respectively. k-1 and S k Then the inter-frame displacement corresponding to the k-th frame, which is the updated inter-frame displacement value, is ΔS. prior =αΔS prior +(1-α)(S k -S k-1 ).

[0105] In another implementation, if the optimized state of the previous frame is the updated optimized state, the inter-frame displacement value between each two frames can be recalculated based on the updated optimized state, and the displacement value between each two frames can be weighted and averaged to obtain the updated inter-frame displacement value.

[0106] It should be noted that there is no specific order between steps S305 and S306. Step S305 can be executed first, or step S306 can be executed first, or both can be executed simultaneously. This disclosure does not impose any restrictions on this.

[0107] Step S307: Update the current point cloud frame to the next point cloud frame, and return to step S302 to continue calculating the optimized state of the next point cloud frame based on the updated current point cloud frame, until the optimized state of the last point cloud frame is calculated. Here, it is first determined whether the last point cloud frame has been predicted. If not, the current point cloud frame is updated to the next point cloud frame.

[0108] In addition to updating the motion model of an object, this disclosure also provides a novel method for updating the 3D point cloud model of an object. Existing 3D reconstruction methods are based on supervised learning, using a deep neural network trained on a CAD model to complete a sparse point cloud into a dense point cloud corresponding to the CAD model. In practical applications, a deep neural network is used to complete the sparse point cloud of each frame into a dense point cloud. This method requires additional CAD models as training data, has poor scalability, and cannot extend to shape categories that are not present in the dataset. Moreover, this scheme is mainly based on single-frame point cloud data, so when the single-frame point cloud is sparse and does not contain sufficient information, it cannot produce high-quality 3D reconstruction.

[0109] Therefore, this disclosure provides a method for achieving high-quality 3D reconstruction using multi-frame, multi-view information from a point cloud sequence, without requiring supervisory data or relying heavily on single-frame point cloud data.

[0110] In one implementation, the optimized shape of the object in the next point cloud frame is determined based on the optimized shape of the object in at least one previous frame and the optimized point cloud of the object in the next point cloud frame. The optimized shape of the initial frame is the initial shape determined from the object detection results of the initial frame.

[0111] This disclosed algorithm, based on the estimated object state and observed point cloud data for each frame, superimposes the point clouds from different frames into a dense point cloud model, and obtains a high-quality 3D reconstruction of the object at the end of the algorithm. The reconstruction effect for vehicles is as follows: Figure 4 As shown in the second column, it can be seen that it can realistically reproduce the 3D form of a vehicle.

[0112] It should be noted that the current point cloud frame and the next point cloud frame can be two adjacent frames in the point cloud sequence, or two frames separated by a predetermined frame order. Object state optimization can be performed every frame, while 3D reconstruction of the object can be performed at predetermined frame intervals. Therefore, before determining the optimized shape of the object in the next point cloud frame, it is necessary to first determine whether the next point cloud frame falls within the frame order of the preset optimization period. If so, the optimized shape of the object in the next point cloud frame is then determined. The frame order of the optimization period is counted from the initial frame; for example, 3D reconstruction is performed every ten frames. In a practical implementation, the 3D model can be updated by overlaying point clouds every second according to the time relationship.

[0113] In summary, this disclosure starts with an initial frame and uses an iterative optimization algorithm to estimate the state of the object in the point cloud of each frame. Key frames are then selected to overlay the point cloud and reconstruct the 3D model. In the estimation of each frame, the predicted state is first determined, followed by iterative filtering of the point cloud and optimization of the state. After optimization, the optimized state is output, and the motion model and 3D model of the object are updated.

[0114] In addition, such as Figure 5 As shown in the figure, this disclosure also provides an object state estimation device 500, including a state prediction module 501, a point cloud prediction module 502, and a state optimization module 503. Wherein,

[0115] The state prediction module 501 calculates the predicted state of the object in the next point cloud frame based on the optimized state of the object in the current point cloud frame. Optionally, the state prediction module 501 calculates the predicted state of the object in the next point cloud frame based on the optimized state of the object in the current point cloud frame and the inter-frame displacement value.

[0116] The point cloud prediction module 502 determines the predicted point cloud of the object in the next point cloud frame based on the predicted state.

[0117] The state optimization module 503 obtains the optimized state of the object in the next point cloud frame by optimizing the estimated point cloud.

[0118] In one implementation, the state optimization module 503 determines an optimization frame set, which includes the next point cloud frame and at least one preceding frame of the next point cloud frame. By optimizing the point cloud within the optimization frame set, the optimization state of the object in the next point cloud frame is obtained, and the optimization state of the object in at least one preceding frame is updated. Optionally, the state optimization module 503 optimizes the point cloud related to the object within the optimization frame set, calculates an optimization loss function, and determines the optimization state of the object in the next point cloud frame based on the loss function.

[0119] like Figure 6 As shown, this embodiment of the present disclosure provides another object state estimation device 600. Device 600 is an optimized version of device 500, including a state prediction module 601, a point cloud prediction module 602, a state optimization module 603, a point cloud optimization module 604, an update module 605, and a shape optimization module 606. The state prediction module 601, point cloud prediction module 602, and state optimization module 603 are implemented in the same way as the state prediction module 501, point cloud prediction module 502, and state optimization module 503 in device 500, and will not be described in detail here.

[0120] The point cloud optimization module 604 determines the optimized point cloud of the object in the next point cloud frame based on the optimization state of the object in the next point cloud frame; and determines the optimized point cloud of the object in at least one previous frame based on the updated optimization state of the object in at least one previous frame.

[0121] In one implementation, the point cloud optimization module 604 generates a second detection box based on the optimized state of the object in the next point cloud frame; and selects point clouds related to the object in the next point cloud frame based on the second detection box as optimized point clouds. Specifically, the first detection box is greater than or equal to the second detection box, and the second detection box is a first multiple of the object's bounding box, which is determined from the target detection results of the initial frame, with the first multiple being greater than or equal to 1. Furthermore, if the current point cloud frame is not the initial frame, the inter-frame displacement value is at least one average displacement value between previous frames, the first detection box is a second multiple of the bounding box, and the second multiple is greater than the first multiple; if the current point cloud frame is the initial frame, the inter-frame displacement value is 0, the first detection box is a third multiple of the bounding box, and the third multiple is greater than the second multiple.

[0122] The update module 605 calculates the displacement value of the object between the current point cloud frame and the next point cloud frame based on the optimized state of the object, and updates the inter-frame displacement value based on the calculated displacement value, so as to calculate the estimated state of the object in other point cloud frames based on the updated inter-frame displacement value.

[0123] The shape optimization module 606 determines the optimized shape of the object in the next point cloud frame based on the optimized shape of the object in at least one previous frame and the optimized point cloud of the object in the next point cloud frame. The optimized shape of the initial frame is the initial shape determined from the target detection results of the initial frame. Optionally, the shape optimization module first determines whether the next point cloud frame is within a preset optimization period; if so, it determines the optimized shape of the object in the next point cloud frame based on the optimized point cloud and the optimized shape of the object in the at least one previous frame.

[0124] It should be noted that the specific implementations of the object state estimation devices 500 and 600 provided in this disclosure embodiment differ in their methods of... Figures 1-4 The details have been disclosed in the description and will not be repeated here.

[0125] In addition, embodiments of this disclosure also provide a computer-readable storage medium including a program or instructions that, when run on a computer, implement the object state estimation method as described above.

[0126] In addition, this disclosure also provides an embodiment as follows: Figure 7The computing device 700 shown includes a memory 701 and one or more processors 702 communicatively connected to the memory. The memory 701 stores instructions executable by the one or more processors 702, which, when executed, cause the one or more processors 702 to implement the object state estimation method described above. The computing device 700 may further include a communication interface 703, which can implement one or more communication protocols (LTE, Wi-Fi, etc.).

[0127] According to the technical solution disclosed herein, the reliance on target detection is eliminated, and the state of the object is directly estimated; no additional prior model is introduced for the shape of the object, so that it can be extended to other categories; higher accuracy state estimation is obtained by directly processing point cloud data; high-quality 3D reconstruction is achieved through multi-frame and multi-view information of point cloud sequence, without the need for supervised data or strong dependence on single-frame point cloud data.

[0128] This disclosure can be used in online perception modules or offline analysis modules for autonomous driving. In online scenarios, this disclosure utilizes general-purpose computing devices (CPUs) to process radar point clouds to obtain estimates of the absolute state of surrounding vehicles, which are then supplied to downstream planning and perception modules. In offline scenarios, this disclosure provides 3D model information and motion state information of surrounding vehicles through offline analysis data, providing benchmark data for planning, fusion, and other parts of the system. It can also be used to create virtual complex radar scenarios.

[0129] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] This disclosure uses specific embodiments to illustrate the principles and implementation methods of this disclosure. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An object state estimation method, comprising: For each current point cloud frame in the point cloud sequence: Based on the optimized state of the object in the current point cloud frame, calculate the estimated state of the object in the next point cloud frame; Determine the estimated point cloud of the object in the next point cloud frame based on the estimated state; The optimized state of the object in the next point cloud frame is obtained by optimizing the estimated point cloud; wherein, both the estimated state and the optimized state include at least one of the object's position parameters and orientation parameters; Determining the estimated point cloud of the object in the next point cloud frame based on the estimated state includes: A first detection box is generated based on the estimated state, and point clouds related to the object are selected within the first detection box in the next point cloud frame as the estimated point cloud.

2. The method according to claim 1, wherein, When the current point cloud frame is the initial point cloud frame, the optimized state of the initial point cloud frame is the annotation state of the object in the initial point cloud frame, and the annotation state is obtained through manual annotation or target detection results.

3. The method according to claim 1, wherein, The current point cloud frame and the next point cloud frame are two adjacent frames in the point cloud sequence or two frames separated by a predetermined frame order.

4. The method according to claim 1, wherein, The optimized state of the object in the next point cloud frame is obtained by optimizing the estimated point cloud, including: Determine an optimized frame set, the optimized frame set including the next point cloud frame and at least one of the previous frames of the next point cloud frame; By optimizing the point cloud within the optimized frame set, the optimized state of the object in the next point cloud frame is obtained.

5. The method according to claim 4, further comprising: By optimizing the point cloud within the optimized frame set, the optimized state of the object in at least one previous frame is updated.

6. The method according to claim 5, further comprising: The optimized point cloud of the object in the next point cloud frame is determined based on the optimized state of the object in the next point cloud frame; as well as The optimized point cloud of the object after being updated in at least one previous frame is determined based on the optimized state of the object after being updated in at least one previous frame.

7. The method according to claim 4, wherein, Optimization is performed within the set of optimized frames, including: Within the set of optimization frames, the point cloud associated with the object is optimized, the loss function of the optimization is calculated, and the optimization state of the object in the next point cloud frame is determined based on the loss function.

8. The method according to claim 7, wherein, The loss function includes at least one of the following: Constraints on inter-frame displacement values ​​between multiple consecutive point cloud frames; Constraints on the change in the height difference between the object and the ground in the initial frame and the next point cloud frame; At least one registration distance constraint for the optimized point cloud between previous frames; The registration distance constraint between at least one optimized point cloud in the previous frame and the estimated point cloud in the next point cloud frame.

9. The method according to claim 1, wherein, Based on the optimized state of the object in the current point cloud frame, calculate the estimated state of the object in the next point cloud frame, including: Based on the optimized state of the object in the current point cloud frame and the inter-frame displacement value, calculate the estimated state of the object in the next point cloud frame.

10. The method according to claim 1, wherein, Based on the optimized state of the object in the next point cloud frame, the point cloud related to the object is determined as the optimized point cloud in the next point cloud frame, including: A second detection box is generated based on the optimized state of the object in the next point cloud frame; The second detection box is used to filter point clouds related to the object in the next point cloud frame, which are then used as the optimized point cloud.

11. The method according to claim 10, wherein, The first detection box is greater than or equal to the second detection box, the second detection box is a first multiple of the bounding box of the object, the bounding box of the object is determined from the target detection results of the initial frame, and the first multiple is greater than or equal to 1.

12. The method according to claim 10, wherein, If the current point cloud frame is not the initial frame, then the inter-frame displacement value is the average displacement value of the at least one frame in the previous frame, and the first detection box is a second multiple of the bounding box, and the second multiple is greater than the first multiple. If the current point cloud frame is the initial frame, then the inter-frame displacement value is 0, the first detection box is a third multiple of the bounding box, and the third multiple is greater than the second multiple.

13. The method according to claim 1, further comprising: Based on the optimized state of the object in the current point cloud frame and the next point cloud frame, calculate the displacement value of the object between the two point cloud frames; The inter-frame displacement value is updated based on the calculated displacement value, so as to calculate the estimated state of the object in other point cloud frames based on the updated inter-frame displacement value.

14. The method of claim 9, wherein, The position parameters include at least one of the first coordinates, second coordinates, and third coordinates of the key points of the object in the spatial coordinate system; The orientation parameters include at least one of pitch angle, yaw angle, and roll angle; The inter-frame displacement value includes at least one of the position parameter change value and the orientation parameter change value.

15. The method according to claim 14, wherein, The key point is the center point of the object or the center point of a specific part of the object.

16. The method of claim 5, further comprising: The optimized shape of the object in the next point cloud frame is determined by comparing the optimized shape of the object in at least one previous frame with the optimized point cloud of the object in the next point cloud frame. The optimized shape of the initial frame is the initial shape determined from the target detection results of the initial frame.

17. The method according to claim 16, wherein, Before determining the optimized shape of the object in the next point cloud frame based on the optimized point cloud and the optimized shape of the object in at least one previous frame, the method further includes: Determine whether the next point cloud frame is within the frame order of a preset optimization period. If so, determine the optimized shape of the object in the next point cloud frame based on the optimized shape of the optimized point cloud and the object in at least one previous frame; wherein the frame order of the optimization period is calculated from the initial frame.

18. An object state estimation device, comprising: The state prediction module is suitable for calculating the predicted state of the object in the next point cloud frame based on the optimized state of the object in the current point cloud frame for each current point cloud frame of the point cloud sequence. The point cloud prediction module is adapted to determine the predicted point cloud of the object in the next point cloud frame based on the prediction state. The state optimization module is adapted to obtain the optimized state of the object in the next point cloud frame by optimizing the estimated point cloud; The estimated state and the optimized state both include at least one of the object's position and orientation; The point cloud prediction module determines the predicted point cloud of the object in the next point cloud frame based on the prediction state, including: A first detection box is generated based on the estimated state, and point clouds related to the object are selected in the next point cloud frame based on the first detection box as the estimated point cloud.

19. A computing device, comprising: Processor, memory, and computer programs stored in memory and capable of running on the processor; When the processor runs the computer program, it performs the method according to any one of claims 1-17.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of claims 1-17.

21. A vehicle comprising the computing device as claimed in claim 19.

Images