A method, apparatus and device for perception data processing

Abstract

The application provides a method, device and equipment for processing perception data. The method comprises: obtaining perception data of a perception object at a current position; predicting data of a position of the perception object at a target time; and performing frame interpolation between the perception data of the perception object at the current position and the data of the position of the perception object at the target time. The application improves the fluency of displaying the motion of the perception object, and improves the authenticity of the displayed motion of the perception object and the user experience.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a method, apparatus, and device for processing sensory data. Background Technology

[0002] Vehicle-road cooperative technology is based on technologies such as wireless communication and sensing to acquire vehicle-road information, and through vehicle-to-vehicle and vehicle-to-infrastructure information interaction and sharing, it realizes intelligent collaboration and cooperation between vehicles and infrastructure, thereby achieving the goals of optimizing the use of system resources, improving road traffic safety, and alleviating traffic congestion.

[0003] In vehicle-road cooperative technology, vehicle-road cooperative sensors collect a large amount of data, such as video data, laser scanning data, microwave sensing data, etc. In other words, the sensors generally acquire the vehicle's perception data (such as the vehicle's specific location data), and then display the vehicle's perception data to truly reflect the vehicle's movement.

[0004] In technologies that reflect vehicle motion, the displayed vehicle motion is not as smooth as expected when the acquired perception data is displayed. In other words, the displayed vehicle motion does not accurately reflect the vehicle's motion in the real scene, affecting the realism of the displayed vehicle motion and the user experience. Summary of the Invention

[0005] In view of this, the present invention proposes a method, apparatus and device for processing perception data. The method for processing perception data proposed in the present invention solves the problem that when displaying the acquired perception data, the displayed vehicle movement is not as smooth as expected, that is, the displayed vehicle movement does not truly reflect the vehicle's movement in the real scene, affecting the authenticity of the displayed vehicle movement and the user experience.

[0006] To achieve the above objectives, one aspect of the present invention provides a method for processing sensing data, comprising: acquiring sensing data of a sensing object at its current location; predicting data of the sensing object's location at a target time; and performing frame interpolation between the sensing data of the sensing object at its current location and the data of its location at the target time.

[0007] In some embodiments, the method further includes fusing the interpolated frame data obtained after interpolation with the acquired sensing data of the next position, so that the position of the sensing object is continuous.

[0008] In some embodiments, the method further includes: obtaining sample data from the sensing data for predicting the location of the sensing object.

[0009] In some embodiments, the step of predicting the location of the sensing object at a target time includes: predicting the location of the sensing object at a target time based on a prediction function and sample data used to predict the sensing object.

[0010] In some embodiments, the step of obtaining sample data for position prediction of the perceived object from the perception data includes: obtaining sample data for position prediction of the perceived object from the perception data based on the scene in which the perceived object is located in the current frame.

[0011] In some embodiments, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene in which the sensing object is located in the current frame includes: in response to the sensing object being in a stationary state in the previous frame of the current frame, obtaining data from the sensing data of several frames starting from the current frame and proceeding backward as sample data for position prediction of the sensing object.

[0012] In some embodiments, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene where the sensing object is located in the current frame further includes: in response to the sensing object being stationary in the next frame after the current frame and in motion in the previous frame of the current frame, obtaining data from the sensing data of several frames forward from the current frame as sample data for position prediction of the sensing object.

[0013] In some embodiments, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene where the sensing object is located in the current frame further includes: in response to the sensing object being in motion in both the previous and next frames of the current frame, obtaining data from the sensing data for several frames forward and several frames backward with the current frame as the threshold as sample data for position prediction of the sensing object.

[0014] In some embodiments, the method further includes: in response to the number of frames of the sensing data of the sensing object that are intermittently lost, determining the size of the number of frames intermittently lost by the sensing object and a preset threshold, and determining whether to supplement the number of frames intermittently lost by the sensing object based on the determination result.

[0015] In some embodiments, the step of determining whether to compensate for the number of frames intermittently lost by the sensing object based on the judgment result includes: compensating for the number of frames intermittently lost by the sensing object in response to the number of frames intermittently lost by the sensing object being less than the preset threshold.

[0016] In another aspect, the present invention provides a sensor data processing apparatus, comprising: a first module configured to acquire sensor data of a sensor object at its current location; a second module configured to predict data of the sensor object's location at a target time; and a third module configured to perform frame interpolation between the sensor data of the sensor object at its current location and the data of its location at the target time.

[0017] In some embodiments, the apparatus further includes a fourth module configured to fuse the interpolated frame data obtained after interpolation with the acquired sensing data of the next position, so that the position of the sensing object is continuous.

[0018] In some embodiments, the apparatus further includes a fifth module configured to acquire sample data from the sensing data for location prediction of the sensing object.

[0019] In some embodiments, the second module is further configured to: predict the position of the sensing object at a target time based on a prediction function and sample data for predicting the sensing object.

[0020] In some embodiments, the fifth module is further configured to: obtain sample data from the perception data for position prediction of the perception object based on the scene in which the perception object is located in the current frame.

[0021] In some embodiments, the fifth module is further configured to: in response to the sensing object being stationary in the previous frame of the current frame, acquire data from the sensing data for several frames starting from the current frame and proceeding backward as sample data for position prediction of the sensing object.

[0022] In some embodiments, the fifth module is further configured to: in response to the sensing object being stationary in the frame following the current frame and in motion in the frame preceding the current frame, acquire data from the sensing data for several frames prior to the current frame as sample data for position prediction of the sensing object.

[0023] In some embodiments, the fifth module is further configured to: in response to the sensing object being in motion in both the previous and next frames of the current frame, acquire data from the sensing data for several frames forward and several frames backward, with the current frame as the critical point, as sample data for position prediction of the sensing object.

[0024] In some embodiments, the apparatus further includes: a sixth module configured to respond to the number of frames of intermittently lost sensing data of the sensing object, determine the size of the number of intermittently lost frames of the sensing object and a preset threshold, and determine whether to supplement the number of intermittently lost frames of the sensing object based on the determination result.

[0025] In some embodiments, the sixth module is further configured to: in response to the number of frames intermittently lost by the sensing object being less than the preset threshold, perform frame supplementation for the number of frames intermittently lost by the sensing object.

[0026] In another aspect of the present invention, a computer device is also provided, including at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of any of the methods described above.

[0027] In another aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements any of the method steps described above.

[0028] The present invention has at least the following beneficial effects: The present invention proposes a method, apparatus and device for processing sensing data. The method for processing sensing data proposed in the present invention performs frame interpolation on the sensing data of the sensing object (such as a vehicle) at its current position and the data of the predicted target position at that time, increasing the number of frames within a preset time (usually in seconds), avoiding the problem of missing frames in the movement position of the sensing object, improving the smoothness of displaying the movement of the sensing object, thereby improving the realism of the displayed movement of the sensing object and the user experience. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0030] Figure 1 The flowchart shown is a method for processing sensory data provided in an embodiment of the present invention;

[0031] Figure 2 The diagram shown is a structural schematic of a sensor data processing device provided in an embodiment of the present invention;

[0032] Figure 3 This diagram illustrates the structure of a computer device according to an embodiment of the present invention.

[0033] Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention is shown. Detailed Implementation

[0034] The following describes embodiments of the present invention. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms.

[0035] Furthermore, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements may include not only those elements but also elements not expressly listed or inherent to such process, method, article, or apparatus.

[0036] One or more embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Based on the above objectives, the first aspect of the present invention provides an embodiment of a method for processing sensing data. Figure 1 The diagram shown is a schematic representation of an embodiment of a perceptual data processing method provided by the present invention. Figure 1 As shown, a method for processing sensory data according to an embodiment of the present invention includes the following steps:

[0038] S1. Obtain the perception data of the perceived object at its current location;

[0039] S2. Data predicting the position of the sensed object at the target time;

[0040] S3. Perform frame interpolation between the sensing data of the sensing object at its current position and the data of its position at the target time.

[0041] According to several embodiments of this application, the method further includes: fusing the interpolated frame data obtained after interpolation with the acquired sensing data of the next position, so that the position of the sensing object is continuous.

[0042] According to several embodiments of this application, the method further includes: obtaining sample data from the sensing data for predicting the location of the sensing object.

[0043] According to several embodiments of this application, the step of predicting the location of a sensed object at a target time includes: predicting the location of the sensed object at the target time based on a prediction function and sample data for predicting the sensed object.

[0044] According to several embodiments of this application, the step of obtaining sample data for position prediction of a perceived object from the perception data includes: obtaining sample data for position prediction of the perceived object from the perception data based on the scene in which the perceived object is located in the current frame.

[0045] According to several embodiments of this application, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene where the sensing object is located in the current frame includes: in response to the sensing object being in a stationary state in the previous frame of the current frame, obtaining data from the sensing data of several frames starting from the current frame and proceeding backward as sample data for position prediction of the sensing object.

[0046] According to several embodiments of this application, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene where the sensing object is located in the current frame further includes: in response to the sensing object being stationary in the next frame after the current frame and in motion in the previous frame of the current frame, obtaining data from the sensing data of several frames forward from the current frame as sample data for position prediction of the sensing object.

[0047] According to several embodiments of this application, the step of obtaining sample data for position prediction of the sensing object from the sensing data based on the scene where the sensing object is located in the current frame further includes: in response to the sensing object being in motion in both the previous frame and the next frame of the current frame, obtaining data from the sensing data for several frames forward and several frames backward with the current frame as the threshold as sample data for position prediction of the sensing object.

[0048] According to several embodiments of this application, the method further includes: responding to the number of frames of the sensing data of the sensing object that are intermittently lost, determining the size of the number of frames intermittently lost by the sensing object and a preset threshold, and determining whether to supplement the number of frames intermittently lost by the sensing object based on the determination result.

[0049] According to several embodiments of this application, the step of determining whether to supplement the number of frames intermittently lost by the sensing object based on the judgment result includes: in response to the number of frames intermittently lost by the sensing object being less than a preset threshold, supplementing the number of frames intermittently lost by the sensing object.

[0050] The following is another embodiment of a method for processing sensory data provided by the present invention.

[0051] In this embodiment, the perception data of the target vehicle on the road is first acquired, such as the vehicle's own perception data, including the vehicle's position data, direction of movement, and speed. Based on the vehicle's current actual position data, the target vehicle's position data (including position data and speed direction) is predicted. Predictive frame interpolation is performed between the current actual position data and the predicted target position data according to the principle of trajectory fitting. In this embodiment, linear fitting is used to predict the vehicle's motion at the target time. Linear fitting is a form of curve fitting. Let x and y be observed quantities, and y be a function of x: y = f(x; b). Curve fitting seeks the best estimate of parameter b using the observed values ​​of x and y, i.e., seeking the best theoretical curve y = f(x; b). When the function y = f(x; b) is a linear function of b, this curve fitting is called linear fitting. Specifically, the linear fitting used in this implementation is second-order linear fitting. For the same vehicle, the prediction sample data is obtained from several frames of the vehicle's perception data. The function for second-order linear fitting is G(t), which is a function of time, and G(t) represents the vehicle's position at time t. The sample data depends on the scene in the current frame of the vehicle's perception data. When the vehicle was stationary in the previous frame, the sample data is taken from the current frame as the starting point and a certain number of sample data are taken forward as the trajectory fitting sample data. When the vehicle was in motion in the previous frame and several frames in the future, the sample data is taken from several frames forward and backward as the trajectory fitting sample data, with the current frame as the critical point. When the vehicle was in motion in the previous frame and stationary for several frames in the next few frames, the current frame is the endpoint and several frames forward are taken as the trajectory fitting sample data. In this system, the current frame is used as the reference frame, the previous frame is the frame before the current frame in chronological order, and the next frame is the frame after the current frame in chronological order. Taking several frames forward from the current frame (using the current frame as the reference frame and taking several frames before the current frame) and several frames backward from the current frame (using the current frame as the reference frame and taking several frames in the future time frame) allows for dynamic acquisition of trajectory fitting sample data based on the different scenarios of the current frame in the vehicle's perception data. Based on this trajectory fitting function, the current position data is continuously updated, and the vehicle's position and orientation at the target time are continuously predicted in the above manner, with frame interpolation performed. Simultaneously, if there are intermittent frame losses in the vehicle's perception data, the number of intermittently lost frames is compared to a preset threshold. When the number of lost frames is less than the preset threshold, the vehicle's position data for several future frames can be predicted based on the above trajectory fitting function, and frame interpolation performed. The interpolated frame data is then fused with the re-emerging perception data to achieve continuous perception object position data, increase the number of frames displayed per unit time, and improve the smoothness of displaying vehicle movement.

[0052] A second aspect of the present invention provides an apparatus for processing sensory data. Figure 2 The diagram shown is a structural schematic of a sensory data processing device provided by the present invention. Figure 2 As shown, the present invention provides a sensory data processing apparatus comprising: a first module 011 configured to acquire sensory data of a sensory object at its current position; a second module 012 configured to predict the position data of the sensory object at a target time; and a third module 013 configured to perform frame interpolation between the sensory data of the sensory object at its current position and the position data at the target time.

[0053] According to several embodiments of the present invention, the apparatus further includes: a fourth module configured to fuse the interpolated frame data obtained after interpolation with the acquired sensing data of the next position, so as to make the position of the sensing object continuous.

[0054] According to several embodiments of the present invention, the apparatus further includes: a fifth module configured to acquire sample data from the sensing data for predicting the location of the sensed object.

[0055] According to several embodiments of the present invention, the second module is further configured to: predict the position of the perceived object at a target time based on a prediction function and sample data for predicting the perceived object.

[0056] According to several embodiments of the present invention, the fifth module is further configured to: obtain sample data for position prediction of the perceived object from the perceived data based on the scene in which the perceived object is located in the current frame.

[0057] According to several embodiments of the present invention, the fifth module is further configured to: in response to the sensing object being stationary in the previous frame of the current frame, acquire data from the sensing data of several frames starting from the current frame and proceeding backward as sample data for position prediction of the sensing object.

[0058] According to several embodiments of the present invention, the fifth module is further configured to: in response to the sensing object being stationary in the next frame after the current frame and in motion in the previous frame of the current frame, acquire data from the sensing data of several frames forward from the current frame as sample data for position prediction of the sensing object.

[0059] According to several embodiments of the present invention, the fifth module is further configured to: in response to the sensing object being in motion in both the previous frame and the next frame of the current frame, acquire data from the sensing data for several frames forward and several frames backward, with the current frame as the critical point, as sample data for position prediction of the sensing object.

[0060] According to several embodiments of the present invention, the device further includes: a sixth module configured to respond to the number of frames lost intermittently in the sensing data of the sensing object, determine the size of the number of frames lost intermittently in the sensing object and a preset threshold, and determine whether to supplement the number of frames lost intermittently in the sensing object based on the determination result.

[0061] According to several embodiments of the present invention, the sixth module is further configured to: in response to the number of frames intermittently lost by the sensed object being less than a preset threshold, perform frame supplementation for the number of frames intermittently lost by the sensed object.

[0062] To achieve the above objectives, a third aspect of the present invention provides a computer device. Figure 3 The diagram shown is a structural schematic of a computer device provided by the present invention. Figure 3 As shown, an embodiment of a computer device provided by the present invention includes the following modules: at least one processor 021; and a memory 022, the memory 022 storing computer instructions 023 that can be executed on the processor 021, the computer instructions 023 implementing the steps of the method described above when executed by the processor 021.

[0063] The present invention also provides a computer-readable storage medium. Figure 4 The diagram shown is a structural schematic of a computer-readable storage medium provided by the present invention. Figure 4 As shown, computer-readable storage medium 031 stores a computer program 032 that, when executed by a processor, performs the steps of the method described above.

[0064] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for setting system parameters can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0065] Furthermore, the method disclosed in the embodiments of the present invention can also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. When the computer program is executed by the processor, it performs the functions defined in the method disclosed in the embodiments of the present invention.

[0066] Furthermore, the above-described method steps and system units can also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to perform the functions of the above-described steps or units.

[0067] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0068] In one or more exemplary designs, functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer. By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store the required program code in the form of instructions or data structures and is accessible to a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection may be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the aforementioned coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0069] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0070] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0071] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0072] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0073] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for processing sensory data, characterized in that, include: Acquire sensory data of the object at its current location; Data to predict the position of the sensed object at the target time; The sensing data of the sensing object at its current position and the data at its position at the target time are interpolated. Based on the different scenes in which the current frame is located in the perception data, sample data for trajectory fitting is obtained, the position data at the current moment is updated, the position and orientation at the target moment are predicted, and frame interpolation is performed; wherein, the sample data for trajectory fitting depends on the scene in which the current frame is located in the perception data. When the motion state of the current frame is stationary, a certain number of sample data are taken from the current frame as the starting point and used as the sample data for trajectory fitting. When the previous frame and several future frames are in motion, the current frame is used as the critical point, and data from several frames forward and backward are taken as sample data for trajectory fitting. When the previous frame is in motion and the next few frames are in a stationary state, take the current frame as the endpoint and take several frames forward as trajectory fitting sample data. In response to the number of frames of intermittently lost sensing data of the sensing object, the number of intermittently lost frames of the sensing object is compared with a preset threshold, and the determination result is used to determine whether to fill in the intermittently lost frames of the sensing object. The interpolated frame data obtained after interpolation is fused with the sensing data of the next position to make the position of the sensing object continuous.

2. The method according to claim 1, characterized in that, The method further includes: Sample data for predicting the location of the perceived object is obtained from the perceived data.

3. The method according to claim 2, characterized in that, The step of predicting the location of the sensed object at the target time includes: Data that predicts the position of the perceived object at the target time based on a prediction function and sample data used to predict the perceived object.

4. The method according to claim 2, characterized in that, The step of obtaining sample data for location prediction of the sensed object from the sensed data includes: Based on the scene in which the perceived object is located in the current frame, sample data for predicting the location of the perceived object is obtained from the perceived data.

5. The method according to claim 1, characterized in that, The step of determining whether to compensate for the intermittently lost frames of the perceived object based on the judgment result includes: In response to the fact that the number of frames intermittently lost by the sensing object is less than the preset threshold, frames intermittently lost by the sensing object are supplemented.

6. A device for processing sensing data, characterized in that, The device includes: The first module configures the acquisition of sensing data of the object at its current location; The second module configures data for predicting the position of the sensed object at a target time; and The third module is configured to perform frame interpolation between the sensing data of the sensing object at its current location and the data at its location at the target time. And the modules configured for the following functions: Based on the different scenes in which the current frame is located in the perception data, sample data for trajectory fitting is obtained, the position data at the current moment is updated, the position and orientation at the target moment are predicted, and frame interpolation is performed; wherein, the sample data for trajectory fitting depends on the scene in which the current frame is located in the perception data. When the motion state of the current frame is stationary, a certain number of sample data are taken from the current frame as the starting point and used as the sample data for trajectory fitting. When the previous frame and several future frames are in motion, the current frame is used as the critical point, and data from several frames forward and backward are taken as sample data for trajectory fitting. When the previous frame is in motion and the next few frames are in a stationary state, take the current frame as the endpoint and take several frames forward as trajectory fitting sample data. In response to the number of frames of intermittently lost sensing data of the sensing object, the number of intermittently lost frames of the sensing object is compared with a preset threshold, and the determination result is used to determine whether to fill in the intermittently lost frames of the sensing object. The interpolated frame data obtained after interpolation is fused with the sensing data of the next position to make the position of the sensing object continuous.

7. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-5.

Images