A sensor data processing method and apparatus, a computing device, and a storage medium

Abstract

The application relates to a sensor data processing method and device, a computing device and a storage medium, and is used for solving the inconsistency problem of sensor data streams transferred to an algorithm module in the prior art. The sensor data processing method comprises the following steps: receiving a plurality of data streams to be processed, each data stream corresponding to a sensor, and a data frame in the data stream carrying a generation timestamp; correcting the generation timestamp of each data frame to obtain a corrected timestamp; and determining the data frame to be sent in each data sending period from each data stream according to the corrected timestamp of each data frame and a preset data sending period.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a sensor data processing method and device, a computing device and a storage medium. BACKGROUND

[0002] In an autonomous driving system, multi-sensor fusion is an indispensable link in the algorithm process. These sensors include but are not limited to satellite positioning systems, inertial measurement units, image sensors, laser radar sensors, millimeter wave radar sensors, etc. Multi-sensor fusion algorithms usually require that the combined incoming data is generated at the same time and / or post-processed for the data generated at the same time to achieve the best algorithm effect. However, in actual application, due to the inconsistent and unpredictable delays of different sensors, such as the exposure time of an image sensor, the scanning time of a laser radar sensor rotating a circle, the data transmission time between a sensor and a server, etc., a scheme is needed to align multiple different input data streams. SUMMARY

[0003] The present application provides a sensor data processing method, device, computing device, storage medium and vehicle, which can align multiple sensor data streams and improve the accuracy of subsequent algorithm module calculation.

[0004] In a first aspect, the present application provides a sensor data processing method, comprising: receiving multiple data streams to be processed, each data stream corresponding to a sensor, and a data frame in the data stream carrying a generation timestamp; correcting the generation timestamp of each data frame to obtain a corrected timestamp; and determining, according to the corrected timestamp of each data frame and a preset data sending period, the data frame that should be sent in each data sending period from each data stream.

[0005] In a second aspect, the present application provides a sensor data processing device, comprising: a data frame receiving module configured to receive multiple data streams to be processed, each data stream corresponding to a sensor, and a data frame in the data stream carrying a generation timestamp; a timestamp correction module configured to correct the generation timestamp of each data frame to obtain a corrected timestamp; and a data frame alignment module configured to determine, according to the corrected timestamp of each data frame and a preset data sending period, the data frame that should be sent in each data sending period from each data stream.

[0006] In a third aspect, the present application provides a computing device, comprising one or more processors; and a memory storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the sensor data processing method according to the present application.

[0007] The application provides a computer readable storage medium in a fourth aspect, which stores a program, and the program realizes the sensor data processing method according to the application when executed by a processor.

[0008] The application provides a vehicle in a fourth aspect, which comprises the computing device described above.

[0009] According to the technical solution of the application, in the case that there are time errors of different sensors, system scheduling influences, and delays caused by data stream processing program calculation, the multiple data streams are aligned to the same trigger frame according to the generation time stamps of the received data streams, so that the data sent to the algorithm module is collected at the same time or is post-processed from the data collected at the same time, and the accuracy of the algorithm module calculation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0011] Figure 1 A schematic diagram of a vehicle 100 according to an example embodiment of the application is shown;

[0012] Figure 2 A flowchart of a sensor data processing method 200 according to an example embodiment of the application is shown;

[0013] Figure 3 A schematic diagram of a data buffer queue of an example of the application is shown;

[0014] Figure 4 A schematic diagram of the mapping relationship of a trigger frame in different data streams of an example of the application is shown;

[0015] Figure 5A A schematic diagram of a generation time stamp of an example of the application is shown;

[0016] Figure 5B A schematic diagram of a correction time stamp of an example of the application is shown;

[0017] Figure 6A A schematic diagram of a frame loss and frame error and high delay check of an example of the application is shown;

[0018] Figure 6B A schematic diagram of a high delay check of another example of the application is shown;

[0019] Figure 7A structural diagram of a sensor data processing apparatus 700 of an example embodiment of the present application is shown; and

[0020] Figure 8 A structural diagram of a computing device 800 according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present specification, those skilled in the art can make various modifications and changes, and all the technical solutions obtained by equivalent changes belong to the scope of protection of the present application.

[0022] In order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items or objects with basically the same function or role. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order.

[0023] The term "and / or" in the present document is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present document generally represents an "or" relationship between the front and rear associated objects.

[0024] The embodiments of the present application provide a sensor data processing method and device, related computing device, storage medium and vehicle, which can align the data generated at the same time and ensure the consistency of the data delivered to the algorithm module.

[0025] In order to make those skilled in the art better understand the present disclosure, some technical terms appearing in the embodiments of the present disclosure will be explained as follows:

[0026] Sensor time: the clock source recorded in the sensor hardware, which is usually synchronized with Wall Time (i.e. real time t 现实 or wall clock time) through satellite time service, etc. with an extremely small error between them.

[0027] Server time: the clock source recorded in the server hardware, which can be obtained by the software installed on the server hardware, and is usually synchronized with Wall Time (i.e. real time or wall clock time) through satellite time service, etc. with an extremely small error between them. The sensor time can be considered to be equal to the server time.

[0028] Hard trigger sensor: a sensor whose time is synchronized to a certain clock source, the error between them is very small, it has a fixed data generation frequency (or trigger frequency), in practical applications, it is usually 50Hz, 20Hz, 10Hz, etc. which can be divided by 1 second, and it will generate data at the beginning of 1 second.

[0029] Sensor data: data obtained by a sensor in conjunction with a sensor driver on a server.

[0030] Sensor driver: a program that can obtain the generation timestamp of each frame of data under normal circumstances, and the error between the timestamp and the actual data generation time is very small, whether it is directly reading the sensor signal to obtain the sensor time, directly taking the server time, or calculating / adjusting through an algorithm.

[0031] Data stream: a continuous string of data in frames.

[0032] Data stream processing program: a program that receives an input data stream, processes each frame to generate a result, takes the result as a new frame, and generates an output data stream.

[0033] Data timestamp: additional information of data, which is a time value generated by observing a certain clock source.

[0034] Generation timestamp: the timestamp attached to sensor data, usually generated by the sensor and obtained by the sensor driver, or directly generated by the sensor driver according to the server time. This timestamp is considered the data generation time, also known as the trigger timestamp.

[0035] Corrected timestamp: the timestamp obtained by correcting the generation timestamp.

[0036] Arrival timestamp: the time observed when the data arrives at the server.

[0037] Data delay: the difference between the arrival timestamp of a data frame and the generation timestamp. For data streams, delay is inevitable and is usually affected by data acquisition and / or calculation time of the data source (sensor, data stream processing program), transmission time, operating system scheduling, etc.

[0038] Trigger frame: a virtual frame, each data transmission cycle will pass the data to be transmitted in this cycle to the algorithm module as a data frame for calculation, and the end time of this data transmission cycle is also an algorithm trigger time, corresponding to a virtual trigger frame.

[0039] Figure 1is a schematic diagram of a vehicle 100 in which various techniques of the present application can be implemented. The vehicle 100 can be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawn mower, backhoe, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment, trolley, golf cart, train, trolley car, or other vehicle. The vehicle 100 can operate entirely or partially in an autonomous driving mode. The vehicle 100, in the autonomous driving mode, can control itself, e.g., the vehicle 100 can determine a current state of the vehicle and a current state of an environment in which the vehicle is located, determine a predicted behavior of at least one other vehicle in the environment, determine a trust level corresponding to a likelihood that the at least one other vehicle will perform the predicted behavior, and control the vehicle 100 itself based on the determined information. While in the autonomous driving mode, the vehicle 100 can operate without human interaction.

[0040] The vehicle 100 can include various vehicle systems, such as a drive system 142, a sensor system 144, a control system 146, a user interface system 148, a control computer system 150, and a communication system 152. The vehicle 100 can include more or fewer systems, and each system can include multiple units. Further, each system and unit of the vehicle 100 can be interconnected. For example, the control computer system 150 can be in data communication with one or more of the systems 142-148 and 152. Thus, one or more of the described functionalities of the vehicle 100 can be divided into additional functional components or physical components, or combined into fewer functional components or physical components. In further examples, additional functional components or physical components can be added to the example shown. Figure 1

[0041] The drive system 142 can include multiple operable components (or units) that provide motive power for the vehicle 100. In one embodiment, the drive system 142 can include an engine or motor, wheels, a transmission, electronics, and a power source (or power source). The engine or motor can be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other forms of engines or motors. In some embodiments, the engine can convert a power source into mechanical energy. In some embodiments, the drive system 142 can include multiple engines or motors. For example, a hybrid vehicle can include a gasoline engine and an electric motor, and other cases are possible.

[0042] ​The wheels of the vehicle 100 can be standard wheels. The wheels of the vehicle 100 can be wheels of various forms, including unicycle, bicycle, tricycle, or quadracycle forms, such as the four wheels of a car or truck. Other numbers of wheels are possible, such as six or more wheels. One or more of the wheels of the vehicle 100 can be operated to rotate in a direction different from the other wheels. The wheels can be at least one wheel that is fixedly connected to the transmission. The wheels can include a combination of metal and rubber, or other combinations of materials. The transmission can include units that are operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission can include a gear box, a clutch, a differential, and a drive shaft. The transmission can also include other units. The drive shaft can include one or more axles that are adapted to the wheels. The electronic system can include units for transmitting or controlling electronic signals of the vehicle 100. These electronic signals can be used to activate various lights, various servos, various electric motors, and other electronic actuators or controls in the vehicle 100. The power source can be an energy source that provides power, in whole or in part, to the engine or electric motor. That is, the engine or electric motor can convert the power source into mechanical energy. Illustratively, the power source can include gasoline, petroleum, petroleum-based fuels, propane, other compressed gas fuels, ethanol, fuel cells, solar panels, batteries, and other sources of electrical energy. The power source can additionally or alternatively include any combination of fuel tanks, batteries, capacitors, or flywheels. The power source can also provide energy to other systems of the vehicle 100.

[0043] The sensor system 144 can include a plurality of sensors for sensing information about the environment and conditions of the vehicle 100. For example, the sensor system 144 can include an inertial measurement unit (IMU), a global positioning system (GPS) transceiver, a RADAR unit, a laser rangefinder / LIDAR unit (or other distance measuring device), an acoustic sensor, and a camera or image capture device. The sensor system 144 can include a plurality of sensors for monitoring the vehicle 100 (e.g., an oxygen (O2) monitor, an oil gauge sensor, an engine oil pressure sensor, etc.). The sensor system 144 can also be configured with other sensors. One or more of the sensors included in the sensor system 144 can be individually actuated or collectively actuated to update the position, orientation, or both of the one or more sensors.

[0044] In some embodiments, each sensor collects data by hardware trigger or software trigger, different sensors have different trigger frequencies, that is, different data collection frequencies, and correspondingly have different data collection periods. For hardware trigger, the trigger source uses the second pulse signal sent by Novatel as the trigger source signal, adjusts according to the trigger frequency required by different sensors, generates a trigger signal and sends it to the corresponding sensor to trigger the corresponding sensor to collect data. Alternatively, the trigger frequency of the camera is 20HZ, the trigger frequency of the LIDAR is 1HZ or 10HZ, and the trigger frequency of the IMU is 100HZ, of course, not limited to this.

[0045] The IMU can include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing changes in position and orientation of the vehicle 100 based on inertial acceleration. The GPS transceiver can be any sensor for estimating the geographical position of the vehicle 100. For this purpose, the GPS transceiver can include a receiver / transmitter to provide position information of the vehicle 100 relative to the earth. It is noted that GPS is an example of a global navigation satellite system, and thus, in some embodiments, the GPS transceiver can be replaced by a Beidou satellite navigation system transceiver or a Galileo satellite navigation system transceiver. The radar unit can use radio signals to sense objects in the environment in which the vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit can also be used to sense the speed and direction of approach of objects to the vehicle 100. The laser rangefinder or LIDAR unit (or other distance measuring device) can be any sensor that uses laser light to sense objects in the environment in which the vehicle 100 is located. In one embodiment, the laser rangefinder / LIDAR unit can include a laser source, a laser scanner, and a detector. The laser rangefinder / LIDAR unit is configured to operate in a continuous (e.g., using heterodyne detection) or discontinuous detection mode. The camera can include a device for capturing a plurality of images of the environment in which the vehicle 100 is located. The camera can be a still image camera or a dynamic video camera.

[0046] The control system 146 is configured to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 can include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0047] The steering unit may be a combination of mechanisms for adjusting the forward direction of vehicle 100. A power control unit (e.g., a throttle) may be used to control the engine speed, thereby controlling the speed of vehicle 100. The braking unit may include a combination of mechanisms for decelerating vehicle 100. The braking unit may utilize friction to decelerate the vehicle in a standard manner. In other embodiments, the braking unit may convert the kinetic energy of the wheels into electrical current. The braking unit may also take other forms. The navigation unit may be any system that determines a driving path or route for vehicle 100. The navigation unit may also dynamically update the driving path as vehicle 100 travels. The control system 146 may also additionally or optionally include other components (or units) not shown or described.

[0048] User interface system 148 can be used to allow vehicle 100 to interact with external sensors, other vehicles, other computer systems, and / or the user of vehicle 100. For example, user interface system 148 may include standard visual display devices (e.g., plasma displays, liquid crystal displays (LCDs), touchscreen displays, head-mounted displays, or other similar displays), speakers or other audio output devices, microphones or other audio input devices. For example, user interface system 148 may also include navigation interfaces and interfaces for controlling the internal environment of vehicle 100 (e.g., temperature, fan, etc.).

[0049] Communication system 152 can provide vehicle 100 with a means of communicating with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, communication system 152 can communicate with one or more devices directly or through a communication network. Communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (e.g., CDMA, EVDO, GSM / GPRS) or 4G cellular communication (e.g., WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (e.g., using...). In some embodiments, the communication system 152 can communicate directly with one or more devices or other vehicles in the vicinity, for example, using infrared light. Or ZigBee. Other wireless protocols, such as various vehicular communication systems, are also within the scope of this application. For example, the communication system may include one or more Dedicated Short Range Communication (DSRC) devices, V2V devices, or V2X devices that conduct public or private data communication with vehicles and / or roadside stations.

[0050] The control computer system 150 can control some or all of the functions of the vehicle 100. An autonomous driving control unit in the control computer system 150 can be used to identify, evaluate, and avoid or navigate around potential obstacles in the environment in which the vehicle 100 is located. Generally, the autonomous driving control unit can be used to control the vehicle 100 without a driver, or to provide assistance to a driver controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from a GPS transceiver, radar data, LIDAR data, camera data, and data from other vehicle systems to determine a travel path or trajectory for the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in an autonomous driving mode.

[0051] The control computer system 150 can include at least one processor (which can include at least one microprocessor) that executes processing instructions (i.e., machine executable instructions) stored in a non-transitory computer readable medium (e.g., a data storage device or memory). The memory stores at least one machine executable instruction that, when executed by the processor, implements functionality including a map engine, a localization module, a perception module, a navigation or path module, and an autonomous control module. The map engine and the localization module are used to provide map information and localization information. The perception module is used to perceive things in the environment in which the vehicle is located based on information obtained from sensor systems and map information provided by the map engine. The navigation or path module is used to plan a travel path for the vehicle based on processing results of the map engine, the localization module, and the perception module. The autonomous control module inputs and parses decision information from the navigation or path module and other modules into control command outputs for the vehicle control system, and sends the control commands to corresponding components in the vehicle control system through a vehicle network (e.g., a vehicle internal electronic network system implemented through a CAN bus, a local area inter-network, a multimedia orientation system transmission, etc.) to achieve autonomous control of the vehicle; the autonomous control module can also obtain information of components in the vehicle through the vehicle network.

[0052] The control computer system 150 can also be multiple computing devices that distributedly control components or systems of the vehicle 100. In some embodiments, the memory can contain processing instructions (e.g., program logic) that are executed by the processor to implement various functions of the vehicle 100. In one embodiment, the control computer system 150 is capable of data communication with the systems 142, 144, 146, 148, and / or 152. Interfaces in the control computer system are used to facilitate data communication between the control computer system 150 and the systems 142, 144, 146, 148, and 152.

[0053] The memory can also include other instructions, including instructions for data transmission, instructions for data reception, instructions for interaction, or instructions for controlling the drive system 142, the sensor system 144, or the control system 146 or the user interface system 148.

[0054] In addition to storing processing instructions, the memory can store a variety of information or data, such as image processing parameters, road maps, and path information. These information can be used by the vehicle 100 and the control computer system 150 during operation of the vehicle 100 in an autonomous manner, a semi-autonomous manner, and / or a manual mode.

[0055] Although the autonomous driving control unit is shown as being separate from the processor and the memory, it should be understood that in some embodiments, some or all of the functionality of the autonomous driving control unit can be implemented with program code instructions that reside in one or more memories (or data storage devices) and are executed by one or more processors, and that the autonomous driving control unit can in some cases be implemented using the same processor and / or memory (or data storage device). In some embodiments, the autonomous driving control unit can be implemented at least partially using various specialized circuit logic, various processors, various field programmable gate arrays (FPGAs), various application specific integrated circuits (ASICs), various real-time controllers, and hardware.

[0056] The control computer system 150 can control the functions of the vehicle 100 in accordance with inputs received from various vehicle systems (e.g., the drive system 142, the sensor system 144, and the control system 146), or inputs received from the user interface system 148. For example, the control computer system 150 can use inputs from the control system 146 to control the steering unit to avoid obstacles detected by the sensor system 144. In one embodiment, the control computer system 150 can be used to control multiple aspects of the vehicle 100 and its systems.

[0057] Although Figure 1 The various components (or units) shown integrated into the vehicle 100 can be on-board the vehicle 100 or separately associated with the vehicle 100. For example, the control computer system can exist partially or entirely separately from the vehicle 100. Thus, the vehicle 100 can exist in the form of separate or integrated equipment units. The equipment units that make up the vehicle 105 can communicate with each other in a wired or wireless manner. In some embodiments, additional components or units can be added to or removed from the various systems (e.g., the LiDAR or radar shown). Figure 1

[0058] ​As described above, in order to realize multi-sensor fusion, a method is needed to align multiple different input data streams, and the aligned multiple data streams can be combined as a data combination. One method is to take one input data stream as a main data stream and other data streams as auxiliary data streams. When the main data stream reaches a frame, find a frame in all auxiliary data streams whose data timestamp is closest to the newly arrived data timestamp of the main data stream, and take it as a data combination. However, due to inconsistent delays between data streams, it may cause the data obtained by a data stream with a larger delay to be a data generated at an earlier time point.

[0059] Another method is to accurately match the arrival time points of different data streams, and take the data with consistent arrival timestamps in different data streams as a data combination. However, due to the clock source error inside the sensor, the error of the sensor trigger time, the sensor driver, the software system scheduling and other influencing factors, it is difficult to achieve consistent data timestamps based on the arrival timestamp, and the error is large. In addition, it is difficult to optimize from the sensor hardware level, and it may have low error reporting sensitivity, and it is difficult to ensure that the maximum delay can still be controlled in the case of unstable system.

[0060] Therefore, the present application provides a data alignment scheme according to the generation timestamp of each sensor data, so as to deliver the data generated at the same time to the algorithm module for calculation as a data combination. Figure 2 is a flowchart of a sensor data processing method 200 according to an example embodiment of the present application. As shown in Figure 2 the method 200 includes:

[0061] Step 202, receiving multiple data streams to be processed, each data stream corresponding to a sensor, and the data frame in the data stream carrying a generation timestamp;

[0062] Step 204, correcting the generation timestamp of each data frame to obtain a corrected timestamp; and

[0063] Step 206, determining the data frame to be sent in each data sending period from each data stream according to the corrected timestamp of each data frame and a preset data sending period.

[0064] In some embodiments, the multiple data streams include at least one of the following: a raw data stream collected by a sensor, a post-processed data stream obtained after calculation on the raw data stream. That is, the multiple data streams include at least one raw data stream and / or at least one post-processed data stream.

[0065] The original data stream is generated by a sensor driver and has a timestamp of the time when the data frame is generated. The post-processed data stream includes a data stream generated by a data stream processing program using the original data stream as input data stream, and also includes a data stream generated by a data stream processing program using the post-processed data stream as input data stream. That is, the post-processed data stream is obtained after the original data stream is processed one or more times.

[0066] The data stream processing program retains the sensor data timestamp of the input data stream and reflects it in the output frame of the data stream processing program, so the post-processed data stream carries the generation timestamp of the corresponding original data stream. If the data frame of the original data stream is an original data frame and the data frame of the post-processed data stream is a post-processed data frame, each post-processed data frame carries the generation timestamp of the corresponding original data frame.

[0067] In some embodiments, the step S202 of receiving the plurality of data streams to be processed includes: setting a buffer queue corresponding to each data stream in a data buffer for temporarily storing data; and storing each data frame in the corresponding buffer queue according to the generation timestamp of each data frame in each data stream.

[0068] Figure 3 FIG. 1 is a schematic diagram of buffer queues of different input data streams according to an example embodiment of the present application. Each data stream has a corresponding data acquisition period (or data generation period or sensor trigger period) and a buffer queue. For example, the sensor trigger frequency of data stream 1 is 50 Hz, and the corresponding data acquisition period is 20 ms; the sensor trigger frequency of data stream 2 is 20 Hz, and the corresponding data acquisition period is 50 ms; and the sensor trigger frequency of data stream n is 10 Hz, and the corresponding data acquisition period is 100 ms.

[0069] In some embodiments, the data buffer has a timeline based on the server time. When data of any data stream enters, the buffer places the data in the buffer queue at the position on the timeline corresponding to the hard trigger timestamp according to the sensor trigger timestamp, so as to temporarily store the data.

[0070] In addition, since the input data streams have the hard trigger feature, multiple data streams will generate data simultaneously every predetermined period (such as 1 s), so multiple data streams will be automatically aligned every predetermined period. It can be seen that at time 0 (i.e., the starting position of one second) on the timeline, each buffer queue will have a data point, and the data points in each buffer queue are equally spaced.

[0071] In some embodiments, the correction of the generation timestamp of each data frame in step S204 includes: correcting the generation timestamp of each data frame in each data stream to the closest integer timestamp according to the tolerance error of the data stream, for example, if the generation timestamp is 17 ms, it is corrected to 20 ms. Generally, the data alignment algorithm can be executed for new data as soon as the new data frame enters the buffer, and the execution of the alignment algorithm can be paused when the algorithm module performs calculation, so as to save resources and ensure real-time performance. In addition, if the generation timestamps of two data frames are corrected to the same integer timestamp at the same time, the data frame that arrives first is retained, or only the data frame with the generation timestamp closer to the integer timestamp is retained.

[0072] Preferably, if the tolerance error is t 误差 , the generation timestamp of a data frame is t 生成 , and the corrected timestamp of the data frame is t , where a is a preset constant, for example, 0.5, but is not limited thereto. The tolerance error of each data stream is a factor of the data acquisition period of the data stream and is less than 1 / 2 of the data acquisition period T, i.e., t 误差 <T / 2. For example, for a data stream of 10 Hz, the data acquisition period is 100 ms, and t 误差 may be set to 5 ms, 10 ms, 20 ms, etc.

[0073] In some embodiments, the determination of the data frame to be sent in each data sending period from each data stream in step 206 includes:

[0074] For any data stream, the target timestamp corresponding to the current data sending period in the data stream is determined; if the corrected timestamp of a data frame in the data stream matches the target timestamp, the data frame is determined as the data frame to be sent in the current data sending period of the data stream; if there is no data frame in the data stream whose corrected timestamp matches the target timestamp, it is determined that the data stream has no data frame to be sent in the current data sending period.

[0075] Here, the data sending period refers to a period in which a data combination is to be sent to the algorithm module, and the aligned data is combined and delivered to the algorithm module for execution at the end time of each period, which corresponds to a virtual trigger frame. One of the purposes of the present application is to determine the data frames corresponding to the same trigger frame in different data streams, and to deliver these data frames as a data large frame to the algorithm module for calculation. For example, a user wants to trigger the module algorithm at a frequency of 20 Hz, so there will be a trigger frame at each position of 0 ms, 50 ms, 100 ms, 150 ms, etc. on the timeline. Considering that the data acquisition periods of different data streams are inconsistent, therefore the same data frame of a certain data stream may correspond to one or more trigger frames. For example Figure 4 The first data frame in the buffer queue 3 is mapped to trigger frame 1 and trigger frame 2 in the mapping relationship of the trigger frames in different data streams.

[0076] In an implementation manner, determining the target timestamp corresponding to the current data sending period in the data stream comprises:

[0077] According to the data acquisition period of the sensor corresponding to the data stream, determining an expected timestamp sequence of the data stream; and selecting the target timestamp corresponding to the current data sending period from the expected timestamp sequence.

[0078] Further, selecting the target timestamp corresponding to the current data sending period from the expected timestamp sequence comprises:

[0079] Selecting at least one candidate timestamp from the expected timestamp sequence, and taking the maximum value in the at least one candidate timestamp as the target timestamp. Wherein, the candidate timestamp is not later than the end time of the current data sending period, which is also the timestamp of the trigger frame.

[0080] Suppose the frequency of the current data stream is f, then its data acquisition period T = 1 / f, and the expected timestamp sequence is Where N is a natural number set. For example, if the data acquisition period is 20 ms, then the expected timestamp sequence is 0 ms, 20 ms, 40 ms, 60 ms, …, etc.

[0081] Suppose the end time of the data sending period of the data stream is t 结束 , then the candidate timestamp is those timestamps less than or equal to t 结束 , and the target timestamp = max{t, t∈L∩t≤t 结束}. Suppose the timestamp of the trigger frame is 60 ms, then the candidate timestamp determined from the expected timestamp sequence is 0 ms, 20 ms, 40 ms, 60 ms, and the maximum value of the candidate timestamp is 60 ms, which is the target timestamp.

[0082] In some embodiments, the matching of the correction timestamp of a data frame with the target timestamp means that the correction timestamp of the data frame is equal to the target timestamp. If the two are equal, the mapping relationship between the data frame and the corresponding trigger frame is established. The present application determines the data frame to be sent for each data stream in each data sending period, and when the data frame to be sent for a data stream in the current data sending period is determined, the mapping relationship between the data frame and the current trigger frame is established. In the case of reasonable parameter setting and accurate data stream transmission, as new data is continuously transmitted in the data stream, a trigger frame will gradually have a mapping relationship with a data frame in each data stream.

[0083] In some embodiments, the multiple sensors simultaneously generate a data frame every predetermined period, and therefore the determination of the data frame to be sent for each data sending period from each data stream in step S206 includes:

[0084] For any data stream, the difference between the correction timestamp of the current data frame and the correction timestamp of the starting data frame of the current predetermined period is calculated. If the ratio of the correction timestamp difference to the data acquisition period of the data stream is not an integer, the data frame is determined to be a wrong frame of the data stream, and the wrong frame is discarded. For example, a hard trigger data stream with a trigger frequency of 10 Hz generates a data frame with a correction timestamp of 370 ms, which is not an integer multiple of 100 ms, and it can be determined that the data at the position of 370 ms is a wrong frame.

[0085] In some embodiments, the determination of the data frame to be sent for each data sending period from each data stream in step S206 includes:

[0086] For any data stream, the difference between the correction timestamp of the current data frame and the correction timestamp of the previous data frame is calculated. The ratio m of the correction timestamp difference to the data acquisition period of the data stream is calculated. If m is not an integer, the data frame is determined to be a wrong frame in the data stream, and the wrong frame is discarded. If m is an integer greater than or equal to 2, it is determined that m-1 frames are lost between the current data frame and the previous data frame.

[0087] Alternatively, the current predetermined period is every 1 s period, and for a data stream, the difference Δt1 between the correction timestamp of the current data frame and the starting time (or the correction timestamp of the starting frame) of the current 1 s period is calculated. If Δt1 / T is not an integer, the data frame is a wrong frame. For example, the difference Δt2 between the correction timestamps of the previous and next frames is calculated. If Δt2 / T is not an integer, the data frame is a wrong frame. If Δt2 / T is not an integer, Δt2 / T-1 frames are lost between the previous and next frames, and the corresponding target timestamp position can be determined as a lost frame.

[0088] For example, the data collection period is 20 ms, but the correction timestamp of the previous frame is 20 ms, and the correction timestamp of the current frame is 80 ms, so it is determined that two frames are lost between the two frames, and the 40 ms and 60 ms positions are determined to be lost frames. Or for a 10 Hz hard trigger data stream, the expected time difference is 100 ms, but after the data correction process, the time difference between the two frames is 160 ms, so the frame at the 160 ms position is considered to be a wrong frame.

[0089] The occurrence scenarios of the lost frames include but are not limited to: no data generated by the sensor, data acquisition failure of the sensor driver, abnormal calculation of the data stream processing program, etc. The lost frame checking system of the present application is established on the basis of the timestamp correction, so that the definition of "approximately equal to" can be avoided, and it is only necessary to determine whether the time difference between the two frames is a non-integer multiple of the expected time difference.

[0090] In some implementations, when it is determined that the current data stream has lost frames, the method 200 can further include at least one of the following steps, so that the system can timely perceive the lost frames and timely process them:

[0091] 1) reporting the abnormality of the current data stream to the outside;

[0092] 2) marking the trigger frame corresponding to the position where the lost frame occurs as a lost frame, and the subsequent operation no longer considers the trigger frame;

[0093] 3) informing the downstream algorithm module of the specific time and position where the lost frame occurs.

[0094] The occurrence scenarios of the wrong frames include but are not limited to: abnormal sensor time, abnormal server time, abnormal time service system, abnormal data acquisition of the sensor driver, abnormal calculation of the data stream processing program, etc. In a data stream with hard trigger characteristics, the time difference between the two frames cannot be corrected by the correction algorithm to be an integer multiple of the expected time difference (the reciprocal of the frequency). The wrong frame checking system of the present application is also established on the basis of the timestamp correction, and it is only necessary to determine whether the time difference between the two frames is a non-integer multiple of the expected time difference.

[0095] In some implementations, when it is determined that the current data stream has a wrong frame, the method 200 can further include at least one of the following steps to ensure the safe operation of the system:

[0096] 1) reporting the abnormality of the current data stream to the outside;

[0097] 2) informing the downstream algorithm module that the data stream may start to have errors;

[0098] 3) after the errors accumulate to a certain extent, switching to a backup system or performing a safety action.

[0099] In some embodiments, the step S206 of determining the data frames to be transmitted in each data transmission cycle from each data stream comprises:

[0100] If the position corresponding to the target timestamp in a data stream is determined to be a missing frame, it is determined that the data stream has no data frame to be transmitted in the current data transmission cycle.

[0101] If a new data frame of a data stream is not received within a predetermined period of time from the target timestamp, it is determined that the data stream is timed out and the data stream has no data frame to be transmitted in the current data transmission cycle.

[0102] In some embodiments, if a timestamp position is determined to be a missing frame, a wrong frame or timed out, the timestamp can be marked as a missing frame, a wrong frame or timed out. The wrong frame can be directly discarded. If a target timestamp position is marked as a missing frame or timed out, it means that the current trigger frame cannot be mapped to the data stream, i.e. the current data transmission cycle has no data frame to be transmitted. In addition, for a trigger frame, the data stream that has been mapped can be marked as "mapped", the data stream whose data frame is timed out can be marked as "timed out", and the data stream whose data frame is missing can be marked as "missing". When all the data streams corresponding to the trigger frame have a marked state (including "mapped", "timed out" and "missing"), it means that the mapping of the trigger frame is completed, the trigger frame is complete, and the data frame of the data transmission cycle is complete, except that some data streams may be mapped to empty frames. The data frame that has been mapped can then be transmitted to the algorithm module as a large frame.

[0103] In some embodiments, the predetermined period of time can be a preset tolerance delay. A global tolerance delay can be set, or a corresponding tolerance delay can be set for each data stream. The present application does not limit this. For each incomplete trigger frame, the expected timestamp of the missing data frame is calculated. If the difference between the current system time and the expected timestamp is greater than the set tolerance error, it is determined that the corresponding data stream is timed out and the data stream has no data frame to be transmitted in the current data transmission cycle. Otherwise, the arrival of the expected data frame is continued to be waited for.

[0104] It should be understood that in some sensor fusion algorithms, there may be situations that need to ensure real-time performance while allowing data loss. For example, an algorithm that fuses the tracking results of multiple sensors to model the road conditions will receive inputs from multiple data streams. Occasionally, the absence of input from a certain data stream will not affect the normal operation of the algorithm. Moreover, according to the execution process of the data frame alignment algorithm, the delay time of the final large data frame is the maximum delay of all data streams. Therefore, when an abnormality occurs in a certain data stream, such as a high delay or no data generation, it will cause the alignment algorithm to have a high delay or even stop running.

[0105] Therefore, the application adds a delay checking system to the data alignment system, which runs independently and can communicate with each other. The delay checking system runs in a loop and can be implemented by using a timer or a delay trigger mechanism to periodically check the timeout status of the data stream.

[0106] In one implementation, a timer is set to check whether there is a data stream timeout every predetermined period.

[0107] In another implementation, a global latest time is calculated for the incomplete trigger frame based on the tolerance delay of each data stream, the observed time of the current server, and the mapping of the data frame. If no new data frame is received before this time, it is determined that the data is timeout.

[0108] Specifically, for the first data sending period of the incomplete data frame (or the first incomplete trigger frame), a global latest time is calculated based on the expected timestamp of each data stream, the tolerance delay, and the real time observed by the server (also known as system time). If no new data frame arrives for each data stream before the global latest time, it is determined that all the data streams are timeout, the data sending period is mapped to empty frame in these data streams, and the current trigger frame is a complete trigger frame with partial data frame timeout. If one or more new data frames arrive for the data streams before the global latest time, the correction timestamp of the new data frame is calculated based on the alignment algorithm, the alignment algorithm is re-executed, and whether the global latest time needs to be updated is determined based on whether there is any missing data frame.

[0109] Here, for each incomplete data sending period, there is a local latest time for each data stream, and the local latest time of the data sending period in data stream k is k，局部 The expected timestamp t ，预期 of the expected data frame is added to the tolerance delay t k,延迟 of the data stream, i.e. k，局部 t k，预期 = t k,延迟 + t 未到 And the global latest time of the data sending period is the maximum of the local latest time of each missing data stream, i.e. Where S 未到 is the set of data streams of all missing data frames.

[0110] For example, assume that no data frame in data stream 2 and 3 arrives in the current data transmission period, if the expected timestamp of data stream 4 is 30 ms and the tolerant delay is 20 ms, then the local latest time of data stream 4 in the current data transmission period is 50 ms. If the expected timestamp of data stream 5 is 60 ms and the tolerant delay is 80 ms, then the local latest time of data stream 5 in the current data transmission period is 90 ms. Therefore, the global latest time of the current data transmission period is the maximum of the two local latest times, i.e. 90 ms.

[0111] In some embodiments, at the current system time of the high delay check, the system only needs to wait for either the "global latest time arrives" or "new data frame arrives" event. If the event that occurs is "global latest time arrives", then all the data frames that have not arrived are directly marked as timeout; if the event that occurs is "new data frame arrives", then the alignment algorithm is re-executed, the corrected timestamp of the new data frame is calculated, and it is determined whether the new data frame can be mapped to the trigger frame. In addition, it is determined whether there are still data frames that have not arrived (i.e. whether there are still data streams that have not been mapped to the current trigger frame). If there are still data frames that have not arrived, it means that the current trigger frame is not complete, and the global latest time needs to be updated according to the expected timestamp of the expected data frame and the tolerant delay of the corresponding data stream until the data frames of the data transmission period are complete.

[0112] In another implementation, when a trigger frame has been mapped in other data streams, the high delay check algorithm can be started to check whether the trigger frame is timeout in the remaining data streams. Preferably, when a trigger frame has been mapped in other data streams, the high delay check algorithm can be started to check whether the trigger frame is timeout in the last data stream.

[0113] After the check is completed, if for a certain trigger frame, it is found that the state of all data streams is "mapped" or "marked as timeout", then the trigger frame can be considered as a complete trigger frame containing empty data. Through the delay check, the maximum delay of the alignment algorithm can be controlled within the tolerant delay, avoiding problems such as system running abnormally caused by too high delay of a single data stream.

[0114] In addition, regarding the integrity judgment of the trigger frame, in one implementation, if the current trigger frame (or the current data sending period) has a corresponding relationship with all data streams, the trigger frame is considered complete; otherwise, if there is a data stream without a corresponding relationship, the trigger frame is considered incomplete. This mode is mainly aimed at algorithm modules with relatively high data integrity requirements. In another implementation, for each trigger frame, data streams without data to be sent are excluded, and if the remaining data streams have determined corresponding data frames (mapping relationships have been established), the trigger frame is determined to be complete, or referred to as a complete trigger frame containing empty data. This mode is mainly aimed at algorithm modules with relatively low data integrity requirements.

[0115] Therefore, the application can determine the judgment requirements of the trigger frame according to the calculation accuracy requirements passed to the algorithm module. For algorithm modules with high requirements, each data stream must have a mapping before being passed to the algorithm module, and for algorithm modules with low requirements, only the mapping relationship between the data stream with data frames to be sent needs to be established.

[0116] For example Figure 4 Trigger frame 1 and trigger frame 2 in the trigger frame 1 and the trigger frame 2 have corresponding data frames in the three buffer queues and are considered complete trigger frames; trigger frame 3 has a mapping relationship with the data frame in the buffer queue 1, but is marked as not timed out by the delay detection algorithm in the buffer queue 3, so that the trigger frame 3 is considered a complete trigger frame containing empty data. Trigger frame 4 has no mapping relationship with any buffer queue and is considered an incomplete trigger frame. Assuming that the corresponding target timestamp positions of trigger frame 4 in the buffer queue 2 and the buffer queue 3 are frame loss or timeout, it is determined that there is no data frame to be sent in the data stream, and at this time, the trigger frame 4 can also be considered a complete trigger frame containing empty data.

[0117] In some embodiments, after step S206, the step of passing the determined data frames to be sent in each data sending period as a combination to the algorithm module for calculation can also be included. Here, the data can be sent to the algorithm module for calculation, or the data can be read by the algorithm module, which is not limited by the application.

[0118] According to the technical solution of the application, the data stream with the hard trigger feature can be aligned, the data frames with the same hard trigger timestamp can be combined into a large frame, and the large frame is passed to the algorithm module for calculation. The calculation has a certain tolerance to errors while ensuring the correctness of the data. Moreover, when the data stream has an exception, including but not limited to an error in the hard trigger timestamp, frame loss, data delay, and the like, the application can timely discover the exception and respond, thereby ensuring the real-time performance and safety of the system.

[0119] The following section will use the three data streams existing in the system as an example to detail the data scheme of this application:

[0120] Data stream 1 is the raw data stream generated by the hard-triggered sensor, and the error between the sensor time and the server time is extremely small; the hardware trigger frequency f1 = 50Hz, that is, the data acquisition cycle is 20ms; the sensor driver can correctly obtain the generation time of each frame of data without error; the delay is about 10ms, fluctuating within a small range of 2ms; data loss may occur.

[0121] Data stream 2 is the raw data stream generated by the hard-triggered sensor, with minimal error between the sensor time and the server time. The sensor driver can accurately obtain the generation time of each frame of data without errors. The hardware trigger frequency f2 = 20Hz, meaning the data acquisition period is 50ms. Due to various reasons, this sensor may generate individual data with larger timestamp deviations, but overall it remains stable. Its latency is approximately 40ms, fluctuating within a 10ms range.

[0122] Data stream 3 is a post-processed data stream generated by the data stream processing program; the hardware trigger frequency f3 = 10Hz, that is, the data acquisition period is 100ms; its latency is unstable, which may be as low as 10ms or as high as 500ms.

[0123] Set the global tolerance error t 误差 =10ms, and the data transmission period (i.e., the algorithm module trigger period) is 50ms. Figure 5A The diagram illustrates the generation timestamps of different data streams. The generation timestamps of the three data frames in the data stream are 0ms, 19ms, and 43ms, respectively. Substituting these timestamps into the... The corresponding corrected timestamps are 0ms, 20ms, and 40ms. The generation timestamps of the two data frames in data stream 2 are 4ms and 49ms, and the corresponding corrected timestamps are 0ms and 50ms. The corrected timestamps for each data stream are as follows: Figure 5B As shown.

[0124] For the second data transmission cycle (i.e., trigger frame 2), the expected timestamp sequence in data stream 2 is L = {0ms, 50ms, 100ms, ...}, therefore the candidate timestamp is {0ms, 50ms}, and the target timestamp is max{0ms, 50ms} = 50ms. Data stream 2 also contains a data frame with a correction timestamp of 50ms, i.e., the second data frame; therefore, a mapping relationship is established between this data frame and trigger frame 2.

[0125] Figure 6AThis is a schematic diagram illustrating frame loss, misaligned frame, and high latency detection in this application. Data stream 1 contains two frames at positions 60ms and 100ms. Since 100ms - 60ms = 40ms = 2 * 20ms, one frame is lost between these two frames at position 80ms. In data stream 2, the frame with a timestamp of 70ms has a time difference of 70ms - 50ms = 20ms = 0.4 × 50ms. Since 0.4 is not an integer multiple, this frame is determined to be a misaligned frame.

[0126] In some embodiments, assuming the system is currently at 181ms, a high-latency check is initiated. Since trigger frame 3 has already completed data mapping at the 100ms position in buffer queues 1 and 2, the high-latency check algorithm directly begins checking the latency of buffer queue 3. The expected timestamp is calculated to be 100ms based on the data acquisition cycle of data stream 3, with a tolerable latency of 80ms. Since 181ms - 100ms > 80ms, buffer queue 3 corresponding to trigger frame 3 is marked as timed out. At this point, the data in all buffer queues for trigger frame 3 is in either the "complete mapping" or "timeout" state, thus trigger frame 3 becomes a complete trigger frame.

[0127] In other embodiments, it is assumed that the server observes the real-time t. 现实 The timeframe is 130ms. The first incomplete trigger frame in the system is trigger frame 3. Data frames for this trigger frame have already arrived at the 100ms position in buffer queues 1 and 2. A data frame is expected to arrive at the 100ms position in buffer queue 3, but this data frame has not yet arrived. At this point, the latest timeframe is calculated. S 未到 Only data stream 3 has a local latest arrival time of 180ms, which is also the latest time for the current system's high-latency check. Assuming the data frame being waited for has not arrived by 180ms, this data frame is marked as timed out. The waiting time of the high-latency check system is calculated as t. 全局 -t 现实 =180ms – 130ms = 50ms. During the waiting period, if no new data frames arrive in data stream 3, all data frames at the waiting position will be marked as timed out. At this time, trigger frame 3 has the marked status in all buffer queues, thus it is a complete trigger frame containing timed-out data, and is passed to the algorithm module for calculation. If a new data frame arrives in the waiting data stream, it is determined whether the newly arrived data frame can be mapped to the trigger frame. If it can, it means that the trigger frame is complete.

[0128] In some other embodiments, such as Figure 6B As shown, assume the server observes the actual time t. 现实The system's first incomplete trigger frame is trigger frame 4, which is 160ms. The data frame at the 100ms position of buffer queue 3 has arrived. Expected data frames are at the 140ms position of buffer queue 1 and the 150ms position of buffer queue 2, but neither has arrived. At this point, the local latest times for data streams 1 and 2 are calculated to be 170ms and 210ms respectively, therefore the global latest time is 210ms. The current waiting time from the global latest time is calculated as 210ms – 160ms = 50ms. During the waiting period, if no new data frames arrive in either data stream 1 or 2, the 140ms position of buffer queue 1 and the 150ms position of buffer queue 2 are both marked as timeout. At this point, trigger frame 4 is determined to be a complete trigger frame containing timeout data and is passed to the algorithm module for calculation.

[0129] If in Figure 6B During the 50ms wait period, for example, in t 现实 At 165ms, a data frame from buffer queue 2 at position 150ms arrives. At this time, one data frame from data stream 1 is still pending. Therefore, the global latest time is updated to the local latest time of data stream 1, 170ms, and the waiting time is updated to 170ms - 165ms = 5ms. The high-latency waiting system then needs to wait another 5ms. If no new data frame from data stream 1 arrives within 5ms, data stream 1 can be marked as timed out.

[0130] Or, in Figure 6B During the 50ms wait period, for example, in t 现实 At 180ms, a data frame from buffer queue 2 at position 150ms arrives. Although the updated global latest time is still 170ms, since the current system time has exceeded this global latest time, there is no need to wait any longer. Trigger frame 4 is determined to be a complete trigger frame containing timeout data (buffer queue 1) and is passed to the algorithm module for calculation.

[0131] According to the technical solution of the present application, the time stamp alignment processing is performed on the multiple input data streams at the software level, so that the data delivered to the algorithm module is generated by multiple sensors at the same time and / or is generated by the post-processing of the data generated at the same time, which can better meet the requirements of multi-sensor fusion on input data and improve the accuracy of the algorithm. The present application realizes an efficient and predictable data alignment system, which can save computing resources, improve algorithm response speed, and the reproducibility of the results makes the debugging work simple when the system has problems. Moreover, the present application also realizes a self-diagnosis system, which can quickly find problems and report externally when problems occur, and the system itself can perform corresponding processing according to different abnormal conditions, such as self-correction, system degradation to an inaccurate but usable state, enabling a backup system, etc., so as to improve the safety of the autonomous driving system, improve the work efficiency of the test personnel, and simplify the work flow of the developers.

[0132] Figure 7 A schematic diagram of a sensor data processing apparatus 700 according to an embodiment of the present application is shown as follows, Figure 7 The sensor data processing apparatus 700 includes:

[0133] A data frame receiving module 702 is configured to receive multiple data streams to be processed, each data stream corresponding to a sensor, and a data frame in the data stream carrying a generation time stamp.

[0134] A time stamp correction module 704 is configured to correct the generation time stamp of each data frame to obtain a corrected time stamp.

[0135] A data frame alignment module 706 is configured to determine the data frame to be sent in each data sending period from each data stream according to the corrected time stamp of each data frame and a preset data sending period.

[0136] In some embodiments, the data frame receiving module 702 sets a buffer queue corresponding to each data stream, and stores each data frame in the corresponding buffer queue according to the generation time stamp of each data frame in each data stream.

[0137] In some embodiments, the time stamp correction module 704 corrects the generation time stamp of each data frame in each data stream to the closest integer time stamp according to the tolerance error of each data stream.

[0138] In some embodiments, for any data stream, the data frame alignment module 706 determines a target timestamp in the data stream corresponding to the current data transmission period; if a data frame of the data stream has a correction timestamp matching the target timestamp, the data frame is determined as the data frame to be transmitted in the current data transmission period; if there is no data frame of the data stream having a correction timestamp matching the target timestamp, it is determined that the data stream has no data frame to be transmitted in the current data transmission period.

[0139] In some embodiments, for any data stream, the data frame alignment module 706 determines a target timestamp in the data stream corresponding to the current data transmission period; if a data frame of the data stream has a correction timestamp matching the target timestamp, the data frame is determined as the data frame to be transmitted in the current data transmission period; if there is no data frame of the data stream having a correction timestamp matching the target timestamp, it is determined that the data stream has no data frame to be transmitted in the current data transmission period.

[0140] In some embodiments, for any data stream, the data frame alignment module 706 calculates a correction timestamp difference between the current data frame and the starting data frame of the current predetermined period; if the ratio of the correction timestamp difference to the data acquisition period of the data stream is not an integer, the data frame is determined as a wrong frame of the data stream, and the wrong frame is discarded.

[0141] In some embodiments, for any data stream, the data frame alignment module 706 calculates a correction timestamp difference between the current data frame and the previous data frame; calculates the ratio m of the correction timestamp difference to the data acquisition period of the data stream; if m is not an integer, the data frame is determined as a wrong frame of the data stream, and the wrong frame is discarded; if m is an integer other than 1, it is determined that m-1 frames are lost between the current data frame and the previous data frame.

[0142] In some embodiments, when the position corresponding to the target timestamp in a data stream is determined as a lost frame, the data frame alignment module 706 determines that the data stream has no data frame to be transmitted in the current data transmission period; and when no new data frame of a data stream is received within a predetermined period, the data frame alignment module 706 determines that the data stream is timed out, and determines that the data stream has no data to be transmitted in each data transmission period within the predetermined period.

[0143] In some embodiments, the apparatus 700 further comprises a data frame transmission module (not shown in the figure) for transmitting the determined data frame to be transmitted in each data transmission period as a combination to the algorithm module for calculation.

[0144] Figure 8FIG. 1 illustrates a diagram of a machine in example form of a computing device 800 within which instructions, when executed, and / or processing logic, when activated, can cause the machine to perform any one or more of the methodologies described and / or claimed herein. In alternative embodiments, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server or a client machine in server-client network environments, or it can act as a peer machine in peer-to-peer (or distributed) network environments. The machine can be a personal computer (PC), a laptop computer, a tablet computing system, a personal digital assistant (PDA), a cellular telephone, a smart phone, a web appliance, a set-top box (STB), a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies described and / or claimed herein.

[0145] The example computing device 800 can include a data processor 802 (e.g., system on a chip (SoC), general purpose processing core, graphics core, and optional other processing logic) and a memory 804 (e.g., internal memory) that can communicate with each other via a bus 806 or other data transfer system. The computing device 800 can also include various input / output (I / O) devices and / or interfaces 810, such as a touchscreen display, audio jacks, voice interface, and optional network interface 812. In example embodiments, the network interface 812 can include one or more radio transceivers configured to operate with any one or more standard wireless and / or cellular protocols or access technologies (e.g., second generation (2G), 2.5 generation, third generation (3G), fourth generation (4G), and next generation wireless access of cellular systems, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), LTE, CDMA2000, WLAN, Wireless Router (WR) mesh, etc.). The network interface 812 can also be configured to operate with various other wired and / or wireless communication protocols, including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, Bluetooth®, IEEE 802.11x, etc. In essence, the network interface 812 can include or support virtually any wired and / or wireless communication and data processing mechanism by which information / data can travel between the computing device 800 and another computing or communication system via the network 814. IEEE 802.11x, etc. In essence, the network interface 812 can include or support virtually any wired and / or wireless communication and data processing mechanism by which information / data can travel between the computing device 800 and another computing or communication system via the network 814.

[0146] Memory 804 can be a machine-readable medium (or computer-readable storage medium) on which is stored one or more sets of instructions, software, firmware, or other processing logic embodying any one or more of the methodologies or functions described and / or claimed herein (e.g., logic 808). The software can also reside, completely or at least partially, within processor 802 during execution thereof by computing device 800, for example (the processor 802 also can constitute machine-readable media). Thus, memory 804 and processor 802 also can constitute machine-readable media (or computer-readable storage media). Logic 808 or portions thereof also can be configured to process logic or logic that is partially implemented in hardware. Logic 808 or portions thereof also can be transmitted or received, for example, over network 814 via network interface 812. While machine-readable medium (or computer-readable storage medium) is shown in an example implementation to be a single medium, the term "machine-readable medium" (or computer-readable storage medium) should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and computing systems) that store the one or more sets of instructions. The term "machine-readable medium" (or computer-readable storage medium) shall also be taken to include any non- transitory medium that is capable of storing, encoding or carrying the sets of instructions for execution by a machine and that cause the machine to perform any one or more of the methodologies of the present embodiments, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term "machine-readable medium" (or computer-readable storage medium) can accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0147] The disclosed and other embodiments, modules and the functions described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in combinations of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0148] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0149] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices of this type are known.

[0150] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, or a computer will be operatively coupled to receive data from or transfer data to or both, a communications network, over a wired or wireless communications link. The application is not limited by the

[0151] While this application contains many specifics, these should not be construed as limiting the scope of any invention or application that can be claimed, but rather as describing features that can be specific to particular embodiments of the application. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described herein in the context of a single embodiment can also be implemented separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0152] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. Additionally, the separation of various system components in the embodiments described herein should not be understood as requiring such separation in all embodiments.

[0153] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this application.

[0154] The description of the embodiments described herein is intended to provide a general understanding of the structure of various embodiments and is not intended to serve as an exhaustive or comprehensive description of all elements and features of apparatuses and systems that can utilize the structures described herein. Many other embodiments will be apparent to those of ordinary skill in the art upon reviewing the description provided herein. Other embodiments can be derived from application of the teachings of the description provided herein to the practice of the structures and logic of the present application. The figures herein are representative only and can not be drawn to scale. Certain proportions can be exaggerated, while other proportions can be minimized. Accordingly, the specification and drawings should be considered illustrative but not restrictive.

[0155] Some embodiments implement the functionality in two or more specific interconnected hardware modules or devices with related control and data signals communicated between and through the modules, or as portions of an application-specific integrated circuit. Accordingly, the example system is applicable to software, firmware, and hardware implementations.

[0156] While example embodiments or examples of the present application have been described with reference to the drawings, it is to be understood that the exemplary discussion is not intended to be exhaustive or to limit the application to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the disclosed subject matter not be limited to any single embodiment or example, but rather that the scope of the application be according to the breadth of the claims and the full range of equivalents to which they are entitled.

Claims

1. A sensor data processing method, comprising: Receive multiple data streams to be processed, each data stream corresponds to a sensor, each data stream has a corresponding data acquisition period, and the data frames in the data stream carry a generation timestamp; The generation timestamps of each data frame are corrected to obtain the corrected timestamps; as well as Based on the correction timestamp of each data frame and the preset data transmission period, determine the data frame to be transmitted in each data transmission period from each data stream; Among these steps, determining the data frame to be sent in each data transmission cycle from each data stream includes: For any data stream, determine the target timestamp in the data stream that corresponds to the current data transmission period; If there is no data frame in the data stream whose correction timestamp matches the target timestamp, then it is determined that there is no data frame to be sent in the current data transmission period.

2. The method according to claim 1, wherein, The plurality of data streams includes at least one of the following: a raw data stream collected by a sensor, and a post-processed data stream obtained by calculating the raw data stream; the post-processed data stream carries a corresponding generation timestamp of the raw data stream.

3. The method according to claim 1, wherein, Receive multiple data streams to be processed, including: Configure a buffer queue for each data stream; Based on the generation timestamp of each data frame in each data stream, each data frame is stored in the buffer queue.

4. The method according to claim 1, wherein, Correct the generation timestamps of each data frame, including: Based on the tolerance error of each data stream, the generation timestamp of each data frame in the data stream is corrected to the nearest integer timestamp. Each data stream has a preset data acquisition period, and the tolerance error of each data stream is a factor of the data acquisition period and is less than 1 / 2 of the data acquisition period.

5. The method according to claim 1, wherein, Determining the data frames to be sent in each data transmission cycle from each data stream also includes: If the corrected timestamp of a data frame in the data stream matches the target timestamp, then the data frame is determined to be the data frame that the data stream should send in the current data transmission cycle.

6. The method according to claim 5, wherein, For any data stream, determine the target timestamp corresponding to the current data transmission period in that data stream, including: Based on the data acquisition cycle of the sensor corresponding to the data stream, determine the expected timestamp sequence of the data stream; Select the target timestamp corresponding to the current data transmission period from the expected timestamp sequence.

7. The method according to claim 6, wherein, Selecting the target timestamp corresponding to the current data transmission period from the expected timestamp sequence includes: At least one candidate timestamp is selected from the expected timestamp sequence, the candidate timestamp being no later than the end time of the current data transmission period; and The maximum value among the at least one candidate timestamp is taken as the target timestamp.

8. The method according to claim 1, wherein, Multiple sensors simultaneously generate a data frame at predetermined intervals. The data frame to be transmitted in each data transmission cycle is determined from each data stream, including: For any data stream, calculate the difference in corrected timestamps between the current data frame and the starting data frame of the current predetermined period in the data stream; If the ratio of the correction timestamp difference to the data acquisition period of the data stream is not an integer, then the data frame is determined to be an incorrect frame of the data stream and is discarded.

9. The method according to claim 5, wherein determining the data frame to be transmitted in each data transmission cycle from each data stream comprises: For any data stream, calculate the difference in corrected timestamps between the current data frame and the previous data frame in the data stream; Calculate the ratio m of the corrected timestamp difference to the data acquisition period of the data stream; If m is not an integer, then the data frame is determined to be an incorrect frame in the data stream and is discarded. as well as If m is a non-1 integer, then it is determined that m-1 frames were lost between the current data frame and the previous data frame.

10. The method according to claim 1, wherein, Determine the data frames to be sent in each data transmission cycle from each data stream, including: If a position in a data stream corresponding to the target timestamp is determined to be a dropped frame, then it is determined that the data stream has no data frames to be sent in the current data transmission period; and If no new data frame is received from a data stream within a predetermined time period starting from the target timestamp, the data stream is determined to have timed out, and the data stream has no data to send in the current data transmission cycle.

11. The method according to claim 1, further comprising: The data frames to be sent in each determined data transmission cycle are combined and passed to the algorithm module for calculation.

12. A sensor data processing device, comprising: The data frame receiving module is used to receive multiple data streams to be processed. Each data stream corresponds to a sensor, each sensor has a corresponding data acquisition period, and the data frames in the data stream carry a generation timestamp. The timestamp correction module is used to correct the generation timestamps of each data frame to obtain the corrected timestamps; as well as The data frame alignment module is used to determine the data frame to be sent in each data transmission period from each data stream based on the correction timestamp of each data frame and the preset data transmission period. The data frame alignment module is further used for: For any data stream, determine the target timestamp in the data stream that corresponds to the current data transmission period; If there is no data frame in the data stream whose correction timestamp matches the target timestamp, then it is determined that there is no data frame to be sent in the current data transmission period.

13. A computing device, comprising: One or more processors; and A memory storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of claims 1-11.

15. A vehicle comprising the computing device as claimed in claim 13.

Images