A method and device for optimizing spatiotemporal fusion of multi-source sensors

By performing coordinate system conversion and data correction in a multi-source sensor system, the problem of data space-time out-of-synchronization after sensor running time is solved, data accuracy and fusion quality are improved, and the needs of intelligent traffic management are met.

CN114359408BActive Publication Date: 2024-11-19BEIJING RONGSHUN ZHIXING TECH DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111678062.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-31
Publication Date
2024-11-19
Estimated Expiration
2041-12-31

AI Technical Summary

Technical Problem

The existing road traffic management methods cannot meet the needs of intelligent management and diversified travel services, resulting in the problem of data space-time and time-space aberration seen by sensors after running time.

Method used

By converting the coordinate systems of multiple sensors to the earth coordinate system and obtaining the perceived data of the target in the overlapping area, the sensor's time and space calibration are corrected to ensure the spatial and temporal consistency of the data.

Benefits of technology

It improves the accuracy and fusion accuracy of sensor data, solves the problem of data space-time and space asymmetry after sensor running time, and improves the quality of subsequent data fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114359408B_ABST
    Figure CN114359408B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for optimizing the spatiotemporal fusion of multi-source sensors. The method can automatically select data with higher confidence from data with lower confidence perceived by the sensor, and then use the data with higher confidence to correct the time calibration and space calibration of the first sensor and the second sensor, thereby improving the correction accuracy, thereby realizing the correction of the time and space calibration of different sensors, effectively solving the problem of the time and space asynchrony of data perceived by different sensors after the sensors have been running for a period of time, and improving the accuracy of subsequent data fusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of time-space calibration, and in particular to a time-space fusion optimization method and device for multi-source sensors. Background Art

[0002] As an important part of security, video surveillance is the main front-end device of the Internet of Things. With the rapid development of social economy, intelligent transportation has become one of the most in-demand areas in the video surveillance market, and video equipment provides a guarantee for traffic safety. However, the road traffic environment is becoming increasingly complex, and the requirements for road traffic management are becoming higher and higher. The existing road management methods can no longer meet the needs of intelligent management and diversified travel services. In order to achieve refined road control, multi-source traffic information holographic perception and high-quality traffic data extraction technology are urgently needed.

[0003] In the related technology, although it is possible to obtain traffic information from multiple sources and calibrate the time and space of different sensors, any sensor will drift after running for a long time, which will lead to the problem of time and space asynchrony of the data sensed by subsequent sensors. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, electronic device and computer storage medium for the spatiotemporal fusion optimization of multi-source sensors, which can realize the time and space calibration of different sensors, effectively solve the problem of spatiotemporal asynchrony of data perceived by different sensors after the sensors have been running for a period of time, and improve the accuracy of subsequent data fusion.

[0005] To achieve the above-mentioned object, in a first aspect, the present invention provides a spatiotemporal fusion optimization method for a multi-source sensor, which is applied to a system including N sensors, where N is greater than or equal to 2, and the N sensors include a first sensor and a second sensor, wherein the first sensor and the second sensor are in the same area, and there is an overlapping area between the sensing area of ​​the first sensor and the sensing area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to the overlapping area between different sensors;

[0006] The method comprises:

[0007] Respectively converting a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system;

[0008] When the distance between the first target and the remaining targets among the n targets sensed by the first sensor and / or the second sensor in the overlapping area is greater than a preset value, first sensing data related to the first target sensed by the first sensor is acquired, where n is greater than or equal to 0, and the first target includes a stationary target or a moving target;

[0009] Based on the first perception data, determining second perception data related to the first target object from the data perceived by the second sensor;

[0010] When the first target object is a stationary target object, based on a first position of the first target object in the first perception data and a second position corresponding to the first position in the second perception data, correcting the spatial calibration of the first sensor and the second sensor;

[0011] When the first target is a moving target, the time calibration of the first sensor and the second sensor is corrected based on the movement time and movement position of the first target in the first perception data and the movement time and movement position of the first target in the second perception data.

[0012] In a possible implementation, the correcting the spatial calibration of the first sensor and the second sensor based on the first position of the first target in the first perception data and the second position corresponding to the first position in the second perception data specifically includes:

[0013] One of the first sensor and the second sensor is randomly selected as a reference object, and the spatial parameters of the sensor that is not selected as the reference object are corrected based on the deviation between the first position and the second position.

[0014] In a possible implementation manner, a sampling frequency of the first sensor is lower than a sampling frequency of the second sensor;

[0015] The correcting the time calibration of the first sensor and the second sensor based on the movement time and the movement position of the first target object in the first perception data and the movement time and the movement position of the first target object in the second perception data specifically includes:

[0016] Randomly selecting a first data frame including the first target object from the first perception data, and determining a first time when the first data frame is collected, and a first position of the first target object in the first data frame;

[0017] Based on the first time, determining a second data frame and a third data frame including the first target object from the second perception data, determining a second time when the second data frame is collected and a third time when the third data frame is collected, and determining a second position of the first target object in the second data frame and a third position of the first target object in the third data frame, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time;

[0018] Determine, based on the first position, the second position, the third position, the second time, and the third time, a fourth time when the second sensor senses the first target when the first target is at the first position;

[0019] The time calibration of the first sensor and the second sensor is corrected according to the time difference between the first time and the fourth time.

[0020] In a possible implementation manner, a sampling frequency of the first sensor is lower than a sampling frequency of the second sensor;

[0021] The determining, based on the first perception data, second perception data related to the first target object from the data perceived by the second sensor specifically includes:

[0022] Based on the first perception data and a pre-calibrated time, a data set corresponding to the first target mark in the first perception data is determined from the second perception data, and the data set constitutes the second perception data.

[0023] In a possible implementation, the method further includes:

[0024] Based on the modified spatial parameters and time parameters, output the target state data of at least one target object included in the first perception data and / or the second perception data

[0025] In a possible implementation manner, outputting target state data of at least one target object included in the first perception data and / or the second perception data based on the corrected spatial parameter and time parameter specifically includes:

[0026] Based on a preset data conversion algorithm, the corrected spatial parameters and time parameters are processed to obtain target state data of the at least one target object.

[0027] In a possible implementation, the target state data includes one or more of the following:

[0028] The spatial and temporal position of the target object, the speed of the target object, the source of sensing the target object, or the dot matrix information of the target object.

[0029] In a possible implementation, the N sensors include one or more of a camera, a millimeter wave radar, a microwave radar, a lidar, a visible light device, an infrared device, or a far infrared device.

[0030] In a second aspect, the present invention provides a spatiotemporal fusion optimization device for a multi-source sensor, which is deployed in a system including N sensors, where N is greater than or equal to 2, and the N sensors include a first sensor and a second sensor, wherein the first sensor and the second sensor are in the same area, and there is an overlapping area between the sensing area of ​​the first sensor and the sensing area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to an overlapping area between different sensors;

[0031] The device comprises:

[0032] A coordinate conversion module, used for converting a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system respectively;

[0033] a data acquisition module, configured to acquire first perception data related to the first target object sensed by the first sensor when the distance between the first target object and the remaining targets in the overlapping area sensed by the first sensor and / or the second sensor is greater than a preset value, wherein n is greater than or equal to 0, wherein the first target object includes a target object in a stationary state or a target object in a moving state, and, based on the first perception data, determine second perception data related to the first target object from data sensed by the second sensor;

[0034] A correction module is used to correct the spatial calibration of the first sensor and the second sensor based on a first position of the first target in the first perception data and a second position corresponding to the first position in the second perception data when the first target is a stationary target, and to correct the time calibration of the first sensor and the second sensor based on a movement time and a movement position of the first target in the first perception data and a movement time and a movement position of the first target in the second perception data when the first target is a moving target.

[0035] In a possible implementation, the correction module is further used to:

[0036] One of the first sensor and the second sensor is randomly selected as a reference object, and the spatial parameters of the sensor that is not selected as the reference object are corrected based on the deviation between the first position and the second position.

[0037] In a possible implementation manner, a sampling frequency of the first sensor is lower than a sampling frequency of the second sensor;

[0038] The correction module is further used for:

[0039] Randomly selecting a first data frame including the first target object from the first perception data, and determining a first time when the first data frame is collected, and a first position of the first target object in the first data frame;

[0040] Based on the first time, determining a second data frame and a third data frame including the first target object from the second perception data, determining a second time when the second data frame is collected and a third time when the third data frame is collected, and determining a second position of the first target object in the second data frame and a third position of the first target object in the third data frame, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time;

[0041] Determine, based on the first position, the second position, the third position, the second time, and the third time, a fourth time when the second sensor senses the first target when the first target is at the first position;

[0042] The time calibration of the first sensor and the second sensor is corrected according to the time difference between the first time and the fourth time.

[0043] In a possible implementation manner, a sampling frequency of the first sensor is lower than a sampling frequency of the second sensor;

[0044] The correction module is further used for:

[0045] Based on the first perception data and a pre-calibrated time, a data set corresponding to the first target object in the first perception data is determined from the second perception data, and the data set constitutes the second perception data.

[0046] In a possible implementation manner, the device further includes:

[0047] an output module, configured to output target state data of at least one target object included in the first perception data and / or the second perception data based on the modified spatial parameter and time parameter;

[0048] In a possible implementation, the output module is further used to:

[0049] Based on a preset data conversion algorithm, the corrected spatial parameters and time parameters are processed to obtain target state data of the at least one target object.

[0050] In a possible implementation, the target state data includes one or more of the following:

[0051] The spatial and temporal position of the target object, the speed of the target object, the source of sensing the target object, or the dot matrix information of the target object.

[0052] In a third aspect, the present invention provides an electronic device, comprising:

[0053] at least one memory for storing a program;

[0054] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed on an electronic device, enables the electronic device to execute the method provided in the first aspect.

[0056] In a fifth aspect, the present invention provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the method provided in the first aspect.

[0057] Compared with the prior art, the spatiotemporal fusion optimization method of multi-source sensors provided in the present invention automatically selects data with higher confidence from data with lower confidence perceived by the sensors, and then uses the data with higher confidence to correct the time calibration and space calibration of the first sensor and the second sensor, thereby improving the correction accuracy, thereby realizing the correction of the time and space calibration of different sensors, effectively solving the problem of spatiotemporal asynchrony of data perceived by different sensors after the sensors have been running for a period of time, and improving the accuracy of subsequent data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a spatiotemporal fusion optimization method for multi-source sensors provided in an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a camera coordinate system provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of a spatiotemporal fusion optimization device for multi-source sensors provided by an embodiment of the present invention;

[0061] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0063] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc. indicate orientations or positional relationships based on the orientations or positional relationships described in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present invention.

[0064] In addition, it should be understood that the use of terms such as "first", "second", and "third" to define components is only for the convenience of distinguishing the above components. If not otherwise stated, the above terms have no special meanings and therefore cannot be understood as limiting the scope of protection of the present invention. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0065] Figure 1 A flow chart of a spatiotemporal fusion optimization method for a multi-source sensor provided in an embodiment of the present invention. It can be understood that the method can be executed by any device, equipment, platform, or equipment cluster with computing and processing capabilities. The method can be applied to a system including N sensors, where N is greater than or equal to 2, and the N sensors include a first sensor and a second sensor, wherein the first sensor and the second sensor are in the same area (for example, the two can be but are not limited to being installed on the same mounting frame, or the distance between the two is within a preset range, etc.), and there is an overlapping area (i.e., an overlapping area) between the perception area of ​​the first sensor and the perception area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to the overlapping area between different sensors. Exemplarily, the first sensor can be a camera or the like, and the second sensor can be a millimeter-wave radar, etc. As Figure 1 Said method may include the following steps:

[0066] S101 , respectively converting a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system.

[0067] Specifically, a first coordinate system with the first sensor itself as the origin can be transformed into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin can be transformed into a geodetic coordinate system respectively by reverse perspective transformation or the like.

[0068] Exemplarily, when the first sensor or the second sensor is a millimeter wave radar, the spherical coordinate system of the millimeter wave radar itself can be converted into a Cartesian coordinate system, and then the coordinate system with the millimeter wave radar itself as the origin can be converted into a geodetic coordinate system.

[0069] When the first sensor or the second sensor is a camera, Figure 2 As shown, the camera coordinate system can be converted to the earth coordinate system by the following conversion formula. The specific conversion formula is:

[0070] x=Δm+(h-Δz)sinα / tanββ; y=Δn+(h-Δz)cosα / tanβ

[0071] Where x and y are the horizontal and vertical coordinates in the geodetic coordinate system, Δm is the lateral offset of the camera relative to the fixed point, Δn is the longitudinal offset relative to the fixed point, (h-Δz) is the height of the target, α is the yaw angle of the camera, and β is the pitch angle of the camera.

[0072] S102. When the distance between the first target and the remaining targets in the overlapping area sensed by the first sensor and / or the second sensor is greater than a preset value, first perception data related to the first target sensed by the first sensor is obtained, where n is greater than or equal to 0, and the first target includes a target in a stationary state or a target in a moving state.

[0073] Specifically, when the distance between the first target and the remaining targets in the overlapping area sensed by the first sensor and / or the second sensor is greater than a preset value, the first sensing data related to the first target sensed by the first sensor can be obtained, n is greater than or equal to 0, wherein the first target includes a target in a stationary state or a target in a moving state. Since it is difficult to determine whether the targets in different data frames are the same target when the distance between the targets in the overlapping area is close, when the distance between one of the targets and the other targets is large, the relevant sensing data of the target is obtained, that is, the time calibration and space calibration of the first sensor and the second sensor can be corrected more accurately. Among them, it can be understood here that the data with higher confidence is automatically selected from the data with lower confidence, and then the time calibration and space calibration of the first sensor and the second sensor are corrected by the data with higher confidence, thereby improving the correction accuracy. Exemplarily, the data with lower confidence can be understood as the data that is difficult to determine the target, and the data with higher confidence can be understood as the data that is easy to determine the target.

[0074] S103. Based on the first perception data, determine second perception data related to the first target object from the data perceived by the second sensor.

[0075] Specifically, based on the first perception data and the pre-calibrated time, a data set corresponding to the first target object in the first perception data is determined from the second perception data, wherein the data set can constitute the second perception data. The data sets of the first perception data and the second perception data can be referred to as a time-space vector set. A certain position of the first target object at a certain time in the first perception data can be found in the second perception data to be the same or similar position.

[0076] As a possible implementation, a first data frame containing a first target object can be randomly selected from the first perception data, and the first time when the first data frame is collected and the first position of the first target object in the first data frame are determined. Then, based on the first time and the pre-calibrated time, the second data frame and the third data frame are determined from the second perception data, and the second time when the second data frame is collected and the third time when the third data frame is collected are determined, and the second position of the first target object in the second data frame and the third position of the first target object in the third data frame are determined, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time. Then, through the interpolation fitting algorithm, according to the sum of the first time and the pre-calibrated time, the second position, the third position, the second time and the third time, the position of the first target object when the time in the data perceived by the second sensor is the sum of the first time and the pre-calibrated time is determined, and the position corresponds to the first position. In this way, the data perceived by the second sensor is processed based on each data frame in the first perception data in turn, and the second perception data can be obtained.

[0077] S104. When the first target object is a stationary target object, based on a first position of the first target object in the first perception data and a second position corresponding to the first position in the second perception data, the spatial calibration of the first sensor and the second sensor is corrected.

[0078] Specifically, after acquiring the first perception data and the second perception data, when the first target object is a stationary object, the spatial calibration of the first sensor and the second sensor can be corrected based on the first position of the first target object in the first perception data and the second position corresponding to the first position in the second perception data.

[0079] As a possible implementation, one of the first sensor and the second sensor can be randomly selected as a reference, and based on the deviation between the first position and the second position, the spatial parameters of the sensor that is not selected as the reference can be corrected. Exemplarily, after the first sensor and the second sensor have been running for a period of time, if neither of them has drifted, the first position and the second position are the same position, and if at least one of them has drifted, the first position and the second position will have a deviation, so one of the sensors can be selected, and it is assumed that the sensor has not drifted, and based on the deviation between the two positions, the spatial parameters of the other sensor can be corrected. Exemplarily, the spatial parameters may include pitch angle and / or yaw angle, etc.

[0080] Exemplarily, a data frame containing a first target object can be randomly selected from the first perception data, and then the position of the first target object in the data frame is used as the first position. Next, based on the time when the data frame is collected and the pre-calibrated time, two adjacent data frames containing the first target object are determined from the second perception data, and the collection time of the two adjacent data frames is determined, wherein the sum of the time when the first sensor collects the first target object at the first position and the pre-calibrated time is between the collection time of the two adjacent data frames collected by the determined second sensor. Finally, the second position corresponding to the first position in the second perception data is determined by an interpolation fitting algorithm from the three determined times and the position of the first target object in the two adjacent data frames. It can be understood that at this time, the sampling frequency of the first sensor can be lower than the sampling frequency of the second sensor.

[0081] S105. When the first target is in motion, the time calibration of the first sensor and the second sensor is corrected based on the motion time and motion position of the first target in the first perception data and the motion time and motion position of the first target in the second perception data.

[0082] Specifically, after acquiring the first perception data and the second perception data, when the first target object is a target in motion, the time calibration of the first sensor and the second sensor can be corrected based on the movement time and movement position of the first target object in the first perception data, and the movement time and movement position of the first target object in the second perception data.

[0083] As a possible implementation, a first data frame containing a first target object can be randomly selected from the first perception data, and the first time when the first data frame is collected and the first position of the first target object in the first data frame are determined. Then, based on the first time, the second data frame and the third data frame are determined from the second perception data, and the second time when the second data frame is collected and the third time when the third data frame is collected are determined, and the second position of the first target object in the second data frame and the third position of the first target object in the third data frame are determined, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time. Then, according to the first position, the second position, the third position, the second time and the third time, the fourth time when the second sensor perceives the first target object when the first target object is in the first position is determined. Finally, according to the time difference between the first time and the fourth time, the time calibration of the first sensor and the second sensor is corrected. Among them, the sampling frequency of the first sensor is lower than the sampling frequency of the second sensor. Exemplarily, the time difference calculated at this time can be compared with the time difference in the pre-calibrated time to determine the time correction parameter.

[0084] Therefore, the spatiotemporal fusion optimization method of multi-source sensors provided in the present invention automatically selects data with higher confidence from data with lower confidence perceived by the sensors, and then corrects the time calibration and space calibration of the first sensor and the second sensor with the data with higher confidence, thereby improving the correction accuracy, thereby realizing the correction of the time and space calibration of different sensors, effectively solving the problem of spatiotemporal asynchrony of data perceived by different sensors after the sensors have been running for a period of time, and improving the accuracy of subsequent data fusion.

[0085] In some embodiments provided by the present invention, before the above S103, it is also possible to

[0086] In some embodiments provided by the present invention, after the time and space calibration of the first sensor and the second sensor are corrected, the target state data of at least one target object included in the first perception data and / or the second perception data can be output based on the corrected space parameters and time parameters. Exemplarily, the target state data may include: the time and space position of the target object, the speed of the target object, the source of the target object (i.e., which sensor perceives it), or one or more of the dot matrix information of the target object.

[0087] As a possible implementation manner, the corrected spatial parameters and time parameters may be processed based on a preset data conversion algorithm to obtain target state data of at least one target object.

[0088] It should be noted that the sensor involved in the embodiments of the present application may be a camera, or a millimeter wave radar, a microwave radar, a lidar, a visible light / infrared / far infrared device, or any combination thereof, which may be determined according to the actual situation and is not limited here. In addition, the overlapping area involved in the embodiments of the present application may be the overlapping area of ​​two sensors, or the overlapping area of ​​three or more sensors, which may be determined according to the actual situation and is not limited here. In addition, the number of sensors involved in the embodiments of the present application may also be greater than two, which may be determined according to the actual situation and is not limited here.

[0089] Based on the method in the above embodiment, the present invention also provides a time-space fusion optimization device for multi-source sensors. The device can be deployed in a system including N sensors, N is greater than or equal to 2, and the N sensors include a first sensor and a second sensor, wherein the first sensor and the second sensor are in the same area, and there is an overlapping area between the sensing area of ​​the first sensor and the sensing area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to the overlapping area between different sensors. Figure 3, the device 300 comprises:

[0090] A coordinate conversion module 301, used to convert a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system;

[0091] The data acquisition module 302 is configured to acquire first perception data related to the first target object sensed by the first sensor when the distance between the first target object and the remaining targets in the overlapping area sensed by the first sensor and / or the second sensor is greater than a preset value, where n is greater than or equal to 0, wherein the first target object includes a target object in a stationary state or a target object in a moving state, and, based on the first perception data, determine second perception data related to the first target object from data sensed by the second sensor;

[0092] The correction module 303 is used to correct the spatial calibration of the first sensor and the second sensor based on the first position of the first target in the first perception data and the second position corresponding to the first position in the second perception data when the first target is a stationary target, and to correct the time calibration of the first sensor and the second sensor based on the movement time and movement position of the first target in the first perception data and the movement time and movement position of the first target in the second perception data when the first target is a moving target.

[0093] In one example, the correction module 303 is further configured to:

[0094] One of the first sensor and the second sensor is randomly selected as a reference object, and the spatial parameter of the sensor that is not selected as the reference object is corrected based on the deviation between the first position and the second position.

[0095] In one example, the sampling frequency of the first sensor is lower than the sampling frequency of the second sensor;

[0096] The correction module 303 is further used for:

[0097] Randomly selecting a first data frame including a first target object from the first perception data, and determining a first time when the first data frame is collected and a first position of the first target object in the first data frame;

[0098] Based on the first time, determining a second data frame and a third data frame including the first target object from the second perception data, determining a second time when the second data frame is collected and a third time when the third data frame is collected, and determining a second position of the first target object in the second data frame and a third position of the first target object in the third data frame, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time;

[0099] Determine, based on the first position, the second position, the third position, the second time, and the third time, a fourth time when the second sensor senses the first target when the first target is at the first position;

[0100] The time calibration of the first sensor and the second sensor is corrected according to the time difference between the first time and the fourth time.

[0101] In one example, the sampling frequency of the first sensor is lower than the sampling frequency of the second sensor;

[0102] The correction module 303 is further used for:

[0103] Based on the first perception data and the pre-calibrated time, a data set corresponding to the first target object in the first perception data is determined from the second perception data, and the data set constitutes the second perception data.

[0104] In one example, the apparatus further comprises:

[0105] An output module (not shown in the figure) is used to output the target state data of at least one target object included in the first perception data and / or the second perception data based on the modified spatial parameters and time parameters.

[0106] In one example, the output module is also used to:

[0107] Based on a preset data conversion algorithm, the corrected spatial parameters and time parameters are processed to obtain target state data of at least one target object.

[0108] In one example, the target state data includes one or more of the following:

[0109] The spatial and temporal position of the target object, the speed of the target object, the source of sensing the target object, or the dot matrix information of the target object.

[0110] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0111] Figure 4Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 4 To implement the above embodiment, an embodiment of the present invention further provides an electronic device, the electronic device 400 includes a memory 401 and a processor 402;

[0112] The processor 402 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 401, so as to implement the spatiotemporal fusion optimization method of the multi-source sensor in the above embodiment.

[0113] In some embodiments, each sensor involved in the embodiments of the present application can be integrated with the electronic device 400 but is not limited to being integrated with the electronic device 400.

[0114] In order to implement the above embodiment, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the spatiotemporal fusion optimization method of multi-source sensors in the above embodiment is implemented.

[0115] In order to implement the above embodiment, the embodiment of the present invention further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the spatiotemporal fusion optimization method of multi-source sensors in the above embodiment.

[0116] It is understandable that the processor in the embodiments of the present invention may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0117] The method steps in the embodiments of the present invention can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0118] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0119] It should be understood that the various numerical numbers involved in the embodiments of the present invention are only used for the convenience of description and are not used to limit the scope of the embodiments of the present invention.

Claims

1. A spatiotemporal fusion optimization method for multi-source sensors, characterized in that: Applicable to a system including N sensors, where N is greater than or equal to 2, the N sensors include a first sensor and a second sensor, the first sensor and the second sensor are in the same area, and there is an overlapping area between the sensing area of ​​the first sensor and the sensing area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to the overlapping area between different sensors; The method comprises: Respectively converting a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system; When the distance between the first target and the remaining targets among the n targets sensed by the first sensor and / or the second sensor in the overlapping area is greater than a preset value, first sensing data related to the first target sensed by the first sensor is acquired, where n is greater than or equal to 0, and the first target includes a stationary target or a moving target; Based on the first perception data, determining second perception data related to the first target object from the data perceived by the second sensor; When the first target object is a stationary target object, based on a first position of the first target object in the first perception data and a second position corresponding to the first position in the second perception data, correcting the spatial calibration of the first sensor and the second sensor; When the first target is a moving target, the time calibration of the first sensor and the second sensor is corrected based on the movement time and movement position of the first target in the first perception data and the movement time and movement position of the first target in the second perception data.

2. The method according to claim 1, characterized in that: The correcting the spatial calibration of the first sensor and the second sensor based on the first position of the first target object in the first perception data and the second position corresponding to the first position in the second perception data specifically includes: One of the first sensor and the second sensor is randomly selected as a reference object, and the spatial parameters of the sensor that is not selected as the reference object are corrected based on the deviation between the first position and the second position.

3. The method according to claim 1, characterized in that The sampling frequency of the first sensor is lower than the sampling frequency of the second sensor; The correcting the time calibration of the first sensor and the second sensor based on the movement time and the movement position of the first target in the first perception data and the movement time and the movement position of the first target in the second perception data specifically includes: Randomly selecting a first data frame including the first target object from the first perception data, and determining a first time when the first data frame is collected, and a first position of the first target object in the first data frame; Based on the first time, determining a second data frame and a third data frame including the first target object from the second perception data, determining a second time when the second data frame is collected and a third time when the third data frame is collected, and determining a second position of the first target object in the second data frame and a third position of the first target object in the third data frame, wherein the second data frame and the third data frame are adjacent data frames, and the first time is greater than the second time and less than the third time; Determine, based on the first position, the second position, the third position, the second time, and the third time, a fourth time when the second sensor senses the first target when the first target is at the first position; The time calibration of the first sensor and the second sensor is corrected according to the time difference between the first time and the fourth time.

4. The method according to any one of claims 1 to 3, characterized in that: The sampling frequency of the first sensor is lower than the sampling frequency of the second sensor; The determining, based on the first perception data, second perception data related to the first target object from the data perceived by the second sensor specifically includes: Based on the first perception data and a pre-calibrated time, a data set corresponding to the first target object in the first perception data is determined from the second perception data, and the data set constitutes the second perception data.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Based on the corrected spatial parameters and temporal parameters, target state data of at least one target object included in the first perception data and / or the second perception data is output.

6. The method according to claim 5, characterized in that The step of outputting target state data of at least one target object included in the first perception data and / or the second perception data based on the corrected spatial parameter and time parameter specifically includes: Based on a preset data conversion algorithm, the corrected spatial parameters and time parameters are processed to obtain target state data of the at least one target object.

7. The method according to claim 5, characterized in that The target state data includes one or more of the following: The spatial and temporal position of the target object, the speed of the target object, the source of sensing the target object, or the dot matrix information of the target object.

8. The method according to claim 1, characterized in that The N sensors include one or more of a camera, a millimeter wave radar, a microwave radar, a lidar, a visible light device, an infrared device or a far infrared device.

9. A device for optimizing the spatiotemporal fusion of multi-source sensors, characterized in that: Deployed in a system including N sensors, N is greater than or equal to 2, the N sensors include a first sensor and a second sensor, the first sensor and the second sensor are in the same area, and there is an overlapping area between the sensing area of ​​the first sensor and the sensing area of ​​the second sensor, and the time and space of the first sensor and the second sensor have been calibrated in advance, wherein the overlapping area refers to the overlapping area between different sensors; The device comprises: A coordinate conversion module, used for converting a first coordinate system with the first sensor itself as the origin into a geodetic coordinate system and a second coordinate system with the second sensor itself as the origin into a geodetic coordinate system; a data acquisition module, configured to acquire first perception data related to the first target object sensed by the first sensor when the distance between the first target object and the remaining targets in the overlapping area sensed by the first sensor and / or the second sensor is greater than a preset value, wherein n is greater than or equal to 0, wherein the first target object includes a target object in a stationary state or a target object in a moving state, and, based on the first perception data, determine second perception data related to the first target object from data sensed by the second sensor; A correction module is used to correct the spatial calibration of the first sensor and the second sensor based on a first position of the first target in the first perception data and a second position corresponding to the first position in the second perception data when the first target is a stationary target, and to correct the time calibration of the first sensor and the second sensor based on a movement time and a movement position of the first target in the first perception data and a movement time and a movement position of the first target in the second perception data when the first target is a moving target.

10. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 8.