Data processing method and device

Abstract

The present invention relates to the field of intelligent transportation technology, and provides a data processing method, device, computer-readable storage medium and electronic device. In an embodiment, the method includes: performing spatial fusion based on the attitude angles of each of the pre-stored multiple sensors to determine the perception results of each frame in the current associated frame set; performing temporal fusion on the current and previous multiple associated frame sets based on the sampling time difference between the pre-stored multiple sensors to determine the temporal fusion result; performing target fusion based on the temporal fusion result to determine the target fusion result; determining overlapping area position data for indicating the positions of several overlapping areas of the multiple sensors; and correcting some or all of the attitude angles and / or some or all of the sampling time differences among the multiple sensors based on the target fusion result, the temporal fusion result and the overlapping area position data. Through the technical solution of the embodiment of the present application, the accuracy of the temporal and spatial calibration of multiple sensors can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of social economy, intelligent transportation is an important means to achieve road safety monitoring. However, the road traffic environment is becoming increasingly complex, and the requirements for road traffic management are becoming higher and higher. The existing road management methods can no longer meet the needs of intelligent management and diversified travel services. In order to achieve refined road control, multi-source traffic information holographic perception and high-quality traffic data extraction technology are urgently needed.

[0003] In the related technology, although it is possible to obtain traffic information from multiple sources and calibrate the time and space of different sensors, any sensor will drift after running for a long time, which will lead to the problem of time and space asynchrony of the data sensed by subsequent sensors. Summary of the invention

[0004] The present invention provides a data processing method, device, computer-readable storage medium and electronic device, which can realize the correction of the time and space calibration of multiple sensors, effectively solve the problem of the time and space asynchrony of data sensed by different sensors after the sensors have been running for a period of time, and improve the accuracy of subsequent data fusion. In addition, since the associated frame sets of multiple sensors are considered, the data of the sensors can be flexibly added or deleted.

[0005] In a first aspect, the present invention provides a data processing method, which pre-stores attitude angles of a plurality of sensors and sampling time differences between the plurality of sensors, wherein detection areas of some of the plurality of sensors overlap, and the method further comprises:

[0006] Based on the attitude angles of the multiple sensors, spatially fuse the current associated frame set to determine the perception results of each frame in the associated frame set; wherein the associated frame set represents the frames collected by the multiple sensors in the same time period;

[0007] Based on the sampling time difference between the multiple sensors, time-fusing the current and previous multiple associated frame sets to determine a time-fusion result; wherein the time-fusion result includes the sampling time and the perception result of each frame in the multiple associated frame sets;

[0008] Perform target fusion based on the time fusion result to determine the target fusion result;

[0009] Determining overlapping region position data; wherein the overlapping region position data indicates the position of at least one overlapping region of the plurality of sensors;

[0010] Based on the target fusion result, the time fusion result and the overlapping area position data, the attitude angles of part or all of the multiple sensors and / or the sampling time differences between part or all of the multiple sensors are corrected.

[0011] In a second aspect, the present invention provides a data processing device, including a perception module, a time fusion module, a target fusion module and a calibration correction module, wherein the perception module pre-stores the attitude angles of a plurality of sensors, and the time fusion module pre-stores the sampling time differences between the plurality of sensors, including:

[0012] A perception module, configured to perform spatial fusion on a current associated frame set based on the respective attitude angles of the multiple sensors, and determine a perception result of each frame in the associated frame set; wherein the associated frame set represents frames collected by the multiple sensors in the same time period;

[0013] A time fusion module, configured to perform time fusion on the current and previous multiple associated frame sets based on the sampling time differences between the multiple sensors, and determine a time fusion result; wherein the time fusion result includes the sampling time and perception result of each frame in the multiple associated frame sets;

[0014] A target fusion module, used for performing target fusion based on the time fusion result and determining the target fusion result;

[0015] A position determination module, configured to determine overlapping area position data; wherein the overlapping area position data indicates the position of at least one overlapping area of ​​the plurality of sensors;

[0016] A correction module is used to correct the attitude angles of some or all of the multiple sensors in the perception module and / or correct the sampling time differences between some or all of the multiple sensors in the time fusion module based on the target fusion result, the time fusion result and the overlapping area position data.

[0017] In a third aspect, the present invention provides a computer-readable storage medium, comprising execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes any method described in the first aspect.

[0018] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes any method described in the first aspect.

[0019] The present invention provides a data processing method, device, computer-readable storage medium and electronic device. The method performs spatial fusion through the pre-stored attitude angles of multiple sensors to determine the perception results of each frame in the current associated frame set; wherein the associated frame set represents the frames collected by multiple sensors in the same period, and the detection areas of some sensors in the multiple sensors overlap; then, based on the pre-stored sampling time difference between the multiple sensors, time fusion is performed to determine the time fusion result; wherein the time fusion result includes the sampling time and perception results of each frame in the current and previous multiple associated frame sets; then, based on the time fusion result, target fusion is performed to determine the target fusion result; then, the overlapping area position data is determined; wherein the overlapping area position data indicates the positions of several overlapping areas of the multiple sensors; then, based on the target fusion result, the attitude angles of some or all of the multiple sensors and / or the sampling time differences between some or all of the multiple sensors are corrected. In summary, through the technical solution of the present invention, the time and space calibration of different sensors can be corrected, which effectively solves the problem of the time and space asynchronism of the data perceived by different sensors after the sensors have been running for a period of time, and improves the accuracy of subsequent data fusion. In addition, since the associated frame sets of multiple sensors are considered, sensors can be flexibly added or deleted, so that the time and space calibration of multiple sensors can be flexibly realized.

[0020] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the yaw angle of the sensor provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the yaw angle of a target object detected by a sensor provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the locations of multiple sensors provided in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the position of the same target object detected by different sensors provided in an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, a method provided by an embodiment of the present invention is provided. The method provided by an embodiment of the present invention can be applied to an electronic device, and can be specifically applied to a server or a general computer. In this embodiment, the method specifically includes the following steps:

[0031] like Figure 1 The present invention provides a data processing method, comprising the following steps:

[0032] Step 101: Based on the respective attitude angles of multiple sensors, spatial fusion is performed on the current associated frame set to determine the perception results of each frame in the associated frame set; wherein the associated frame set represents frames collected by multiple sensors in the same time period.

[0033] In a possible implementation, the attitude angles of the multiple sensors indicate the attitude angles of the multiple sensors after the last spatial calibration. The attitude angle indicates the sum of the installation angle and the detection angle based on the positive direction of the world coordinate system. The detection angle indicates the detection range of the sensor, and the installation angle indicates the angle between the edge of the detection angle close to the positive direction of the world coordinate system and the positive direction of the world coordinate system.

[0034] It should be noted that for the sensor, the detection angle will not change, but some sensors may move slightly during use, causing the installation angle to change.

[0035] In one example, for a sensor, the attitude angle includes a yaw angle A and a pitch angle B. Figure 2As shown, the roll angle A is the sum of the installed roll angle a1 and the detected roll angle a2 of the sensor. The pitch angle B is similar to the pitch angle A, and the pitch angle B is the sum of the installed pitch angle b1 and the detected pitch angle b2 of the sensor.

[0036] As a possible implementation, for a frame collected by each of the multiple sensors, the perception result of the frame includes the position of each of the multiple target objects in the world coordinate system and the relative position between the target object and the sensor. The relative position indicates the positional relationship between the target object and the sensor. Furthermore, multiple static features of the target object may also be included, such as vehicle type, color, size, lane, etc. It should be noted that the target object is a moving object on the road, usually various types of vehicles, such as cars, bicycles, road sweepers, trucks, etc.

[0037] In one example, an electronic device pre-stores the positions of multiple sensors in a world coordinate system. For frames captured by each of the multiple sensors, target detection is performed on the frames, and the relative position of the detected target is determined based on the attitude angle of the sensor. The position of the target in the world coordinate system is determined based on the relative position and the preset position of the sensor in the world coordinate system.

[0038] It should be noted that spatial fusion can be understood as converting the positions of the target objects detected by multiple sensors into the same coordinate system, such as the world coordinate system.

[0039] In a possible implementation, frames collected by multiple sensors are synchronized to obtain associated frame sets of the multiple sensors.

[0040] It should be noted that frame synchronization can be understood as finding the data with the shortest time interval between each frame and the reference sensor through the nearest matching method. For example, a reference sensor among multiple sensors is determined, the sampling time of the frame last collected by the reference sensor is determined, and a frame collected by other sensors with the smallest difference from the sampling time is found through the nearest matching method, thereby obtaining the frames currently collected by multiple sensors. For example, the reference sensor can be understood as the sensor with the highest sampling frequency, or the sensor that senses the target the latest; correspondingly, the frames collected by multiple sensors can be understood as the last frame collected by multiple sensors, that is, the current frame.

[0041] It should be noted that the end time of the time period indicated by the associated frame set is the current time, and the duration is equal to the sampling time difference of the sensor with the largest sampling frequency among multiple sensors, so that the frames collected by each sensor can be obtained. In some possible cases, the associated frame set may include all frames collected by multiple sensors within a preset time period. The specific needs are determined in combination with actual conditions, and this embodiment does not make specific limitations on this. It should be understood that the associated frame set is used to represent the situation of the area currently detected by multiple sensors.

[0042] As a feasible implementation manner, a plurality of sensors are arranged on a gantry, and the plurality of sensors form an overlapping area.

[0043] As another feasible implementation, multiple sensors are arranged on different gantries; wherein the detection areas of the multiple sensors on adjacent gantries overlap to form an overlapping area; or, the detection areas of the multiple sensors on one gantry overlap to form an overlapping area. The overlapping area can be understood as a continuous overlapping area among the multiple sensors, and different overlapping areas are discontinuous.

[0044] For example, Figure 4 As shown, the detection areas of sensor 1, sensor 2 and sensor 3 overlap to form overlapping area 1. The detection areas of sensor 4, sensor 5, sensor 6 and sensor 7 overlap to form overlapping area 2.

[0045] Step 102: Based on the sampling time difference between multiple sensors, time fusion is performed on the current and previous multiple associated frame sets to determine the time fusion result; wherein the time fusion result includes the sampling time and perception result of each frame in the multiple associated frame sets.

[0046] In a possible implementation, the sampling time difference between the multiple sensors is the sampling time difference between different sensors after the last time calibration.

[0047] In one example, the sampling time difference between the multiple sensors indicates the time difference between the sampling times of the reference sensor and the other sensors in the multiple sensors.

[0048] For example, the reference sensor may be a sensor having the highest sampling frequency among the multiple sensors.

[0049] It should be noted that time fusion can be understood as unifying multiple frames collected by multiple sensors into the same timestamp, that is, aligning the time starting points of different sensors based on the sampling time difference between multiple sensors, thereby achieving time unification.

[0050] In actual applications, multiple keyframe sets within a period of time, for example, 1 second, are cached.

[0051] Step 103: Perform target fusion based on the time fusion result to determine the target fusion result.

[0052] Target fusion refers to fusing the targets sensed by different sensors at different sampling times to determine a more reliable target detection result. Target fusion technology is an existing technology and will not be described in detail here.

[0053] As a possible implementation method, the target fusion result includes the positions of several targets in the world coordinate system at different sampling times, their relative positions to the sensor, and dynamic features. The dynamic features may include whether they are moving, the speed of movement, acceleration, etc. Furthermore, multiple static features of the target may also be included, such as vehicle type, color, size, etc. It should be noted that the target is a moving object on the road, usually various types of vehicles, such as cars, bicycles, road sweepers, trucks, etc.

[0054] As a feasible implementation method, the electronic device obtains the weather conditions of the target road section; performs target fusion based on the weather conditions, the sampling time of each frame in the frame set and the respective perception results, and determines the target fusion results of multiple sensors.

[0055] In this implementation, the frames collected by different sensors are screened by considering the weather conditions, thereby ensuring the accuracy of the target fusion result.

[0056] Step 104: Determine overlapping area position data; wherein the overlapping area position data indicates the position of at least one overlapping area of ​​the multiple sensors.

[0057] The overlapping area position data includes positions of a plurality of overlapping areas of the plurality of sensors in a world coordinate system.

[0058] As a feasible implementation manner, the electronic device determines the overlapping area position data based on the sensor position data of the multiple sensors and the attitude angles of the multiple sensors respectively.

[0059] The sensor position data indicates the positions of the multiple sensors, which may be the positions of the multiple sensors in the world coordinate system, or the relative position relationship between the multiple sensors, preferably the positions in the world coordinate system. In practical applications, the positions of the multiple sensors in the world coordinate system are manually input into the electronic device, so that the electronic device stores the positions of the multiple sensors in the world coordinate system.

[0060] Furthermore, based on the positions of each of the multiple sensors in the world coordinate system and the attitude angles of each of the multiple sensors, the positions of each of the multiple overlapping regions in the world coordinate system are determined, thereby obtaining overlapping region position data.

[0061] If the attitude angle of the sensor changes, the position of the overlapping area may also change. Therefore, in practical applications, it is necessary to correct the position of the overlapping area in real time based on the attitude angle of the sensor, that is, to correct the position data of the overlapping area.

[0062] Step 105: Based on the target fusion result, the time fusion result and the overlapping area position data, correct the attitude angles of part or all of the multiple sensors and / or correct the sampling time differences between part or all of the multiple sensors.

[0063] In this embodiment, the attitude angles of some or all of the multiple sensors and / or the sampling time differences between some or all of the multiple sensors are corrected through the target fusion results and the time fusion results, and the frames collected by the multiple sensors are processed based on the corrected attitude angles and / or sampling time differences to ensure the accuracy of the subsequent target fusion results.

[0064] Furthermore, for the sensor with corrected attitude angle, the position of the target object detected by the sensor in the target fusion result is corrected based on the corrected attitude angle of the sensor.

[0065] For multiple sensors that correct sampling time differences, the sampling moments of the targets sensed by these sensors in the target fusion result are corrected based on the sampling time differences between these sensors.

[0066] As a feasible implementation method, the electronic device can correct the attitude angle and sampling time difference in the following implementation method:

[0067] Based on the target fusion result and the overlapping area position data, at least one first target and at least one second target are determined; wherein the first target is a stationary or moving target in any area of ​​the at least one overlapping area that meets the low-speed motion condition, and the second target is a moving target in any area of ​​the at least one overlapping area that does not meet the low-speed motion condition; for each target of the at least one first target, based on the position of the first target sensed by each of the multiple first sensors in the time fusion result, the attitude angles of each of the multiple first sensors are corrected; wherein the multiple first sensors are multiple sensors among the multiple sensors that sense the first target in the same time period; based on the attitude angles of each of the multiple first sensors after correction of each of the at least one first target, the positions of some or all of the targets in the target fusion result are corrected; for each target of the at least one second target, based on the position of the second target sensed by each of the multiple second sensors at different sampling times in the corrected target fusion result, the sampling time difference between the multiple second sensors is corrected; wherein the multiple second sensors are multiple sensors among the multiple sensors that sense the second target in the same time period.

[0068] In one example, the electronic device may specifically determine the first target object and / or the second target object in the following implementation manner:

[0069] For each area of ​​at least one overlapping area, when it is determined that at least one target object exists in the overlapping area based on the overlapping area position data and the target fusion result, at least one first target object and / or at least one second target object are determined.

[0070] Among them, the first target object is stationary or traveling at a lower speed in the overlapping area, preferably stationary. Among them, the lower speed can be understood as the walking speed or the speed of a sweeper, for example, 5km / h. It should be understood that the sensor is not prone to drift when traveling at a lower speed. Therefore, stationary or low-speed targets can be used for spatial calibration. It should be noted that for the first target object traveling at a low speed, since the first target object travels at a uniform slow speed, the difference between the positions of the first target object at different sampling times is small, and it can be considered to be stationary.

[0071] Furthermore, one overlapping area may determine one first target object. The distance between the first target object and other targets other than the first target object in the overlapping area is greater than a preset value, or the first target object is the only target object in the overlapping area, or the first target object is the only target object in any lane in the overlapping area.

[0072] The second target is a moving target in the overlapping area, which usually moves at a higher speed.

[0073] Furthermore, one overlapping area may determine one second target object. The distance between the second target object and other targets other than the second target object in the overlapping area is greater than a preset value, or the second target object is the only target object in the overlapping area, or the second target object is the only target object in any lane in the overlapping area.

[0074] For example, Figure 4 As shown, if the target fusion result includes the positions of target 1 and target 2 at sampling times T0 to T2, target 1 in overlapping area 1 is stationary, and target 1 is used as the first target in overlapping area 1; target 2 in overlapping area 2 is moving, and target 2 is used as the second target in overlapping area 2. In addition, for target 1, sensors 1 to 3 are all target sensors; for target 2, sensors 4 to 7 are all target sensors.

[0075] It is worth noting that in this embodiment, there is no need to consider whether the targets in different overlapping areas are the same. If the overlapping areas where the first target is located are different, they can be considered to be different targets. Therefore, the position of the first target should be understood as the position of the first target in the overlapping area where it is located, and the second target is similar. In actual application, for the overlapping area where the first target and / or the second target exist, if there is a first target in the overlapping area, based on the relevant data corresponding to the first target in the overlapping area, such as the respective positions at different sampling times in the time fusion result, and the respective positions at different sampling times in the target fusion result, the multiple first sensors detecting the overlapping area are spatially calibrated; if there is a second target in the overlapping area, based on the relevant data corresponding to the second target in the overlapping area, such as the respective positions at different sampling times in the time fusion result, and the respective positions at different sampling times in the target fusion result, the multiple second sensors detecting the overlapping area are time-calibrated.

[0076] Of course, if some targets move faster and appear in multiple overlapping areas in the time fusion result, then the target can be used as the second target in any of the multiple overlapping areas. In other words, the first target is only located in one overlapping area, and the second target is also only located in one overlapping area. In practical applications, the distance between adjacent sensors set in the longitudinal direction of the road is usually more than 100 meters, and road monitoring is mainly for highways or roads in the city. Therefore, the maximum speed of the target is 33.3m / s. The method provided in this embodiment requires real-time monitoring of the road, and the setting of the overlapping area will also be reasonably arranged based on the road conditions. Therefore, in a short period of time, a target will generally only appear in one overlapping area, and only in very rare cases will it appear in multiple overlapping areas.

[0077] In one example, for any first target object, the electronic device can implement spatial calibration in the following manner:

[0078] Determine the position of the first target object sensed by each of the multiple first sensors in the time fusion result in the world coordinate system, and determine the spatial calibration position of the first target object; for any sensor of the multiple first sensors, determine the first target attitude angle of the first sensor based on the position of the first sensor in the world coordinate system, the spatial calibration position and the positive direction of the world coordinate system; determine the second target attitude angle of the first sensor based on the position of the first target object relative to the first sensor in the time fusion result and the positive direction of the world coordinate system; determine the attitude angle difference based on the first target attitude angle and the second target attitude angle; and correct the attitude angle of the first sensor based on the attitude angle difference and the preset attitude angle of the first sensor.

[0079] It should be noted that the first sensor is used to detect the overlapping area where the first target is located, and multiple first sensors detect the same overlapping area. In addition, multiple first sensors sense the first target in the same time period, and the same time period here can be understood as the time period when the first target passes through the overlapping area of ​​the detection areas of multiple first sensors.

[0080] It should also be noted that when the first target is stationary, the spatial calibration position can be the position of the first target in the world coordinate system in the target fusion result, or the weighted average of the positions of the first target in the world coordinate system sensed by multiple first sensors in the time fusion result. When the first target moves at a low speed, the spatial calibration position of the first target is determined based on the position of the first target in the world coordinate system at the target sampling moment in the target perception result and the positive direction of the world coordinate system; for any of the multiple first sensors, the relative position of the first target relative to the first sensor at the target sampling moment is determined based on the position of the first target detected by the target sensor in the world coordinate system at different sampling moments in the time fusion result and the position of the first sensor in the world coordinate system, and the second target attitude angle is determined based on the relative position and the positive direction of the world coordinate system. Among them, the target sampling moment should be any sampling moment in the intersection of the time periods of the first target detected by multiple first sensors.

[0081] It is worth noting that the position of the first target relative to the first sensor can be determined based on the position of the first sensor in the world coordinate system and the spatial calibration position. In addition, in some possible cases, when the first target moves at a low speed, since the sampling time difference between the multiple first sensors is small, for any first sensor, the relative position of the first target relative to the target sensor at the sampling time closest to the target sampling time in the time fusion result can be used as the relative position of the first target relative to the target sensor at the target sampling time.

[0082] For any sensor, the calculation formula of the geodetic projection coordinates of the target object in the world coordinate system is as follows:

[0083]

[0084] Where x represents the coordinate of the geodetic projection of the horizontal axis in the world coordinate system; y represents the coordinate of the geodetic projection of the vertical axis in the world coordinate system; z represents the coordinate of the geodetic projection of the vertical axis in the world coordinate system; s represents the horizontal distance between the target object and the center point of the sensor; Figure 4 As shown, h represents the vertical distance between the sensor and the road; Figure 4As shown, α represents the sum of the installation yaw angle a1 of the sensor and the detected yaw angle a3 of the first target; β represents the sum of the installation pitch angle b1 of the sensor and the detected pitch angle b3 of the first target.

[0085] It should be noted that knowing x, y, s, and h can solve α and β in formula (1). For the camera, the horizontal distance s between the target and the sensor can be determined by inverse perspective transformation. It should be noted that, if Figure 3 As shown in the figure, the horizontal distance s can be understood as the horizontal distance between the target and the center point of the sensor. The horizontal distance s remains unchanged even if the attitude angle of the sensor changes. Figure 4 h1 to h7 in FIG. 7 respectively represent the vertical distances between sensor 1 to sensor 7 and the road.

[0086] Wherein, the first target attitude angle and the second target attitude angle both include α and β; however, since different data are used to calculate the first target attitude angle and the second target attitude angle, the first target attitude angle and the second target attitude angle are different. The attitude angle difference includes the yaw angle difference and the pitch angle difference. Since the detected yaw angle a3 and the detected pitch angle b3 are constant, the yaw angle difference indicates the difference between the installed yaw angle a1, and the pitch angle difference indicates the difference between the installed pitch angles b1. It should be noted that the relative position of the first target object from the sensor can be determined based on the spatial calibration position and the position of the sensor in the world coordinate system.

[0087] Furthermore, for the sensor, the corrected attitude angle of the first sensor can be determined based on the attitude angle difference and the preset attitude angle of the first sensor. Considering that the vertical distance h between the sensor and the road is constant, the position of each target object detected by the sensor can be corrected based on the corrected attitude angle.

[0088] In one example, for any second target object, the electronic device can implement time calibration in the following manner:

[0089] Determine a third sensor among the multiple second sensors; for each sensor other than the third sensor among the multiple second sensors, determine the time difference between the second sensor and the third sensor based on the time-calibrated position of the second target object sensed by the second sensor in the corrected target fusion result, and the passed position of the second target object sensed by the third sensor relative to the time-calibrated position, the sampling time corresponding to the passed position, the position to be passed, and the sampling time corresponding to the position to be passed; wherein the time-calibrated position is located between the passed position and the position to be passed.

[0090] like Figure 5As shown, for each sensor other than the third sensor in the plurality of second sensors, the occurrence sampling time of the time calibration position can be calculated specifically by the following formula (2):

[0091]

[0092] Among them, P′ represents the time-calibrated position of the second target object sensed by the second sensor; t′ represents the sampling time of the position P′ of the first target sensed by the second sensor; P0 represents the passed position of the second target object sensed by the third sensor; P represents the to-pass position of the second target object sensed by the third sensor; t0 represents the sampling time of the passed position P0 of the second target sensed by the third sensor; t represents the sampling time of the to-pass position P of the second target sensed by the third sensor.

[0093] In practical applications, t' is within the time period formed by t0 and t; the position of the second target at P' is between the passed position P0 and the position to be passed P. The third sensor can be the sensor with the highest sampling frequency among the multiple second sensors. The passed position P0 and the position to be passed P are the two positions of the second target sensed by the third sensor that are closest to the time calibration position P'.

[0094] It should be noted that the second sensor can be understood as a sensor in the multiple sensors that detects the overlapping area where the second target is located, and the multiple second sensors detect the same overlapping area. In addition, the multiple second sensors sense the second target in the same time period, and the same time period here can be understood as the time period when the second target passes through the overlapping area of ​​the detection areas of the multiple second sensors.

[0095] For example, the detection areas of the multiple sensors overlap, that is, the multiple sensors form an overlapping area, and the third sensor is the sensor with the largest sampling frequency among the multiple sensors.

[0096] For example, the detection areas of the multiple sensors form multiple overlapping areas; for each overlapping area, a third sensor is determined from the multiple second sensors that can detect the second target object in the overlapping area, and then the sampling time differences between each second sensor other than the third sensor and the third sensor are determined.

[0097] In one possible scenario, if the same second sensor has a sampling time difference with different third sensors, for example, the detection range of the second sensor includes multiple overlapping areas, then the sampling difference fusion of different third sensors can be performed based on this, for example, a third sensor is reselected to determine the sampling time difference between other second sensors and the third sensor. It should be noted that if the detection range of the second sensor is far, the detection area of ​​the second sensor may include multiple overlapping areas, usually only one, which needs to be determined in combination with actual conditions. For example, for sections of road where accidents often occur, the detection range of the sensor can be appropriately increased so that one area can be detected by more sensors.

[0098] In some possible cases, the third sensors in different overlapping areas are the same, for example, if the detection range of the third sensor includes at least two overlapping areas.

[0099] In some possible cases, the sampling time differences among multiple sensors may be fused based on the difference between the sampling times of different third sensors in the last frame. For example, a final third sensor is determined from multiple third sensors, and the sampling time differences between other second sensors and the third sensor are determined.

[0100] Furthermore, if the first target object does not exist, time calibration can be performed directly based on the target fusion result.

[0101] In addition, there can be multiple first targets and multiple second targets. The overlapping areas where multiple first targets are located are usually different. If the overlapping areas where some of the first targets are located are the same, for the multiple first targets located in the same overlapping area, the attitude angles of the multiple sensors forming the overlapping area are obtained by spatial calibration based on the multiple first targets, and then the attitude angles of the multiple sensors forming the overlapping area are fused, so that the attitude angles of the multiple sensors forming the overlapping area can be determined more accurately. The overlapping areas where multiple second targets are located are usually different. If the overlapping areas where some of the second targets are located are the same, for the multiple second targets located in the same overlapping area, the sampling time differences between the multiple sensors forming the overlapping area are obtained by time calibration based on the multiple first targets, and then the sampling time differences between the multiple sensors forming the overlapping area are fused, so that the sampling time differences between the multiple sensors forming the overlapping area can be determined more accurately.

[0102] In practical applications, after the attitude angle of the sensor is corrected, the target detected by the sensor will be processed according to the corrected attitude angle, thereby correcting the target fusion result. After the sampling time difference between multiple sensors is corrected, the appearance sampling time of the target detected by these sensors will be processed according to the corrected sampling time difference, further correcting the target fusion result.

[0103] In practical applications, the method provided in this embodiment needs to monitor the road in real time. Therefore, a set of key frames will be collected at preset time intervals, and multiple key frame sets within the most recent time period, such as 1s, will be time-fused and then target fused. Then, the sensor's attitude angle correction and the sampling time difference between different sensors will be corrected. In other words, the method provided in this embodiment is executed in real time at preset time intervals to achieve real-time road monitoring.

[0104] Through the above technical solution, it can be seen that the beneficial effects of this embodiment are:

[0105] On the one hand, the time and space calibration of different sensors can be corrected through the time fusion results and the perception results of different sensors, which effectively solves the problem of time and space asynchrony of data perceived by different sensors after the sensors have been running for a period of time, and improves the accuracy of subsequent data fusion.

[0106] On the other hand, since the data processing method does not limit the number of sensors, in practical applications, sensors can be flexibly added or removed.

[0107] Based on the same concept as the embodiment of the method of the present invention, please refer to Figure 6 The embodiment of the present invention further provides a data processing device, including a perception module, a time fusion module, a target fusion module and a calibration correction module, wherein the perception module pre-stores the attitude angles of a plurality of sensors, and the time fusion module pre-stores the sampling time differences between the plurality of sensors, including:

[0108] A perception module 601 is used to perform spatial fusion on a current associated frame set based on the attitude angles of each of the multiple sensors to determine a perception result of each frame in the associated frame set; wherein the associated frame set represents frames collected by each of the multiple sensors in the same time period;

[0109] A time fusion module 602 is used to perform time fusion on the current and previous multiple associated frame sets based on the sampling time differences between the multiple sensors to determine a time fusion result; wherein the time fusion result includes the sampling time and perception result of each frame in the multiple associated frame sets;

[0110] A target fusion module 603 is used to perform target fusion based on the time fusion result and determine a target fusion result;

[0111] A position determination module 604 is used to determine overlapping area position data; wherein the overlapping area position data indicates the position of at least one overlapping area of ​​the multiple sensors;

[0112] The correction module 605 is used for the correction module, which is used to correct the attitude angles of part or all of the multiple sensors in the perception module and / or correct the sampling time differences between part or all of the multiple sensors in the time fusion module based on the target fusion result, the time fusion result and the overlapping area position data.

[0113] As a feasible implementation method, the target fusion result is at least used to describe the position of at least one target object at different sampling times, and the perception result is at least used to describe the position of at least one target object;

[0114] The correction module includes: a target object determination unit, a space calibration unit and a time calibration unit; wherein,

[0115] The target object determination unit is used to determine at least one first target object and at least one second target object based on the target fusion result and the overlapping area position data; wherein the first target object is a stationary or moving target object in any area of ​​the at least one overlapping area and satisfies a low-speed motion condition, and the second target object is a moving target object in any area of ​​the at least one overlapping area and does not satisfy the low-speed motion condition;

[0116] The spatial calibration unit is used to correct the attitude angle of each of the multiple first sensors for each of the at least one first target object based on the position of the first target object sensed by each of the multiple first sensors in the time fusion result; wherein the multiple first sensors are multiple sensors among the multiple sensors that sense the first target object in the same time period;

[0117] The target fusion module 603 is further configured to correct the positions of some or all of the target objects in the target fusion result based on the corrected attitude angles of the plurality of first sensors of the at least one first target object;

[0118] The time calibration unit is used to correct the sampling time difference between the multiple second sensors for each of the at least one second target object based on the multiple second sensors in the corrected target fusion result each sensing the position of the second target object at different sampling times; wherein the multiple second sensors are multiple sensors among the multiple sensors that sense the second target object in the same time period.

[0119] As a feasible implementation method, the target object determination unit is used to determine at least one first target object and / or at least one second target object for each area of ​​the at least one overlapping area when it is determined that there is at least one target object in the overlapping area based on the overlapping area position data and the target fusion result.

[0120] As a feasible implementation manner, the distance between the first target and other targets other than the first target in the overlapping area is greater than a preset value, the first target is the only target in the overlapping area, or the first target is the only target in any lane in the overlapping area.

[0121] As a feasible implementation manner, the distance between the second target and other targets other than the second target in the overlapping area is greater than a preset value, the second target is the only target in the overlapping area, or the second target is the only target in any lane in the overlapping area.

[0122] As a feasible implementation method, the space calibration unit includes: a position fusion subunit and a space correction subunit; wherein,

[0123] The position fusion subunit is used to determine the position of the first target object sensed by each of the multiple first sensors in the target fusion result in the world coordinate system, and determine the spatial calibration position of the first target object;

[0124] The spatial correction subunit is used to determine, for any sensor of the multiple first sensors, a first target attitude angle of the first sensor based on the position of the first sensor in the world coordinate system, the spatial calibration position and the positive direction of the world coordinate system; determine a second target attitude angle of the first sensor based on the position of the first target object relative to the first sensor in the time fusion result and the positive direction of the world coordinate system; determine an attitude angle difference based on the first target attitude angle and the second target attitude angle; and correct the attitude angle of the first sensor based on the attitude angle difference and a preset attitude angle of the first sensor.

[0125] As a feasible implementation, the time calibration unit includes: a sensor determination subunit and a time correction subunit; wherein,

[0126] The sensor determination subunit is used to determine a third sensor among the plurality of second sensors;

[0127] The time correction subunit is used to determine the time difference between the second sensor and the third sensor for each sensor other than the third sensor among the multiple second sensors based on the time-calibrated position of the second target object sensed by the second sensor in the corrected target fusion result, and the passed position of the second target object sensed by the third sensor relative to the time-calibrated position, the sampling time corresponding to the passed position, the position to be passed, and the sampling time corresponding to the position to be passed; wherein the time-calibrated position is located between the passed position and the position to be passed.

[0128] As a feasible implementation method, the third sensor is the sensor with the largest sampling frequency among the multiple second sensors, and the passed position and the to-be-passed position are the positions of the second target object sensed by the third sensor that are closest to the time calibration position.

[0129] As a feasible implementation manner, the time correction subunit is further used to correct the time difference between different third sensors based on the time difference between the same first sensor and different third sensors if it is determined that there is a time difference between the same first sensor and different third sensors.

[0130] As a feasible implementation, the associated frame set includes the frame last collected by a fourth sensor among the multiple sensors, and frames collected by other sensors other than the fourth sensor with the smallest sampling time compared with the frame last collected by the fourth sensor.

[0131] As a feasible implementation manner, the target fusion module 603 is further used to perform target fusion based on the weather conditions of the multiple sensors and the target fusion results, and determine the target fusion results of the multiple sensors.

[0132] As a feasible implementation manner, the position determination module 604 is used to determine the overlapping area position data based on the attitude angles of the multiple sensors and the positions of the multiple sensors.

[0133] Figure 7It is a structural diagram of an electronic device provided by an embodiment of the present invention. At the hardware level, the electronic device includes a processor 701 and a memory 702 storing execution instructions, and optionally also includes an internal bus 703 and a network interface 704. Among them, the memory 702 may include a memory 7021, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 7022 (non-volatile memory), such as at least one disk storage, etc.; the processor 701, the network interface 704 and the memory 702 can be interconnected through an internal bus 703, and the internal bus 703 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc.; the internal bus 703 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. Of course, the electronic device may also include hardware required for other services. When the processor 701 executes the execution instruction stored in the memory 702, the processor 701 executes the method in any embodiment of the present invention, and is at least used to execute the following steps: Figure 1 or Figure 2 The method shown.

[0134] In a possible implementation, the processor reads the corresponding execution instruction from the non-volatile memory into the memory and then runs it, and can also obtain the corresponding execution instruction from other devices to form a data processing device at the logical level. The processor executes the execution instruction stored in the memory to implement a data processing method provided in any embodiment of the present invention through the execution of the execution instruction.

[0135] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0136] The embodiment of the present invention further provides a computer-readable storage medium, including an execution instruction. When a processor of an electronic device executes the execution instruction, the processor executes the method provided in any embodiment of the present invention. The electronic device may be specifically as follows: Figure 7 The electronic device shown; the execution instruction is a computer program corresponding to a data processing device.

[0137] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may be implemented in the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.

[0138] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0140] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data processing method, characterized in that: Pre-storing the attitude angles of a plurality of sensors and the sampling time differences between the plurality of sensors, where the detection areas of some of the plurality of sensors overlap, the method comprises: Based on the attitude angles of the multiple sensors, spatially fuse the current associated frame set to determine the perception results of each frame in the associated frame set; wherein the associated frame set represents the frames collected by the multiple sensors in the same time period; Based on the sampling time difference between the multiple sensors, time-fusing the current and previous multiple associated frame sets to determine a time-fusion result; wherein the time-fusion result includes the sampling time and the perception result of each frame in the multiple associated frame sets; Perform target fusion based on the time fusion result to determine the target fusion result; Determining overlapping region position data; wherein the overlapping region position data indicates the position of at least one overlapping region of the plurality of sensors; Based on the target fusion result, the time fusion result and the overlapping area position data, correct the attitude angles of part or all of the multiple sensors and / or correct the sampling time differences between part or all of the multiple sensors; The target fusion result is at least used to describe the position of at least one target object at different sampling times, and the perception result is at least used to describe the position of at least one target object; The method of correcting the attitude angles of part or all of the multiple sensors and / or correcting the sampling time differences between part or all of the multiple sensors based on the target fusion result, the time fusion result and the overlapping area position data includes: Based on the target fusion result and the overlapping area position data, at least one first target and at least one second target are determined; wherein the first target is a stationary or moving target in any area of ​​the at least one overlapping area and satisfies a low-speed motion condition, and the second target is a moving target in any area of ​​the at least one overlapping area and does not satisfy the low-speed motion condition; For each of the at least one first target, based on the position of the first target sensed by each of the multiple first sensors in the time fusion result, correct the attitude angle of each of the multiple first sensors; wherein the multiple first sensors are multiple sensors among the multiple sensors that sense the first target in the same time period; Based on the corrected attitude angles of the plurality of first sensors of the at least one first target object, correcting the positions of some or all of the target objects in the target fusion result; For each of the at least one second target object, based on the positions of the second target object sensed by multiple second sensors at different sampling times in the corrected target fusion result, the sampling time differences between the multiple second sensors are corrected; wherein the multiple second sensors are multiple sensors among the multiple sensors that sense the second target object in the same time period.

2. The method according to claim 1, characterized in that The determining, based on the target fusion result and the overlapping area position data, at least one first target object and at least one second target object comprises: For each area of ​​the at least one overlapping area, when it is determined based on the overlapping area position data and the target fusion result that at least one target object exists in the overlapping area, at least one first target object and / or at least one second target object is determined.

3. The method according to claim 2, characterized in that The distance between the first target and other targets other than the first target in the overlapping area is greater than a preset value, the first target is the only target in the overlapping area, or the first target is the only target in any lane in the overlapping area; or, The distance between the second target and other targets other than the second target in the overlapping area is greater than a preset value, the second target is the only target in the overlapping area, or the second target is the only target in any lane in the overlapping area.

4. The method according to claim 1, characterized in that: The correcting the attitude angles of the plurality of first sensors based on the positions of the first target objects sensed by the plurality of first sensors in the time fusion result includes: Determine the position of the first target object sensed by each of the multiple first sensors in the time fusion result in a world coordinate system, and determine the spatial calibration position of the first target object; For any sensor of the multiple first sensors, a first target attitude angle of the first sensor is determined based on the position of the first sensor in the world coordinate system, the spatial calibration position and the positive direction of the world coordinate system; a second target attitude angle of the first sensor is determined based on the position of the first target object relative to the first sensor in the time fusion result and the positive direction of the world coordinate system; an attitude angle difference is determined based on the first target attitude angle and the second target attitude angle; and the attitude angle of the first sensor is corrected based on the attitude angle difference and a preset attitude angle of the first sensor.

5. The method according to claim 1, characterized in that The method of correcting the sampling time difference between the plurality of second sensors based on the position of the second target object at different sampling times sensed by the plurality of second sensors in the corrected target fusion result includes: determining a third sensor among the plurality of second sensors; For each sensor other than the third sensor among the multiple second sensors, the time difference between the second sensor and the third sensor is determined based on the time-calibrated position of the second target object sensed by the second sensor in the corrected target fusion result, and the passed position of the second target object relative to the time-calibrated position, the sampling time corresponding to the passed position, the position to be passed, and the sampling time corresponding to the position to be passed, wherein the time-calibrated position is located between the passed position and the position to be passed.

6. The method according to claim 5, characterized in that The third sensor is a sensor with the largest sampling frequency among the plurality of second sensors, and the passed position and the to-be-passed position are positions of the second target object sensed by the third sensor that are closest to the time calibration position; or, The method further comprises: If it is determined that there is a time difference between the same second sensor and different third sensors, the time difference between different third sensors is corrected based on the time difference between the same second sensor and different third sensors.

7. The method according to claim 1, characterized in that The associated frame set includes a frame last collected by a fourth sensor among the multiple sensors, and frames collected by other sensors except the fourth sensor and having the smallest sampling time with respect to the frame last collected by the fourth sensor.

8. The method according to claim 1, characterized in that The determining, based on the target fusion result, at least one first target and at least one second target from the at least one target, comprises: Performing target fusion based on the weather conditions of the multiple sensors and the target fusion results to determine the target fusion results of the multiple sensors; or, The determining of the overlapping area position data comprises: Based on the attitude angles of the plurality of sensors and the positions of the plurality of sensors, overlapping area position data are determined.

9. A data processing device, characterized in that: It includes a perception module, a time fusion module, a target fusion module and a calibration correction module. The perception module stores the attitude angles of multiple sensors in advance, and the time fusion module stores the sampling time differences between the multiple sensors in advance, including: A perception module, configured to perform spatial fusion on a current associated frame set based on the respective attitude angles of the multiple sensors, and determine a perception result of each frame in the associated frame set; wherein the associated frame set represents frames collected by the multiple sensors in the same time period; A time fusion module, configured to perform time fusion on the current and previous multiple associated frame sets based on the sampling time differences between the multiple sensors, and determine a time fusion result; wherein the time fusion result includes the sampling time and perception result of each frame in the multiple associated frame sets; A target fusion module, used for performing target fusion based on the time fusion result and determining the target fusion result; A position determination module, configured to determine overlapping area position data; wherein the overlapping area position data indicates the position of at least one overlapping area of ​​the plurality of sensors; A correction module, used for correcting the attitude angles of part or all of the multiple sensors in the perception module and / or correcting the sampling time differences between part or all of the multiple sensors in the time fusion module based on the target fusion result, the time fusion result and the overlapping area position data; The target fusion result is at least used to describe the position of at least one target object at different sampling times, and the perception result is at least used to describe the position of at least one target object; The method of correcting the attitude angles of part or all of the multiple sensors and / or correcting the sampling time differences between part or all of the multiple sensors based on the target fusion result, the time fusion result and the overlapping area position data includes: Based on the target fusion result and the overlapping area position data, at least one first target and at least one second target are determined; wherein the first target is a stationary or moving target in any area of ​​the at least one overlapping area and satisfies a low-speed motion condition, and the second target is a moving target in any area of ​​the at least one overlapping area and does not satisfy the low-speed motion condition; For each of the at least one first target, based on the position of the first target sensed by each of the multiple first sensors in the time fusion result, correct the attitude angle of each of the multiple first sensors; wherein the multiple first sensors are multiple sensors among the multiple sensors that sense the first target in the same time period; Based on the corrected attitude angles of the plurality of first sensors of the at least one first target object, correcting the positions of some or all of the target objects in the target fusion result; For each of the at least one second target object, based on the positions of the second target object sensed by multiple second sensors at different sampling times in the corrected target fusion result, the sampling time differences between the multiple second sensors are corrected; wherein the multiple second sensors are multiple sensors among the multiple sensors that sense the second target object in the same time period.

Images