Multi-sensor target matching and data fusion method and device

Abstract

The invention discloses a multi-sensor target matching and data fusion method and device, and the method comprises the steps: obtaining mixed multi-source sensor data at a current moment, and enabling different sensors to detect the same target to obtain the mixed multi-source sensor data; performing target matching on sensor data with different sources in the mixed multi-source sensor data at the current moment to obtain multi-source sensor data of each target at the current moment; the optimization weights of the sensors are updated according to real-time quality scores of the sensors, the optimization weight of each sensor at the current moment is obtained, the real-time quality scores are determined according to measurement errors, noise and environmental influences of the sensors, and the more stable the real-time quality scores of the sensors are, the higher the optimization weights are; and according to the optimization weight of each sensor at the current moment, performing weighted filtering on the multi-source sensor data of each target at the current moment to obtain fused sensor data of each target at the current moment. The method is better in robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and particularly relates to a method and device for target matching and data fusion of multi-sensors. Background Art

[0002] In the fields of intelligent transportation systems, autonomous driving, security monitoring, robot perception, etc., multi-sensor data fusion has become a core technology for improving the reliability, accuracy, and robustness of perception systems. Due to the physical characteristics of a single sensor and environmental adaptability issues, it is difficult to meet the comprehensive perception requirements in complex dynamic scenarios. Therefore, by enabling multiple sensors to work together and comprehensively utilizing the advantages of different sensors to improve the capabilities of target detection, tracking, and recognition has become a research hotspot. For example, millimeter-wave radar can accurately measure the distance and speed of targets and has strong anti-interference ability in bad weather, but its angular resolution is low and it is difficult to accurately distinguish multiple close-range targets; lidar can provide high-resolution three-dimensional point cloud data and is suitable for obstacle detection and map construction, but it is restricted by weather, cost, and computing resources; cameras can provide rich texture, color, and other information, which is convenient for target classification and recognition, but lack direct depth measurement ability and are sensitive to changes in illumination; infrared thermal imagers are suitable for night or low visibility environments, but have relatively low resolution and are difficult to accurately locate targets; ultrasonic sensors have advantages in short-distance obstacle detection, but their ranging range is limited. Therefore, in intelligent perception systems, how to efficiently fuse the information of these sensors to give full play to their respective advantages has become the key to improving the performance of perception systems.

[0003] Multi-sensor target matching and data fusion technologies have been widely applied in many fields. For example, in an autonomous driving system, the fusion of in-vehicle cameras, lidar, and millimeter-wave radar can improve the vehicle's perception ability of the surrounding environment and ensure the safety of decision-making and path planning; in intelligent transportation, vehicle-road cooperation technology uses the data fusion of roadside sensors and in-vehicle sensors to achieve real-time traffic flow optimization and traffic light scheduling; in the field of UAV monitoring, fusing the data of radar, optical cameras, and infrared sensors can improve the reliability of long-distance target recognition; in robot navigation, multi-sensor fusion improves the accuracy of environmental modeling and path planning, enabling robots to avoid obstacles autonomously and move efficiently; in security monitoring, by fusing camera and millimeter-wave radar data, the accuracy of night target detection can be improved and the false detection rate caused by changes in illumination can be reduced.

[0004] However, in complex traffic scenarios (such as bad weather, lighting changes, and target occlusion), current methods are difficult to maintain a stable target matching effect. The solutions that simply rely on single sensors or simple weighted fusion lack dynamic adjustment strategies, resulting in performance degradation in specific environments. In a multi-target, high-density traffic environment, due to target occlusion or trajectory crossing, current matching algorithms are difficult to ensure correct target association.

[0005] Therefore, the current data fusion methods have poor robustness. Summary of the Invention

[0006] Embodiments of the present invention provide a multi-sensor target matching and data fusion method and apparatus, which can solve the problem of poor robustness of current data fusion methods.

[0007] In a first aspect, a multi-sensor target matching and data fusion method provided by an embodiment of the present invention includes:

[0008] Obtain mixed multi-source sensor data at the current moment, where the mixed multi-source sensor data is obtained by different sensors detecting the same target;

[0009] Perform target matching on sensor data from different sources in the mixed multi-source sensor data at the current moment to obtain multi-source sensor data of each target at the current moment;

[0010] Update the optimization weight of the sensor according to the real-time quality score of the sensor to obtain the optimization weight of each sensor at the current moment, where the real-time quality score is determined according to the measurement error, noise, and environmental impact of the sensor, and the more stable the real-time quality score of the sensor, the higher the optimization weight;

[0011] Perform weighted filtering on the multi-source sensor data of each target at the current moment according to the optimization weight of each sensor at the current moment to fuse sensor data from different sources of the same target and obtain the fused sensor data of each target at the current moment.

[0012] In a second aspect, an embodiment of the present invention provides a multi-sensor target matching and data fusion apparatus, including:

[0013] An acquisition module, configured to acquire mixed multi-source sensor data at the current moment, where the mixed multi-source sensor data is obtained by different sensors detecting the same target;

[0014] A matching module, configured to perform target matching on sensor data from different sources in the mixed multi-source sensor data at the current moment to obtain multi-source sensor data of each target at the current moment;

[0015] A weight optimization module, which is used to update the optimization weight of a sensor according to the real-time quality score of the sensor, so as to obtain the optimization weight of each sensor at the current moment. Wherein, the real-time quality score is determined according to the measurement error, noise and environmental impact of the sensor, and the more stable the real-time quality score of the sensor is, the higher the optimization weight is.

[0016] A fusion module, which is used to perform weighted filtering on the multi-source sensor data of each target at the current moment according to the optimization weight of each sensor at the current moment, so as to fuse the sensor data from different sources of the same target and obtain the fused sensor data of each target at the current moment.

[0017] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: Since the factors for determining the real-time quality score include environmental impact, measurement error and noise of the sensor, the present invention dynamically adjusts the optimization weight of each sensor through the real-time quality score reflecting the sensor performance, and then performs weighted filtering on the multi-source sensor data of the target at the current moment according to the optimization weight of each sensor; it can improve the adaptability of the data fusion method to environmental changes, improve the stability of data fusion, and the present invention can make full use of the advantages of each sensor, avoid redundant or missing fusion information, and further improve the fusion stability. Description of the Drawings

[0018] Figure 1 It is a flowchart of the implementation of a multi-sensor target matching and data fusion method provided by an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of the implementation of a method for target matching of sensor data from different sources provided by an embodiment of the present invention;

[0020] Figure 3 It is a flowchart of the implementation of a local Kalman filtering method provided by an embodiment of the present invention;

[0021] Figure 4 It is a schematic structural diagram of a multi-sensor target matching and data fusion device provided by an embodiment of the present invention. Detailed Embodiment

[0022] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0023] It should be understood that when used in the specification of the present invention and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0024] It should also be understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0025] As used in the specification of the present invention and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrases "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" depending on the context.

[0026] In addition, in the description of the specification of the present invention and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and cannot be understood as indicating or implying relative importance.

[0027] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present invention means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present invention. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear at different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0028] The present invention will be further described in detail below in conjunction with specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0029] The method for target matching and data fusion of multi-sensors provided by the embodiments of the present invention can be applied to electronic devices such as mobile terminals, personal laptop computers, supercomputers, etc. The specific types of electronic devices are not limited in the embodiments of the present invention.

[0030] Figure 1The figure shows a flowchart of the implementation of a method for target matching and data fusion of multiple sensors provided by an embodiment of the present invention. By way of example and not limitation, the method may include steps S101 - S104, which are described below.

[0031] S101, Obtain the mixed multi - source sensor data at the current moment.

[0032] Exemplarily, the mixed multi - source sensor data may be obtained by different sensors detecting the same target.

[0033] For example, the target cluster detected by sensor 1 is R = {r1, r2, …, r m′}, and the target cluster detected by sensor 2 is V = {v1, v2, …, v n}, R and V may be the same target cluster, and m′ is also equal to n. Just when receiving the data of these two sensors, the corresponding relationship of each target in R and V is unknown (that is, whether r i′ and v j are the same target).

[0034] S102, Perform target matching on the sensor data from different sources in the mixed multi - source sensor data at the current moment to obtain the multi - source sensor data of each target at the current moment.

[0035] In a possible implementation manner, the sensor data from different sources can be matched in stages according to different matching parameters. For the initial matching, a quick screening is performed to obtain the initial matching pairs; when there are still remaining targets after the initial matching, the remaining targets are rematched to obtain the remaining matching pairs; finally, the sensor data from different sources of the same target are associated to obtain the multi - source sensor data of each target at the current moment.

[0036] In an example, since traditional geometric - feature - based methods (such as nearest - neighbor search) are sensitive to noise and measurement errors and are prone to false matching during the target movement. Probability - statistical methods (such as Kalman filtering, joint - probability data - association matching) have a decreased matching reliability when the data is incomplete or the target undergoes mutations. Therefore, the initial matching can be based on the nearest - neighbor matching algorithm to improve the calculation efficiency, and the re - matching can be based on the Hungarian algorithm to improve the matching accuracy; thus, combining the advantages of these two algorithms can flexibly adapt to different scenarios and data volumes while taking into account real - time performance and accuracy, fully meeting the various requirements of the multi - sensor fusion system for target matching.

[0037] S103, Update the optimization weights of the sensors according to the real - time quality scores of the sensors to obtain the optimization weights of each sensor at the current moment.

[0038] Exemplarily, the real-time quality score of a sensor can reflect the performance of the sensor, which can be calculated based on the measurement error, noise, and environmental impact of the sensor. The more stable the performance of the sensor, the higher the optimization weight.

[0039] In a possible implementation, the confidence of each sensor at the current moment can be determined according to its real-time quality score, and then the entropy of the sensor at the current moment can be determined according to its confidence; the basic weight of each sensor at the current moment can be determined according to the entropy of each sensor at the current moment, and finally, the basic weight can be optimized to obtain the optimized weight of each sensor at the current moment.

[0040] In an example, the confidence of the i-th sensor at the current moment can satisfy the following formula:

[0041]

[0042] where C i is the confidence of the i-th sensor S i at the current moment; Q i is the real-time quality score of the sensor S i ; α is the first smoothing factor, which is used to control the sensitivity of C i to Q i .

[0043] In an example, the entropy weight method can be used to perform normalization calculation on the sensor confidence, so that sensors with lower uncertainty have higher weights.

[0044] Exemplarily, the smaller the entropy of the sensor, the more stable the measurement data of the sensor and the higher the credibility.

[0045] Specifically, the entropy of the i-th sensor at the current moment can satisfy the following formula:

[0046]

[0047] where H i is the entropy of S i at the current moment, p ij represents the normalized contribution of the sensor S i to the measurement data j (i.e., the data obtained by measuring the j-th target), and n is the total number of targets;

[0048] where:

[0049]

[0050] where C ij is the confidence of the i-th sensor S i in the measurement data j, and m is the total number of sensors.

[0051] In one example, the basic weight of the i-th sensor may satisfy the following formula:

[0052]

[0053] where W i is the basic weight of the i-th sensor, H i is the entropy of the i-th sensor S i at the current moment, and m is the total number of sensors.

[0054] In one example, the optimized weight of the i-th sensor at the current moment may satisfy the following formula:

[0055]

[0056] where is the optimized weight of the i-th sensor at the current moment, γ is the second smoothing factor, is the optimized weight of the i-th sensor at the (here representing the previous moment) t-1 moment;

[0057] where:

[0058]

[0059] where P(D|W i ) is the Bayesian weight of the i-th sensor at the current moment, P(D) is the comprehensive weight of the mixed multi-source sensor data D at the current moment, and W i is the basic weight of the i-th sensor at the current moment.

[0060] S104. According to the optimized weight of each sensor at the current moment, perform weighted filtering on the multi-source sensor data of each target at the current moment to fuse the sensor data from different sources of the same target, and obtain the fused sensor data of each target at the current moment.

[0061] In a possible implementation manner, based on the weighted Kalman filtering algorithm, perform weighted filtering on the multi-source sensor data of each target at the current moment according to the optimized weight.

[0062] The weighted Kalman data fusion algorithm is based on the Kalman filtering theory and realizes the joint estimation of multi-source information through a two-step recursive process of state prediction and measurement update. Its core idea is to dynamically adjust the weights of the observation data according to the confidence levels of different sensors at different time periods, thereby enhancing the adaptive ability of data fusion.

[0063] Exemplarily, based on the weighted Kalman fusion algorithm, local Kalman filtering can be first performed on the multi-source sensor data of each target at the current moment to calculate the local state and local error covariance of each sensor. Then, based on the optimized weight of each sensor at the current moment, all local states and local error covariances are weighted and fused to obtain the fused sensor data of the target at the current moment.

[0064] In one example, the fused sensor data of the target at the current moment may include the fused global state estimate and global error covariance.

[0065] Exemplarily, the fused global state estimate may satisfy the following formula:

[0066]

[0067] where, is the fused global state estimate of the target at the t-th moment (here is the current moment), and α i (t) is equal to is the optimized weight of the i-th sensor at the current moment, is the updated value of the local state of the i-th sensor at the t-th moment.

[0068] Exemplarily, the fused global error covariance may satisfy the following formula:

[0069]

[0070] where, P(t|t) is the fused global error covariance of the target at the t-th moment, and P i (t|t) is the updated value of the local error covariance of the i-th sensor at the t-th moment, which can reflect the contribution of each sensor to the global error.

[0071] Exemplarily, the optimized weight of the i-th sensor at the t-th moment satisfies the constraint condition:

[0072] Since the factors for determining the real-time quality score include environmental impact, measurement error and noise of the sensor, the present invention dynamically adjusts the optimized weight of each sensor through the real-time quality score reflecting the sensor performance, and then performs weighted filtering on the multi-source sensor data of the target at the current moment according to the optimized weight of each sensor; it can improve the adaptability of the data fusion method to environmental changes, improve the stability of data fusion, and the present invention can make full use of the advantages of each sensor, avoid redundant or missing fusion information, and further improve the fusion stability.

[0073] Figure 2The figure shows a flowchart of an implementation of a method for target matching of sensor data from different sources provided by an embodiment of the present invention. By way of example and not limitation, this method may be a possible implementation of the above step S102. This method may include steps S201 - S206, which are described below.

[0074] S201, compare the number of targets with a preset target number threshold to determine whether the scene at the current moment is a complex scene.

[0075] In one example, if the number of targets is not greater than the preset target number threshold, the scene at the current moment is a simple scene, and the following steps S202 and S203 can be performed sequentially.

[0076] In another example, if the number of targets is greater than the preset target number threshold, the scene at the current moment is a complex scene, and the following steps S204 and S205 can be performed sequentially.

[0077] S202, perform an initial match on the sensor data from different sources according to the first matching parameter to obtain initial match pairs.

[0078] In a possible implementation, based on the nearest neighbor matching algorithm, preliminary pairing can be performed according to the first matching parameter to quickly determine the potential relationship between targets.

[0079] Exemplarily, since the initial match pairs may not include all targets, step S203 can be performed after the initial match to perform another match.

[0080] Exemplarily, the first matching parameter can be extracted from the mixed multi-source sensor data at the current moment, and may include, for example, the target speed and target position of each target.

[0081] In one example, the comprehensive distance between any pair of targets in the sensor data to be retrieved can be calculated based on the target speed and target position. If the nearest comprehensive distance of the main retrieval target is less than the first cost threshold, the secondary retrieval target with the nearest comprehensive distance to the main retrieval target can be determined as the matching target of the main retrieval target. After matching all the main retrieval targets and all groups of sensor data, initial match pairs can be obtained.

[0082] Exemplarily, the sensor data to be retrieved can include any two groups of sensor data with different sources in the mixed multi-source sensor data at the current moment.

[0083] For example, the mixed multi-source sensor data at the current moment includes data detected by 5 sensors. Any two groups of data can be selected as the data to be retrieved. For example, the data of sensor 1 and the data of sensor 2 are selected as the data to be retrieved.

[0084] Exemplarily, a set of homologous data can be selected from the data to be retrieved as the main retrieval data, and another set as the secondary retrieval data; the target in the main retrieval data is the main retrieval target, and the target in the secondary retrieval data is the secondary retrieval target.

[0085] Taking sensors 1 and 2 as an example again, the targets {r1, r2, …, r m′} in sensor 1 can be used as the main retrieval targets, and the targets {v1, v2, …, v n} in sensor 2 can be used as the secondary retrieval targets.

[0086] Exemplarily, the main retrieval target and the matching target of the main retrieval target form a pair of initial matching pairs. For example, the j-th target v j in sensor 2 has the smallest comprehensive distance from the i'-th target r i′ in sensor 1. Therefore, v j and r i′ can be used as a pair of initial matching pairs.

[0087] Exemplarily, the comprehensive distance between the i'-th main retrieval target and the j-th secondary retrieval target can satisfy the following formula:

[0088]

[0089] where d i′j is the comprehensive distance between the i'-th main retrieval target and the j-th secondary retrieval target, are the two-dimensional coordinate values of the position of the i'-th main retrieval target respectively, are the two-dimensional coordinate values of the velocity of the i'-th main retrieval target respectively, and α' is the weight of the velocity information, which is used to balance the importance of the position and velocity features; are the two-dimensional coordinate values of the position of the j-th secondary retrieval target respectively, are the two-dimensional coordinate values of the velocity of the j-th secondary retrieval target respectively.

[0090] Correspondingly, the process of finding the matching target of the main retrieval target based on the nearest neighbor algorithm can be expressed as:

[0091]

[0092] where j * is the matching target of the main retrieval target.

[0093] S203. Re-match the sensor data from different sources according to the first matching parameter to obtain the remaining matching pairs.

[0094] In a possible implementation, the remaining targets in the sensor data to be retrieved can be globally optimized based on the Hungarian algorithm to obtain remaining matching pairs, and the target motion information and appearance features can be comprehensively used to construct a global cost matrix to improve the matching accuracy.

[0095] Exemplarily, after performing step S203, step S206 can be carried out.

[0096] In one example, a first cost matrix can be constructed for the remaining targets in the sensor data to be retrieved according to the first motion characteristic cost of the remaining targets; then the Hungarian algorithm is used to globally optimize the first cost matrix to obtain remaining matching pairs.

[0097] Exemplarily, the essence of the first motion characteristic cost can be the comprehensive distance between the remaining main retrieval targets and the remaining secondary retrieval targets.

[0098] Exemplarily, the element in the i''-th row and j'-th column of the first cost matrix can satisfy C i″j′ = d motion (r i″ , v j′ ), d motion (r i″ , v j′ ) is the first motion characteristic cost between the i''-th remaining main retrieval target and the j'-th remaining secondary retrieval target. The value ranges of i'' and j' are determined by the numbers of the remaining main retrieval targets and the remaining secondary retrieval targets respectively.

[0099] Exemplarily, the process of globally optimizing using the Hungarian algorithm can be expressed as:

[0100]

[0101] where, is the remaining matching pair, and C is the first cost matrix.

[0102] S204, perform an initial matching on the sensor data from different sources according to the second matching parameter to obtain initial matching pairs.

[0103] Exemplarily, the first matching parameter and the second matching parameter are not completely the same, and the second matching parameter can include target position, target speed, target acceleration, and target historical trajectory information.

[0104] In a possible implementation, when the detection scenario is relatively complex, based on the idea of layer-by-layer optimization, first, based on the nearest neighbor algorithm, perform an initial matching at a coarse-grained level according to the target position, target speed, and target acceleration to obtain initial matching pairs to ensure the efficiency of the matching.

[0105] In one example, the second characteristic cost of any pair of targets among the data to be retrieved can be determined based on the target position, target speed, and target acceleration. Similarly, if the minimum second characteristic cost of the primary retrieval target is less than the second cost threshold, the secondary retrieval target with the minimum second characteristic cost of the primary retrieval target can be determined as the matching target of the primary retrieval target. After matching all the primary retrieval targets and all groups of sensor data, the initial matching pairs can be obtained.

[0106] Exemplarily, the second characteristic cost can be calculated by the following formula:

[0107]

[0108] where d′ motion (r i′ , v j ) is the second characteristic cost between the i'-th primary retrieval target and the j-th secondary retrieval target, are respectively the speed and acceleration of the i'-th primary retrieval target, are respectively the speed and acceleration of the j-th secondary retrieval target, and β is the weight coefficient of the acceleration, which is used to balance the influence of the acceleration on the matching result.

[0109] Exemplarily, the target acceleration can be obtained by the speed transformation between two consecutive frames:

[0110]

[0111] where v xr (t + 1), v yr (t + 1) are respectively the x-axis and y-axis components of the speed of r i′ at the (t + 1)-th moment, v xr (t), v yr (t) are respectively the x-axis and y-axis components of the speed of r i′ at the t-th moment, and Δt is the time difference between the (t + 1)-th moment and the t-th moment.

[0112] Correspondingly, the process of finding the matching target of the primary retrieval target based on the nearest neighbor algorithm can be expressed as:

[0113]

[0114] where j * is the matching target of the primary retrieval target.

[0115] S205, rematch the sensor data from different sources according to the second matching parameter to obtain the remaining matching pairs.

[0116] In a possible implementation, in the fine-grained matching stage, to further optimize the accuracy of target matching, in addition to considering the basic information of the target, such as position, speed, and acceleration, the historical trajectory information of the target can also be introduced to utilize the richer dynamic features of the target to determine whether the target belongs to the same object, especially when the target is occluded or lost for a short time.

[0117] Exemplarily, after step S205 is executed, step S206 can be performed.

[0118] In one example, the predicted position of each target in the sensor data to be retrieved can be calculated based on the historical trajectory information of the target. Similarly, a second cost matrix is constructed for the remaining targets in the sensor data to be retrieved according to the predicted position of the target and the second motion characteristic cost; finally, the Hungarian algorithm is used to globally optimize the second cost matrix to obtain the remaining matching pairs.

[0119] Exemplarily, the second cost matrix can satisfy the following formula:

[0120] C′ i″j′ =λ·d′ motion (r i″ ,v j′ )+(1-λ)·d trajectory (r i″ ,v j′ ) (1.15)

[0121] C′ i″j′ is the element in the i″-th row and j′-th column of the second cost matrix, d′ motion (r i″ ,v j′ ) is the second motion characteristic cost between the i″-th remaining main retrieval target and the j′-th remaining secondary retrieval target, and λ is a weight parameter.

[0122] Where:

[0123]

[0124] Among them, are the two-dimensional coordinate values of the predicted position of the i″-th remaining main retrieval target respectively, are the two-dimensional coordinate values of the actual position of the j′-th remaining secondary retrieval target respectively, is the two-dimensional coordinate value of the actual position of the i″-th remaining main retrieval target from the previous moment to the start moment.

[0125] Exemplarily, the process of globally optimizing using the Hungarian algorithm can be expressed as:

[0126]

[0127] Among them, is the remaining matching pairs, and C is the second cost matrix.

[0128] S206, associate the initial matching pairs of the sensor data to be retrieved with the sensor data of different sources for each target in the remaining matching pairs; after associating the sensor data of all sources, obtain the multi-source sensor data of each target at the current moment.

[0129] In one example, it is possible to associate the sensor data of different sources for each target in the final matching pairs. After associating the sensor data of all sources, obtain the multi-source sensor data of each target at the current moment.

[0130] For example, referring to the above example, if the target r1 detected by sensor 1 and the target v1 detected by sensor 2 are targets within the same matching pair, this indicates that the targets r1 and v1 are the same target. The data detected by sensor 1, such as the position and speed of target r1, and the data detected by sensor 2, such as the position and speed of target v1, can be associated and determined as the relevant data of target r1.

[0131] Traditional optimization algorithms have a slow convergence speed in large-scale target matching problems, which affects the online processing ability. The joint probabilistic data association calculates all possible target-measurement assignment probabilities, and when the number of targets increases, the calculation time grows exponentially and it is difficult to execute in real-time systems. However, the present invention adopts an adaptive matching strategy and takes different matching methods according to the different amounts of target data, which can reduce redundant calculations and meet the real-time requirements.

[0132] Figure 3 The following shows a flowchart of the implementation of a local Kalman filtering method provided by an embodiment of the present invention. By way of example and not limitation, this method can be a specific possible implementation of the local filtering process mentioned in step S104 above. This method may include steps S301 - S303, and the following will explain each step.

[0133] S301, perform local Kalman filtering on each set of sensor data of the target at the current moment according to the state transition model and the observation model to obtain the predicted values of the local state and error covariance of the i-th sensor.

[0134] In one example, the multi-sensor fusion system can be a discrete-time linear stochastic control system, and its state transition model and observation model can be calculated according to the following formulas respectively:

[0135] x(t + 1) = Φ(t)x(t) + B(t)u(t) + w(t) (1.17)

[0136] y i (t) = Hi (t)x(t)+v i (t), i = 1, 2, …, m (1.18)

[0137] Where x(t + 1) and x(t) are the system states at the (t + 1)-th and t-th moments respectively, Φ(t) is the state transition matrix at the t-th moment, B(t) is the input matrix at the t-th moment, u(t) is the known control input at the t-th moment, w(t) is the process noise at the t-th moment, and it follows a Gaussian distribution Q(t) is its noise variance; y i (t) is the observation value of the i-th sensor at the t-th moment, H i (t) is the i-th observation matrix, v i (t) is the measurement noise at the t-th moment, and it also follows a Gaussian distribution R i (t) is its noise variance.

[0138] In an example, the Kalman filter can be independently run for each sensor, and the predicted state and error covariance can be calculated respectively according to the following formulas:

[0139]

[0140] P i (t|t - 1) = Φ(t - 1)P i (t - 1|t - 1)Φ T (t - 1) + Q(t - 1)

[0141] Where, P i (t|t - 1) are the predicted values of the local state and error covariance of the i-th sensor at the t-th moment respectively, P i (t - 1|t - 1) are the updated values of the local state of the i-th sensor at the (t - 1)-th moment respectively.

[0142] S302. Calculate the gain of the i-th sensor at the current moment according to the predicted values of the local state and error covariance of the i-th sensor.

[0143] Exemplarily, the gain K i (t) of the i-th sensor at the t-th moment can satisfy the following formula:

[0144]

[0145] S303. Determine the updated values of the local state and error covariance of the i-th sensor at the current moment according to the gain of the i-th sensor at the current moment.

[0146] Exemplarily, the updated values of the local state and error covariance of the i-th sensor at the t-th moment can be calculated by the following formulas respectively:

[0147]

[0148] P i (t|t) = [I - K i (t)H i (t)]P i (t|t - 1).

[0149] Through the above solution, the present invention adopts an adaptive multi-sensor weight allocation method (i.e., the above steps S101 - S104 and S301 - S303), combines entropy weight optimization, Bayesian update, and spatio-temporal consistency constraints, and dynamically adjusts the contribution weights of different sensors, which can improve the stability of target matching and data fusion. This method can perform adaptive weight optimization according to real-time environmental changes (such as light changes, bad weather, target occlusion, etc.), sensor measurement errors (such as noise interference, data loss, etc.), and the complexity of multi-target scenarios (such as target overlap, occlusion, or non-rigid motion), so that high-confidence sensor information occupies a greater weight in the fusion process, while the influence of low-quality data is effectively suppressed.

[0150] Furthermore, on the basis of ensuring high real-time performance and high precision, the present invention adopts a multi-stage multi-sensor target matching algorithm (i.e., the above steps S201 - S206) and an adaptive multi-sensor weight allocation method, which can flexibly adapt to different sensor configurations, complex environmental changes, and large-scale data streams, and effectively cope with challenges such as dynamic occlusion, measurement noise, non-linear motion, and environmental interference in multi-target tracking. At the same time, the present invention ensures efficient operation on high-data-throughput and resource-constrained embedded platforms by optimizing computing resource allocation, reducing computational complexity, and parallel processing strategies, thus fully meeting the real-time target tracking requirements of multi-sensor fusion systems in scenarios such as intelligent transportation, autonomous driving, security monitoring, and UAV navigation.

[0151] In addition, the present invention adopting the adaptive multi-sensor weight allocation method can also combine spatio-temporal alignment optimization to achieve efficient fusion and accurate matching of different sensor data. Aiming at the differences in data format, feature space, coordinate system, and time synchronization among sensors such as millimeter-wave radar, lidar, and camera, the present invention dynamically allocates the contribution weights of different modality data according to the confidence, environmental adaptability, and measurement stability of sensor data through an adaptive weight adjustment strategy, improving the accuracy and reliability of cross-modal matching.

[0152] Figure 4The following is a schematic structural diagram of a multi-sensor target matching and data fusion device provided by an embodiment of the present invention. By way of example and not limitation, device 400 may include an acquisition module 410, a matching module 420, a weight optimization module 430, and a fusion module 440.

[0153] Exemplarily, the acquisition module 410 is configured to acquire mixed multi-source sensor data at the current moment, where the mixed multi-source sensor data is obtained by different sensors detecting the same target; the matching module 420 is configured to perform target matching on sensor data from different sources in the mixed multi-source sensor data at the current moment to obtain the multi-source sensor data of each target at the current moment; the weight optimization module 430 is configured to update the optimization weight of the sensor according to the real-time quality score of the sensor to obtain the optimization weight of each sensor at the current moment, where the real-time quality score is determined according to the measurement error, noise, and environmental impact of the sensor, and the more stable the real-time quality score of the sensor, the higher the optimization weight; the fusion module 440 is configured to perform weighted filtering on the multi-source sensor data of each target at the current moment according to the optimization weight of each sensor at the current moment to fuse the sensor data from different sources of the same target and obtain the fused sensor data of each target at the current moment.

[0154] Since the factors for determining the real-time quality score include environmental impact, measurement error, and noise of the sensor, the present invention dynamically adjusts the optimization weight of each sensor through the real-time quality score reflecting the sensor performance, and then performs weighted filtering on the multi-source sensor data of the target at the current moment according to the optimization weight of each sensor; it can improve the adaptability of the data fusion method to environmental changes, improve the stability of data fusion, and the present invention can make full use of the advantages of each sensor, avoid redundant or missing fusion information, and further improve the fusion stability.

[0155] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Claims

1. A target matching and data fusion method for multi-sensors, characterized in that, Including: Obtaining the mixed multi-source sensor data at the current moment, where the mixed multi-source sensor data is obtained by different sensors detecting the same target; Performing target matching on the sensor data from different sources in the mixed multi-source sensor data at the current moment to obtain the multi-source sensor data of each target at the current moment; Updating the optimization weight of the sensor according to the real-time quality score of the sensor to obtain the optimization weight of each sensor at the current moment, where the real-time quality score is determined according to the measurement error, noise and environmental impact of the sensor, and the more stable the real-time quality score of the sensor, the higher the optimization weight; Performing weighted filtering on the multi-source sensor data of each target at the current moment according to the optimization weight of each sensor at the current moment to fuse the sensor data from different sources of the same target and obtain the fused sensor data of each target at the current moment.

2. The method according to claim 1, wherein The performing target matching on the sensor data from different sources in the mixed multi-source sensor data at the current moment to obtain the multi-source sensor data of each target at the current moment includes: Comparing the number of targets with a preset target number threshold to determine whether the scene at the current moment is a complex scene; If the scene at the current moment is a simple scene, performing initial matching and re-matching on the sensor data from different sources in turn according to the first matching parameter to obtain the multi-source sensor data of each target at the current moment; If the scene at the current moment is a complex scene, performing initial matching and re-matching on the sensor data from different sources in turn according to the second matching parameter to obtain the multi-source sensor data of each target at the current moment, where the first matching parameter and the second matching parameter are not completely the same.

3. The method according to claim 2, wherein The first matching parameter includes the target position and the target speed.

4. The method according to claim 2, wherein The second matching parameter includes the target position, the target speed, the target acceleration and the target historical trajectory information; Among them, the performing initial matching and re-matching on the sensor data from different sources in turn according to the second matching parameter to obtain the multi-source sensor data of each target at the current moment includes: Determining the second motion characteristic cost of each pair of targets among the sensor data to be retrieved according to the target position, the target speed and the target acceleration, where the sensor data to be retrieved includes any two sets of sensor data from different sources in the mixed multi-source sensor data at the current moment; Performing initial matching on each target in the sensor data to be retrieved according to the second motion characteristic cost to obtain the initial matching pairs of the sensor data to be retrieved, and the initial matching pairs do not necessarily include all the targets in the sensor data to be retrieved; Determining the predicted position of each target in the sensor data to be retrieved according to the target historical trajectory information; Performing re-matching on the remaining targets in the sensor data to be retrieved according to the predicted position and the second motion characteristic cost to obtain the remaining matching pairs of the sensor data to be retrieved, where the targets in the same pair in the initial matching pairs and the remaining matching pairs are the same targets detected by different sensors; Associate the sensor data of different sources for each pair of targets in the initial matching pairs and the remaining matching pairs of the to-be-retrieved sensor data; after associating the sensor data of all sources, obtain the multi-source sensor data of each target at the current moment.

5. The method according to claim 1, wherein The updating the optimized weight of the sensor according to the real-time quality score of the sensor to obtain the optimized weight of each sensor at the current moment includes: Determine the confidence of each sensor at the current moment according to the real-time quality score of each sensor; Determine the entropy of each sensor at the current moment according to the confidence of each sensor at the current moment; Determine the basic weight of each sensor at the current moment according to the entropy of each sensor at the current moment; Optimize the basic weight of each sensor at the current moment to obtain the optimized weight of each sensor at the current moment.

6. The method according to claim 5, wherein The confidence of the i-th sensor at the current moment satisfies the following formula: Where Ci is the confidence of the i-th sensor Si at the current moment, α is the first smoothing factor, and Qi is the real-time quality score of the sensor Si.

7. The method according to claim 5, wherein The entropy of the i-th sensor at the current moment satisfies the following formula: Where Hi is the entropy of the i-th sensor Si at the current moment, pij represents the normalized contribution of the sensor Si to the measurement data j, and n is the total number of targets; Where: Cij is the confidence of the i-th sensor Si in the measurement data j, and m is the total number of sensors.

8. The method according to claim 5, characterized in that, The basic weight of the i-th sensor satisfies the following formula: Where Wi is the basic weight of the i-th sensor, Hi is the entropy of the i-th sensor Si at the current moment, and m is the total number of sensors.

9. The method according to claim 5, characterized in that, The optimized weight of the i-th sensor at the current moment satisfies the following formula: wherein, is the optimized weight of the i-th sensor at the current moment, γ is the second smoothing factor, is the optimized weight of the i-th sensor at the previous moment; Where: Where P(D|Wi) is the Bayesian weight of the i-th sensor, P(D) is the comprehensive weight of the mixed multi-source sensor data at the current moment, and Wi is the basic weight of the i-th sensor.

10. A target matching and data fusion device for multiple sensors, characterized in that, Including: An acquisition module, the acquisition module is used to acquire the mixed multi-source sensor data at the current moment, where the mixed multi-source sensor data is obtained by different sensors detecting the same target; A matching module, the matching module is used to perform target matching on the sensor data of different sources in the mixed multi-source sensor data at the current moment to obtain the multi-source sensor data of each target at the current moment; A weight optimization module, the weight optimization module is used to update the optimized weight of the sensor according to the real-time quality score of the sensor to obtain the optimized weight of each sensor at the current moment, where the real-time quality score is determined according to the measurement error, noise and environmental impact of the sensor, and the more stable the real-time quality score of the sensor, the higher the optimized weight; A fusion module, the fusion module is used to perform weighted filtering on the multi-source sensor data of each target at the current moment according to the optimized weight of each sensor at the current moment to fuse the sensor data of the same target from different sources and obtain the fused sensor data of each target at the current moment.

Images