Multi-sensor trajectory fusion methods and electronic devices

By selecting the global track with the closest spatiotemporally normalized distance in multi-sensor track fusion and enhancing the matching strength, the problem of erroneous association caused by noise in existing methods is solved, achieving higher fusion accuracy and robustness.

CN123069259APending Publication Date: 2026-11-10RUICHUANG MICROELECTRONICS (YANTAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611229071.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-13
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multi-sensor track fusion methods are susceptible to noise, leading to incorrect or missed associations. This is especially true when targets are dense or sensor accuracy is limited, where the mismatch rate increases significantly, affecting the accuracy of the fusion results.

Method used

By determining the first global track that has the closest spatiotemporal normalized distance between the current sensor track and the global track, and enhancing the matching strength when both long-term consistency and instantaneous highest similarity are satisfied, the first global track is updated using the current sensor track, reducing erroneous updates caused by factors such as noise and temporary occlusion.

Benefits of technology

It improves the robustness and accuracy of multi-sensor track fusion, effectively reduces erroneous updates caused by factors such as noise, target intersection or temporary occlusion, and enhances the reliability and accuracy of fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069259A_ABST
    Figure CN123069259A_ABST
Patent Text Reader

Abstract

This application provides a multi-sensor track fusion method and electronic device. The method includes: processing target data from sensors to be processed to obtain updated current sensor tracks; determining candidate global tracks that satisfy association candidate conditions with the current sensor track within the current association matching period based on the distance between the current sensor track and global tracks; selecting a first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track; if the current sensor track and the first global track are already matched, enhancing the matching strength between the current sensor track and the first global track, and updating the first global track using the current sensor track. This method improves the robustness and accuracy of multi-sensor track fusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of sensor perception fusion technology, and in particular to a multi-sensor trajectory fusion method and electronic device. Background Technology

[0002] In the field of multi-sensor track fusion, existing methods generally adopt a direct association strategy based on distance metrics. Specifically, after obtaining the local track from the sensors, the system calculates the distance between the track and each global track, selects the global track with the smallest distance as the association object, and then directly performs fusion and update.

[0003] This fusion method is simple to implement and computationally efficient, and is therefore widely used in engineering. However, sensor measurements inevitably contain noise, and this calculation method is susceptible to random errors, leading to incorrect or missed associations. Especially when targets are dense or sensor accuracy is limited, the mismatch rate increases significantly, resulting in abnormal fusion results. Summary of the Invention

[0004] To address the existing technical problems, this application provides a multi-sensor track fusion method, electronic device, storage medium, and program that can improve the accuracy of multi-sensor track fusion.

[0005] Firstly, a multi-sensor trajectory fusion method is provided, the method comprising: Process the target data from the sensor to be processed to obtain the updated current sensor track; Based on the distance between the current sensor track and the global track, candidate global tracks that meet the association candidate conditions with the current sensor track within the current association matching period are determined. Select the first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track; If the current sensor track is already matched with the first global track, the matching strength between the current sensor track and the first global track is increased, and the first global track is updated using the current sensor track.

[0006] In a second aspect, an electronic device is provided, comprising multiple sensors of different types, a processor connected to the sensors, and a memory connected to the processor, wherein the memory stores a computer program executable by the processor, and the computer program, when executed by the processor, implements the steps of the multi-sensor track fusion method described above.

[0007] The multi-sensor track fusion method provided in the above embodiments, after the current sensor track is updated, determines candidate global tracks that meet the association candidate conditions within the current association matching period based on the current sensor track and the global track. From the candidate global tracks, the first global track with the closest spatiotemporal normalized distance to the current sensor track is selected. If the current sensor track and the first global track are already matched, it indicates that they have been identified as the same target in the past multiple association matching periods, demonstrating long-term consistency. Furthermore, if their spatiotemporal normalized distance is the closest in the current association matching period, it confirms that their overall difference in time and space dimensions is minimal compared to other global tracks in the current association matching period, i.e., the highest instantaneous similarity. When both long-term consistency and the highest instantaneous similarity are satisfied, the matching relationship between the current sensor track and the first global track is deemed reliable, thus strengthening their matching strength and updating the first global track using the current sensor track. In this way, the enhancement and update of matching strength relies on both the accumulation of historical matching relationships and the verification of the shortest spatiotemporal normalized distance in the current period, thereby effectively reducing erroneous updates caused by factors such as noise, target intersection or temporary occlusion, and improving the robustness and accuracy of multi-sensor track fusion.

[0008] The electronic devices provided in the above embodiments belong to the same concept as the corresponding multi-sensor trajectory fusion method embodiments, and thus have the same technical effects as the corresponding multi-sensor trajectory fusion method embodiments, which will not be repeated here. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the functional modules of an electronic device in one embodiment.

[0010] Figure 2 This is a flowchart of a multi-sensor track fusion method in one embodiment.

[0011] Figure 3 This is a flowchart of a multi-sensor track fusion method in another embodiment.

[0012] Figure 4 This is a schematic diagram illustrating the state transition between the matching state machine and the associated state machine in one embodiment.

[0013] Figure 5 This is a flowchart of the steps for determining candidate global tracks in one embodiment.

[0014] Figure 6 This is a flowchart of a multi-sensor track fusion method in another embodiment. Figures 7 to 9 This is a schematic diagram showing the correlation status of the flight path at different times. Detailed Implementation

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0018] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] The first aspect of this application provides a multi-sensor track fusion method. For example... Figure 1 As shown, this multi-sensor trajectory fusion method is applied to electronic devices with multiple sensors, such as drones, autonomous vehicles, robots, intelligent monitoring systems, and intelligent driving vehicles.

[0020] like Figure 1 As shown, the electronic device 100 includes multiple sensors 10, such as a first sensor, a second sensor, a third sensor, and so on. These sensors include, but are not limited to, visible light cameras, infrared cameras, millimeter-wave radar, lidar, acoustic sensors, inertial measurement units, and ultrasonic sensors. The sensors can be synchronized or asynchronous in time or space; that is, the detection time, data rate, sampling start time, installation position, and attitude of each sensor may differ. Regardless of whether the sensor data is synchronized in time and space, the multi-sensor trajectory fusion method of this application can be directly used for fusion.

[0021] like Figure 1 As shown, the electronic device 100 also includes a processor 20 connected to each sensor 10, and a memory 30 connected to the processor 20. The memory 30 stores a computer program that can be executed by the processor 20. When the computer program is executed by the processor 20, it implements the multi-sensor track fusion method of this application.

[0022] like Figure 2As shown, a multi-sensor track fusion method includes: Step 202: Process the target data of the sensor to be processed to obtain the updated current sensor track.

[0023] The sensor to be processed refers to any sensor in an electronic device, such as any one of a visible light camera, radar, or infrared camera. This sensor independently acquires raw data and identifies target data using a target detection algorithm. Target data typically includes the target's motion state and attribute information. Motion state includes position and velocity. Attribute information includes target ID and sensor type.

[0024] In some embodiments, if the sensors have achieved spatiotemporal synchronization through hardware timing and unified coordinate calibration, then the local track of the target corresponding to that sensor is updated using Kalman filtering or other recursive filters. For targets with existing tracks, a prediction-update cycle is performed; for new targets, a new track is initialized. This filtering process relies only on the sensor's own historical data and current measurements, and does not involve data from other sensors.

[0025] In some embodiments, if the sensors are not synchronized in time and space, such as due to different data rates, random sampling times, or different installation locations, the spatial coordinates (e.g., latitude, longitude, altitude, and local coordinates of the sensor) in the target data of the sensor to be processed are first transformed to a preset reference coordinate system (e.g., the ENU Northeast-Sky Coordinate System with a reference point as its origin). This transformation does not depend on whether other sensors are synchronized; even if the installation locations and orientations of the sensors are different, all data after transformation are represented under the same spatial reference. Then, the local track of the target corresponding to the sensor is updated using Kalman filtering or other recursive filters. For targets with existing tracks, a prediction-update cycle is performed; for new targets, a new track is initialized. This filtering process relies only on the historical data and current measurements of the sensor itself and does not involve data from other sensors.

[0026] Step 204: Based on the distance between the current sensor track and the global track, determine the candidate global track that meets the association candidate conditions with the current sensor track within the current association matching period.

[0027] The association matching period refers to one processing cycle of the prediction, and is usually related to the detection period. In one embodiment, the length of the association matching period is denoted as... It can be based on the sensor's nominal detection period. and allowed time fluctuation value Settings, for example The current association matching period refers to the association matching period at the current moment.

[0028] In one embodiment, for sensor tracks and a certain global trajectory The distance between the two is calculated and compared with a preset correlation threshold. If the distance is less than or equal to the correlation threshold, a global track is determined. Candidate global tracks that meet the association candidate conditions within the current association matching period; if the distance is greater than the association threshold, it is determined that the conditions are not met.

[0029] In one embodiment, the type of distance can be determined based on whether the sensor data is time-synchronized. If the sensor data is time-synchronized through hardware timing or interpolation alignment, the distance can be spatial, such as Euclidean distance or Mahalanobis distance. If the sensor data is not time-synchronized, i.e., the detection times and data rates of each sensor are inconsistent, the distance can be spatiotemporal, i.e., simultaneously considering spatial differences (position, velocity) and timestamp differences, for example, by normalizing the time difference and then weighting it with the spatial distance. This allows for a reasonable measurement of the similarity between tracks in asynchronous situations.

[0030] Step 206: Select the first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track.

[0031] Spatiotemporally normalized distance is calculated based on temporal and spatial distances. After normalization, the closest spatiotemporally normalized distance means that, considering both temporal and spatial dimensions, this global track is the candidate global track that best matches and is most likely to be associated with the current sensor track.

[0032] Step 208: If the current sensor track is already matched with the first global track, then the matching strength between the current sensor track and the first global track is increased, and the first global track is updated using the current sensor track.

[0033] If the current sensor track is already matched with the first global track, it means that the two have been consistently identified as the same target in the past multiple association matching cycles, that is, there is long-term consistency.

[0034] Furthermore, if the spatiotemporal normalized distance between the sensor track and the first global track is shorter than the spatiotemporal normalized distance between other candidate global tracks and the sensor track within the current association matching period, it indicates that the overall difference between the two in the time and space dimensions is the smallest in the current period, that is, the instantaneous similarity is the highest.

[0035] When both long-term consistency and instantaneous highest similarity are satisfied, the matching relationship is determined to be highly reliable. Therefore, the matching strength between the current sensor track and the first global track is enhanced, for example, by increasing the matching strength by one level from the current value.

[0036] Based on this, updating the first global track using the current sensor track can effectively reduce the risk of erroneous updates caused by noise, target intersection, temporary occlusion, etc., thereby significantly improving the robustness and accuracy of multi-sensor track fusion.

[0037] The multi-sensor track fusion method in this embodiment, after updating the current sensor track, determines candidate global tracks that meet the association candidate conditions within the current association matching period based on the current sensor track and the global track. From these candidate global tracks, it selects the first global track with the closest spatiotemporal normalized distance to the current sensor track. If the current sensor track and the first global track are already matched, it indicates that they have been identified as the same target in the past multiple association matching periods, demonstrating long-term consistency. Furthermore, if their spatiotemporal normalized distance is the closest in the current association matching period, it confirms that their overall difference in time and space dimensions is minimal compared to other global tracks in the current association matching period, i.e., their instantaneous similarity is highest. When both long-term consistency and the highest instantaneous similarity are satisfied, the matching relationship between the current sensor track and the first global track is deemed reliable. Therefore, the matching strength is enhanced, and the first global track is updated using the current sensor track. In this way, the enhancement and update of matching strength relies on both the accumulation of historical matching relationships and the verification of the shortest spatiotemporal normalized distance in the current period, thereby effectively reducing erroneous updates caused by factors such as noise, target intersection or temporary occlusion, and improving the robustness and accuracy of multi-sensor track fusion.

[0038] In one embodiment, a matching relationship between sensor tracks and global tracks is defined, and the matching relationship is divided into multiple matching intensities from low to high. If a sensor track is matched with a global track, it indicates that they belong to the same target; therefore, the matching relationship is unique. In one association matching cycle, a sensor track within the same sensor can only be matched with one global track, and a global track can also only be matched with one sensor track within the same sensor; that is, the matching relationship is unique.

[0039] In one embodiment, the association relationship between sensor tracks and global tracks is also defined. The association relationship indicates that the two tracks are candidate matches. When the association relationship is strengthened and certain conditions are met, the sensor track and global track change from an association relationship to a match relationship. A sensor track within the same sensor can be associated with multiple global tracks, and a global track can be associated with multiple sensor tracks within the same sensor.

[0040] In one example, the matching strength of a relationship can be divided into first matching strength, second matching strength, and no matching, from high to low. If a global track and a sensor track are matched and the matching strength is first matching strength, then they are in a matching relationship with the highest matching strength. In another example, the matching strength of a relationship can be divided into bound, strong matching, weak matching, and no matching, from high to low. If a global track and a sensor track are matched and the matching strength is bound, then they are in a matching relationship with the highest matching strength.

[0041] The different levels of matching intensity can be determined based on actual needs, the computing power of electronic devices, sensor characteristics, and the complexity of the application scenario. For example, in scenarios with ample computing resources and high tracking accuracy requirements, finer levels (such as five or six levels) can be used to more precisely reflect the gradual change in matching intensity. In scenarios with limited resources or extremely high real-time requirements, fewer levels can be used to reduce the complexity of matching state switching.

[0042] In one embodiment, the association strength of the association state includes no association and at least two associated strengths, with the highest association strength being among the at least two associated strengths; the matching strength of the matching relationship includes no matching and at least two matched strengths, with the highest matching strength being among the at least two matched strengths.

[0043] In one example, the association strength, from highest to lowest, includes first association strength, second association strength, and no association. In another example, the association strength, from highest to lowest, includes strong association, weak association, and no association. The different association strengths can be categorized based on actual needs, the computing power of electronic devices, sensor characteristics, and the complexity of the application scenario. For example, in scenarios with ample computing resources and high tracking accuracy requirements, finer levels (such as five or six levels) can be used to more precisely reflect the gradual change in matching degree. In scenarios with limited resources or extremely high real-time requirements, fewer levels can be used to reduce the complexity of association state switching.

[0044] In one embodiment, after obtaining the updated current sensor track, the method further includes: identifying tracks in the sensor track and global track whose update time exceeds their corresponding association matching period as update timeout tracks; Reduce the correlation strength between other tracks that are associated with the update timeout track, and reduce the matching strength between other tracks that are matched with the update timeout track; the correlation represents the degree of candidate matching between sensor tracks and global tracks.

[0045] Specifically, after obtaining the updated current sensor track, all active sensor tracks and the global track are traversed, and the last update timestamp of each track is obtained, and the time difference with the current time is calculated. The time difference is compared with its associated matching period. If the time difference exceeds its associated matching period, the sensor track is marked as an update timeout track.

[0046] The association matching period is different for each track, or the association matching period is different for different types of tracks.

[0047] For each track marked as having timed out of update, reduce the association strength and matching strength.

[0048] Specifically, for a sensor track, its associated global track is searched; for a global track, its associated sensor track is searched. Specifically, if sensor track A and global track C have both an association and a matching relationship, the matching strength decreases progressively (binding → strong matching → weak matching → no matching), while the association strength is directly reduced to no association (strong association / weak association → no association). If the two only have an association relationship, the association strength is simply reduced to no association.

[0049] In this embodiment, the outdated relationships are gradually decayed by adjusting the association and matching relationships of tracks that have not been updated within the association matching period.

[0050] One embodiment of the multi-sensor track fusion method, such as Figure 3 As shown, it includes the following steps: Step 300: Process the target data of the sensor to be processed to obtain the updated current sensor track.

[0051] Step 302: Determine whether the update time of all sensor tracks and the global track exceeds their corresponding association matching period.

[0052] If yes, proceed to step 301; otherwise, proceed to step 304.

[0053] Step 301: Determine the update timeout track, reduce the association strength between other tracks that are related to the update timeout track, and reduce the matching strength between other tracks that are matched with the update timeout track.

[0054] Specifically, if sensor track A and global track C have both a correlation and a matching relationship, the matching strength decreases progressively (binding → strong matching → weak matching → no matching), while the correlation strength is directly reduced to no correlation (strong correlation / weak correlation → no correlation). If the two only have a correlation relationship, the correlation strength is directly reduced to no correlation.

[0055] After step 301, proceed to step 304.

[0056] Step 304: Based on the distance between the current sensor track and the global track, determine whether there is a candidate global track that meets the association candidate conditions with the current sensor track within the current association matching period. If so, proceed to step 306.

[0057] Step 306: Select the first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track.

[0058] That is, in the current association matching period, the current sensor track and the first global track have the highest similarity in time and space.

[0059] Step 308: Determine whether the current sensor track and the first global track are already in a matching relationship.

[0060] If yes, meaning that the two have been confirmed to belong to the same target in the past association matching cycle, then proceed to step 310; otherwise, proceed to step 314.

[0061] Step 310: Enhance the matching strength between the current sensor track and the first global track.

[0062] If the judgment result in step 308 is yes, it means that the current sensor track and the first global track are closest in spatiotemporal normalized distance in the current association matching period and are in a matching state. Then, the matching strength between the two is further enhanced. The higher the matching strength, the higher the confidence that the two belong to the same target and the less likely they are to be interfered with by other matching or association.

[0063] In one example, enhancing the matching strength between the current sensor track and the first global track can be achieved through a step-by-step approach. The matching strength, from highest to lowest, can be categorized as bound, strong match, weak match, and no match. For instance, if the current sensor track and the candidate global track have a strong match, the relationship is upgraded to bound; if the current match is weak, it is upgraded to strong match. If the current sensor track and the candidate global track already have a bound relationship (the highest level), then the bound relationship is maintained.

[0064] After step 310, step 312 is executed to update the first global track using the current sensor track.

[0065] By fusing the latest sensor observations into the global track based on high-confidence matching relationships, the trajectory can be effectively smoothed, prediction errors reduced, and the accuracy of fusion tracking improved.

[0066] In one embodiment, if the determination result in step 308 is negative, it means that within the current association matching period, the first global track is the track with the closest spatiotemporal normalized distance among all candidate global tracks of the current sensor track, but the current sensor was not determined to belong to the same target in the past association matching period, i.e., it was not matched. Then, step 314 is executed to determine whether the first global track and other sensor tracks are in the matching relationship with the highest matching strength, such as binding.

[0067] If the result of step 314 is yes, it means that the first global track has been occupied by another sensor track with the highest confidence. Since this global track has formed a strong binding with another sensor track, it should not be matched or associated with the current sensor track to avoid conflict.

[0068] If the result of step 314 is negative, it means that the first global track is not occupied by other sensor tracks with the highest matching strength. Then, step 316 is executed to determine whether the current sensor track is in a matching relationship with other global tracks with the highest matching strength.

[0069] Among them, the first global track is not in a matching relationship with other sensor tracks with the highest matching strength, including the first global track not being in a matching relationship with other sensor tracks, or the first global track being in a matching relationship with other sensor tracks but the matching strength is not the highest matching strength, such as a weak match.

[0070] If the result of step 316 is negative, meaning the current sensor track is not occupied by any other global track with the highest matching strength, and the first global track is also not occupied by any other sensor track with the highest matching strength, then although the two were not matched in the past, they are closest in the current period and neither has a higher priority occupancy, indicating that they have the potential to become the same target. In this case, step 318 is executed to strengthen the correlation between the current sensor track and the first global track.

[0071] In one example, the association strength, from highest to lowest, includes strong association, weak association, and no association. If the current sensor track and the first global track are not associated, then strengthening their association results in a weak association. If the current sensor track and the first global track are weakly associated, then strengthening their association results in a strong association. If the current sensor track and the first global track are strongly associated, then maintaining their association results in a strong association strength.

[0072] Thus, through this gradual enhancement mechanism, if the association strength reaches the preset conversion condition after multiple association matching cycles, the association relationship can be further upgraded to a matching relationship, and an initial matching strength, such as the minimum matching strength, can be assigned. In this way, the gradual enhancement of association strength serves as a transitional stage for establishing a matching relationship, effectively reducing the risk of mismatches caused by direct matching at the nearest distance in a single instance, and improving the reliability of track fusion.

[0073] In one embodiment, such as Figure 3 As shown, if the judgment result of step 316 is yes, that is, although the current sensor track has the closest spatiotemporal normalized distance to the first global track, the two are not in a matching relationship, while the current sensor track is in a matching relationship with other global tracks with the highest matching strength, such as being bound, it means that the current sensor track has been occupied by other global tracks with the highest confidence. In this case, even if it is closest to the first global track in the current period, it should not establish an association or match with the first global track again, otherwise it will lead to the contradiction of one sensor track matching two different global tracks at the same time.

[0074] Therefore, if the judgment result in step 316 is yes, step 317 is executed to reduce the correlation between the current sensor track and the relevant global track to no correlation.

[0075] By reducing existing associations of the current sensor track to unassociations, including the association between the current sensor track and the first global track, while preserving the high-strength match between the current sensor track and its original global track, the uniqueness and stability of the matching relationship are maintained. This reduces the probability of accidentally switching to a matching relationship due to proximity, thus ensuring that existing high-strength matching is not disturbed.

[0076] In another embodiment, such as Figure 3 As shown, if the judgment result of step 314 is yes, that is, if the first global track and other sensor tracks are in the highest matching strength matching relationship, then step 320 is further executed.

[0077] Step 320: Determine whether the current sensor is associated with or matched with any global track.

[0078] If the result of step 320 is negative, meaning the current sensor track has no historical association or matching record, it indicates that the sensor track is a completely new target. In the current association and matching cycle, although it has the closest spatiotemporal normalized distance to the first global track, the first global track is already occupied by another sensor track with the highest matching strength, making it impossible to establish an association or match with it. Since this sensor track has no historical association, it is more likely that it corresponds to a new target that has not yet been tracked by the system, rather than the same target as the first global track. Therefore, step 322 is executed: a new global track is started with the current sensor track, and a matching relationship with the highest matching strength is established between the current sensor track and this new global track, such as binding, to quickly capture the new target.

[0079] In one embodiment, if the judgment result of step 304 is negative, that is, based on the distance between the current sensor track and the global track, it is determined that there is no candidate global track that meets the association candidate conditions with the current sensor track in the current association matching period, it means that the current sensor track is far away from all existing global tracks and cannot be associated or matched with any existing target.

[0080] At this point, step 320 is executed to determine whether the current sensor track is associated with or matched with any global track.

[0081] If the result of step 320 is negative, meaning there is no historical correlation record, it indicates that the sensor track represents a completely new target that has not been tracked by the system before. Therefore, step 322 is executed: a new global track is started from the current sensor track, and a matching relationship with the highest matching strength is established between the current sensor track and this new global track. This allows for rapid target acquisition.

[0082] In this way, when there is no historical correlation, a new global track can be started in a timely manner, reducing the probability of missing new targets. When there is historical correlation, the failure of correlation / matching relationships can be reasonably decayed to prevent long-term resource occupation due to short-term distance.

[0083] In another embodiment, such as Figure 3 As shown, after step 318, i.e. after strengthening the relationship between the first sensor track and the first global track, step 326 is executed to determine the following two conditions: Condition 1: The first global track has the highest correlation strength with the current sensor track, and is matched with other sensor tracks but not with the highest matching strength, such as weak matching; Condition 2: Among the sensor tracks that have the highest correlation strength or the lowest matching strength with the first global track, the current sensor track is closest to the spatiotemporal normalized distance of the first global track.

[0084] In this embodiment, step 326 determines whether the conversion conditions for upgrading from an association relationship to a matching relationship are met by using conditions 1 and 2.

[0085] If both conditions 1 and 2 are met, it indicates that the first global track and the current sensor track have established the highest correlation strength, such as a strong correlation, meaning they are very close at the candidate level. Simultaneously, the first global track has a matching relationship with another sensor track, but it is not the highest matching strength, such as a weak match, meaning this matching relationship is fragile and easily replaced. Furthermore, among all sensor tracks with the highest or lowest correlation strength with the first global track, the current sensor track has the shortest spatiotemporal normalized distance, indicating that it is currently the optimal candidate.

[0086] Therefore, if both conditions 1 and 2 are satisfied, it means that the current sensor track has been enhanced through multiple associations and has become the candidate most closely associated with the first global track. The original matching strength of the first global track is very low, and it should be replaced by the superior current sensor track. Therefore, if both conditions 1 and 2 are satisfied, then step 328 is executed, establishing a matching relationship between the first global track and the current sensor track with the lowest matching strength, and dissolving the matching relationships between the first global track and other sensor tracks.

[0087] Specifically, a matching relationship is established between the first global track and the current sensor track, and the lowest matching strength is assigned, such as weak matching, while the matching relationship between the first global track and the original sensor track is deactivated.

[0088] Then, step 312 is executed to update the first global track using the current sensor track.

[0089] Thus, this embodiment utilizes the transformation condition from association to matching. It achieves a transition from high association strength to low matching strength, allowing sensor tracks to re-establish matching relationships with more reliable global tracks, thereby enabling target updates.

[0090] In one embodiment, after obtaining the updated current sensor track, the method further includes: if the difference between the current time and the last update time of the sensor track exceeds a preset sensor track termination period, then the sensor track is terminated, and the association and matching relationships associated with the terminated sensor track are cleared.

[0091] In this embodiment, after the current sensor track is updated, if the difference between the current time and the last update time of the sensor track exceeds the preset sensor track termination period, it indicates that the sensor track has expired. The sensor track is then deleted from the active track list or marked as terminated. At the same time, all global tracks are traversed to clear the association and matching relationship between the sensor track and any global track.

[0092] In this way, outdated sensor tracks generated due to targets leaving the sensor's field of view, sensor malfunction, or false alarms can be cleared in a timely manner, preventing them from occupying system resources for a long time and interfering with subsequent correlation matching. By clearing correlation / matching relationships, invalid sensor tracks are prevented from incorrectly affecting the update or matching strength of global tracks.

[0093] In one embodiment, after obtaining the updated current sensor track, the method further includes: if the difference between the current time and the last update time of the global track exceeds a preset global track termination period, then the global track is terminated, and the association and matching relationships associated with the terminated global track are cleared.

[0094] Generally speaking, the duration of the global track termination cycle is longer than the duration of the sensor track termination cycle.

[0095] In this embodiment, after the current sensor track is updated, if the difference between the current time and the last update time of the global track exceeds the preset global track termination period, it indicates that the global track has failed. In this case, the global track is deleted from the global track list or marked as terminated, and all associations and matching relationships related to the global track are cleared.

[0096] In this way, outdated global tracks that have not been updated by any sensors for a long time can be automatically cleared.

[0097] Track termination refers to the removal of a sensor track or global track from the track list due to reasons such as the target leaving the detection range, prolonged lack of updates, or sensor failure. Track termination means that the physical target represented by the track no longer exists or is no longer credible. At this time, the associated status of all related sensor tracks or candidate global tracks is set to unassociated to clear recent association information of invalid tracks and free up resources.

[0098] To better illustrate the switching between matching and association states in this application, in one embodiment, the matching state can be divided into binding, strong matching, weak matching and no matching from high to low, and the association state can be divided into strong association, weak association and no association from high to low.

[0099] Switching conditions include: Condition A: Initial candidate global track for sensor tracks; Condition B: Within the current association matching period, the sensor track and the candidate global track satisfy the association candidate condition.

[0100] Condition C: The current sensor track is already matched with the first global track, which is the track with the closest spatiotemporal normalized distance to the current sensor track.

[0101] Condition D: Within the current association matching period, the sensor track and a certain global track do not meet the association candidate conditions; Condition E: The current sensor track is not matched with the first global track.

[0102] Condition F: The first global track is bound to its sensor track.

[0103] Condition G: The first global track is not bound to its sensor track.

[0104] Condition H: Sensor tracks are bound to other global tracks; Condition I: Sensor tracks are not bound to other global tracks. Condition J: The sensor track has not established a correlation or matching relationship with any global track; Condition K: The sensor track is currently in a weak matching relationship with a certain global track (denoted as G1), while there is another global track (denoted as G2) that is in a strong correlation with the sensor track, and the spatiotemporal normalized distance from the sensor track to G2 is the shortest.

[0105] Condition L: Track update timeout.

[0106] Condition M: The sensor track or global track is terminated.

[0107] Within a given time period, a global track can be associated with multiple sensor tracks, and a sensor track can be associated with multiple global tracks. This allows for the simultaneous maintenance of various associations between sensor tracks and global tracks. For example... Figure 4 As shown, the matching state machine controls the switching of matching states, including: 1. When condition A is met, i.e., when the sensor track starts a candidate global track, the matching status between the sensor track and the candidate global track is set to binding. One way to start a global track is: when conditions D and J are met simultaneously, or when conditions B, E, F and J are met simultaneously, a new global track is started with the current sensor track.

[0108] 2. When conditions B and C are met simultaneously, if the current matching state is weak, it will be upgraded to strong matching; if the current matching state is strong matching, it will be upgraded to binding; if it is already bound, it will remain bound.

[0109] 3. When condition L is met, the matching relationship between the two decreases step by step. If the current matching state of the sensor track and the candidate global track is bound, it decreases to a strong match; if the current matching state is a strong match, it decreases to a weak match; if the current matching state is a weak match, it switches to no match; if the current state is no match, the current matching state remains no match.

[0110] 4. When condition M is met, that is, when the sensor track or global track is terminated, regardless of the current matching status value, the matching status related to the terminated track is determined to be unmatched.

[0111] 5. When condition K is met, that is, the sensor track is currently in a weak match with a certain global track, and there is another global track that is in a strong correlation with the sensor track and is closest in distance (such as spatiotemporal distance), then the matching status of the original weak match pair changes from weak match to unmatch, and the new match pair changes from unmatch to weak match.

[0112] like Figure 4 As shown, the associated state machine controls the switching of associated states, including: 1. When condition L is met, if the correlation strength is strong or weak, it will decrease to no correlation.

[0113] 2. When conditions B, E, G and H are met simultaneously, if the correlation between the sensor track and the candidate global track is weak or strong, then they are all reduced to uncorrelated.

[0114] 3. When conditions B, E, G and I are met simultaneously, if the association status of the sensor track is not associated, it will be upgraded to weak association; if the association status of the sensor track is weakly associated, it will be upgraded to strong association; if the association status of the sensor track is strong association, it will remain strong association.

[0115] 4. If condition M is met, i.e. the sensor track or global track is terminated, then the associated states related to the terminated track are all reduced to unassociated.

[0116] By using a matching state machine to manage the long-term cumulative relationships (bound / strong / weak / unmatched) of the track, and an association state machine to manage the recent association activity (strong / weak / unmatched), and by making the two influence each other (e.g., clearing other associations when binding, and strong associations participating in weak match competition switching), the following effects are achieved: 1. The matching status gradually decreases while retaining historical trust, and the association status is directly cleared to reflect the latest interruption; after the target reappears, the association status gradually increases, and the matching status jumps directly from weak / strong matching back to binding, balancing continuity and response speed.

[0117] 2. Meeting the candidate condition once only improves the associated state. Multiple cycles of accumulation are required to upgrade the matching state (or directly bind it at the beginning), reducing the probability of erroneous fusion triggered by instantaneous noise.

[0118] 3. Weak matching competition switching requires that the new candidate is in a strong correlation and the closest distance. By utilizing historical evidence of the correlation state, the probability of incorrect switching caused by the minimum distance in a single frame is reduced.

[0119] 4. When the matching status reaches the binding state, the association status of other global tracks will be forcibly cleared to ensure that a sensor track only updates one global track and prevent data pollution.

[0120] 5. When the global track is unmatched, since there is no matching conflict, when the current sensor track meets the candidate conditions and neither of them is bound, the association strength can steadily increase step by step (unmatched → weakly matched → strongly matched), providing a clear confidence accumulation path for subsequent upgrade to a matched relationship.

[0121] The multi-sensor track fusion method described in the above embodiments is used in scenarios where sensor data has been spatiotemporally synchronized or has not yet been synchronized. After obtaining the updated sensor tracks, prediction-update is performed on the existing tracks, and tracks are initialized for the new targets.

[0122] In practical applications, electronic devices typically integrate multiple functions to meet the needs of their applications, and their sensors are often heterogeneous. Taking drones as an example, with the booming development of the low-altitude economy and the widespread application of drone technology, drones are increasingly used in logistics, surveying, agriculture, and emergency rescue. At the same time, the safety hazards they bring are becoming increasingly prominent. To regulate industry development, the state has issued a series of standards, such as the "Safety Requirements for Civil Unmanned Aerial Vehicle Systems," to promote the drone industry into a standardized and large-scale application stage. Against this backdrop, efficient, accurate, and reliable drone detection technology has become crucial for ensuring low-altitude safety and airspace management.

[0123] Currently, UAV detection systems generally rely on the collaborative operation of multiple types of sensors. While collaborative detection using various sensor types can improve detection performance, the inconsistent data rates provided by heterogeneous sensors pose challenges to UAV detection. In existing technologies, most methods correct the data from all sensors to the same timestamp through time synchronization for collaborative detection. However, when the data rates of all sensors are relatively low, time synchronization using interpolation methods can increase the inherent error of the data and reduce fusion accuracy. For example, some existing methods require track data to have the same timestamp and to align the collected sensor data in time. Some existing methods also require time synchronization of data from various sensors before using deep learning methods for UAV detection. This approach is highly dependent on labeled data and has high engineering implementation costs. Another approach is to address the data rate inconsistency problem by using an asynchronous association system optimal allocation algorithm. However, this approach has high computational complexity. For example, it accumulates the Cauchy distribution membership degrees between all trajectory point pairs through time dynamic regularization and performs trajectory association allocation based on the system optimal principle. When the number of trajectories increases, combinatorial explosion occurs during the association allocation process, reducing the real-time performance of the algorithm. Furthermore, this approach does not consider the real-time processing of asynchronous trajectories within the same sensor detection period.

[0124] To address the issue of heterogeneous multi-sensor architecture, this application proposes a method for accurately measuring and determining candidate global tracks under spatiotemporal asynchrony when updating data from multiple heterogeneous sensors in an electronic device.

[0125] For heterogeneous sensors, when the target data of the sensor is acquired, the target data of the sensor is first transformed to achieve spatial synchronization.

[0126] Specifically, the target data of the sensor to be processed is processed and the sensor track is updated, including: acquiring the target data of the sensor and transferring the target data of the sensor to a preset reference coordinate system; tracking the target data of the sensor after coordinate transformation and updating the sensor track.

[0127] Specifically, from electronic devices Choose one sensor from the sensors. The reference sensor is denoted as , and its LLA latitude and longitude coordinates are used as reference coordinates. The reference coordinates can also be set manually, among which, Indicates the number of sensors. Indicates the sensor index. , and These represent the longitude, latitude, and altitude of the reference coordinates, respectively. At any given time, upon receiving target data from the sensor, the LLA coordinates in the target data are transformed to ENU (northeast-east) coordinates relative to the reference coordinates, and the target data is recorded as... ,in This indicates the number of data received by the data processing unit in the global scope. The detection timestamp of each data point This indicates the target's motion state information, such as position and velocity. This indicates the target's attribute information, such as target ID, sensor ID, sensor type, target type, etc.

[0128] The data processing unit of the electronic device will receive the target data sequentially. Then through The sensor type, sensor ID, and target ID in the data can be used to determine the unique identifier of the target from multi-sensor target data: (sensor type, sensor ID, target ID). This information is then accumulated over multiple time points to form a track. During track formation, Kalman filtering is used to estimate the target's motion state, as detailed below: pass The sensor type and sensor ID are used to determine the data. From sensors , This indicates that the data processing unit receives data from the sensor. The first received in There are 1 data point, and the corresponding timestamp is 1. .according to In the (sensor type, sensor ID, target ID) field, you can retrieve the sensor information. Extract sensor tracks from the track list ,in Indicates sensor Index of the Chinese flight path Indicates sensor The number of flight paths. If no such path exists... For the corresponding track, a new track will be created. If it exists with The corresponding track update process is as follows: track The last state was estimated to be its own number 1. The next estimate, the state, and the covariance are expressed as: , where state Given the position, velocity, and angular velocity along the X, Y, and Z axes in three-dimensional space, then in the prediction step: in, This represents the predicted state and the predicted state covariance. Indicates the flight path exist The state transition matrix at time t, Indicates the flight path exist The process noise matrix of the motion model at any given time. Received data. Afterwards, the flight path was... exist Estimate the state at time: in, Represents the covariance matrix of the predicted measurement. Denotes its inverse matrix, Represents the measurement equation, when When linear, If it is a linear observation matrix, then it is the Jacobian matrix of the measurement equation. express The noise covariance matrix, Indicates Kalman gain, Indicates the flight path exist The estimated state and the estimated state covariance at time t.

[0129] Through the recursive process of prediction and update using Kalman filtering, the state and covariance are iteratively corrected based on the motion model and measurement information each time new measurement data is received, so that the estimated value gradually approaches the actual motion trajectory of the target, while the covariance reflects the reliability of the estimate.

[0130] Thus, single-sensor tracking is achieved, in which state estimation is performed. This represents the optimal estimate of the target's motion state (position, velocity, acceleration, etc.) at the current moment from the sensor track, while the estimated state covariance matrix... This indicates the uncertainty (variance and correlation) of the estimate.

[0131] After the sensor track is updated, the time of the sensor track and the global track is synchronized. Based on the spatiotemporal synchronization, candidate global tracks that meet the association candidate conditions with the sensor track within the current association matching period are determined.

[0132] Specifically, based on the distance between the sensor track and the global track, candidate global tracks that meet the association candidate conditions with the sensor track within the current association matching period are determined, such as... Figure 5 As shown, it includes: Step 502: The timestamps of the global trajectory prediction and the sensor trajectory are used to achieve time alignment.

[0133] Maintaining the global track in the data processing unit And sequentially acquire the tracks of currently active sensors. and link it to the global track Then, they are associated and then merged. Index representing the global track. This indicates the number of global tracks at the current moment.

[0134] Make global trajectory The last state fusion was its own first The corresponding timestamp for this fusion is Using the state transition matrix, all timestamps Less than Global track extrapolation of state to Global track The The state and covariance of the sub-fusion are represented as follows: Then, in the prediction step, we have: in, Represents the global track The predicted state and the predicted state covariance, Represents the global track exist The state transition matrix at time t, Represents the global track exist The process noise matrix of the motion model at any given time.

[0135] This step uses Kalman filtering to extrapolate the state of the global track to the precise timestamp of the sensor track. After prediction alignment, theoretically, both represent the target state at the same moment.

[0136] Step 504: Based on the state of the sensor track and the global track in the reference coordinate system, calculate the spatial distance between the sensor track and the global track. The spatial distance includes the horizontal position track distance, the vertical position track distance, the horizontal velocity track distance, and the vertical velocity track distance.

[0137] In this embodiment, spatial distance is represented by multiple distance components of horizontal position / velocity and vertical position / velocity.

[0138] Step 506: Determine whether the corresponding candidate association conditions are met based on the spatial distance.

[0139] Spatial distance (position difference, velocity difference) directly determines whether two tracks could be the same target in physical space. If the spatial distance is too large (e.g., a difference of hundreds of meters), correlation is absolutely impossible regardless of temporal synchronization. Temporal distance mainly reflects the degree of temporal asynchrony; it does not directly negate the possibility of correlation. Even with a large time difference, if the target's movement is gentle, the spatial distance may still be small after prediction compensation. Therefore, in the single correlation candidate screening stage, four independent distance components (horizontal position, vertical position, horizontal velocity, and vertical velocity) are used as hard threshold conditions. This multi-dimensional distance component screening mechanism can quickly eliminate a large number of irrelevant global tracks, significantly reducing the number of candidates required for subsequent fine calculations. At the same time, evaluating the velocity component independently of the position component reduces the probability of incorrectly eliminating candidates due to sudden changes in the overall distance caused by abrupt changes in target velocity, improving the correlation success rate in complex scenarios. After completing the single candidate screening, the correlation candidate situation of multiple correlation matching cycles is further utilized. Historical evidence is accumulated through a matching state machine to finally determine the first global track that achieves the highest matching strength with the sensor track.

[0140] In one embodiment, the spatial distance between the sensor track and the global track is calculated based on their states in a reference coordinate system. This includes: acquiring the state sequence of the sensor track and the state sequence of the global track within a set time window; for each state point in the state sequence of the global track, finding adjacent sensor state points in the state sequence of the sensor track that meet the interpolation conditions at adjacent time points; obtaining corresponding interpolation points by linear interpolation of the adjacent sensor state points; and calculating the horizontal position distance, vertical position distance, horizontal velocity distance, and vertical velocity distance between the global track state point and the interpolation point; for each state point in the state sequence of the sensor track, in... Find adjacent global track state points in the global track state sequence that meet the interpolation conditions at adjacent time points. Obtain the corresponding interpolation points by linear interpolation of adjacent global track state points. Calculate the horizontal position distance, vertical position distance, horizontal velocity distance, and vertical velocity distance between the sensor track state points and the interpolation points. Obtain the horizontal position track distance based on the average horizontal position distance from the interpolation points, the vertical position track distance based on the average vertical position distance from the interpolation points, the horizontal velocity track distance based on the average horizontal velocity distance from the interpolation points, and the vertical velocity track distance based on the average vertical velocity distance from the interpolation points.

[0141] Specifically, setting a time window During the time window Internal computing sensor track Spatial distance to each global track. Set within a time window. Sensor tracks The state sequence is The corresponding time series is Global track The state sequence is The corresponding time series is The distance value for each state pair is calculated using bidirectional interpolation.

[0142] First, the sensor trajectory was analyzed. Perform interpolation to calculate the global trajectory. Each state to sensor track The distance. If for , The time condition for interpolation is satisfied. , Then it can be found in the sensor track Two adjacent sensor state points and Interpolation between The interpolation formula is: Then calculate and Represented by the distance between To sensor track The distance. The formula for calculating the location distance is: in, and They represent and The horizontal and vertical distances between them , and They represent The X, Y, and Z coordinates in the data. , and They represent interpolation. The X, Y, and Z coordinates can be used to calculate the horizontal velocity and distance. and vertical velocity distance .

[0143] Similarly, for the global trajectory Perform interpolation to calculate sensor tracks From each state to the global track The distance can be used to obtain the sensor track. middle To global track Horizontal position distance Vertical position distance Horizontal speed and distance Vertical velocity and distance .

[0144] Therefore, the global trajectory can be obtained. With sensor tracks Distance between tracks: in, , , and Representing the global trajectory and sensor tracks The horizontal position track distance, vertical position track distance, horizontal velocity track distance, and vertical velocity track distance between them. and Representing the global trajectory and sensor tracks The number of states that satisfy the time conditions for interpolation.

[0145] Traditional spatial distance calculations are typically based on Mahalanobis distance, which requires an accurate covariance matrix and real-time updates to its inverse matrix, resulting in high computational costs. Furthermore, the statistical characteristics of noise in real-world systems are difficult to model accurately, easily leading to distance distortion. Additionally, Mahalanobis distance only uses the latest state (or a predicted single point) for judgment, making it susceptible to transient noise interference and resulting in insufficient correlation reliability. To address these shortcomings, this embodiment employs four independent spatial distance components (horizontal position, vertical position, horizontal velocity, and vertical velocity) and calculates the average distance of multiple historical state points through bidirectional interpolation within a time window. This approach reduces dependence on the covariance matrix, significantly decreasing computational costs and making it suitable for real-time embedded systems. Moreover, the multi-point averaging naturally filters sensor noise and transient jitter, significantly improving correlation reliability.

[0146] In one embodiment, the associated candidate conditions include: The horizontal position track distance between the global track and the sensor track is less than the horizontal position track association threshold; The vertical position track distance between the global track and the sensor track is less than the vertical position track association threshold; The horizontal velocity track distance between the global track and the sensor track is less than the horizontal velocity track association threshold; The vertical velocity track distance between the global track and the sensor track is less than the vertical velocity track association threshold; The horizontal position distance between the latest sensor track time and the corresponding interpolation point of the global track is less than the horizontal position track association threshold; The vertical distance between the latest sensor track and the corresponding interpolation point of the global track is less than the vertical position track association threshold.

[0147] Specifically, global trajectory and sensor tracks The global trajectory is determined if the following conditions are met. Entering the sensor track The associated candidates are determined based on the following conditions: in, Indicates the horizontal position track association threshold, Indicates the vertical position track association threshold, This indicates the horizontal speed trajectory association threshold. These parameters represent the vertical velocity trajectory correlation thresholds and need to be set in conjunction with the sensor's detection accuracy. (Regarding the global trajectory...) With sensor tracks If the association candidate conditions are not met, then switch to the next global track, continue to calculate the spatial distance, and determine whether the association candidate conditions are met.

[0148] Based on this, the first global track with the closest spatiotemporal normalized distance to the current sensor track is selected from the candidate global tracks. This includes: calculating the time distance between the sensor track and the global track; calculating the spatiotemporal normalized distance of the spatial distance and the time distance; the spatial distance includes the horizontal position track distance, the vertical position track distance, the horizontal velocity track distance, and the vertical velocity track distance; and determining the global track with the smallest spatiotemporal normalized distance to the sensor track as the closest candidate global track.

[0149] The multi-sensor track fusion method of this application still needs to filter out the closest candidate global track during the switching process between the associated state machine and the matching state machine (e.g., weak matching competition switching) and when there are multiple candidate global tracks that meet the conditions for the same sensor track.

[0150] This embodiment uses spatiotemporally normalized distance to quantify the similarity between tracks. This distance comprehensively considers five components: horizontal position, vertical position, horizontal velocity, vertical velocity, and time difference, and then normalizes each component before weighted summation. By calculating the spatiotemporally normalized distance, the global track that is closest to the current sensor track can be objectively and accurately selected as the basis for state machine switching or competition arbitration.

[0151] By employing the above approach, the reliability of the fusion decision is ensured through multi-cycle cumulative confirmation, and the best matching result is provided at the moment of selection using spatiotemporal normalized distance, thus organically combining historical trust with current observation. This method reduces the noise sensitivity problem caused by relying solely on single-frame distance, and can effectively handle scenarios where targets are re-associated after brief separation through a step-by-step descent and step-by-step ascent mechanism of matching state. This significantly improves the accuracy and robustness of multi-sensor track fusion in complex dynamic environments.

[0152] In one embodiment, calculating the time distance between the sensor track and the global track includes: The number of missing cycles is determined based on the relationship between the difference between the timestamps of the sensor track and the global track and the integer multiple of the sensor detection period. The time distance is then determined based on the number of missing cycles. The time distance increases stepwise as the number of missing cycles increases.

[0153] Specifically, sensor tracks The latest timestamp is Global track The latest timestamp is Therefore, the time difference is: Set sensor The detection cycle is In a real-world system, without any missed alarms, the time interval between two consecutive detections of the same target by the same sensor will be... It fluctuates up and down, so a float value is set. Therefore, the sensor track With global track The time interval is: in, , You can choose flexibly according to the actual situation, usually .

[0154] when Sometimes, the timestamp of the sensor track is earlier than that of the global track. This should not normally happen because the global track is aligned to the sensor time through prediction, but as a fallback, the maximum value is assigned. This indicates that the association is extremely unreliable.

[0155] when When the time difference is less than one complete cycle (after deducting fluctuations), it indicates that the sensor's timing is very close to the global track's timing, and is assigned a value. , is a relatively small constant.

[0156] when At that time, the time difference is approximately equal to One whole cycle (allowing for ±ΔT_s of oscillation). At this point, the value is assigned... ,For example Time assignment , Time assignment And so on. This means that it has been lost. For each detection cycle, the associated cost increases linearly with the number of missing cycles.

[0157] when At this time, the time difference is in the middle range between two integer periods. The value assigned at this time is... ,For example Time assignment , Time assignment And so on. The penalty in this area is slightly higher than the previous area, but lower than the next full cycle area.

[0158] when If the time difference exceeds the set maximum consideration range, assign a value. This indicates that the time deviation is too large, and the possibility of correlation is extremely low.

[0159] Existing methods for calculating time distance typically use the timestamp difference directly as the time distance. This method is highly sensitive to time shifts; for example, even a small time shift caused by network latency or processing jitter can drastically degrade the association quality. In contrast, this application's time distance calculation method maps continuous timestamp differences to discrete step values ​​corresponding to the number of missing detection cycles. This directly reveals the physical-level association information—how many detection cycles were lost—rather than simply using the absolute value of the time difference. By pre-setting a period fluctuation tolerance range (i.e., allowing a small range of time jitter), this scheme is insensitive to small shifts within the period. The step quantization value is only increased when the time difference exceeds the expected range, thus significantly enhancing the robustness of asynchronous sensor data fusion and solving the problem of misjudgment or missed associations caused by small time asynchrony in traditional methods.

[0160] In one embodiment, calculating the spatiotemporally normalized distance of spatial and temporal distance includes: The normalized value of the horizontal position distance is obtained by the ratio of the horizontal position track distance to the maximum horizontal position track distance threshold; the normalized value of the vertical position distance is obtained by the ratio of the vertical position track distance to the maximum vertical position track distance threshold; the normalized value of the horizontal speed track distance is obtained by the ratio of the horizontal speed track distance to the maximum horizontal speed track distance threshold; the normalized value of the vertical speed track distance is obtained by the ratio of the vertical speed track distance to the maximum vertical speed track distance threshold; and the normalized value of the time distance is obtained by the ratio of the time distance to the maximum time distance threshold. Based on each normalized value and its preset weight, a weighted sum is calculated to obtain the spatiotemporal normalized distance.

[0161] Specifically, global trajectory and sensor tracks The formula for calculating the spatiotemporal normalized distance between them is: in, This is the maximum distance threshold over time. This is the maximum distance threshold for the horizontal position track; This is the maximum distance threshold for the vertical position track; This is the maximum distance threshold for the horizontal speed track; This is the maximum distance threshold for the vertical velocity trajectory.

[0162] horizontal position track distance Vertical position track distance Horizontal speed and track distance Vertical speed and track distance and time distance Divide each by its corresponding maximum threshold ( , , , , This yields five normalized values. A preset weight is assigned to each normalized component. (and the sum of the weights is 1), calculate the weighted sum, and then obtain the spatiotemporally normalized distance.

[0163] Among them, the sum of the preset weights of each normalized value These weighting parameters can be flexibly selected, for example... , , , , , Indicates the maximum distance threshold of the horizontal position track. Indicates the maximum distance threshold of the vertical position track. This indicates the maximum distance threshold for the horizontal speed track. These parameters represent the maximum distance threshold for the vertical velocity track, and can be set in conjunction with the sensor's detection accuracy.

[0164] The spatiotemporal normalized distance calculation method normalizes five components—horizontal position, vertical position, horizontal velocity, vertical velocity, and time difference—independently. The weights of each component can be flexibly configured according to sensor characteristics and application scenarios, thereby adaptively emphasizing high-precision dimensions and suppressing low-confidence dimensions during the fusion process, significantly improving the adaptability to heterogeneous sensors and complex dynamic scenarios.

[0165] In order to leverage the detection characteristics of different types of sensors to achieve asynchronous fusion of multi-source heterogeneous sensors and improve the detection performance of electronic devices, this application proposes a multi-sensor track fusion method. This method can achieve stable fusion of electronic devices' target detection from completely asynchronous multi-source data.

[0166] like Figure 6 As shown, the method includes the following steps: Step 602: When the target data of the sensor is received, the target data of the sensor to be processed is processed and the sensor track is updated.

[0167] Specifically, the target data of the sensor is acquired and transferred to a preset reference coordinate system; the target data of the sensor after coordinate transformation is tracked and processed to update the sensor track.

[0168] Specifically, from electronic devices Choose one sensor from the sensors. The reference sensor is denoted as , and its LLA latitude and longitude coordinates are used as reference coordinates. The reference coordinates can also be set manually, among which, Indicates the number of sensors. Indicates the sensor index. , and These represent the longitude, latitude, and altitude of the reference coordinates, respectively. At a certain time, the data processing unit receives a target data segment. Through coordinate transformation, it converts the LLA coordinates in the target data to ENU (northeast) coordinates relative to the reference coordinates, and records the target data as... ,in This indicates the number of data received by the data processing unit in the global scope. The detection timestamp of each data point It represents the target's motion state information, such as position and velocity. This indicates the target's attribute information, such as target ID, sensor ID, sensor type, target type, etc.

[0169] The data processing unit will receive the target data sequentially. Then through The sensor type, sensor ID, and target ID in the data can be used to determine the unique identifier of the target from multi-sensor target data: (sensor type, sensor ID, target ID). This information is then accumulated over multiple time points to form a track. During track formation, Kalman filtering is used to estimate the target's motion state, as detailed below: pass The sensor type and sensor ID are used to determine the data. From sensors , This indicates that the data processing unit receives data from the sensor. The first received in There are 1 data point, and the corresponding timestamp is 1. .according to In the (sensor type, sensor ID, target ID) field, you can retrieve the sensor information. Extract sensor tracks from the track list ,in Indicates sensor Index of the Chinese flight path Indicates sensor The number of flight paths. If no such path exists... For the corresponding track, a new track will be created. If it exists with The corresponding track update process is as follows: track The last state was estimated to be its own number 1. The next estimate, the state, and the covariance are expressed as: , where state Given the position, velocity, and angular velocity along the X, Y, and Z axes in three-dimensional space, then in the prediction step: in, This represents the predicted state and the predicted state covariance. Indicates the flight path exist The state transition matrix at time t, Indicates the flight path exist The process noise matrix of the motion model at any given time. The data processing unit receives the data. Afterwards, the flight path was... exist Estimate the state at time: in, Represents the covariance matrix of the predicted measurement. Denotes its inverse matrix, Represents the measurement equation, when When linear, If it is a linear observation matrix, then it is the Jacobian matrix of the measurement equation. express The noise covariance matrix, Indicates Kalman gain, Indicates the flight path exist The estimated state and the estimated state covariance at time t.

[0170] Step 604: Determine the associated global track and update the global track using sensor tracks.

[0171] Specifically, the global track is maintained in the data processing unit. And sequentially acquire the tracks of currently active sensors. and link it to the global track The data is correlated and then merged, where 'o' represents the index of the global track in the data processing unit. This indicates the number of global tracks at the current moment.

[0172] like Figure 6 As shown, determining the associated global track and updating the global track using sensor tracks includes the following steps: Step 6041: The timestamps of the global trajectory prediction and the sensor trajectory are used to achieve time alignment.

[0173] Specifically, make the global trajectory The last state fusion was its own first The corresponding timestamp for this fusion is Using the state transition matrix, all timestamps Less than Global track extrapolation of state to Global track The The state and covariance of the sub-fusion are represented as follows: Then, in the prediction step, we have: in, Represents the global track The predicted state and the predicted state covariance, Represents the global track exist The state transition matrix at time t, Represents the global track exist The process noise matrix of the motion model at any given time.

[0174] Step 6042: By calculating the spatial distance between the sensor track and the global track, the association candidate is judged, the spatiotemporal normalized distance is calculated, and the first global track that can be updated is determined based on the association candidate judgment, the spatiotemporal normalized distance, the association status and the matching status.

[0175] Specifically, it includes the following steps 1 to 6: Step 1: Calculate the spatial distance.

[0176] Spatial distance can include horizontal position track distance, vertical position track distance, horizontal velocity track distance, and vertical velocity track distance.

[0177] Among them, setting a time window During the time window Internal computing sensor track Spatial distance to each global track. Set within a time window. Sensor tracks The state sequence is The corresponding time series is Global track The state sequence is The corresponding time series is The distance value for each state pair is calculated using bidirectional interpolation.

[0178] First, the sensor trajectory was analyzed. Perform interpolation to calculate the global trajectory. Each state to sensor track The distance. If for , The time condition for interpolation is satisfied. , Then it can be found in the sensor track two states and Interpolation between The interpolation formula is: Then calculate and Represented by the distance between To sensor track The distance. The formula for calculating the location distance is: in, and They represent and The horizontal and vertical distances between them , and They represent The X, Y, and Z coordinates in the data. , and They represent interpolation. The X, Y, and Z coordinates can be used to calculate the horizontal velocity and distance. and vertical velocity distance .

[0179] Similarly, for the global trajectory Perform interpolation to calculate sensor tracks From each state to the global track The distance can be used to obtain the sensor track. middle To global track Horizontal position distance Vertical position distance Horizontal speed and distance Vertical velocity and distance .

[0180] Therefore, the global trajectory can be obtained. With sensor tracks Distance between tracks: in, , , and Representing the global trajectory and sensor tracks The horizontal position track distance, vertical position track distance, horizontal velocity track distance, and vertical velocity track distance between them. and Representing the global trajectory and sensor tracks The number of states that satisfy the time conditions for interpolation.

[0181] Step 2: Determine whether the spatial distance meets the candidate association conditions.

[0182] If global trajectory and sensor tracks The global trajectory is determined if the following conditions are met. Entering the sensor track The associated candidates are determined based on the following conditions: in, Indicates the horizontal position track association threshold, Indicates the vertical position track association threshold, This indicates the horizontal speed trajectory association threshold. These parameters represent the vertical velocity trajectory correlation thresholds and need to be set in conjunction with the sensor's detection accuracy. (Regarding the global trajectory...) Entering the sensor track If the associated candidate is found, proceed to step 3; otherwise, change to the next global track and continue repeating steps 1 and 2.

[0183] Step 3: Calculate the time distance.

[0184] Sensor tracks The latest timestamp is Global track The latest timestamp is Therefore, the time difference is: Set sensor The detection cycle is In a real-world system, without any missed alarms, the time interval between two consecutive detections of the same target by the same sensor will be... It fluctuates up and down, so a float value is set. Therefore, the sensor track With global track The time interval is: in, , You can choose flexibly according to the actual situation, usually .

[0185] Step 4: Calculate the spatiotemporal normalized distance.

[0186] Global Track and sensor tracks The formula for calculating the spatiotemporal normalized distance between them is: in, These weighting parameters can be flexibly selected, for example... , , , , , Indicates the maximum distance threshold of the horizontal position track. Indicates the maximum distance threshold of the vertical position track. This indicates the maximum distance threshold for the horizontal speed track. These parameters represent the maximum distance threshold for the vertical velocity track, and need to be set in conjunction with the sensor's detection accuracy.

[0187] Step 5: Based on the association candidate judgment, spatiotemporal normalized distance, association status and matching status, determine the first global track that can be updated.

[0188] Specifically, let Represents global track and sensors The correlation matching period of the flight track, global flight track and sensor tracks The relationships between them are described using two types of state machines: associative state machines and matching state machines.

[0189] Among them, the state transitions of the associated state machine and the matching state machine are as follows: Figure 4 As shown: The matching state machine has four states: bound, strong match, weak match, and unmatched. Within the same time period, the global track... Only with sensors One of the sensor tracks is in a bound, strongly matched, or weakly matched state, and the sensor track A sensor track can only be bound, strongly matched, or weakly matched with one global track. A sensor track in a bound or matched state cannot be used to start a global track. State transition. Figure 3 and Figure 4 As shown.

[0190] like Figure 3 As shown, the first global track that can be used for updating must meet any of the following conditions: 1. Matched and closest: Within the current association matching period, the first global track and the current sensor track have the closest spatiotemporal normalized distance, and the two are already in a matching relationship.

[0191] 2. Upgrade successful: Within the current association matching period, the first global track and the current sensor track have the shortest spatiotemporal normalized distance and meet the conversion conditions for upgrading from association relationship to matching relationship (the first global track and the current sensor track are at the highest association strength, and the first global track and other sensor tracks are at the lowest matching strength, and the current sensor track is the shortest distance), and the lowest matching strength is successfully established.

[0192] Step 6043: Update the first global track using sensor tracks.

[0193] Set sensor trajectory With global track Once the binding state is entered, then: in, Represents the covariance matrix of the predicted measurement. Denotes its inverse matrix, Indicates Kalman gain, Indicates the flight path exist The fusion state and fusion state covariance at each moment. If the sensor track... If no global track becomes a candidate for association, then the sensor track is used. Create a new global track.

[0194] The multi-sensor track fusion method of this application enables asynchronous multi-sensor track fusion. It fully considers the data characteristics and asynchronous temporal features of various heterogeneous sensors. A spatiotemporally normalized distance is designed to measure the similarity between the global track and the local tracks of the sensors under asynchronous time conditions. An association matching state machine is used to manage the association matching relationship between the global track and the local tracks of the sensors. Finally, the nearest neighbor association algorithm is combined to solve the data association problem under asynchronous time conditions. Furthermore, by processing data from various sensors, the tracking-filtered sensor tracks are sequentially fused with the global track, improving the stability and accuracy of electronic equipment detection and achieving stable fusion of electronic equipment targets. This method has low computational complexity, is easy to deploy on devices with low computing resources, and is practical. It can be effectively applied in the field of electronic equipment detection, and even in other multi-sensor fusion applications.

[0195] like Figures 7 to 9 The image shows an application example using the multi-sensor track fusion method proposed in the application. Figure 7As shown, global track F1 is in a strong match with sensor track P1-30157 with sensor ID 30157, and the latest timestamp is 54941.5578. Global track F2 is in a strong match with sensor track P2-30157 with sensor ID 30157, and the latest timestamp is 54941.3530. It is also in a strong match with sensor track R107 with sensor ID 30142, and the latest timestamp is 54938.9850.

[0196] like Figure 8 As shown, the next sensor track data P2-30157 is transmitted to the system at timestamp 54942.9360. By calculating the spatiotemporal distances of P2-30157 with F1 and F2 respectively, it is found that P2-30157 is closer to F2, and P2-30157 and F2 are in a strong match state. Therefore, it is considered that P2-30157 matches F2 again, and the state of F2 is updated using P2-30157, as shown below. Figure 9 The trajectory is shown. Without an association matching state machine to manage and restrict the association matching state of the trajectory, when P1-30157 and P2-30157 first appear, F1 will appear first because P1-30157 appears before P2-30157, and F2 will not appear. When P2-30157 appears, when calculating the distance between P2-30157 and all global trajectories, because F2 is absent, P2-30157 will be found to be closest to F1. According to the nearest neighbor algorithm, P2-30157 is associated with F1, so F2 cannot start, and F1 will jump back and forth between P1-30157 and P2-30157. Therefore, using spatiotemporal normalized distance and an association matching state machine can effectively compensate for the shortcomings of the nearest neighbor association algorithm in handling asynchronous data.

[0197] This method has the following technical effects: 1. Easy to deploy.

[0198] This method employs an asynchronous association strategy within the sequential Kalman estimation framework. It does not require prior spatiotemporal synchronization between sensors, nor does it rely on a large amount of labeled data or deep learning training. It is easier to deploy in environments where real-world data is scarce and system resources are limited.

[0199] 2. High real-time performance.

[0200] By aligning multiple trajectory points using the target motion model and linear interpolation, the similarity between trajectories is obtained by calculating the spatiotemporally normalized distance. Then, the association relationship between different sensor tracks and the global track is determined using an association matching state machine and the nearest neighbor principle. This association process only involves comparing the nearest neighbors of the current sensor track and the existing global track, avoiding combinatorial explosion and ensuring strong real-time performance.

[0201] 3. Highly scalable.

[0202] This proposed asynchronous nearest neighbor data association incorporates asynchronous time factors into the spatiotemporal distance construction and combines a dual mechanism of association matching state machine to solve the asynchronous association problem. Furthermore, this method is fully compatible with synchronous association scenarios, requires no algorithm switching, and exhibits good scalability.

[0203] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0205] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-sensor trajectory fusion method, characterized in that, The method includes: Process the target data from the sensor to be processed to obtain the updated current sensor track; Based on the distance between the current sensor track and the global track, candidate global tracks that meet the association candidate conditions with the current sensor track within the current association matching period are determined. Select the first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track; If the current sensor track is already matched with the first global track, the matching strength between the current sensor track and the first global track is increased, and the first global track is updated using the current sensor track.

2. The multi-sensor trajectory fusion method according to claim 1, characterized in that, Following the step of obtaining the updated current sensor track, the method further includes: Tracks whose update time exceeds their corresponding association matching period in sensor tracks and global tracks are identified as update timeout tracks; Reduce the association strength between other tracks that are associated with the updated timeout track, and reduce the matching strength between other tracks that are matched with the updated timeout track; the association represents the degree of candidate matching between sensor tracks and global tracks.

3. The multi-sensor trajectory fusion method according to claim 2, characterized in that, The method further includes: If the current sensor track and the first global track are not matched, and the following conditions are met simultaneously: The first global track is not in a matching relationship with other sensor tracks of the highest matching strength; The current sensor track is not in a matching relationship with other global tracks with the highest matching strength; This strengthens the correlation between the current sensor track and the first global track.

4. The multi-sensor trajectory fusion method according to claim 3, characterized in that, The method further includes: If there is no matching relationship between the current sensor track and the first global track, and the following conditions are met: The first global track has the highest matching strength with other sensor tracks; The current sensor track is not associated with or matched with any global track. Then, a new global track is started from the current sensor track, and a matching relationship with the highest matching strength is established between the current sensor track and the new global track.

5. The multi-sensor trajectory fusion method according to claim 2, characterized in that, If there is no candidate global track that meets the association candidate conditions with the current sensor track in the current association matching period, then check whether the current sensor track has no association or matching relationship with any global track. If so, a new global track is started with the current sensor track, and a matching relationship with the highest matching strength is established between the current sensor track and the new global track.

6. The multi-sensor trajectory fusion method according to claim 3, characterized in that, The method further includes: After enhancing the correlation strength between the current sensor track and the first global track, if the following conditions are met simultaneously: The first global track has the highest correlation strength with the current sensor track, and is matched with other sensor tracks but not with the highest matching strength. Among the sensor tracks that have the highest correlation strength with the first global track or are not at the highest matching strength, the current sensor track is closest to the spatiotemporal normalized distance of the first global track. Then, a matching relationship is established between the first global track and the current sensor track with the lowest matching strength, the matching relationship between the first global track and other sensor tracks is removed, and the first global track is updated using the current sensor track.

7. The multi-sensor trajectory fusion method according to claim 3, characterized in that, If the current sensor track is not matched with the first global track, but is matched with other global tracks with the highest matching strength, then the association between the current sensor track and the first global track will be reduced to unassociated.

8. The multi-sensor track fusion method according to any one of claims 1 to 7, characterized in that, Following the step of obtaining the updated current sensor track, the method further includes: If the difference between the current time and the last update time of the sensor track exceeds the preset sensor track termination period, the sensor track is terminated, and the association and matching relationships associated with the terminated sensor track are cleared. And / or, If the difference between the current time and the last update time of the global track exceeds the preset global track termination period, the global track is terminated, and the association and matching relationships associated with the terminated global track are cleared.

9. The multi-sensor track fusion method according to any one of claims 1 to 7, characterized in that, The step of determining candidate global tracks that satisfy the association candidate conditions with the current sensor track within the current association matching period based on the distance between the current sensor track and the global track includes: The timestamp of the global trajectory prediction is used to match the current sensor trajectory, thus achieving time alignment. Based on the state of the current sensor track and the global track in the reference coordinate system, calculate the spatial distance between the current sensor track and the global track; the spatial distance includes the horizontal position track distance, the vertical position track distance, the horizontal velocity track distance, and the vertical velocity track distance; The spatial distance is used to determine whether the corresponding candidate association conditions are met.

10. The multi-sensor trajectory fusion method according to claim 9, characterized in that, The step of calculating the spatial distance between the current sensor track and the global track based on their states in the reference coordinate system includes: Obtain the status sequence of the current sensor track and the status sequence of the global track within a set time window; For each state point in the state sequence of the global track, find the adjacent sensor state point that is adjacent to the current sensor track state sequence and satisfies the interpolation condition. Obtain the corresponding interpolation point by linear interpolation of the adjacent sensor state point. Calculate the horizontal position distance, vertical position distance, horizontal velocity distance, and vertical velocity distance between the global track state point and the interpolation point. For each state point in the current sensor track state sequence, find adjacent global track state points in the global track state sequence that are adjacent time points and satisfy the interpolation conditions. Obtain the corresponding interpolation point by linear interpolation of the adjacent global track state points. Calculate the horizontal position distance, vertical position distance, horizontal velocity distance, and vertical velocity distance between the sensor track state point and the interpolation point. The horizontal position track distance is obtained based on the average horizontal position distance from the interpolation point, the vertical position track distance is obtained based on the average vertical position distance from the interpolation point, the horizontal velocity track distance is obtained based on the average horizontal velocity distance from the interpolation point, and the vertical velocity track distance is obtained based on the average vertical velocity distance from the interpolation point.

11. The multi-sensor trajectory fusion method according to claim 10, characterized in that, The associated candidate conditions include: The horizontal position track distance between the global track and the sensor track is less than the horizontal position track association threshold; The vertical position track distance between the global track and the sensor track is less than the vertical position track association threshold; The horizontal velocity track distance between the global track and the sensor track is less than the horizontal velocity track association threshold; The vertical velocity track distance between the global track and the sensor track is less than the vertical velocity track association threshold; The horizontal position distance between the latest sensor track time and the corresponding interpolation point of the global track is less than the horizontal position track association threshold; The vertical distance between the latest sensor track and the corresponding interpolation point of the global track is less than the vertical position track association threshold.

12. The multi-sensor track fusion method according to claim 1 or 11, characterized in that, Selecting the first global track from the candidate global tracks that has the closest spatiotemporal normalized distance to the current sensor track includes: Calculate the time distance between the sensor track and the global track; Calculate the spatiotemporally normalized distance of the spatial distance and the temporal distance; the spatial distance includes the horizontal position track distance, the vertical position track distance, the horizontal velocity track distance, and the vertical velocity track distance; The global track with the smallest spatiotemporal normalized distance to the current sensor track is determined as the nearest candidate global track.

13. The multi-sensor trajectory fusion method according to claim 12, characterized in that, The calculation of the spatiotemporally normalized distance of the spatial distance and the temporal distance includes: The normalized value of the horizontal position distance is obtained based on the ratio of the horizontal position track distance to the maximum horizontal position track distance threshold; the normalized value of the vertical position distance is obtained based on the ratio of the vertical position track distance to the maximum vertical position track distance threshold; the normalized value of the horizontal speed distance is obtained based on the ratio of the horizontal velocity track distance to the maximum horizontal velocity track distance threshold; the normalized value of the vertical speed distance is obtained based on the ratio of the vertical velocity track distance to the maximum vertical velocity track distance threshold; and the normalized value of the time distance is obtained based on the ratio of the time distance to the maximum time distance threshold. The spatiotemporal normalized distance is obtained by calculating a weighted sum based on each normalized value and its preset weight.

14. An electronic device, characterized in that, The system includes multiple sensors of different types, a processor connected to the sensors, and a memory connected to the processor, the memory storing a computer program executable by the processor, the computer program being executed by the processor to implement the steps of the multi-sensor track fusion method as described in any one of claims 1 to 13.