A Multi-Source Data Spatiotemporal Synchronization Fusion Method for Multi-Unmanned Payload Inspection Scenarios

Abstract

This invention proposes a spatiotemporal synchronous fusion method for multi-source data in multi-unmanned payload inspection scenarios. During collaborative inspection by multiple unmanned payloads, target detection is first performed using YOLOv5, followed by feature extraction using a convolutional neural network. Firstly, at a certain time point, based on the motion and visual information obtained by each of the multiple unmanned payloads, it is determined whether the detected data from different positions and perspectives belong to the same target. If they are the same target, single-point fusion is performed on the acquired data. Then, at any consecutive time points, data association and matching are performed to determine whether the multiple unmanned payloads still detect the same target under different spatiotemporal states. If they are still the same target, multi-point fusion is performed on the acquired data. Finally, a feedback correction mechanism is introduced to correct data with poor fusion results.

Description

Technical Field

[0001] This invention relates to the field of data monitoring, specifically to a method for spatiotemporal synchronization fusion of multi-source data for multi-unmanned payload inspection scenarios. Background Technology

[0002] With the continuous development of IoT technology and the rapid growth of big data, swarm intelligence has become a new research hotspot, among which multi-source data spatiotemporal synchronization fusion technology has become one of the key technologies. In swarm intelligence scenarios, multiple intelligent agents form a highly self-organizing and adaptive system through continuous interaction and collaboration. In this system, a large amount of data comes from various sensors, including visual sensors, sound sensors, accelerometers, GPS, etc., and the data volume is enormous, with significant differences in time and space. Therefore, how to effectively process and fuse this data is crucial to achieving the goal of swarm intelligence. By collecting, preprocessing, synchronizing, and fusing data from multiple sensors, comprehensive monitoring and control of the swarm intelligence system can be achieved. At the same time, this technology can improve the reliability and accuracy of data, improve the system's response speed and decision-making accuracy, and thus better adapt to complex and ever-changing swarm intelligence scenarios.

[0003] In existing multi-source data spatiotemporal synchronization fusion algorithms, most of the difficulties encountered are related to the characteristics of spatiotemporal sequence data, such as data class imbalance, correlation between features, time dependence between index values, spatiotemporal dependence of spatiotemporal sequences, and the need to consider the interrelationship between different features. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a spatiotemporal synchronous fusion method for multi-source data in multi-unmanned payload inspection scenarios. During the collaborative inspection of multiple unmanned payloads, this method combines current scenario parameters to accurately process multi-source data obtained during the multi-unmanned payload inspection process under different spatiotemporal states and perspectives.

[0005] A method for spatiotemporal synchronization fusion of multi-source data for multi-unmanned payload inspection scenarios includes the following steps:

[0006] S1. After using the YOLOv5 algorithm for target detection, the Deepsort multi-target tracking algorithm is used for target association and target tracking, and a convolutional neural network is used for feature extraction. The extracted features are the spatial state information of the target, which includes position information, velocity information and visual information.

[0007] S2. At any given moment, determine whether the target detected by the multiple unmanned payloads is the same target in the same area. If so, perform weighted fusion of the position and velocity information obtained by the multiple unmanned payloads respectively, and stitch together the visual information obtained from different perspectives.

[0008] S3. Within a continuous time period, determine whether multiple unmanned payloads detect the same target. If so, perform multi-point fusion of the spatial state information obtained by multiple unmanned payloads.

[0009] Furthermore, the method also includes step S4, which introduces a result feedback correction mechanism to perform feedback correction on the fusion result.

[0010] Furthermore, step S2 includes the following sub-steps:

[0011] S2.1, at time t, when In this case, it is assumed that n unmanned payloads at different locations have detected targets in the same area; where:

[0012]

[0013]

[0014] Where, d (l) D represents the distance between the l-th unmanned payload and the target. cos d This represents the overall distance similarity between n unmanned payloads located at different positions and the same target.

[0015] S2.2 At time t, for the targets in the same area detected by the n unmanned vehicles at different locations in step S2.1, further when H cos n ≤K n In this case, it is assumed that n unmanned payloads at different locations have detected the same target; where:

[0016]

[0017]

[0018]

[0019] Where, r l ,r k These are the feature vectors with a magnitude of 1 output by the l-th and k-th unmanned payloads to target j, respectively. l T r k It is the cosine similarity between the l-th unmanned payload and the k-th unmanned payload, n (l) H represents the feature similarity obtained by integrating the visual information of the l-th unmanned payload with that of the other unmanned payloads. cos n This represents the comprehensive feature similarity obtained by associating and fusing the appearance features of n unmanned payloads at time t.

[0020] S2.3. At time t, to determine whether the n unmanned payloads at different locations are approaching the same target in the same area, the position and velocity information of the n unmanned payloads at different locations are weighted and fused separately, and the visual information obtained from different perspectives is stitched and fused together; where:

[0021]

[0022]

[0023]

[0024]

[0025] Where, d t l v represents the target distance detected by the l-th unmanned payload at time t. t l Let represent the target velocity detected by the l-th unmanned payload at time t, and cos θ. l θ is the cosine of the angle between the l-th unmanned payload and the target, ω l V represents the speed fusion weight of the l-th unmanned payload. t This represents the velocity vector of the multi-unmanned payload relative to the target at time t after weighted fusion. The modulus of D t R represents the position of the multi-unmanned payload relative to the target at time t after weighted fusion. t r represents the visual information obtained by stitching and fusing the visual information of multiple unmanned payloads at time t. l It is the feature vector with a magnitude of 1 output by the l-th unmanned payload to the target.

[0026] Furthermore, step S3 includes the following sub-steps:

[0027] S3.1. Within a continuous time period, for multiple unmanned payloads, cosine distance is used to associate position information, velocity angle vector is used to associate velocity information, and cosine similarity is used to associate visual information.

[0028]

[0029]

[0030]

[0031] Among them, D euclidean (t, t-1) represents the correlation distance between times t and t-1, θ(t, t-1) represents the velocity correlation angle between the two times, and R similarity(t, t-1) represents the associated visual feature vectors at the preceding and following times. D represents the velocity vector of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 R represents the position of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 This represents the visual information obtained after stitching and fusing the visual information of multiple unmanned payloads at time t-1;

[0032] When D euclidean (t, t-1) ≤ K t α And θ(t,t-1)≤K t β And R similarity (t, t-1) ≤ K t γ If the multi-unmanned payload detects the same target at times t and t-1, then:

[0033]

[0034]

[0035]

[0036] S3.2 When multiple unmanned payloads detect the same target at times t and t-1, the spatial state information of the unmanned payloads at time t is fused from multiple points. The specific formula is as follows:

[0037]

[0038]

[0039] in, X represents the predicted state information obtained at time t-1 for the current time. t-1 F' represents the spatial state information after multi-point fusion at time t-1. t-1 Let N be the state transition matrix at time t-1. t-1 H represents the process noise at time t-1. t-1 Let W be the observation matrix at time t-1. t-1 To observe noise.

[0040] Further, step S4 includes the following sub-steps:

[0041] S4.1. The root mean square error (RMSE) is used to measure the data fusion effect. The formula for RMSE is as follows:

[0042]

[0043] Among them, Xs+1 ′ represents the spatial state information after multi-point fusion at time s+1, X s+1 This represents the spatial state information obtained by the unmanned payload at time s+1;

[0044] S4.2, when X t+1 When it exceeds RMSE, for X t+1 'Perform feedback correction, where feedback correction includes: first, X t+1 The process involves normalization, data augmentation of the normalization result, and finally, denormalization of the data augmentation result.

[0045] Furthermore, the formula for data augmentation is as follows:

[0046]

[0047] in, X represents the result of data augmentation. norm X represents t+1 The normalized result of ′, where a is the coefficient.

[0048] This invention also provides a multi-source data spatiotemporal synchronization fusion system for multi-unmanned payload inspection scenarios, comprising:

[0049] The object detection module is used for object detection using the YOLOv5 algorithm.

[0050] The target tracking module is used for target association and target tracking using the Deepsort multi-target tracking algorithm;

[0051] The feature extraction module is used to extract features using a convolutional neural network;

[0052] The weighted and stitched fusion module is used to determine at any given time whether the targets detected by multiple unmanned payloads are the same targets in the same area. If so, the position and velocity information obtained by the multiple unmanned payloads are weighted and fused respectively, and the visual information obtained from different perspectives is stitched and fused.

[0053] The multi-point fusion module is used to determine whether multiple unmanned payloads are detecting the same target within a continuous time period. If so, it performs multi-point fusion on the spatial state information obtained by the multiple unmanned payloads.

[0054] Furthermore, the system also includes a feedback correction mechanism for correcting the fusion results.

[0055] The present invention also provides a computer-readable storage medium for storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method described above.

[0056] The present invention also provides a multi-source data spatiotemporal synchronization fusion device for multi-unmanned payload inspection scenarios, including one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the methods described above.

[0057] Beneficial effects

[0058] 1) Improved data accuracy: This invention can acquire data from multiple data sources and integrate them through spatiotemporal synchronization. By fusing information from multiple data sources, the accuracy and reliability of the data can be improved.

[0059] 2) Enhanced Knowledge Integration: Multi-source data spatiotemporal synchronous fusion technology can integrate data from different fields or sources to generate more comprehensive and richer knowledge. By integrating information from multiple data sources, it is possible to discover correlations and patterns between data, generate new knowledge, and provide a more comprehensive perspective and stronger support for decision-making and problem-solving.

[0060] 3) Intelligent Decision Optimization: Through multi-source data spatiotemporal synchronous fusion technology, more accurate data and more comprehensive knowledge can be obtained, thus providing a better foundation for decision-making in intelligent scenarios. By integrating information from multiple sources such as sensors, social media, and meteorological data, decisions on urban traffic scheduling, environmental monitoring, and resource allocation can be made more accurately, thereby optimizing and intelligentizing urban operations.

[0061] 4) Enhanced service experience: Multi-source data spatiotemporal synchronization fusion technology can provide more accurate and real-time data analysis and prediction, thereby improving the user's service experience. Attached Figure Description

[0062] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0063] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0064] A spatiotemporal synchronization fusion technology for multi-source data in swarm intelligence scenarios, such as Figure 1 As shown, the specific operation steps are as follows:

[0065] Step 1: After using the YOLOv5 algorithm for target detection, the Deepsort multi-target tracking algorithm is used for target association and tracking, and a convolutional neural network is used for feature extraction. The extracted features are the spatial state information of the target, which includes position information, velocity information and visual information.

[0066] Step 2: At a certain moment, multiple unmanned payloads make an initial judgment based on their respective position information to determine whether they are seeing the same area. Then, based on their respective viewpoint information, they make a further judgment to determine whether they are still the same target. If they are still the same target, the position and velocity information of the multiple unmanned payloads are weighted and fused respectively, and the visual features obtained from different viewpoints are stitched and fused together.

[0067] Step 3: Within a continuous time period, use cosine distance to associate position information, velocity angle vector to associate velocity information, and cosine similarity to associate visual information to determine whether multiple unmanned payloads detect the same target within a continuous time period. If they are the same target, then perform multi-point fusion of the spatial state information obtained by multiple unmanned payloads under different spatiotemporal states and different perspectives.

[0068] Step 4: Introduce a result feedback correction mechanism, using mean squared error to measure the fusion effect, and perform feedback correction on fusion results with poor fusion effect: first, perform normalization processing, then perform data augmentation, and finally perform inverse normalization processing for feedback correction.

[0069] Furthermore, in step 1, the YOLOv5 algorithm is used for target detection, followed by the improved Deepsort algorithm for multi-target tracking. Kalman filtering is used to predict the target's motion state information for the next time step and update the target trajectory information. The formula is as follows:

[0070] Assume that at time t, the l-th unmanned payload detects the target state as follows: The target state at time t-1 is have:

[0071]

[0072] Where a t l d represents acceleration. t l v represents the location of the target at time t. t l F represents the target velocity at time t. t Let B be the state transition matrix. tThis represents the control matrix. The Hungarian algorithm is used to match the current tracking result with the detection result at the next time step, and uses the data information as the cost matrix to achieve target association matching.

[0073] Furthermore, in step 2: assuming that multiple unmanned payloads initially determine whether they are seeing the same area based on their respective acquired location information, the formula is as follows:

[0074]

[0075]

[0076] Where, d (l) D represents the distance between the l-th unmanned payload and the target. cos d This represents the overall distance similarity between n unmanned payloads located at different positions and the same target.

[0077] When D cos d No more than K d At that time, it was initially assumed that the n unmanned payloads in different locations were seeing the same target area.

[0078] Furthermore, multiple unmanned payloads determine whether they are seeing the same target based on their respective perspective information, using the following formula:

[0079]

[0080]

[0081]

[0082] Where, r l ,r k These are the feature vectors with a magnitude of 1 of the output of the l-th and k-th unmanned payloads to the target, respectively. l T r k It is the cosine similarity between the l-th unmanned payload and the k-th unmanned payload, n (l) H represents the feature similarity obtained by integrating the visual information of the l-th unmanned payload with that of the other unmanned payloads. cos n This represents the comprehensive feature similarity obtained by associating and fusing the appearance features of n unmanned vehicles at time t.

[0083] When H cos n No more than K n At that time, it was assumed that n unmanned payloads in different positions would see the same target from different perspectives.

[0084] At time t, if multiple unmanned payloads detect the same target in the same area, then the visual information obtained from different perspectives is stitched and fused. The position and velocity information of the multiple unmanned payloads are weighted and fused separately, and the data obtained from different perspectives are also stitched and fused. The formula is as follows:

[0085]

[0086]

[0087]

[0088]

[0089] Where, d t l v represents the target distance detected by the l-th unmanned payload at time t. t l Let represent the target velocity detected by the l-th unmanned payload at time t, and cos θ. l θ is the cosine of the angle between the l-th unmanned payload and the target, ω l V represents the speed fusion weight of the l-th unmanned payload. t This represents the velocity vector of the multi-unmanned payload relative to the target at time t after weighted fusion. The modulus of D t R represents the position of the multi-unmanned payload relative to the target at time t after weighted fusion. t r represents the visual information obtained by stitching and fusing the visual information of multiple unmanned payloads at time t. l It is the feature vector with a magnitude of 1 output by the l-th unmanned payload to the target.

[0090] Furthermore, in step 3, the unmanned payload has different spatiotemporal states at different times. At any time before or after a continuous time period, the position information is associated with cosine distance, the velocity angle vector is associated with velocity information, and the visual information is associated with cosine similarity. The formula is as follows:

[0091]

[0092]

[0093]

[0094] Among them, D euclidean (t, t-1) represents the correlation distance between times t and t-1, θ(t, t-1) represents the velocity correlation angle between the two times, and R similarity (t, t-1) represents the associated visual feature vectors at the preceding and following times. D represents the velocity vector of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 R represents the position of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 This represents the visual information obtained after stitching and fusing the visual information of multiple unmanned payloads at time t-1.

[0095] When D euclidean (t, t-1) ≤ K t α And θ(t,t-1)≤K t β And R similarity (t, t-1) ≤ K t γ If the multi-unmanned payload detects the same target at times t and t-1, then:

[0096]

[0097]

[0098]

[0099] When multiple unmanned payloads detect the same target at times t and t-1, the spatial state information of the unmanned payloads at time t is fused at multiple points (i.e., the fused data X from the previous time step is fused). t ′ and prediction data for the current moment The data obtained by fusing data from multiple unmanned payloads over a continuous time period is fused at multiple points, and the specific formula is as follows:

[0100]

[0101]

[0102] in, X represents the predicted state information obtained at time t-1 for the current time. t-1 F' represents the spatial state information after multi-point fusion at time t-1. t-1 Let N be the state transition matrix at time t-1. t-1 H represents the process noise at time t-1. t-1 Let W be the observation matrix at time t-1. t-1 To observe noise.

[0103] Furthermore, in step 4, the root mean square error is used to measure the effectiveness of data fusion:

[0104]

[0105] Among them, Xs+1 ′ represents the spatial state information after multi-point fusion at time s+1, X s+1 This represents the spatial state information obtained by the unmanned payload at time s+1.

[0106] RMSE represents the prediction accuracy of the Kalman filter. A smaller RMSE indicates less difference between data points, suggesting better data fusion. A smaller RMSE indicates poor fusion performance (X...). t+1 Data exceeding RMSE undergoes augmentation and feedback correction: first, normalization is performed, then data augmentation is performed, and finally, denormalization is performed.

[0107]

[0108]

[0109]

[0110] Among them, X norm (k) This indicates that the fused data has been normalized. This indicates that the normalized data is augmented to strengthen the correlation between data that are otherwise weakly correlated after merging. X t+1 (k)′ X represents the spatiotemporal synchronized fusion data of multiple unmanned payloads obtained by performing inverse normalization on the augmented data. max and X min These represent the maximum and minimum values ​​of the target state information detected by the unmanned payload, respectively.

[0111] This invention also provides a multi-source data spatiotemporal synchronization fusion system for multi-unmanned payload inspection scenarios, comprising:

[0112] The object detection module is used for object detection using the YOLOv5 algorithm.

[0113] The target tracking module is used for target association and target tracking using the Deepsort multi-target tracking algorithm;

[0114] The feature extraction module is used to extract features using a convolutional neural network;

[0115] The weighted and stitched fusion module is used to determine at any given time whether the targets detected by multiple unmanned payloads are the same targets in the same area. If so, the position and velocity information obtained by the multiple unmanned payloads are weighted and fused respectively, and the visual information obtained from different perspectives is stitched and fused.

[0116] The multi-point fusion module is used to determine whether multiple unmanned payloads are detecting the same target within a continuous time period. If so, it performs multi-point fusion on the spatial state information obtained by the multiple unmanned payloads.

[0117] Furthermore, the system also includes a feedback correction mechanism for correcting the fusion results.

[0118] The technical solution of the multi-source data spatiotemporal synchronization fusion system for multi-unmanned payload inspection scenarios is similar to the aforementioned method, and will not be repeated here.

[0119] Based on the same technical solution, the present invention also discloses a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to execute the above-described method for spatiotemporal synchronization fusion of multi-source data for multi-unmanned payload inspection scenarios.

[0120] Based on the same technical solution, the present invention also discloses a computing device, including one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the above-described method for spatiotemporal synchronization fusion of multi-source data for multi-unmanned payload inspection scenarios.

[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for spatiotemporal synchronization fusion of multi-source data in multi-unmanned payload inspection scenarios, characterized in that, Includes the following steps: S1. After using the YOLOv5 algorithm to perform target detection on the images acquired by the multi-unmanned payload inspection, the Deepsort multi-target tracking algorithm is used for target association and target tracking, and a convolutional neural network is used to extract features from the targets. The extracted features are the spatial state information of the targets, which includes position information, velocity information and visual information. S2. At any given moment, determine whether the target detected by the multiple unmanned payloads is the same target in the same area. If so, perform weighted fusion of the position and velocity information obtained by the multiple unmanned payloads respectively, and stitch together the visual information obtained from different perspectives. S3. Within a continuous time period, determine whether multiple unmanned payloads detect the same target. If so, perform multi-point fusion of the spatial state information obtained by the multiple unmanned payloads. Step S2 includes the following sub-steps: S2.1 At time t, when D cos d ≤K d In this case, it is assumed that n unmanned payloads at different locations have detected targets in the same area; where: Where, d (l) D represents the distance between the l-th unmanned payload and the target. cos d This represents the overall distance similarity between n unmanned payloads located at different positions and the same target. S2.2 At time t, for the targets in the same area detected by the n unmanned vehicles at different locations in step S2.1, further when H cos n ≤K n In this case, it is assumed that n unmanned payloads at different locations have detected the same target; where: Where, r l ,r k These are the feature vectors with a magnitude of 1 output by the l-th and k-th unmanned payloads to target j, respectively. l T r k It is the cosine similarity between the l-th unmanned payload and the k-th unmanned payload, n (l) H represents the feature similarity obtained by integrating the visual information of the l-th unmanned payload with that of the other unmanned payloads. cos n This represents the comprehensive feature similarity obtained by associating and fusing the appearance features of n unmanned payloads at time t. S2.

3. At time t, if n unmanned payloads at different locations detect the same target in the same area, then the position and velocity information of the n unmanned payloads at different locations are weighted and fused separately, and the visual information obtained from different perspectives is stitched and fused together; where: Where, d t l v represents the target distance detected by the l-th unmanned payload at time t. t l Let represent the target velocity detected by the l-th unmanned payload at time t, and cos θ. l θ is the cosine of the angle between the l-th unmanned payload and the target, ω l V represents the speed fusion weight of the l-th unmanned payload. t This represents the velocity vector of the multi-unmanned payload relative to the target at time t after weighted fusion. The modulus of D t R represents the position of the multi-unmanned payload relative to the target at time t after weighted fusion. t r represents the visual information obtained by stitching and fusing the visual information of multiple unmanned payloads at time t. l It is the feature vector with a magnitude of 1 output by the l-th unmanned payload to the target.

2. The method for spatiotemporal synchronization fusion of multi-source data in multi-unmanned payload inspection scenarios according to claim 1, characterized in that, The method also includes step S4, which introduces a result feedback correction mechanism to perform feedback correction on the fusion result.

3. The method for spatiotemporal synchronization fusion of multi-source data in multi-unmanned payload inspection scenarios according to claim 1, characterized in that, Step S3 includes the following sub-steps: S3.

1. Within a continuous time period, for multiple unmanned payloads, cosine distance is used to associate position information, velocity angle vector is used to associate velocity information, and cosine similarity is used to associate visual information. Among them, D euclidean (t, t-1) represents the correlation distance between times t and t-1, θ(t, t-1) represents the velocity correlation angle between the two times, and R similarity (t, t-1) represents the associated visual feature vectors at the preceding and following times. D represents the velocity vector of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 R represents the position of the multi-unmanned payload relative to the target at time t-1 after weighted fusion. t-1 This represents the visual information obtained after stitching and fusing the visual information of multiple unmanned payloads at time t-1; When D euclidean (t, t-1) ≤ K t α And θ(t,t-1)≤K t β And R similarity (t, t-1) ≤ K t γ If the multi-unmanned payload detects the same target at times t and t-1, then: S3.2 When multiple unmanned payloads detect the same target at times t and t-1, the spatial state information of the unmanned payloads at time t is fused from multiple points. The specific formula is as follows: in, X represents the predicted state information obtained at time t-1 for the current time. t-1 F' represents the spatial state information after multi-point fusion at time t-1. t-1 Let N be the state transition matrix at time t-1. t-1 H represents the process noise at time t-1. t-1 Let W be the observation matrix at time t-1. t-1 To observe noise.

4. The method for spatiotemporal synchronization fusion of multi-source data for multi-unmanned payload inspection scenarios according to claim 2, characterized in that, Step S4 includes the following sub-steps: S4.

1. The root mean square error (RMSE) is used to measure the data fusion effect. The formula for RMSE is as follows: Among them, X s+1 ′ represents the spatial state information after multi-point fusion at time s+1, X s+1 This represents the spatial state information obtained by the unmanned payload at time s+1; S4.2, when X t+1 When it exceeds RMSE, for X t+1 'Perform feedback correction, where feedback correction includes: first, X t+1 The process involves normalization, data augmentation of the normalization result, and finally, denormalization of the data augmentation result.

5. The method for spatiotemporal synchronization fusion of multi-source data in multi-unmanned payload inspection scenarios according to claim 4, characterized in that, The formula for data augmentation is as follows: in, X represents the result of data augmentation. norm X represents t+1 The normalized result of ′, where a is the coefficient.

6. A multi-source data spatiotemporal synchronization fusion system for multi-unmanned payload inspection scenarios, using the method described in any one of claims 1 to 5, characterized in that, include: The object detection module is used for object detection using the YOLOv5 algorithm. The target tracking module is used for target association and target tracking using the Deepsort multi-target tracking algorithm; The feature extraction module is used to extract features using a convolutional neural network; The weighted and stitched fusion module is used to determine at any given time whether the targets detected by multiple unmanned payloads are the same targets in the same area. If so, the position and velocity information obtained by the multiple unmanned payloads are weighted and fused respectively, and the visual information obtained from different perspectives is stitched and fused. The multi-point fusion module is used to determine whether multiple unmanned payloads are detecting the same target within a continuous time period. If so, it performs multi-point fusion on the spatial state information obtained by the multiple unmanned payloads.

7. A multi-source data spatiotemporal synchronization fusion system for multi-unmanned payload inspection scenarios according to claim 6, characterized in that, The system also includes a feedback correction mechanism for correcting the fusion results.

8. A computer-readable storage medium storing one or more programs, said one or more programs comprising instructions, characterized in that, When the instruction is executed by the computing device, it causes the computing device to perform the method as described in any one of claims 1 to 5.

9. A multi-source data spatiotemporal synchronization fusion device for multi-unmanned payload inspection scenarios, characterized in that, It includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method as described in any one of claims 1 to 5.

Images