Image acquisition card multi-mode identification method and system based on intelligent security and protection

Abstract

The invention provides an image acquisition card multi-mode identification method and system based on intelligent security and protection. The method comprises the following steps: acquiring a multi-mode original data stream according to a global clock signal of an image acquisition card; performing space-time calibration on the multi-modal original data stream to generate a synchronous multi-modal data queue; extracting a multi-modal feature tensor of the synchronous multi-modal data queue through data preprocessing; performing hierarchical attention fusion on the multi-modal feature tensor to generate a fusion feature matrix; performing channel pruning on the fusion feature matrix through a lightweight convolutional neural network, and constructing a target detection model; and determining a detection result corresponding to the multi-modal original data stream according to the target detection model. Through the synergistic effect of a global clock signal and a dynamic space-time calibration algorithm, a time synchronization and space alignment compensation mechanism is constructed in a multi-modal data stream, and the problem of multi-modal information complementary advantage attenuation caused by space-time mismatch is effectively solved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to a multi-modal recognition method and system for an image acquisition card based on intelligent security. Background Art

[0002] The multi-modal target recognition of intelligent security systems faces severe challenges in complex scenarios. Especially in dynamic environments and large-scale heterogeneous sensor networks, ensuring the spatio-temporal consistency of multi-source data has become the core bottleneck restricting detection accuracy. When dealing with multi-modal data such as visible light, infrared, and radar, the problem of inconsistent spatio-temporal benchmarks caused by differences in the physical characteristics of sensors has become increasingly prominent: the timing misalignment between the high frame rate characteristics of visible light sensors and the low refresh rate of radar results in a millisecond-level time offset in the representation of moving targets in different modalities; the difference in spatial resolution between infrared thermal imaging and optical images leads to pixel-level spatial deviations during cross-modal feature matching. Such spatio-temporal mismatch phenomena are particularly severe when the target moves rapidly or the sensor network vibrates, causing detection anomalies such as trajectory breaks and identity jumps. Traditional methods using fixed delay compensation or single-modal dominant fusion strategies are difficult to adapt to the non-linear changes of spatio-temporal errors in dynamic scenarios, resulting in the inability to fully utilize the complementary advantages of multi-modal information. Summary of the Invention

[0003] This application provides a multi-modal recognition method and system for an image acquisition card based on intelligent security, which is used to solve the problem that the complementary advantages of multi-modal information cannot be fully utilized in related technologies.

[0004] In the first aspect of this application, a multi-modal recognition method for an image acquisition card based on intelligent security is provided. The multi-modal recognition method for an image acquisition card based on intelligent security includes: Collecting a multi-modal raw data stream according to the global clock signal of the image acquisition card; Performing spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue; Extracting multi-modal feature tensors of the synchronized multi-modal data queue through data preprocessing; Performing hierarchical attention fusion on the multi-modal feature tensors to generate a fusion feature matrix; Performing channel pruning on the fusion feature matrix through a lightweight convolutional neural network to construct a target detection model; Determining the detection result corresponding to the multi-modal raw data stream according to the target detection model.

[0005] Optionally, in the first implementation manner of the first aspect of this application, the step of performing spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue includes: Perform timestamp interpolation compensation on the first data stream of the multi-modal original data stream through the dynamic time warping algorithm to determine a time-compensated data stream; wherein, the first data stream is the data stream in the multi-modal original data stream that fails to perform time synchronization based on the global clock signal. Perform mapping transformation on the spatial coordinates of the second data stream according to the preset calibration parameters to generate a spatially aligned data stream; wherein, the second data stream is the data stream in the multi-modal original data stream that is time-synchronized. Generate a synchronized multi-modal data queue through a cross-modal alignment algorithm according to the time-compensated data stream and the spatially aligned data stream.

[0006] Optionally, in the second implementation manner of the first aspect of the present application, the synchronized multi-modal data queue includes a visible light data stream, an infrared data stream, and a radar data stream. The step of extracting the multi-modal feature tensor of the synchronized multi-modal data queue through data preprocessing includes: Perform dynamic range adjustment on the visible light data stream to generate enhanced visible light data. Perform non-uniformity correction on the infrared data stream to generate corrected infrared data. Perform structured processing on the radar data stream, and extract a target area point cloud set through density screening. Perform feature extraction on the enhanced visible light data, the corrected infrared data, and the target area point cloud set to generate a multi-modal feature tensor.

[0007] Optionally, in the third implementation manner of the first aspect of the present application, the step of performing hierarchical attention fusion on the multi-modal feature tensor to generate a fusion feature matrix includes: Adjust the weights of the visible light feature map in the multi-modal feature tensor through spatial attention to generate spatially weighted visible light features. Perform channel dimension screening on the infrared feature map in the multi-modal feature tensor through channel attention to generate channel-weighted infrared features. Perform cross-modal cross-attention calculation on the radar feature map in the multi-modal feature tensor to generate radar interaction features associated with the visible light feature map and the infrared feature map. Generate a fusion feature matrix corresponding to the spatially weighted visible light features, the channel-weighted infrared features, and the radar interaction features through multi-scale feature fusion.

[0008] Optionally, in the fourth implementation manner of the first aspect of the present application, the step of constructing a target detection model by performing channel pruning on the fusion feature matrix through a lightweight convolutional neural network includes: Determine the weight coefficients of each channel in the fusion feature matrix, and perform dynamic threshold calibration on the weight coefficients through feature distribution statistics; Perform structured pruning on the redundant channels of the lightweight convolutional neural network according to the dynamic threshold calibration result, and perform weight compensation on the pruned network through gradient reparameterization; Align the feature distributions of the remaining channels of the pruned network to generate a normalized feature map, and transfer the classification decision boundary of the lightweight convolutional neural network to the pruned network through a knowledge distillation strategy; Construct a target detection model corresponding to the pruned network according to the normalized feature map and the classification decision boundary.

[0009] Optionally, in the fifth implementation manner of the first aspect of this application, the step of determining the detection result corresponding to the multi-modal original data stream according to the target detection model includes: Perform multi-scale feature traversal on the fusion feature matrix through the convolutional kernel sliding window of the target detection model to generate an initial detection box set and the corresponding class confidence distribution; Perform cross-modal trajectory tracking on the initial detection box set according to the spatio-temporal correlation of the multi-modal original data stream; Perform spatio-temporal joint screening on the overlapping detection boxes of the target motion trajectory chain determined by cross-modal trajectory tracking to generate an optimized detection box set; Determine the detection result based on the product relationship between the class confidence distribution of the optimized detection box set and the radar point cloud reflection intensity value.

[0010] Optionally, in the sixth implementation manner of the first aspect of this application, the method further includes: Determine the sensitive area of privacy protection according to the target class label and position coordinates in the detection result; Perform local differential privacy perturbation on the feature vector corresponding to the sensitive area through a Laplace noise generator to generate a differentially private constrained desensitized data stream; Determine the sandbox output data of the desensitized data stream through privacy sandbox field isolation; Based on multiple edge nodes of the sandbox output data, perform weighted aggregation on the gradient tensors of the target detection model through the federated averaging algorithm to generate a global model parameter set; Generate an encrypted parameter update package through ciphertext conversion of the global model parameter set; Distribute the encrypted parameter update package to each of the edge nodes to reconstruct the target detection model.

[0011] The second aspect of the present application provides a multi-modal recognition device for an image acquisition card based on intelligent security. The multi-modal recognition device for an image acquisition card based on intelligent security includes: An acquisition module, configured to acquire a multi-modal raw data stream according to the global clock signal of the image acquisition card; A calibration module, configured to perform spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue; An extraction module, configured to extract a multi-modal feature tensor of the synchronized multi-modal data queue through data preprocessing; A fusion module, configured to perform hierarchical attention fusion on the multi-modal feature tensor to generate a fusion feature matrix; A construction module, configured to perform channel pruning on the fusion feature matrix through a lightweight convolutional neural network to construct an object detection model; A determination module, configured to determine a detection result corresponding to the multi-modal raw data stream according to the object detection model.

[0012] The third aspect of the embodiments of the present application provides an electronic device, including a memory and a processor. The processor is configured to execute a computer program stored on the memory. When the processor executes the computer program, each step in the multi-modal recognition method for an image acquisition card based on intelligent security provided in the first aspect of the embodiments of the present application is implemented.

[0013] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step in the multi-modal recognition method for an image acquisition card based on intelligent security provided in the first aspect of the embodiments of the present application is implemented.

[0014] In summary, according to a multi-modal recognition method and system for an image acquisition card based on intelligent security provided by the solution of the present application, a multi-modal raw data stream is acquired according to the global clock signal of the image acquisition card; spatio-temporal calibration is performed on the multi-modal raw data stream to generate a synchronized multi-modal data queue; a multi-modal feature tensor of the synchronized multi-modal data queue is extracted through data preprocessing; hierarchical attention fusion is performed on the multi-modal feature tensor to generate a fusion feature matrix; channel pruning is performed on the fusion feature matrix through a lightweight convolutional neural network to construct an object detection model; a detection result corresponding to the multi-modal raw data stream is determined according to the object detection model. Through the synergistic effect of the global clock signal and the dynamic spatio-temporal calibration algorithm, a sub-millisecond time synchronization accuracy and a sub-pixel spatial alignment compensation mechanism are constructed in the multi-modal data stream, effectively solving the problem of the attenuation of the complementary advantages of multi-modal information caused by spatio-temporal mismatch. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1Schematic flowchart of the multi-modal recognition method for an image acquisition card based on intelligent security provided by an embodiment of the present application; Figure 2 Schematic diagram of program modules of the multi-modal recognition device for an image acquisition card based on intelligent security provided by an embodiment of the present application; Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0016] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0017] To solve the problem that the complementary advantages of multi-modal information cannot be fully utilized in the related art, an embodiment of the present application provides a multi-modal recognition method for an image acquisition card based on intelligent security, as Figure 1 Schematic flowchart of the multi-modal recognition method for an image acquisition card based on intelligent security provided by this embodiment. The multi-modal recognition method for an image acquisition card based on intelligent security includes the following steps: Step 110: Collect multi-modal raw data streams according to the global clock signal of the image acquisition card.

[0018] Specifically, in the security inspection area of a large airport, the behaviors and states of passengers are monitored through multi-modal sensors to identify potential security risks. According to the global clock signal built in the image acquisition card in the security inspection area, data synchronization collection is performed on the deployed visible light cameras, infrared thermal imagers, millimeter wave radars, and distributed microphone arrays. An FPGA (Field Programmable Gate Array, a programmable logic chip that can complete general functions) module with an integrated multi-channel hardware trigger interface is used to control the sampling periods of various sensors through the PTP protocol, so that devices such as visible light cameras (30fps), infrared sensors (15fps), and millimeter wave radars (10Hz) maintain an error of less than 1ms on the time basis. In addition, a laser calibration board is used to perform spatial joint calibration on the cameras and radar sensors to establish a unified three-dimensional coordinate system, so that the sensing data of different modalities are accurately aligned in space, thereby forming a multi-modal raw data stream containing visual, thermal imaging, point cloud, and audio data, providing a basis for subsequent security inspection anomaly detection.

[0019] Step 120: Perform spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue.

[0020] Specifically, perform spatio-temporal calibration on the collected multi-modal raw data stream. Since the data refresh rates of the millimeter-wave radar and the microphone array are low or there is asynchrony, a timestamp interpolation compensation algorithm based on dynamic time warping (DTW) is used to synchronize the motion trajectories detected by the radar with the visible light video stream. In addition, by analyzing the voiceprint features collected by the microphone, the emotional state of the passengers is inferred and time-matched with the visual information to ensure that the recognition of abnormal behaviors has a complete temporal logic. At the same time, the joint calibration data is used to perform spatial mapping on the sensor data of different modalities, so that the target position information of visible light, infrared, and radar is consistent, thereby generating a spatio-temporally aligned multi-modal data queue to provide unified input data for the security inspection anomaly detection system.

[0021] In an optional implementation manner of this embodiment, the step of performing spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue includes: performing timestamp interpolation compensation on the first data stream of the multi-modal raw data stream through the dynamic time warping algorithm to determine the time-compensated data stream; wherein, the first data stream is the data stream in the multi-modal raw data stream that cannot be time-synchronized based on the global clock signal; performing mapping transformation on the spatial coordinates of the second data stream according to the preset calibration parameters to generate a spatially aligned data stream; wherein, the second data stream is the data stream in the multi-modal raw data stream that is time-synchronized; and generating a synchronized multi-modal data queue through a cross-modal alignment algorithm according to the time-compensated data stream and the spatially aligned data stream.

[0022] Specifically, in this embodiment, the first data stream in the multi-modal raw data stream that cannot obtain accurate timestamps relying on the global clock signal is interpolated and compensated through the dynamic time warping algorithm. The core idea is to optimize the objective function to minimize the cost between the timestamp error and the feature inconsistency, thereby realizing the time compensation of the data stream. Specifically, let the timestamp of the k-th sample in the first data stream be and the corresponding interpolation timestamp in the reference global time series be , and at the same time define the feature difference of the sampling point as and the feature change rate as , then the following formula is proposed: , where, P represents the alignment path, K is the number of samples, 、 、 They are weight coefficients respectively, used to balance the mismatches of time error, feature difference and change rate. This formula is used to quantify the time and feature costs between each pair of sampling points, and obtain the optimal alignment path through iterative optimization. Thus, in, for example, an airport monitoring system, the timing deviation caused by sensor clock drift is compensated to generate a time-compensated data stream. Then, for the second data stream in the multi-modal original data stream that realizes time synchronization depending on the global clock signal, a spatial coordinate mapping transformation is performed through preset calibration parameters, and the correction matrix obtained by using a laser calibration board For the homogeneous coordinates of each sampling point in the second data stream Perform the mapping transformation, specifically X (where represents matrix multiplication operation, and x, y, and z respectively represent the coordinates of points in space). This process converts the original coordinates from the local coordinate systems of their respective sensors to a unified three-dimensional space coordinate system to ensure precise spatial alignment of the data. Subsequently, the obtained time-compensated data stream and the spatially aligned data stream are fused through a cross-modal alignment algorithm. Using the reference vector space mapping mechanism, both are mapped to a unified feature space, and the best match is achieved according to the mutual information maximization criterion. Let and represent the feature representations after time compensation and spatial alignment respectively, then the following alignment objective function can be constructed: , where is the cross-modal alignment parameter, and I represents the mutual information, whose role is to measure the statistical correlation between two feature representations. By continuously adjusting to achieve the maximum mutual information, ensuring the best fit of the two data streams in semantic features, and finally generating a synchronized multi-modal data queue. Each data unit in this queue contains information that has been time-compensated and spatially aligned, providing high-quality, unified, and strongly correlated input data for subsequent multi-modal recognition and analysis.

[0023] Step 130: Extract the multi-modal feature tensors of the synchronized multi-modal data queue through data preprocessing.

[0024] Specifically, preprocess the spatio-temporally aligned multimodal data to extract feature information applicable to the security inspection scenario. For visible light data, an improved image enhancement algorithm is used for illumination compensation to reduce the influence of complex illumination conditions (such as backlight and shadows) in the airport security inspection area. Infrared data is corrected for non-uniformity (NUC) to remove thermal noise, and bilinear interpolation is used to align the visible light resolution to ensure the scale consistency of cross-modal features. Millimeter-wave radar data is processed by the VoxelNet (an end-to-end point cloud object detection network) algorithm for point cloud sparsification to extract the gait features of passengers and identify abnormal gaits or people carrying suspicious items. At the same time, a parallel pipeline architecture is deployed on the FPGA module of the image acquisition card to synchronously preprocess visible light, infrared, and radar data, reduce transmission and calculation delays, and generate a lightweight multimodal feature tensor to provide an optimized input for subsequent behavior analysis and target recognition.

[0025] In an optional implementation manner of this embodiment, the steps of extracting the multimodal feature tensor of the synchronous multimodal data queue through data preprocessing include: adjusting the dynamic range of the visible light data stream to generate enhanced visible light data; correcting the non-uniformity of the infrared data stream to generate corrected infrared data; performing structured processing on the radar data stream and extracting the point cloud set of the target area through density screening; and extracting features from the enhanced visible light data, corrected infrared data, and the point cloud set of the target area to generate a multimodal feature tensor.

[0026] Specifically, the synchronous multimodal data queue includes a visible light data stream, an infrared data stream, and a radar data stream. For the dynamic range adjustment of the visible light data stream, an algorithm combining adaptive histogram equalization and local contrast enhancement is adopted. Its core lies in performing local statistical analysis on the brightness of each pixel in the input image and constructing a non-linear mapping function to expand the details of the dark and bright parts in the image. Let the brightness distribution of the original image be L(x, y), and after being transformed by the mapping function the enhanced image is generated, and its expression is: , where integrates local mean, standard deviation, and gradient information to achieve balanced adjustment of the brightness distribution. This formula can be used in the airport security inspection scenario to improve the recognizability of details of targets (such as faces or luggage) in low-light environments and reduce the loss of information in overexposed areas.

[0027] For the infrared data stream, considering the response non-uniformity caused by factors such as manufacturing process errors and temperature drift in the sensor, a non-uniformity correction method based on statistical regression is adopted to correct the original infrared image. Let the original infrared image be I(x, y), and construct the correction function such that the corrected infrared image is expressed as: , This method establishes an infrared sensor background noise model and uses local mean filtering and variance smoothing techniques to suppress the brightness deviation caused by sensor instability, ensuring that the target thermal signal is accurately retained, thereby achieving precise identification of thermal anomaly regions in fire warning or night monitoring.

[0028] For the radar data stream, the millimeter-wave echo signal is structurally processed to reconstruct a three-dimensional point cloud, and the position information of each point in space is obtained using the time delay and reflection intensity of the echo signal. Denote the coordinates of each point as After that, a density-based clustering method is used to screen the target region. Define the neighborhood density function as: , where, is the Gaussian kernel function, represents the Euclidean distance between two points, and set the density threshold to determine whether the local point cloud belongs to the target region. Only the point cloud that satisfies ≥ is included in the target region point cloud set. This method effectively filters noise and extracts a point group with significant target features. For example, it can be used to detect hidden vehicles or abnormal objects in border monitoring.

[0029] Input the enhanced visible light data after dynamic range adjustment, the infrared data after non-uniformity correction, and the target region point cloud set obtained through density screening into the deep feature extraction module, and use an architecture combining a convolutional neural network (CNN) and a graph convolutional network (GCN) to perform feature encoding on each modality respectively. Let the visible light feature be , the infrared feature be and the point cloud feature be . Through an innovative cross-modal feature fusion algorithm, map the features of each modality to the same feature space to generate a multi-modal feature tensor, which can be expressed as: , where the function H performs weighted summation and non-linear activation operations on different modality features based on the attention mechanism to retain the complementary semantic information between modalities to the greatest extent. This method helps to improve the detection accuracy of suspicious behaviors or objects in an intelligent video surveillance system and realizes deep fusion and joint analysis of multi-source data.

[0030] Step 140: Perform hierarchical attention fusion on the multi-modal feature tensor to generate a fusion feature matrix.

[0031] Specifically, a hierarchical attention fusion method is adopted to fuse multi-modal feature tensors to enhance the ability to recognize abnormal behaviors in airport security scenarios. First, in the early fusion stage, the visible light and infrared data channels are concatenated and input into a lightweight CNN to extract the basic contour features of passengers. Then, in the mid-term fusion stage, a cross-modal attention mechanism based on Transformer is adopted. Through deformable attention, different modal features are dynamically aligned, and the weights of each modality are adjusted using environmental state information (such as the illumination intensity and noise level in the airport), so that the contribution of infrared thermal imaging increases in low-light environments, while the weight of the voiceprint feature is enhanced in noisy environments. In addition, a temporal convolutional network (TCN) is introduced to smooth the continuous frames, reducing detection fluctuations caused by the rapid movement or occlusion of passengers, thereby generating a stable fused feature matrix to provide high-confidence fused data for security anomaly behavior detection.

[0032] In an optional implementation manner of this embodiment, the steps of performing hierarchical attention fusion on the multi-modal feature tensor to generate a fused feature matrix include: adjusting the weights of the visible light feature map in the multi-modal feature tensor through spatial attention to generate spatially weighted visible light features; screening the infrared feature map in the multi-modal feature tensor through channel attention to generate channel-weighted infrared features; performing cross-modal cross-attention calculation on the radar feature map in the multi-modal feature tensor to generate radar interaction features associated with the visible light feature map and the infrared feature map; generating a fused feature matrix corresponding to the spatially weighted visible light features, channel-weighted infrared features, and radar interaction features through multi-scale feature fusion.

[0033] Specifically, by introducing an adaptive diffusion mechanism based on partial differential equations, the spatial attention adjustment of the visible light feature map is realized. The core lies in calculating the diffusion weight using the local gradient information of the image, thereby highlighting the regions with rich structural information in the image. The specific method is to use the original visible light feature map By solving the non-linear diffusion equation: , where represents the weight distribution at the diffusion time t, and the function J is an edge-preserving function, and represent the gradient and divergence operators respectively. By obtaining the steady-state solution after (T is a specific value), multiplying the original feature map by this weight distribution, the spatially weighted visible light feature This formula innovatively uses partial differential equations to characterize the diffusion effect of spatial information, where the parameter t reflects the dynamic evolution of the diffusion process and can adaptively adjust the regional importance in complex scenarios. At the same time, channel attention calculation is performed on the infrared feature map, and by constructing the covariance matrix between channels, the screening and compression of redundant information are realized. Let the infrared feature map be denoted as the feature vector of the i-th channel , and the elements of the channel covariance matrix are defined as: , Furthermore, the principal components are extracted by the eigenvalue decomposition method, a channel selection matrix is constructed, and the features of each channel are weighted and reconstructed using this matrix to generate the channel-weighted infrared feature . This method uses covariance analysis in statistics to break through the traditional global pooling method and realizes the accurate capture of the semantic correlation between different channels in infrared data. The parameters i, j represent the channel indices respectively and can effectively suppress the influence of noise channels. For the radar feature map, a cross-modal cross-attention mechanism is introduced. By constructing the radar query matrix and the jointly constructed visible light and infrared key-value matrix , on this basis, mutual information constraints are incorporated to maximize the semantic correlation between radar features and other modal features. The following optimization objective is constructed: , where A represents the cross-modal attention matrix, and the operator represents element-wise multiplication, is the mutual information function. This objective aims to make the radar features after attention weighting and the fused visual and infrared features achieve the optimal match in statistical distribution, generating the radar feature with interactive characteristics. This method effectively captures the deep-level correlation between different modalities and improves the overall recognition robustness. Finally, through the multi-scale feature fusion module, the space-weighted visible light feature , the channel-weighted infrared feature and the interactive radar feature are synergistically integrated at different resolutions. Using multi-scale decomposition based on wavelet transform, each modal feature is decomposed into low-frequency and high-frequency sub-bands, and then the information of each scale sub-band is adaptively weighted and fused through the hierarchical attention mechanism to construct the fused feature matrix: , where the function Designed as a composite structure of multi-layer convolution and non-linear activation, with an embedded spatial and channel adaptive adjustment unit, it can achieve effective interaction between scales while retaining the key semantic information of each modality. This fusion strategy can significantly improve the accuracy and robustness of target recognition and abnormal behavior detection in actual monitoring scenarios, such as airport security checks or urban video surveillance.

[0034] Step 150: Perform channel pruning on the fused feature matrix through a lightweight convolutional neural network to construct an object detection model.

[0035] Specifically, for the fused feature matrix, a lightweight convolutional neural network is used to perform channel pruning on it to construct an efficient object detection model. In the backbone network of the object detection model, a lightweight backbone (such as the MobileNetV3 variant) is adopted, and a dynamic depthwise separable convolution (DDS-Conv) module is introduced to adaptively adjust the number of network layers according to the complexity of the input data. For example, for the standard passing behavior of ordinary passengers, only the shallower convolutional calculations are enabled, while for passengers carrying suspicious items or with abnormal gaits, the deeper network is activated to improve the detection accuracy. In addition, a multi-task learning head is designed at the output end of the detection network to simultaneously perform object detection (such as detecting the size of the luggage carried by passengers), behavior classification (such as abnormal gait, sudden running), and anomaly scoring (such as safety level assessment based on audio emotion analysis), so as to achieve multi-task fusion in the airport security check scenario and improve the applicability and inference efficiency of the model.

[0036] In an optional implementation manner of this embodiment, the step of performing channel pruning on the fused feature matrix through a lightweight convolutional neural network to construct an object detection model includes: determining the weight coefficients of each channel in the fused feature matrix, and dynamically calibrating the weight coefficients through feature distribution statistics; performing structured pruning on the redundant channels of the lightweight convolutional neural network according to the dynamic calibration result, and compensating the weights of the pruned network through gradient reparameterization; aligning the feature distributions of the remaining channels of the pruned network to generate a normalized feature map, and migrating the classification decision boundary of the lightweight convolutional neural network to the pruned network through a knowledge distillation strategy; constructing an object detection model corresponding to the pruned network according to the normalized feature map and the classification decision boundary.

[0037] Specifically, in the fused feature matrix, the determination of the weight coefficients of each channel depends on the in-depth analysis of the feature distribution statistics. The core is to use a dynamic index based on Tsallis entropy to quantitatively evaluate the probability distribution of the feature values in each channel. Let the feature vector of the i-th channel be , and the feature vector of the k-th channel be , and its normalized probability distribution is (where j is the feature index), then the Tsallis entropy is defined as: , wherein, is the entropy index parameter, and the entropy value reflects the diversity and sparsity of channel information. Subsequently, the normalization formula is adopted: , to obtain the dynamic threshold calibration parameter Evaluate each channel. For channels below the preset threshold, they are identified as redundant. Next, based on the dynamic threshold calibration result, perform structured pruning on the corresponding redundant channels in the lightweight convolutional neural network, that is, remove the entire set of convolutional filters to reduce the computational complexity and memory occupancy. To recover the information loss caused by pruning, the gradient reparameterization technique is used for weight compensation. The specific scheme is to calculate the mean activation energy of each channel before and after pruning. Let the original activation mean of the i-th channel be and after pruning be , then apply the reparameterization compensation factor to the remaining weights such that , where is a small positive number to prevent division by zero. This formula aims to make each channel maintain the original activation energy distribution characteristics after pruning, thereby maintaining the discriminative ability of the network. After pruning, perform feature distribution alignment on the remaining channels. By solving the optimal transport problem, minimize the distance between the feature activation distribution P of each channel and the predetermined target distribution Q, that is, solve the optimization problem , wherein, represents the set of joint distributions that satisfy the marginal constraints. This method can generate normalized feature maps to ensure the statistical consistency of each remaining channel. At the same time, by constructing a teacher-student structure, adopt the knowledge distillation strategy to transfer the classification decision boundary of the original lightweight convolutional neural network to the pruned network. In this process, the soft labels output by the teacher network are generated by the temperature-adjusted softmax function to obtain the probability distribution, while the student network realizes the fine correction of the decision boundary by minimizing the Kullback-Leibler divergence loss between the two. Finally, based on the normalized feature map and the classification decision boundary optimized by the knowledge distillation strategy, construct a model for the object detection task. This model uses the fused features for boundary regression and class prediction during the detection process, and can meet the requirements of high-efficiency computing and accurate positioning in practical applications, such as real-time detection of abnormal behaviors in a monitoring system, thereby realizing the overall solution for high-performance object detection.

[0038] Step 160: Determine the detection result corresponding to the multi-modal original data stream according to the object detection model.

[0039] Specifically, based on the output of the object detection model, the abnormal behavior detection result of the multi-modal original data stream is determined. By combining the object detection box, the behavior classification label, and the abnormal score data, the behavior state of the passengers is automatically analyzed to identify potential security risks in real time. For example, if an abnormal thermal imaging feature (high-temperature object), a suspicious gait (abnormal gait caused by carrying prohibited items), and a tense voice emotion are detected for a certain passenger when passing through the security check, the system will automatically trigger an alarm, mark the passenger as a high-risk object, and notify the security personnel for further inspection. In addition, the detection results can be used for airport big data analysis, optimizing the security check process, improving the overall security and response speed of the system, thus constructing a complete and efficient intelligent security solution to enhance the security guarantee ability of airport security checks.

[0040] In an alternative implementation manner of this embodiment, the steps of determining the detection result corresponding to the multi-modal original data stream according to the object detection model include: performing multi-scale feature traversal on the fused feature matrix through the convolutional kernel sliding window of the object detection model to generate an initial detection box set and the corresponding category confidence distribution; performing cross-modal trajectory tracking on the initial detection box set according to the spatio-temporal correlation of the multi-modal original data stream; performing spatio-temporal joint screening on the overlapping detection boxes of the target motion trajectory chain determined by the cross-modal trajectory tracking to generate an optimized detection box set; and determining the detection result based on the product relationship between the category confidence distribution of the optimized detection box set and the radar point cloud reflection intensity value.

[0041] Specifically, the convolutional kernels in the target detection model adopt a sliding window mechanism to perform multi-scale feature traversal on the fused feature matrix, realizing the scanning of regions with different scales and resolutions, generating local candidate regions and predicting the corresponding initial detection box set and class confidence distribution. In this process, the fused feature matrix, as the output after multi-modal data fusion, contains the joint semantic information of visual, infrared, and radar data. A bounding box and a class probability distribution normalized by softmax will be output within each sliding window, indicating the possibility of the target appearing in this local area. For example, in a security monitoring scenario, a detection box may be given a high confidence for the "pedestrian" class. Then, using the spatio-temporal correlation contained in the multi-modal original data stream, a cross-modal trajectory tracking algorithm is used to associate and match the initial detection boxes within consecutive time frames. This algorithm comprehensively considers the spatial overlap rate between detection boxes (such as using the intersection over union IoU) and the Euclidean distance of the center points. By constructing a joint matching matrix, the detection results from different sensors such as visible light, infrared, and radar are linked into a target motion trajectory chain, so that the detection results can maintain continuity and consistency in the time dimension. Subsequently, spatio-temporal joint screening is performed on the overlapping detection boxes in the formed target motion trajectory chain, fusing spatial position and time continuity information, and using an optimization algorithm based on dynamic clustering and spatio-temporal non-maximum suppression to merge and filter the overlapping parts to generate an optimized detection box set. This process effectively removes the redundant boxes introduced by environmental noise or accidental misdetections and retains the candidate regions with high continuity and semantic consistency. Finally, by establishing a non-linear fusion relationship between the class confidence distribution in the optimized detection box set and the radar point cloud reflection intensity value, the final detection result is determined. Specifically, using the radar point cloud reflection intensity as a physical attribute to supplement visual information, the following decision formula is constructed: , where S is the fusion score, C represents the class confidence after normalization in the optimized detection box, and I represents the normalized value of the radar reflection intensity in the corresponding region. The symbol is used to compress the large numerical range and highlight the low-value part. When the fusion score S is greater than the preset threshold, the target is confirmed as a valid detection. This decision-making mechanism improves the robustness and accuracy of target detection in complex environments by jointly considering the complementary advantages of visual semantics and radar physical attributes. For example, in airport monitoring, when the detection box shows a high probability of the "suspicious person" class and the radar reflection intensity indicates the existence of an entity structure, the fusion score will increase significantly, triggering subsequent alarm or response measures. The entire implementation process constructs a target detection system that considers both the spatio-temporal consistency of multi-modal data and fully utilizes their respective advantages through multi-scale traversal, spatio-temporal trajectory tracking, joint screening, and non-linear fusion, effectively improving the detection accuracy and real-time performance.

[0042] In an alternative implementation of this embodiment, a sensitive area for privacy protection is determined based on the target category label and location coordinates in the detection result; the feature vector corresponding to the sensitive area is locally differentially privately perturbed by a Laplace noise generator to generate a differentially private constrained desensitized data stream; the sandbox output data of the desensitized data stream is determined through privacy sandbox field isolation; based on multiple edge nodes of the sandbox output data, the gradient tensors of the target detection model are weighted and aggregated through the federated averaging algorithm to generate a global model parameter set; the global model parameter set is ciphertext-converted to generate an encrypted parameter update package; and the encrypted parameter update package is distributed to each edge node to reconstruct the target detection model.

[0043] Specifically, the target category label and location coordinates output by the target detection model are first used to identify sensitive areas containing personal identity information in the surveillance video. For example, in crowded public places, if a bounding box covering the face or license plate area is detected, it can be identified as a privacy-sensitive area, and these areas represent personal feature data that must be protected. Next, the feature vectors extracted for the sensitive areas are locally differentially privately perturbed using a Laplace noise generator. The core idea is to introduce a non-linear preprocessing function to weaken the prominent outliers in the original features and use the random noise generated by the Laplace distribution to achieve perturbation. The formula is defined as: , where, represents the i-th feature component in the sensitive area, and the function is a non-linear transformation function, and its inverse function is used to restore the data structure, while the noise obeys the Laplace distribution, and its scale parameter is determined by the feature sensitivity and privacy budget. The purpose of this formula is to introduce sufficient random perturbation while ensuring data usability, so as to generate a desensitized data stream that conforms to differential privacy constraints. Subsequently, the sensitive fields in the desensitized data stream are separated through the privacy sandbox field isolation mechanism. The privacy sandbox, as a logical isolation layer, stores and transmits data containing sensitive information separately from ordinary data, ensuring that only authorized processing modules can access the sensitive part, thus forming the sandbox output data. This data not only meets the local privacy protection requirements but also provides a secure data interface for subsequent model updates. Based on multiple edge nodes of the sandbox output data, the federated averaging algorithm is used to weight and aggregate the gradient tensors of the target detection model. In this process, after each edge node calculates the local gradient tensor , the adaptive weight is calculated according to the node data volume and the degree of noise influence. The global model parameter set is calculated through the formula: , Obtained, where N is the total number of participating nodes, Z is the normalization factor, and the weight reflects the confidence levels brought by the data quality of each node and the privacy perturbation. This step ensures that the global update integrates the information of each node without revealing the original data of a single node. Then, perform ciphertext conversion on the generated global model parameter set, and use the ciphertext conversion function E based on elliptic curve cryptography to encrypt the parameters, that is, for each parameter Calculate the ciphertext , where G(r) is a random mapping function and r is a random number. This process makes the parameter update package meet the requirements of irreversibility and collision resistance, thereby generating an encrypted parameter update package. Finally, distribute the encrypted parameter update package to each edge node. After receiving the update package, the edge node decrypts it using the pre-shared key and reconstructs the target detection model based on the decrypted global parameters. This process not only realizes the secure transmission and collaborative update of model parameters, but also ensures that the overall target detection system meets high standards in terms of privacy protection and data security. For example, in a smart security system, it can continuously optimize the accuracy and robustness of abnormal behavior detection while maintaining personal privacy.

[0044] According to a multi-modal recognition method for an image acquisition card based on smart security provided by the solution of the present application, collect a multi-modal raw data stream according to the global clock signal of the image acquisition card; perform spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue; extract the multi-modal feature tensors of the synchronized multi-modal data queue through data preprocessing; perform hierarchical attention fusion on the multi-modal feature tensors to generate a fused feature matrix; perform channel pruning on the fused feature matrix through a lightweight convolutional neural network to construct a target detection model; determine the detection result corresponding to the multi-modal raw data stream according to the target detection model. Through the synergistic effect of the global clock signal and the dynamic spatio-temporal calibration algorithm, a time synchronization and space alignment compensation mechanism is constructed in the multi-modal data stream, effectively solving the problem of the attenuation of the complementary advantages of multi-modal information caused by spatio-temporal mismatch.

[0045] Figure 2 A multi-modal recognition device for an image acquisition card based on smart security provided by an embodiment of the present application. This multi-modal recognition device for an image acquisition card based on smart security can be used to implement the multi-modal recognition method for an image acquisition card based on smart security in the foregoing embodiment. As Figure 2 shown, this multi-modal recognition device for an image acquisition card based on smart security mainly includes: An acquisition module, configured to collect a multi-modal raw data stream according to the global clock signal of the image acquisition card; A calibration module, configured to perform spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue; An extraction module, configured to extract a multi-modal feature tensor of a synchronous multi-modal data queue through data preprocessing; A fusion module, configured to perform hierarchical attention fusion on the multi-modal feature tensor to generate a fusion feature matrix; A construction module, configured to perform channel pruning on the fusion feature matrix through a lightweight convolutional neural network to construct an object detection model; A determination module, configured to determine a detection result corresponding to the multi-modal original data stream according to the object detection model.

[0046] In an optional implementation manner of this embodiment, the calibration module is specifically configured to: perform timestamp interpolation compensation on the first data stream of the multi-modal original data stream through a dynamic time warping algorithm to determine a time-compensated data stream; where the first data stream is the data stream in the multi-modal original data stream that fails to perform time synchronization based on a global clock signal; perform mapping transformation on the spatial coordinates of the second data stream according to preset calibration parameters to generate a spatially aligned data stream; where the second data stream is the data stream in the multi-modal original data stream that is time-synchronized; generate a synchronous multi-modal data queue through a cross-modal alignment algorithm according to the time-compensated data stream and the spatially aligned data stream.

[0047] In an optional implementation manner of this embodiment, the extraction module is specifically configured to: perform dynamic range adjustment on the visible light data stream to generate enhanced visible light data; perform non-uniformity correction on the infrared data stream to generate corrected infrared data; perform structured processing on the radar data stream and extract a target area point cloud set through density screening; perform feature extraction on the enhanced visible light data, the corrected infrared data, and the target area point cloud set to generate a multi-modal feature tensor.

[0048] In an optional implementation manner of this embodiment, the fusion module is specifically configured to: adjust the weights of the visible light feature map in the multi-modal feature tensor through spatial attention to generate spatially weighted visible light features; perform channel dimension screening on the infrared feature map in the multi-modal feature tensor through channel attention to generate channel-weighted infrared features; perform cross-modal cross-attention calculation on the radar feature map in the multi-modal feature tensor to generate radar interaction features associated with the visible light feature map and the infrared feature map; generate a fusion feature matrix corresponding to the spatially weighted visible light features, the channel-weighted infrared features, and the radar interaction features through multi-scale feature fusion.

[0049] In an alternative implementation of this embodiment, the construction module is specifically configured to: determine the weight coefficients of each channel in the fused feature matrix, and perform dynamic threshold calibration on the weight coefficients through feature distribution statistics; perform structured pruning on the redundant channels of the lightweight convolutional neural network according to the dynamic threshold calibration result, and perform weight compensation on the pruned network through gradient reparameterization; align the feature distributions of the remaining channels of the pruned network to generate a normalized feature map, and transfer the classification decision boundary of the lightweight convolutional neural network to the pruned network through a knowledge distillation strategy; construct a target detection model corresponding to the pruned network according to the normalized feature map and the classification decision boundary.

[0050] In an alternative implementation of this embodiment, the determination module is specifically configured to: perform multi-scale feature traversal on the fused feature matrix through the convolutional kernel sliding window of the target detection model to generate an initial detection box set and the corresponding class confidence distribution; perform cross-modal trajectory tracking on the initial detection box set according to the spatio-temporal correlation of the multi-modal original data stream; perform spatio-temporal joint screening on the overlapping detection boxes of the target motion trajectory chain determined by the cross-modal trajectory tracking to generate an optimized detection box set; determine the detection result based on the product relationship between the class confidence distribution of the optimized detection box set and the radar point cloud reflection intensity value.

[0051] In an alternative implementation of this embodiment, the multi-modal recognition device of the image acquisition card further includes: a reconstruction module. The reconstruction module is used to: determine the sensitive area for privacy protection according to the target class label and position coordinates in the detection result; perform local differential privacy perturbation on the feature vector corresponding to the sensitive area through a Laplace noise generator to generate a differentially private constrained desensitized data stream; determine the sandbox output data of the desensitized data stream through privacy sandbox field isolation; perform weighted aggregation on the gradient tensors of the target detection model through the federated averaging algorithm based on multiple edge nodes of the sandbox output data to generate a global model parameter set; perform ciphertext conversion on the global model parameter set to generate an encrypted parameter update package; distribute the encrypted parameter update package to each edge node to reconstruct the target detection model.

[0052] A multi-modal recognition device for an image capture card based on intelligent security provided by the solution of the present application collects a multi-modal raw data stream according to the global clock signal of the image capture card; performs spatio-temporal calibration on the multi-modal raw data stream to generate a synchronized multi-modal data queue; extracts multi-modal feature tensors of the synchronized multi-modal data queue through data preprocessing; performs hierarchical attention fusion on the multi-modal feature tensors to generate a fused feature matrix; performs channel pruning on the fused feature matrix through a lightweight convolutional neural network to construct an object detection model; determines the detection result corresponding to the multi-modal raw data stream according to the object detection model. Through the synergistic effect of the global clock signal and the dynamic spatio-temporal calibration algorithm, a time synchronization and spatial alignment compensation mechanism is constructed in the multi-modal data stream, effectively solving the problem of attenuation of the complementary advantages of multi-modal information caused by spatio-temporal mismatch.

[0053] Provided by the solution of the present application Figure 3 An electronic device provided by an embodiment of the present application. This electronic device can be used to implement the multi-modal recognition method for an image capture card based on intelligent security in the foregoing embodiments, and mainly includes: A memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are communicatively connected. When the processor 302 executes the computer program 303, it implements the multi-modal recognition method for an image capture card based on intelligent security in the foregoing embodiments. Among them, the number of processors can be one or more.

[0054] The memory 301 can be a high-speed random access memory (RAM, Random Access Memory) or a non-unstable memory (non-volatile memory), such as a disk memory. The memory 301 is used to store executable program codes, and the processor 302 is coupled to the memory 301.

[0055] Further, an embodiment of the present application also provides a computer-readable storage medium. This computer-readable storage medium can be disposed in the electronic device in the foregoing embodiments. This computer-readable storage medium can be the memory in the foregoing Figure 3 illustrated embodiments.

[0056] A computer program is stored on this computer-readable storage medium. When the program is executed by the processor, it implements the multi-modal recognition method for an image capture card based on intelligent security in the foregoing embodiments. Further, this computer-readable storage medium can also be various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a RAM, a magnetic disk, or an optical disc that can store program codes.

[0057] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0058] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0059] The above is the case. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of this application.

Claims

1. A multi-modal recognition method for image acquisition cards based on intelligent security, characterized in that: include: Collect multi-modal raw data streams according to the global clock signal of the image acquisition card; Performing time and space calibration on the multimodal original data stream to generate a synchronous multimodal data queue; Extracting a multimodal feature tensor of the synchronous multimodal data queue through data preprocessing; Performing hierarchical attention fusion on the multimodal feature tensor to generate a fused feature matrix; Perform channel pruning on the fusion feature matrix through a lightweight convolutional neural network to build a target detection model; Determine a detection result corresponding to the multimodal original data stream according to the target detection model.

2. The multimodal recognition method of image acquisition card based on intelligent security according to claim 1, characterized in that: The step of performing spatiotemporal calibration on the multimodal original data stream to generate a synchronous multimodal data queue comprises: Performing timestamp interpolation compensation on the first data stream of the multimodal original data stream through a dynamic time warping algorithm to determine a time-compensated data stream; wherein the first data stream is a data stream in the multimodal original data stream that fails to be time-synchronized based on a global clock signal; Mapping and transforming the spatial coordinates of the second data stream according to preset calibration parameters to generate a spatially aligned data stream; wherein the second data stream is a time-synchronized data stream in the multimodal original data stream; According to the time compensation data stream and the space alignment data stream, a synchronous multimodal data queue is generated through a cross-modal alignment algorithm.

3. The multimodal recognition method of image acquisition card based on intelligent security according to claim 1, characterized in that: The synchronous multimodal data queue includes a visible light data stream, an infrared data stream, and a radar data stream, and the step of extracting a multimodal feature tensor of the synchronous multimodal data queue through data preprocessing includes: Performing dynamic range adjustment on the visible light data stream to generate enhanced visible light data; Performing non-uniformity correction on the infrared data stream to generate corrected infrared data; Performing structured processing on the radar data stream and extracting a point cloud set of the target area through density screening; Feature extraction is performed on the enhanced visible light data, the corrected infrared data, and the target area point cloud set to generate a multimodal feature tensor.

4. The multimodal recognition method of image acquisition card based on intelligent security according to claim 3 is characterized in that: The step of performing hierarchical attention fusion on the multimodal feature tensor to generate a fusion feature matrix includes: weighting the visible light feature map in the multimodal feature tensor by spatial attention to generate a spatially weighted visible light feature; Performing channel dimension screening on the infrared feature map in the multimodal feature tensor through channel attention to generate channel weighted infrared features; Performing cross-modal cross-attention calculation on the radar feature map in the multimodal feature tensor to generate radar interaction features associated with the visible light feature map and the infrared feature map; A fusion feature matrix corresponding to the spatially weighted visible light feature, the channel-weighted infrared feature, and the radar interaction feature is generated through multi-scale feature fusion.

5. The multimodal recognition method of image acquisition card based on intelligent security according to claim 4 is characterized in that: The step of performing channel pruning on the fusion feature matrix through a lightweight convolutional neural network to construct a target detection model includes: Determine the weight coefficient of each channel in the fusion feature matrix, and perform dynamic threshold calibration on the weight coefficient through feature distribution statistics; Performing structured pruning on redundant channels of the lightweight convolutional neural network according to the dynamic threshold calibration result, and performing weight compensation on the pruned network through gradient reparameterization; Performing feature distribution alignment on the remaining channels of the pruned network to generate a normalized feature map, and migrating the classification decision boundary of the lightweight convolutional neural network to the pruned network through a knowledge distillation strategy; According to the normalized feature map and the classification decision boundary, a target detection model corresponding to the pruned network is constructed.

6. The multimodal recognition method of image acquisition card based on intelligent security according to claim 1, characterized in that: The step of determining the detection result corresponding to the multimodal original data stream according to the target detection model includes: Performing multi-scale feature traversal on the fused feature matrix through the convolution kernel sliding window of the target detection model to generate an initial detection frame set and a corresponding category confidence distribution; According to the spatiotemporal correlation of the multimodal original data stream, performing cross-modal trajectory tracking on the initial detection frame set; Generate an optimized detection frame set by performing spatiotemporal joint screening on the overlapping detection frames of the target motion trajectory chain determined by cross-modal trajectory tracking; The detection result is determined based on the product relationship between the category confidence distribution of the optimized detection frame set and the radar point cloud reflection intensity value.

7. The multi-modal recognition method of image acquisition card based on intelligent security according to claim 6, characterized in that: The method further comprises: Determine a sensitive area for privacy protection according to the target category label and location coordinates in the detection result; The Laplace noise generator is used to perform local differential privacy perturbations on the feature vectors corresponding to the sensitive areas to generate desensitized data streams with differential privacy constraints. Determining the sandbox output data of the desensitized data flow through privacy sandbox field isolation; Based on multiple edge nodes of the sandbox output data, weighted aggregation is performed on the gradient tensor of the target detection model through a federated averaging algorithm to generate a global model parameter set; Generate an encrypted parameter update package by performing ciphertext conversion on the global model parameter set; The encrypted parameter update package is distributed to each of the edge nodes to reconstruct the target detection model.

8. An image acquisition card multimodal recognition device based on intelligent security, characterized in that: The image acquisition card multimodal recognition device based on intelligent security includes: An acquisition module, used for acquiring multi-modal raw data streams according to a global clock signal of an image acquisition card; A calibration module, used for performing spatiotemporal calibration on the multimodal original data stream to generate a synchronous multimodal data queue; An extraction module, configured to extract a multimodal feature tensor of the synchronous multimodal data queue through data preprocessing; A fusion module, used for performing hierarchical attention fusion on the multimodal feature tensor to generate a fused feature matrix; A construction module, used to perform channel pruning on the fusion feature matrix through a lightweight convolutional neural network to construct a target detection model; A determination module is used to determine the detection result corresponding to the multimodal original data stream according to the target detection model.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the multimodal recognition method of the image acquisition card based on smart security described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the multimodal recognition method of the image acquisition card based on smart security described in any one of claims 1 to 7 are implemented.

Images