Living body target intelligent identification method based on multi-modal heterogeneous data collaboration

Abstract

The invention discloses a living body target intelligent identification method based on multi-mode heterogeneous data collaboration, and particularly relates to the technical field of target intelligent identification. Multi-mode heterogeneous data are acquired in real time by using multiple types of sensors, and high-precision timestamps of all modes are recorded to construct multi-mode asynchronous mismatch coefficients; the time sequence mismatch risk caused by collection and transmission asynchronization is dynamically recognized, a cross-modal similarity matrix is constructed, joint modeling is conducted on the cross-modal similarity matrix and a modal feature significance conflict area to calculate a cross-modal feature conflict enhancement coefficient, the enhancement effect of cross-modal feature conflicts is quantitatively evaluated from the feature level, and the time sequence mismatch risk caused by collection and transmission asynchronization is evaluated. The cross-modal feature conflict enhancement coefficient and the multi-modal asynchronous mismatch coefficient form a complementary output multi-modal fusion risk assessment index, the stability and risk level of the current multi-modal fusion state are quantified, and the potential time sequence and feature collaborative failure risk in the fusion process is accurately reflected; and dynamically adjusting the fusion weight of each mode based on a risk assessment result.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent target recognition, and more specifically, to a method for intelligent recognition of living targets based on multimodal heterogeneous data collaboration. Background Art

[0002] With the rapid development of artificial intelligence and sensor fusion technologies, live target recognition is gaining widespread application in fields such as security, finance, transportation, and edge computing. Traditional single-modality recognition methods (such as those based solely on images or infrared) exhibit significant robustness deficiencies in scenarios such as complex lighting, occlusion, and attack deception. Therefore, live target recognition methods based on the collaborative use of multimodal heterogeneous data have become a research hotspot. These methods achieve more comprehensive target perception and discrimination capabilities by fusing multiple sources of information, including visible light images, infrared, depth point clouds, audio, and micro-motion radar.

[0003] However, while multimodal fusion improves the recognition accuracy and anti-interference capabilities of the system, it also introduces new systemic challenges. Among them, multimodal asynchronous mismatch and increased cross-modal feature conflict have become core issues that restrict the stability and practicality of such methods. On the one hand, there are inherent sampling frequency differences and transmission delays in the acquisition and transmission of modal data, which can easily lead to time alignment failures and semantic misalignment; on the other hand, misaligned multimodal features may produce information redundancy or representation conflicts during the fusion stage, causing the attention mechanism or weighting strategy to shift, forming the so-called "cross-modal pseudo-activation misjudgment effect." This effect manifests itself as the system falsely triggering abnormal alarms when it is not under real attack, or misjudging it as a normal living organism when it is under attack, seriously affecting the security and practicality of the system.

[0004] Therefore, it is urgent to construct a living target recognition method with a cross-modal temporal coordination mechanism and feature consistency constraint capability to solve the multimodal fusion instability problem caused by the superposition of the above-mentioned temporal mismatch and feature conflict, thereby improving the recognition stability and credibility of the system in dynamic and complex scenarios. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a method for intelligent recognition of living targets based on multimodal heterogeneous data collaboration to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: A method for intelligent recognition of living objects based on multimodal heterogeneous data collaboration includes the following steps: Step S1, acquiring heterogeneous modal data in real time through multiple types of sensors, and recording high-precision timestamps for each modal data acquisition; Step S2, calculating the multimodal asynchronous mismatch coefficient based on the high-precision timestamps and timing features obtained from each modality, and evaluating the timing mismatch risk of each modality; Step S3: construct a cross-modal similarity matrix, and jointly model the cross-modal similarity matrix with the modal feature significant conflict area to calculate the cross-modal feature conflict enhancement coefficient and evaluate the cross-modal feature conflict enhancement risk; Step S4: construct a multimodal fusion risk assessment model based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, output a multimodal fusion risk assessment index, and evaluate the current multimodal fusion risk level; Step S5: Dynamically adjust the fusion weights of each modality according to the current multimodal fusion risk level.

[0007] In a preferred embodiment, the multimodal asynchronous mismatch coefficient is calculated based on the high-precision timestamps and timing features obtained from each modality, as follows: Construct a high-precision timestamp set based on the high-precision timestamps obtained from each modality ,in is the high-precision timestamp of the n-th sample of the ith mode, I is the total number of modes, and N is the total number of samples of modal data. Based on the reference clock Tref of the identification system, the high-precision timestamp of the standardized mode is: ,in It is a standardized high-precision timestamp; Compute the mean of the modal pairs with time offsets: ,in is the mean time offset of the mode pair, (p, q) is the matching frame pair of the sample data frames of mode i and mode j, and YZ is the number of valid frame pairs; Constructing the modal pair dynamic time shift matrix based on the modal pair time shift mean ; Calculate the modal pair mismatch uncertainty: ,in is the modal pair mismatch uncertainty; Fit the time series disturbance response curve of each modal dynamic characteristic on the time axis: ,in is the disturbance response value of the ith mode at time t, is the time basis function, is the fitting coefficient of the mth time basis function of mode i; Compute the mode-to-disturbance residuals: ,in is the mode pair disturbance residual, is the disturbance response value of the jth mode at time t; Calculate the modal pair mismatch coefficient: ,in is the modal pair mismatch coefficient, rdc is the mean of the disturbance residual, represent the preset proportional coefficients of the modal to time offset mean and disturbance residual mean, respectively, and are all greater than 0; the calculation formula for the mean of the disturbance residual is as follows: , where TW is the time window width, ; Calculate the multimodal asynchronous mismatch coefficient: , where Nasy is the multimodal asynchronous mismatch coefficient.

[0008] In a preferred embodiment, in step S3, the acquired modal data is input into a pre-trained deep neural network to extract the feature embedding vectors of each modality. ,in is the feature embedding vector of the i-th modality, is the m-th dimension feature of the feature embedding vector of the i-th modality, M is the total dimension; calculate the feature cosine similarity of modality i and modality j ,in is the feature embedding vector of the jth modality; based on the feature cosine similarity between each modality, a cross-modal similarity matrix is ​​constructed .

[0009] In a preferred embodiment, the cross-modal similarity matrix and the modal feature saliency conflict region are jointly modeled to calculate the cross-modal feature conflict enhancement coefficient, specifically as follows: For each modality i, the saliency branch in the pre-trained deep neural network is used to generate a saliency feature mask vector ,in is the significance probability of the m-th dimension feature of the i-th modality; Calculate the significance conflict vector between modality i and modality j , where YCj is the salient feature mask vector of the j-th modality; Calculate the characteristic energy conflict vector ,in represents element-wise multiplication, is the characteristic square sum vector of the corresponding dimensions of mode i and mode j; Calculate the characteristic energy conflict mean ,in is the characteristic energy conflict vector The actual value of the m-th dimension element in; Calculate the cumulative potential energy of the conflict ,in is the residual of the mode pair disturbance, TW is the time window width, ; Calculate dynamic interaction attention weights: ,in is the dynamic interaction attention weight of modality i and modality j, is the feature cosine similarity of the modality pair (u, v), is the mean of the disturbance residuals of mode i and mode j, is the mean of the disturbance residuals of the mode pair (u, v); Calculate the cross-modal feature conflict enhancement coefficient: , where Cend is the cross-modal feature conflict enhancement coefficient.

[0010] In a preferred embodiment, a multimodal fusion risk assessment model is constructed based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, and a multimodal fusion risk assessment index is output. The formula of the multimodal fusion risk assessment model is as follows: , where dmt is the multimodal fusion risk assessment index, Nasy is the multimodal asynchronous mismatch coefficient, and Cend is the cross-modal feature conflict enhancement coefficient. Respectively represent the preset proportional coefficients of the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, and Both are greater than 0.

[0011] In a preferred embodiment, the multimodal fusion risk assessment index is compared with a preset multimodal fusion risk assessment index threshold to assess the current multimodal fusion risk level, as follows: If the multimodal fusion risk assessment index is greater than the multimodal fusion risk assessment index threshold, the current multimodal fusion risk level is marked as high multimodal fusion risk; If the multimodal fusion risk assessment index is less than or equal to the multimodal fusion risk assessment index threshold, it means that the current multimodal fusion risk level is marked as low multimodal fusion risk.

[0012] In a preferred embodiment, in step S5, the fusion weights of each modality are dynamically adjusted according to the current multimodal fusion risk level, as follows: If the current multimodal fusion risk level is marked as high, the fusion weights of each modality are dynamically adjusted according to the multimodal fusion risk assessment index output by the current multimodal fusion risk assessment model: ,in is the fusion weight after dynamic adjustment of mode i, is the initial fusion weight of modality i.

[0013] Technical effects and advantages of the present invention: 1. The intelligent live object recognition method based on multimodal heterogeneous data collaboration provided by this invention can significantly improve the robustness, accuracy, and security of liveness recognition in complex dynamic environments, and has significant technical effects and advantages. This method addresses the timing mismatch and feature conflict issues faced by traditional liveness recognition in the multimodal fusion process, and proposes a systematic risk perception and adaptive control mechanism, comprehensively improving the practicality and reliability of multimodal recognition systems in dynamic and complex scenarios. First, make full use of multiple types of sensors to acquire multimodal heterogeneous data such as visible light images, infrared, depth point clouds, audio, and micro-motion radar in real time, and accurately record high-precision timestamps of each modality to effectively support the accuracy of subsequent multimodal time series collaborative analysis and risk perception. Based on high-precision timestamps, a multimodal asynchronous mismatch coefficient is constructed to dynamically identify the timing mismatch risk caused by asynchronous acquisition and transmission. Secondly, by constructing a cross-modal similarity matrix and jointly modeling the cross-modal similarity matrix with the modal feature significance conflict area to calculate the cross-modal feature conflict enhancement coefficient, the enhancement effect of cross-modal feature conflict is quantitatively evaluated from the feature level, effectively revealing the differences between different modalities in a specific spatiotemporal region due to inconsistent feature expression or attention bias. The system accurately detects the degree of conflict caused by shifts, helping the system accurately perceive potential risks that could lead to false activations or misjudgments in fusion. The cross-modal feature conflict enhancement coefficient and the multimodal asynchronous mismatch coefficient complement each other, creating a collaborative model and outputting a multimodal fusion risk assessment index. This dynamically quantifies the stability and risk level of the current multimodal fusion state, accurately reflecting the potential risk of timing and feature coordination failure during the fusion process. This enables hierarchical management and dynamic alerting of fusion risks. Based on the risk assessment results, the fusion weights of each modality are dynamically adjusted, enabling the system to adaptively reduce the weight of high-risk modalities and enhance the contribution of low-risk modalities, effectively suppressing the effects of cross-modal false activation misjudgments and improving the accuracy, robustness, and anti-interference capabilities of fusion decisions. This weight control mechanism exhibits excellent dynamic adaptability in scenarios with complex lighting, occlusion, and attack interference, ensuring the credibility and practicality of the system's output results to the greatest extent possible. Achieving a technological breakthrough in multimodal live target recognition from passive fusion to active risk perception and dynamic optimization effectively solves the problem of traditional multimodal fusion methods being prone to instability and misjudgment in complex application scenarios, and provides a safer, more efficient and reliable solution for intelligent recognition of live targets in security, finance, transportation, edge computing and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 Flowchart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0016] Example: Figure 1 The present invention provides a method for intelligently identifying living objects based on multimodal heterogeneous data collaboration, which includes the following steps: Step S1, acquiring heterogeneous modal data in real time through multiple types of sensors, and recording high-precision timestamps for each modal data acquisition; Step S2, calculating the multimodal asynchronous mismatch coefficient based on the high-precision timestamps and timing features obtained from each modality, and evaluating the timing mismatch risk of each modality; Step S3: construct a cross-modal similarity matrix, and jointly model the cross-modal similarity matrix with the modal feature significant conflict area to calculate the cross-modal feature conflict enhancement coefficient and evaluate the cross-modal feature conflict enhancement risk; Step S4: construct a multimodal fusion risk assessment model based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, output a multimodal fusion risk assessment index, and evaluate the current multimodal fusion risk level; Step S5: dynamically adjust the fusion weights of each modality according to the current multimodal fusion risk level; In step S1, heterogeneous modal data, including visible light images, infrared images, depth point clouds, audio, and micro-motion radar signals, are acquired in real time by identifying multiple types of sensors configured in the system (e.g., cameras supporting high frame rates, cold infrared thermal imagers, high-sensitivity microphone arrays, and micro-motion detection radars). All sensors are interface driven, data cached, and time-series managed through a unified data acquisition control board (optional FPGA+ARM architecture or high-performance embedded SoC). During initialization, all sensors are synchronously started using a unified system clock. At the moment each modal data packet is generated or acquired, a high-precision timestamp for each modal data acquisition is recorded in real time, and the high-precision timestamp is stored in the data structure of each modality as metadata. Step S2, calculating the multimodal asynchronous mismatch coefficient based on the high-precision timestamps and timing features obtained from each modality, and evaluating the timing mismatch risk of each modality; In the present invention, the multimodal asynchronous mismatch coefficient is a key indicator for measuring the acquisition timing consistency and dynamic synchronization accuracy of multimodal heterogeneous data in the time dimension. This coefficient reflects the accuracy of semantic alignment and the degree of deviation from time synchronization of different modal data in the same time window, and is a basic parameter for evaluating multimodal fusion risk. Specifically, the larger the value of the multimodal asynchronous mismatch coefficient, the more obvious the time alignment deviation of each modality during data acquisition and transmission. The physical action or state representation corresponding to the key features between the modalities is misaligned, crossed, or lagged on the time axis. This will directly lead to a decrease in semantic relevance in the feature fusion stage, a shift in attention focus, and may even induce problems such as cross-modal feature misjudgment and unstable fusion discrimination. A larger multimodal asynchronous mismatch coefficient indicates that the fusion timing risk of the current modal combination is at a high level. The system is prone to adverse consequences such as judgment delay, state false triggering, false alarm or missed alarm during the process of living target recognition, which is not conducive to achieving high robustness and high reliability real-time recognition targets. On the contrary, when the multimodal asynchronous mismatch coefficient is in a smaller range, it indicates that the temporal consistency of each modal data during acquisition and transmission is good, and the physical states corresponding to each modal feature can be aligned and expressed collaboratively with high precision on the time axis. In this case, the semantic interaction between features in the modal fusion stage is closer, the fusion network can effectively play the advantages of different modalities, and the attention mechanism and weighting strategy can accurately focus on the complementary information area of ​​the multimodal features, significantly improving the accuracy and stability of liveness recognition. A smaller multimodal asynchronous mismatch coefficient indicates that the current system has a lower multimodal fusion timing risk, and the recognition process has a stronger anti-interference ability and dynamic self-adaptation ability, which is more conducive to dealing with challenges such as dynamic changes, occlusion interference, and human attacks in complex environments. Assessing the current multimodal fusion risk level based on the multimodal asynchronous mismatch coefficient has important technical value and application significance. On the one hand, this coefficient can quantify the degree of timing mismatch in the multimodal system in real time, providing a quantitative and dynamic risk perception basis for the system fusion strategy, breaking through the limitations of traditional reliance on fixed timing alignment or static weight fusion; on the other hand, the present invention uses a risk assessment mechanism based on the multimodal asynchronous mismatch coefficient to enable the recognition system to actively adjust the fusion weights of each modality or trigger a compensation alignment mechanism when high-risk timing deviations are detected, thereby achieving dynamic self-repair of timing alignment and reducing fusion errors caused by time mismatch. In addition, during the process of living target recognition, this mechanism can effectively suppress the cross-modal pseudo-activation effect caused by modal data delay, buffer lag or sudden frame loss, significantly improving the stability, accuracy and security of the system in complex environments. The present invention not only enhances the perception and control capabilities of dynamic timing risks in the multimodal fusion stage, but also provides more robust technical support for the application scenarios of living target intelligent recognition systems in edge computing, real-time security, unmanned access control, face-swiping payment, etc., which will help promote the large-scale application and deployment of multimodal recognition systems in actual scenarios.

[0017] The multimodal asynchronous mismatch coefficient is calculated based on the high-precision timestamps and timing features obtained from each modality, as follows: Construct a high-precision timestamp set based on the high-precision timestamps obtained from each modality: ,in is the high-precision timestamp of the n-th sample of the ith mode, I is the total number of modes, and N is the total number of samples of modal data. Based on the reference clock Tref of the identification system, the high-precision timestamp of the standardized mode is: ,in It is a standardized high-precision timestamp; Compute the mean of the modal pairs with time offsets: ,in is the mean time offset of the modal pair, (p, q) is the matching frame pair of the sample data frames of modality i and modality j, based on the nearest neighbor matching, and YZ is the number of valid frame pairs; Constructing the modal pair dynamic time shift matrix based on the modal pair time shift mean ; Calculate the modal pair mismatch uncertainty: ,in is the modal pair mismatch uncertainty; the time series disturbance response curve of each modal dynamic characteristic is fitted on the time axis: ,in is the disturbance response value of the ith mode at time t, is a time basis function (such as Gaussian basis function, high-order B-spline, Fourier basis function, etc., adapted to different characteristic waveforms), is the fitting coefficient of the m-th time basis function of mode i, which is used to describe the contribution of the basis function to the dynamic change of the mode; It should be noted that the fitting coefficient The solution can be obtained by least squares fitting: ,in is the original dynamic eigenvalue curve of the i-th mode; Compute the mode-to-disturbance residuals: ,in is the mode pair disturbance residual, is the disturbance response value of the jth mode at time t; Calculate the modal pair mismatch coefficient: ,in is the modal pair mismatch coefficient, rdc is the mean of the disturbance residual, represent the preset proportional coefficients of the modal to time offset mean and disturbance residual mean, respectively, and are all greater than 0; the calculation formula for the mean of the disturbance residual is as follows: , where TW is the time window width, ; It should be noted that Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations, for example, It can be 0.5, 0.5; Calculate the multimodal asynchronous mismatch coefficient: , where Nasy is the multimodal asynchronous mismatch coefficient; It should be noted that the above formulas are all dimensionless and calculated numerically. Common dimensionless methods include Min-Max normalization and Z-Score normalization, which will not be described here. In step S3, the acquired modal data is input into the pre-trained deep neural network to extract the feature embedding vector of each modality. ,in is the feature embedding vector of the i-th modality, tzi m is the m-th dimension feature of the feature embedding vector of the i-th modality, and M is the total dimension; Calculate the feature cosine similarity between modality i and modality j ,in is the feature embedding vector of the jth modality; based on the feature cosine similarity between each modality, a cross-modal similarity matrix is ​​constructed ; It should be noted that pre-trained deep neural networks refer to deep neural network structures trained on standard large-scale datasets (such as ImageNet, VGGFace, CASIA-WebFace, VoxCeleb, KITTI, NuScenes, etc.) for various modal task requirements. They are used to generate highly expressive feature embedding vectors. These networks may include: convolutional neural networks (CNNs), temporal convolutional networks (TCNs), transformers, temporal transformers, point cloud feature extraction networks (such as PointNet and PointNet++), multi-layer perceptron (MLP) embedding heads, and other structures. These network structures and training methods are widely disclosed in existing literature and patents and will not be elaborated on here. The cross-modal feature conflict enhancement coefficient in the present invention is a comprehensive indicator used to measure the significance of conflicts and the degree of conflict energy enhancement at the feature level of multimodal data. It reflects the comprehensive intensity of feature conflicts caused by factors such as physical properties, sensor characteristics or environmental disturbances in the feature embedding space of different modalities, and can effectively characterize the potential risk of feature inconsistency and hidden dangers of fusion failure in the multimodal fusion process. The larger the cross-modal feature conflict enhancement coefficient, the more significant the conflict area between different modalities in the feature space, and the higher the degree of enhancement of feature energy distribution in the conflict area. This often means that there are stronger feature inconsistencies in the multimodal fusion process, the stability and reliability of the fusion results are reduced, and the misjudgment risk of the fusion model, the probability of abnormal response and the volatility of the system output are significantly increased. Therefore, a larger cross-modal feature conflict enhancement coefficient is an important early warning indicator of increased multimodal fusion risk, suggesting that a higher-weighted conflict suppression, feature reconstruction or modality de-weighting mechanism needs to be introduced into the fusion strategy to alleviate the negative impact caused by enhanced feature conflict. Conversely, a smaller cross-modal feature conflict enhancement coefficient indicates that the energy level of the feature space representations of different modalities in the conflicting regions is lower, and the modalities maintain good compatibility and consistency within the key feature regions. This leads to better synergy between the different modalities during the fusion process, and higher precision, stability, and robustness of the fusion results. This lays the foundation for achieving high-confidence fusion decisions and stable perception and recognition in dynamic environments. Therefore, dynamically assessing the current multimodal fusion risk level through the cross-modal feature conflict enhancement coefficient can significantly improve the security, adaptability, and risk prevention and control capabilities of the multimodal fusion system.

[0018] The cross-modal similarity matrix and the modal feature saliency conflict region are jointly modeled to calculate the cross-modal feature conflict enhancement coefficient, as follows: For each modality i, the saliency branch in the pre-trained deep neural network is used to generate saliency Sexual characteristics mask vector ,in is the significance probability of the m-th dimension feature of the i-th modality; Calculate the significance conflict vector between modality i and modality j ,in is the salient feature mask vector of the j-th modality; Calculate the characteristic energy conflict vector ,in represents element-wise multiplication, is the characteristic square sum vector of the corresponding dimensions of mode i and mode j, specifically:

[0019] Calculate the characteristic energy conflict mean ,in is the actual value of the m-th dimension element in the characteristic energy conflict vector XLij; Calculate the cumulative potential energy of the conflict ,in is the residual of the mode pair disturbance, TW is the time window width, ; Calculate dynamic interaction attention weights: ,in is the dynamic interaction attention weight of modality i and modality j, is the feature cosine similarity of the modality pair (u, v), is the mean of the disturbance residuals of mode i and mode j, is the mean of the disturbance residuals of the mode pair (u, v); Calculate the cross-modal feature conflict enhancement coefficient: , where Cend is the cross-modal feature conflict enhancement coefficient; It should be noted that the above formulas are all dimensionless and numerical calculations. Common dimensionless methods include Min-Max normalization and Z-Score normalization, which will not be described here. Step S4: construct a multimodal fusion risk assessment model based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, output a multimodal fusion risk assessment index, and evaluate the current multimodal fusion risk level; A multimodal fusion risk assessment model is constructed based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, and the multimodal fusion risk assessment index is output. The formula based on the multimodal fusion risk assessment model is as follows: , where dmt is the multimodal fusion risk assessment index, Nasy is the multimodal asynchronous mismatch coefficient, and Cend is the cross-modal feature conflict enhancement coefficient. Respectively represent the preset proportional coefficients of the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, and All greater than 0; It should be noted that the above formulas are all dimensionless and calculated numerically. Common dimensionless methods include Min-Max normalization and Z-Score normalization, which will not be described here. Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations, for example, It can be 0.5, 0.5; From the above calculation expression, it can be seen that the larger the multimodal asynchronous mismatch coefficient and the larger the cross-modal feature conflict enhancement coefficient, the larger the multimodal fusion risk assessment index, indicating that the current multimodal data has a higher degree of mismatch and conflict risk at the time synchronization level and feature synergy level. This means that in the multimodal fusion process, due to the asynchrony of the acquisition time series, the perturbation instability of the dynamic feature evolution, or the conflict enhancement of the feature space expression, different modalities may cause inconsistency in the fusion results, a decrease in the confidence of the output decision, and abnormal fluctuations in the perception model, thereby significantly increasing the probability of functional deviation, recognition failure, false alarms, and missed alarms in the multimodal fusion system. Conversely, the smaller the multimodal asynchronous mismatch coefficient and the smaller the cross-modal feature conflict enhancement coefficient, the smaller the multimodal fusion risk assessment index, indicating that the current multimodal data performs well in terms of time synchronization characteristics and feature synergy, and that different modalities can effectively cooperate and complement each other's advantages, providing a strong guarantee for the fusion system to achieve high stability, high precision, and high robustness in comprehensive perception and recognition capabilities. As an important indicator for dynamic monitoring of multimodal fusion risks, the multimodal fusion risk assessment index can provide real-time risk perception basis in fusion decision-making, drive the execution of adaptive fusion weight adjustment, abnormal modality suppression and risk avoidance strategies, and thus realize the safe and stable operation and accurate intelligent perception of the multimodal fusion system in complex, dynamic and heterogeneous environments.

[0020] The multimodal fusion risk assessment index is compared with the preset multimodal fusion risk assessment index threshold to evaluate the current multimodal fusion risk level, as follows: If the multimodal fusion risk assessment index is greater than the multimodal fusion risk assessment index threshold, it means that the current multimodal data has significant risks in terms of time synchronization and feature synergy, that is, the degree of multimodal asynchronous mismatch is large and cross-modal feature conflict is significantly enhanced, which may lead to information inconsistency, feature expression interference, and unstable fusion decision-making in the fusion process of different modal data. In this case, the multimodal fusion system faces a high probability of failure or perception bias risk, making it difficult to ensure the accuracy and reliability of the output results. The current multimodal fusion risk level is marked as high multimodal fusion risk; If the multimodal fusion risk assessment index is less than or equal to the multimodal fusion risk assessment index threshold, it means that the current multimodal data has good overall performance in terms of time synchronization and feature synergy, that is, the degree of multimodal asynchronous mismatch is small, the degree of cross-modal feature conflict is low, different modal data can effectively cooperate at the time series and feature levels, the information complementarity in the fusion process is good, and the fusion decision is stable and reliable, indicating that the current fusion risk level is within a controllable range, the multimodal perception results and decision confidence output by the system are high, and the fusion process has good robustness and adaptability. The current multimodal fusion risk level is marked as low multimodal fusion risk; In step S5, the fusion weights of each modality are dynamically adjusted according to the current multimodal fusion risk level, as follows: If the current multimodal fusion risk level is marked as high, the fusion weights of each modality are dynamically adjusted according to the multimodal fusion risk assessment index output by the current multimodal fusion risk assessment model: ,in is the fusion weight after dynamic adjustment of mode i, is the initial fusion weight of modality i; The intelligent live object recognition method based on multimodal heterogeneous data collaboration provided by this invention can significantly improve the robustness, accuracy, and security of liveness recognition in complex dynamic environments, demonstrating significant technical benefits and advantages. This method addresses the timing mismatch and feature conflict issues faced by traditional liveness recognition during multimodal fusion, proposing a systematic risk perception and adaptive control mechanism. This method comprehensively improves the practicality and reliability of multimodal recognition systems in dynamic and complex scenarios. First, make full use of multiple types of sensors to acquire multimodal heterogeneous data such as visible light images, infrared, depth point clouds, audio, and micro-motion radar in real time, and accurately record high-precision timestamps of each modality to effectively support the accuracy of subsequent multimodal time series collaborative analysis and risk perception. Based on high-precision timestamps, a multimodal asynchronous mismatch coefficient is constructed to dynamically identify the timing mismatch risk caused by asynchronous acquisition and transmission. Secondly, by constructing a cross-modal similarity matrix and jointly modeling the cross-modal similarity matrix with the modal feature significance conflict area to calculate the cross-modal feature conflict enhancement coefficient, the enhancement effect of cross-modal feature conflict is quantitatively evaluated from the feature level, effectively revealing the differences between different modalities in a specific spatiotemporal region due to inconsistent feature expression or attention bias. The system accurately detects the degree of conflict caused by shifts, helping the system accurately perceive potential risks that could lead to false activations or misjudgments in fusion. The cross-modal feature conflict enhancement coefficient and the multimodal asynchronous mismatch coefficient complement each other, creating a collaborative model and outputting a multimodal fusion risk assessment index. This dynamically quantifies the stability and risk level of the current multimodal fusion state, accurately reflecting the potential risk of timing and feature coordination failure during the fusion process. This enables hierarchical management and dynamic alerting of fusion risks. Based on the risk assessment results, the fusion weights of each modality are dynamically adjusted, enabling the system to adaptively reduce the weight of high-risk modalities and enhance the contribution of low-risk modalities, effectively suppressing the effects of cross-modal false activation misjudgments and improving the accuracy, robustness, and anti-interference capabilities of fusion decisions. This weight control mechanism exhibits excellent dynamic adaptability in scenarios with complex lighting, occlusion, and attack interference, ensuring the credibility and practicality of the system's output results to the greatest extent possible. Achieving a technological breakthrough in multimodal live target recognition from passive fusion to active risk perception and dynamic optimization effectively solves the problem of traditional multimodal fusion methods being prone to instability and misjudgment in complex application scenarios, and provides a safer, more efficient and reliable solution for intelligent recognition of live targets in security, finance, transportation, edge computing and other fields.

[0021] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0022] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for intelligent recognition of living objects based on multimodal heterogeneous data collaboration, characterized by: The steps include: Step S1, acquiring heterogeneous modal data in real time through multiple types of sensors, and recording high-precision timestamps for each modal data acquisition; Step S2, calculating the multimodal asynchronous mismatch coefficient based on the high-precision timestamps and timing features obtained from each modality, and evaluating the timing mismatch risk of each modality; Step S3: construct a cross-modal similarity matrix, and jointly model the cross-modal similarity matrix with the modal feature significant conflict area to calculate the cross-modal feature conflict enhancement coefficient and evaluate the cross-modal feature conflict enhancement risk; Step S4: construct a multimodal fusion risk assessment model based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, output a multimodal fusion risk assessment index, and evaluate the current multimodal fusion risk level; Step S5: Dynamically adjust the fusion weights of each modality according to the current multimodal fusion risk level.

2. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 1 is characterized by: The multimodal asynchronous mismatch coefficient is calculated based on the high-precision timestamps and timing features obtained from each modality, as follows: Construct a high-precision timestamp set based on the high-precision timestamps obtained from each modality: , where STi (n) is the high-precision timestamp of the nth sample of the ith mode, I is the total number of modes, and N is the total number of samples of modal data; Based on the identification system reference clock Tref, the high-precision timestamp of the standardized mode is: ,in, It is a standardized high-precision timestamp; Compute the mean of the modal pairs with time offsets: ,in is the mean time offset of the mode pair, (p, q) is the matching frame pair of the sample data frames of mode i and mode j, and YZ is the number of valid frame pairs; Constructing the modal pair dynamic time shift matrix based on the modal pair time shift mean ; Calculate the modal pair mismatch uncertainty: , where UMij is the modal pair mismatch uncertainty; Fit the time series disturbance response curve of each modal dynamic characteristic on the time axis: , where RDi(t) is the disturbance response value of the ith mode at time t, is the time basis function, is the fitting coefficient of the mth time basis function of mode i; Compute the mode-pair perturbation residuals: , where ERij(t) is the residual error of the mode to disturbance, and RDj(t) is the disturbance response value of the jth mode at time t; Calculate the modal pair mismatch coefficient: , where SPij is the modal pair mismatch coefficient, rdc is the disturbance residual mean, a1 and a2 represent the preset proportional coefficients of the modal pair time offset mean and the disturbance residual mean, respectively, and a1 and a2 are both greater than 0; the calculation formula for the disturbance residual mean is as follows , where TW is the time window width, t ∈ [t0,t0 + TW]; Calculate the multimodal asynchronous mismatch coefficient: , where Nasy is the multi-modal asynchronous mismatch coefficient.

3. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 2, characterized in that: In step S3, the acquired modal data is input into the pre-trained deep neural network to extract the feature embedding vector of each modality. , where QZi is the feature embedding vector of the i-th modality, tzi m is the m-th dimension feature of the feature embedding vector of the i-th modality, and M is the total dimension; calculate the feature cosine similarity simij of modality i and modality j: simij = , where QZj is the feature embedding vector of the jth modality; according to the feature cosine similarity between each modality, a cross-modal similarity matrix is ​​constructed .

4. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 3 is characterized by: The cross-modal similarity matrix and the modal feature saliency conflict region are jointly modeled to calculate the cross-modal feature conflict enhancement coefficient, as follows: For each modality i, the saliency branch in the pre-trained deep neural network is used to generate a saliency feature mask vector YCi: YCi = [gl im] = [gl i 1 , gl i 2 , . . . , gl iM], where gl im is the saliency probability of the m-th dimension feature of the i-th modality: Calculate the saliency conflict vector YCi cj onfict of modality i and modality j: YCi cj onfict = |YCi − YCj|, where YCj is the saliency feature mask vector of the j-th modality; Calculate the characteristic energy conflict vector ,in Represents dimension-wise element-wise multiplication, TZHij is the characteristic square sum vector of the corresponding dimension of mode i and mode j; Calculate the characteristic energy conflict mean ,in is the characteristic energy conflict vector The actual value of the m-th dimension element in; Calculate the cumulative potential energy of the conflict , where ERij(t) is the residual error of the mode pair disturbance, TW is the time window width, ; Calculate dynamic interaction attention weights: ,in is the dynamic interaction attention weight of modality i and modality j, is the feature cosine similarity of the modality pair (u, v), is the mean of the disturbance residuals of mode i and mode j, is the mean of the disturbance residuals of the mode pair (u, v); Calculate the cross-modal feature conflict enhancement coefficient: , where Cend is the cross-modal feature conflict enhancement coefficient.

5. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 4 is characterized by: A multimodal fusion risk assessment model is constructed based on the multimodal asynchronous mismatch coefficient and the cross-modal feature conflict enhancement coefficient, and the multimodal fusion risk assessment index is output.

6. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 5, characterized in that: The multimodal fusion risk assessment index is compared with the preset multimodal fusion risk assessment index threshold to evaluate the current multimodal fusion risk level, as follows: If the multimodal fusion risk assessment index is greater than the multimodal fusion risk assessment index threshold, the current multimodal fusion risk level is marked as high multimodal fusion risk; If the multimodal fusion risk assessment index is less than or equal to the multimodal fusion risk assessment index threshold, it means that the current multimodal fusion risk level is marked as low multimodal fusion risk.

7. The method for intelligently identifying living objects based on multimodal heterogeneous data collaboration according to claim 6, characterized in that: In step S5, the fusion weights of each modality are dynamically adjusted according to the current multimodal fusion risk level, as follows: If the current multimodal fusion risk level is marked as high, the fusion weights of each modality are dynamically adjusted according to the multimodal fusion risk assessment index output by the current multimodal fusion risk assessment model: ,in is the fusion weight after dynamic adjustment of mode i, is the initial fusion weight of modality i.

Images