Multi-modal behavior recognition method based on modal credibility scheduling and conflict reconstruction

Abstract

This invention discloses a multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction, comprising the following steps: acquiring video and skeletal modal data of behavior samples, completing time alignment and standardization preprocessing to obtain time-synchronized bimodal input tensors; feature encoding the bimodal input tensors respectively to obtain global semantic features and behavior category prediction probability distributions for each modality; calculating the credibility weights of video and skeletal modalities based on the uncertainty of the prediction probability distributions; calculating the comprehensive conflict intensity of the bimodal prediction distributions to generate continuous conflict adjustment factors; modulating the credibility weights using the conflict adjustment factors to complete the adaptive fusion of bimodal probabilities, and outputting the final behavior recognition result. This method solves the technical problems in existing multimodal behavior recognition technologies, such as the lack of explicit quantitative modeling of modal reliability and the lack of active intervention mechanisms for cross-modal prediction conflicts, leading to insufficient recognition accuracy and system stability in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically involving a multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction. Background Technology

[0002] Human behavior recognition is one of the core research directions in the fields of computer vision and pattern recognition, and it is widely used in many scenarios such as intelligent security, human-computer interaction, health monitoring, and sports motion analysis. With the rapid development of deep learning technology, human behavior recognition technology based on multimodal information fusion has gradually become the mainstream technology path in this field due to its comprehensive ability to characterize behavioral features. In practical engineering applications, single-modal behavior recognition schemes are difficult to adapt to complex and ever-changing application environments and have inherent performance bottlenecks: recognition methods based solely on RGB video image modalities are easily affected by factors such as sudden changes in illumination, complex background interference, camera viewpoint shifts, and human target occlusion, resulting in insufficient feature robustness; recognition methods based solely on human skeleton modalities are easily affected by problems such as key point detection errors, pose estimation fluctuations, and missing joint data, leading to unstable behavioral feature representation and difficulty in ensuring consistent recognition accuracy.

[0003] To address the technical bottlenecks of single-modal approaches, the industry commonly employs multimodal fusion schemes that combine video and skeletal modalities. These schemes integrate multi-dimensional information through feature stitching, weighted fusion, and attention-based fusion mechanisms, aiming to leverage the complementary features between different modalities to improve recognition performance. However, in real-world, complex scenarios, existing multimodal fusion schemes suffer from three core technical shortcomings, failing to meet the demands for high stability and accuracy: First, they lack explicit quantitative modeling of modal reliability, making it impossible to avoid interference from low-quality features. Most existing schemes assume stable input data quality and constant feature representation capabilities for each modality. However, in real-world scenarios, the representation capabilities of each modality fluctuate dynamically due to environmental interference. When the data quality of a particular modality deteriorates due to environmental interference, its noisy feature information still participates in the fusion decision process, easily negatively impacting the final recognition result and leading to insufficient system robustness. Second, they lack explicit detection and reconstruction mechanisms for cross-modal prediction conflicts, making them prone to error aggregation. When significant semantic discrepancies arise in the prediction results of different modalities for the same behavioral sample, most existing solutions rely on implicit weight adjustments based on network structure or employ simple weighted averaging strategies. These solutions fail to proactively identify and intervene in cross-modal conflicts, and the fusion process may even amplify the prediction errors of a single modality, reducing the stability and reliability of the system's decisions. Third, the dynamic adaptability of fusion strategies is insufficient, resulting in limited adaptability to complex scenarios. Most existing fusion solutions use fixed network structures or weight allocation strategies, failing to dynamically select the dominant modality based on the real-time characteristics of the input data. This makes it difficult to adapt to actual working conditions involving fluctuating data quality and changing scene environments, leading to a significant decrease in recognition accuracy and generalization ability under complex interference environments.

[0004] In summary, there is an urgent need in the industry for a multimodal behavior recognition method that can dynamically quantify and evaluate modal credibility and explicitly intervene in and reconstruct cross-modal prediction conflicts, in order to address the aforementioned shortcomings of existing technologies and improve the recognition accuracy and operational stability of multimodal behavior recognition systems in complex environments. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal behavior recognition method based on modal reliability scheduling and conflict reconstruction, which solves the technical problems of insufficient recognition accuracy and system stability in complex scenarios caused by the lack of explicit quantitative modeling of modal reliability, lack of active intervention mechanism for cross-modal prediction conflict, and insufficient dynamic adaptation capability of fusion strategy in existing multimodal behavior recognition technologies.

[0006] The technical solution adopted in this invention is a multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction, comprising the following steps:

[0007] S1: Acquire video and skeletal modal data of behavioral samples, complete time alignment and standardization preprocessing, and obtain time-synchronized bimodal input tensors; S2: Encode the features of the dual-modal input tensor separately to obtain the global semantic features and behavior category prediction probability distribution of each modality; S3: Calculate the credibility weights of video and skeletal modality based on the uncertainty of the predicted probability distribution; S4: Calculate the comprehensive conflict intensity of the bimodal prediction distribution and generate a continuous conflict adjustment factor; S5: Modulate the credibility weight using the conflict adjustment factor to complete the dual-modal probability adaptive fusion and output the final behavior recognition result.

[0008] The invention is further characterized by: S1 specifically includes: S1.1: Modeling video modal data, the original video sequence consists of T frames, represented as follows: ,in: For the RGB image of frame t, , The image height and width are represented by 3, and the number of color channels is 3; after uniform length processing: a fixed time length is set. , Equal-interval sampling is used. The normalized video input tensor is obtained by repeatedly interpolating frames. ; S1.2: Modeling the skeletal modal data. The original skeletal sequence consists of T frames of 3D keypoints, represented as follows: ,in: Let J be the 3D coordinates of the Jth skeletal keypoint in frame t; after uniform length processing aligned with the video modal time, a standardized skeletal input tensor is obtained. ; S1.3: Dual-mode time synchronization is achieved through a time index mapping function, which is: ,in: Represents a standardized time index. To perform the floor function, a mapping function is used to map the original video frames and skeleton frames to a unified time scale, ensuring that the bimodal data correspond one-to-one in the time dimension.

[0009] S2 specifically includes: S2.1: Perform feature encoding on the video modalities, and define the video modal feature extraction function. ,in, The video feature vector dimension is used to extract global semantic features of the video modality. ; S2.2: Perform feature encoding on the skeletal modalities. Let the skeletal modal feature extraction function be... ,in, Using the skeletal feature vector dimension, global semantic features of the skeletal modality are obtained through feature extraction. ; S2.3: The classification score vector for the corresponding modality is obtained through linear mapping: , ,in, , For the corresponding modality classifier parameter matrix, For the corresponding bias vector The total number of behavior categories; S2.4: The classification score vector is mapped to a normalized predicted probability distribution using the Softmax function. The calculation formula is as follows: ,in, Modal identifier, corresponding to the video modality or skeletal modality , The corresponding modal prediction is the first The probabilities of class behaviors are ultimately used to obtain video modal probability vectors that satisfy normalization constraints. Skeletal modal probability vector ,and .

[0010] In S2, the video modality uses a 3D convolutional network for spatiotemporal feature extraction and encoding, while the skeletal modality uses a temporal graph structure network for spatial topology and temporal dynamic modeling. The feature encoding networks of the two modalities are trained independently until convergence, and then connected to the credibility calculation and conflict adjustment fusion module for joint training.

[0011] S3 specifically includes: S3.1: Information entropy is used to characterize the uncertainty of the prediction probability distribution of each modality. The formula for calculating the entropy value is: ,in, For each modal prediction distribution, the entropy value is used to calculate the video modal uncertainty. Skeletal modal uncertainty The lower the entropy value, the higher the certainty of modal prediction; S3.2: Construct an inverse entropy index that is inversely proportional to the entropy value. The calculation formula is as follows: , where ε is a small positive constant to prevent the denominator from being zero; S3.3: Normalize the anti-entropy index to obtain the dual-modal confidence weight. ,in, , These are the credibility weights for video and skeletal modalities, respectively, satisfying... .

[0012] The calculation of the overall conflict intensity in S4 specifically includes: S4.1 Calculate the overall bias of the bimodal prediction distribution using the following formula: ,in, The overall deviation between the two modal prediction distributions, with a value range of [value missing]. ; S4.2: Construct a primary category consistency enhancement term to determine the primary prediction category for the bimodal model: ,in, , These are the category indices with the highest prediction probabilities for video and skeletal modalities, respectively; a principal category conflict indicator function is constructed as follows: ; S4.3 Combines the overall deviation and the main category conflict term to obtain the comprehensive conflict intensity: ,in, This is a comprehensive conflict intensity value. Primary category conflict enhancement coefficient; S4.4: Set the conflict triggering threshold τ and construct a binary conflict determination function: ,in, In a low-conflict state, This is a state of high conflict.

[0013] In S4, the integrated conflict intensity is mapped to a continuously differentiable conflict adjustment factor using the Sigmoid function. The calculation formula is as follows: ,in, As a conflict moderating factor, For the Sigmoid function, This is the slope control parameter. The conflict trigger threshold must be met. ;when hour This corresponds to a low-conflict state; when hour This corresponds to a high-conflict state; slope parameter Conflict trigger threshold Primary category conflict enhancement coefficient All were determined through validation set optimization.

[0014] In S5, the credibility weight is modulated using a conflict adjustment factor to obtain the modulated effective weight. The calculation formula is as follows: , ,in, , The effective weights for the modulated video and the skeletal modality are respectively, satisfying... ;when When the effective weights approach the original credibility weights; when At that time, the exponential enhancement effect amplifies the weight of the high-confidence mode, strengthening its decision-making dominance.

[0015] In S5, dual-modal probability adaptive fusion is performed based on the modulated effective weights, and the calculation formula is as follows: ,in, The prediction after fusion is the first The final probability of class behavior, the fusion probability satisfies the normalization constraint. The final behavior recognition result is the category corresponding to the maximum fusion probability, i.e.: ,in, The predicted behavior category output by the system.

[0016] During the training phase, the method uses cross-entropy loss function for supervision, and the optimizer is stochastic gradient descent. The initial learning rate is set to 0.001, and a segmented decay strategy is used for updating. The batch size is set to 128, and the total number of training rounds is set to 80. During the inference phase, Top-1 accuracy and Top-5 accuracy are used as the core evaluation metrics.

[0017] The beneficial effects of this invention are: This invention constructs a dynamic evaluation and scheduling mechanism for modal credibility based on the uncertainty of prediction distribution. This mechanism achieves sample-level adaptive quantization modeling of the reliability of video and skeletal modalities. It can automatically reduce the impact of single-modal interference from factors such as illumination changes, target occlusion, and keypoint detection errors on the final decision, effectively avoiding the negative impact of low-quality features on the recognition results and significantly improving the robustness and stability of the system in complex environments. At the same time, by designing an explicit detection and continuous reconstruction mechanism for cross-modal prediction conflicts, it can proactively identify semantic prediction discrepancies between bimodalities and specifically strengthen the decision-making dominance of the high-credibility modality. This avoids the error superposition problem that is prone to occur in traditional fixed-weight fusion, ensuring the consistency and reliability of the final decision results. Moreover, the entire mechanism does not rely on a specific feature extraction network model, has excellent versatility and scalability, and can be adapted to various mainstream behavior recognition frameworks. Without significantly increasing the model complexity, it stably improves the overall recognition accuracy and adaptability to complex scenarios of multimodal behavior recognition, and has extremely high engineering application value. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction of the present invention. Figure 2 This is a block diagram of the adaptive fusion structure in the multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction of the present invention; Figure 3This is a schematic diagram of the conflict adjustment mechanism in the multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Example 1 like Figure 1-3 As shown ( Figure 1 It demonstrates the overall processing flow from multimodal data input, feature extraction, adaptive fusion to behavior recognition result output. Figure 2 This represents the weight allocation and feature fusion structure of visual modality and skeletal modality features in the fusion module. Figure 3 This refers to the process of adaptively adjusting the fusion result through conflict detection and weight adjustment mechanisms when there are differences in the recognition results of different modalities. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction disclosed in this embodiment includes the following steps: S1: Acquire video and skeletal modal data of behavioral samples, complete time alignment and standardization preprocessing, and obtain time-synchronized bimodal input tensors; S2: Encode the features of the dual-modal input tensor separately to obtain the global semantic features and behavior category prediction probability distribution of each modality; S3: Calculate the credibility weights of video and skeletal modality based on the uncertainty of the predicted probability distribution; S4: Calculate the comprehensive conflict intensity of the bimodal prediction distribution and generate a continuous conflict adjustment factor; S5: Modulate the credibility weight using the conflict adjustment factor to complete the dual-modal probability adaptive fusion and output the final behavior recognition result.

[0021] Example 2 Based on Example 1, S1 specifically includes: In this embodiment, the publicly available large-scale multimodal behavior recognition dataset NTURGB+D is used as the training and validation data source. This dataset contains RGB video sequences and corresponding human 3D skeletal keypoint data, and is suitable for multimodal behavior recognition tasks.

[0022] Let a single behavioral sample selected from the dataset be represented as:

[0023] in, Represents video modal data, Represents skeletal modal data, This represents the corresponding behavior category label. (Variable) ,in This indicates the total number of behavior categories.

[0024] S1.1: For video modalities, assuming the original video sequence consists of T frames, the video data is represented as follows:

[0025] in, This represents the RGB image of frame t, where H and W represent the height and width of the image, respectively, and 3 represents the number of color channels.

[0026] To ensure consistency in input dimensions, the original video sequences are processed to have a uniform length. Let the fixed time length be... When the original number of frames Frame selection is performed using an equal-interval sampling strategy; when The length is extended using repeated frame interpolation. The processed video tensor representation is as follows:

[0027] in, This represents the standardized video input tensor; S1.2: The NTURGB+D dataset simultaneously provides 3D coordinate information of human keypoints. Assuming J keypoints are extracted from each frame, the skeletal modality sequence is represented as:

[0028] in, , represents the set of three-dimensional coordinates of the skeletal keypoints in frame t; J is the number of keypoints, and 3 represents the three-dimensional spatial coordinates (x,y,z).

[0029] Similarly, to ensure consistency between the temporal dimension and the video modality, the skeletal sequence is temporally aligned to obtain a uniform length representation:

[0030] in, This represents the time-aligned skeletal input tensor. S1.3: To ensure strict consistency between video modalities and skeletal modalities in the time dimension, a time index mapping function is constructed:

[0031] in, Represents a standardized time index. Indicates the length of the original sequence, symbol This indicates a round-down operation.

[0032] Through mapping function (t)\phi(t) (t) can map the original video frames and skeleton frames to a unified time scale, thereby ensuring:

[0033] This indicates a one-to-one correspondence between two modes in the time dimension.

[0034] S1.4: After the above processing, each behavioral sample can be uniformly represented as:

[0035] in, Input tensors for the video. The skeleton is input tensor, and y is the behavior category label. This unified representation provides a standardized input basis for subsequent modality feature extraction, credibility modeling, and conflict detection mechanisms.

[0036] Example 3 Based on Example 1, S2 specifically includes: After obtaining bimodal input samples of uniform length, feature encoding is performed on the video modality and the skeletal modality respectively, and corresponding initial classification probability distributions are generated, providing a foundation for subsequent credibility modeling and conflict detection.

[0037] S2.1: Let the video modal feature extraction function be:

[0038] Among them, the function This represents a video feature coding network. This represents the dimension of the video feature vector.

[0039] For input video tensor Feature extraction is performed to obtain the global semantic representation of the video:

[0040] in, A high-dimensional semantic feature vector representing a video modality.

[0041] To achieve behavior category prediction, a video modality classifier parameter matrix is ​​introduced. With bias vector Construct a linear mapping:

[0042] in, This represents the classification score vector for the video modality.

[0043] The scores are mapped to a probability distribution using the Softmax function:

[0044] in, Indicates that the video modality prediction is the first The probability of class behavior, This indicates the total number of behavior categories.

[0045] The final video modal probability vector is obtained as follows:

[0046] Where, vector And satisfy:

[0047] S2.2: Let the skeletal modal feature extraction function be:

[0048] Among them, the function This represents a skeletal temporal feature encoding network. This represents the dimension of the skeletal feature vector.

[0049] Input tensors to the skeleton Encoding is performed to obtain the semantic representation of the skeletal modality:

[0050] in, Represents the global feature vector of the skeletal modality.

[0051] Similarly, construct the parameter matrix for the skeletal modality classifier. With bias vector The skeletal classification score is obtained as follows:

[0052] in, This represents the classification score vector for the skeletal modality.

[0053] The skeletal modality probability distribution is obtained through Softmax mapping:

[0054] in, Indicates that the skeletal modality prediction is the first The probability of class behavior.

[0055] The final skeletal modality probability vector is obtained as follows:

[0056] Where, vector And satisfy:

[0057] S2.3: After processing in S2, each behavioral sample yields the following intermediate variable: video modal feature vector. Skeletal modal feature vectors Video modality prediction probability vector Skeletal modality prediction probability vector The above variables form the input basis for subsequent modal credibility calculation and conflict detection.

[0058] In the feature encoding stage, the video modality employs a 3D convolutional network structure for spatiotemporal feature extraction, while the skeletal modality uses a temporal graph network structure for spatial topology and temporal dynamic modeling. After the two modality networks are trained independently until convergence, they are then jointly trained using the credibility calculation and conflict mitigation fusion module proposed in this invention.

[0059] Example 4 Based on Example 1, S3 specifically includes: The video modal probability distribution has been obtained in S2. With skeletal modality probability distribution Although both modalities output normalized probabilities, their ability to discriminate the same behavioral sample varies significantly in complex real-world scenarios. Therefore, a quantifiable reliability evaluation mechanism is needed to measure the reliability of each modality's prediction results in the current sample.

[0060] S3.1: For any modality prediction probability vector Its uncertainty can be characterized by information entropy:

[0061] in, This represents the entropy value of the predicted distribution of this mode.

[0062] Information entropy reflects the degree of dispersion of a probability distribution. When the probability of a certain category is close to 1, the entropy value approaches 0, indicating that the prediction is highly certain; when the probabilities of each category are close to a uniform distribution, the entropy value reaches its maximum value, indicating that the prediction is highly uncertain.

[0063] Therefore, the calculations are performed separately for the video modality and the skeletal modality:

[0064] in, and These represent the prediction uncertainties for the video modality and the skeletal modality, respectively.

[0065] Step 3.2, Modal credibility normalization construction, Since a higher entropy value indicates greater uncertainty, credibility should be inversely proportional to the entropy value. To achieve comparability between different modes, an inverse entropy index is constructed:

[0066] in, To prevent small positive constants with a denominator of zero.

[0067] The anti-entropy index was then normalized to obtain the modal confidence weights:

[0068] in, The credibility weights of the video modalities are represented. Represents the credibility weight of the skeletal modality, and satisfies Through the above construction, sample-level adaptive confidence allocation can be achieved. When the prediction of a certain modality becomes more concentrated (lower entropy), its weight will automatically increase.

[0069] S3.3: After processing in S3, each sample has an additional variable based on the original variables: modal uncertainty. , Modal credibility weights , Thus, the bimodal system no longer uses fixed-weight fusion, but establishes a sample adaptive credibility allocation mechanism, providing a theoretical basis for subsequent conflict detection and dynamic fusion.

[0070] Example 5 Based on Example 1, the modal confidence weights have been obtained in S3. , However, relying solely on credibility assignment is insufficient to address the severe semantic discrepancies in bimodal prediction results. Therefore, a modal prediction conflict measurement mechanism is needed to determine whether structural contradictions exist between the two modal outputs.

[0071] This step continuously characterizes the conflict from the perspective of probability distribution and constructs the conflict triggering function.

[0072] S4.1: Given the video modal probability vector With skeletal modal probability vector The difference in distribution between the two is defined as:

[0073] in, It indicates the overall degree of deviation between the two modal prediction distributions.

[0074] This metric directly characterizes the sum of the absolute differences between two probability distributions across each category dimension. When the two modalities are highly consistent in their classifications of each category, this metric is effective. The value is relatively small; when the prediction directions of the two modalities are significantly different or even the main classes are completely inconsistent. Significantly increased.

[0075] Since each probability vector satisfies the normalization constraint, we know that:

[0076] therefore, It can be used as a measure of conflict intensity under a unified scale.

[0077] S4.2: Using only the overall distribution difference may not adequately reflect the conflict in main category predictions. To further enhance sensitivity to core category disagreements, a main category prediction index is introduced:

[0078] in, and These represent the category indices with the highest prediction probabilities for the video modality and the skeletal modality, respectively.

[0079] Construct the main category conflict indicator function:

[0080] When the main predicted classes of the two modalities are consistent =0; when the main category diverges. .

[0081] S4.3: Overall distribution differences Conflict with the main category Perform continuous fusion to construct a comprehensive conflict intensity function:

[0082] in, This indicates the final conflict intensity value. The primary category conflict enhancement coefficient.

[0083] When the overall difference between the two modalities is small and their main categories are consistent At a low level; when overall differences increase or main categories conflict. A significant increase.

[0084] S4.4: Set the conflict trigger threshold Construct a conflict determination function:

[0085] in, This indicates that the current sample is in a low-collision state. This indicates that the current sample is in a high-conflict state. Thus, the calculation of continuous conflict intensity based on probability distribution has been achieved, and the conflict state classification has been completed through a threshold mechanism.

[0086] Example 6 Based on Example 1, modal confidence weights have been obtained in S3. and In step 4, the conflict determination variables are obtained. This step constructs a unified fusion function, enabling the system to stably output the final prediction result under both low-conflict and high-conflict states.

[0087] To avoid the discontinuity problem caused by discrete piecewise functions, the conflict intensity is first determined. The mapping is a continuous adjustment factor.

[0088] S5.1: Define the conflict resolution function:

[0089] in, For the Sigmoid function, This is the slope control parameter. The conflict threshold defined in S4, To assess the overall intensity of the conflict.

[0090] From the properties of the Sigmoid function, we know that:

[0091] when hour, This indicates that the system is in a low-conflict state. hour, This indicates that the system has entered a high-conflict state. Therefore, It can be viewed as a weight of the degree of conflict continuity, rather than a simple binary variable.

[0092] S5.2: In low-conflict situations, the fusion should be mainly based on the credibility weights obtained in S3; while in high-conflict situations, the dominant role of the high-credibility modality should be strengthened.

[0093] Therefore, the effective weights after conflict modulation are constructed as follows:

[0094]

[0095] As can be seen, when hour,

[0096] when At that time, the exponential enhancement effect amplifies the larger weights, giving the more credible modal a stronger dominant position.

[0097] Therefore, this structure implements a conflict-driven adaptive weight reinforcement mechanism.

[0098] S5.3: Based on the modulated effective weights, perform weighted fusion of the probability distributions of the two modes:

[0099] in, This represents the final probability of being predicted as class i after fusion. , These are the modal prediction probabilities obtained in S2.

[0100] because And since the probabilities of each mode are normalized, we can obtain:

[0101] The final classification result is defined as:

[0102] in, The predicted behavior category output by the system.

[0103] Based on the continuous construction from S1 to S5, this invention has formed a complete bimodal adaptive fusion decision mechanism. For any input sample, its final fusion probability can be uniformly expressed as:

[0104] Among them, the fusion weight and The decision function is no longer a fixed constant, but is jointly determined by the uncertainty of modal prediction and the intensity of conflict between modalities. The higher the degree of conflict, the more obvious the weighting reinforcement effect, and the stronger the dominance of the more credible modality in the final decision. Since the conflict adjustment process is constructed using a continuous mapping method, the entire decision function remains mathematically smooth and differentiable, and there is no discrete jump problem, thus ensuring the stability of the system during the training and inference phases.

[0105] In low-conflict scenarios, the bimodal prediction distributions tend to be consistent, the conflict adjustment factor automatically weakens, and the system degenerates into a confidence-based stable weighted fusion mode, which can fully utilize the complementary information of the two modalities. In high-conflict scenarios, the system automatically amplifies the weight ratio of the high-confidence modality and suppresses the interference of the low-confidence modality on the final result, enabling the model to maintain stable output even in complex scenes, occluded environments, or local modal distortion. Compared with traditional fixed-weight fusion methods, the weights in this method are dynamically generated from the sample-level prediction structure, achieving real-time perception and control of modal divergence. This significantly enhances the robustness and generalization ability of the bimodal behavior recognition system in complex environments and forms a complete closed-loop decision-making mechanism from probability distribution analysis and conflict detection to weight modulation.

[0106] During training, cross-entropy loss is used for supervision, and stochastic gradient descent is selected as the optimizer. The initial learning rate is set to 0.001, and a piecewise decay strategy is used for updates. The batch size is set to 128, and the total number of training epochs is set to 80. The slope parameter κ and the conflict threshold τ in the conflict adjustment function are determined through optimization using the validation set. During inference, Top-1 accuracy and Top-5 accuracy are used as the main evaluation metrics. Confidence calculation and conflict adjustment weights are adjusted based on the predicted probability distribution.

[0107] The technical effects of this invention were verified through the following experiments: Experimental environment: Hardware NVIDIA 3060 GPU, software PyTorch 2.0, CUDA 12.0.

[0108] To verify the effectiveness of this method, experiments were conducted on the publicly available NTURGB+D dataset. The standard Cross-Subject partitioning protocol was used for training and testing to ensure comparability of results. This experimental configuration ensures stable model training, and the fusion module can be directly embedded into existing bimodal recognition frameworks.

[0109] To verify the performance improvement of this method, it is compared with the following three benchmark methods: single-video modality recognition method, single-skeleton modality recognition method, and fixed-weight bimodal fusion method. The fixed-weight fusion method uses an equal-weighted summation of probabilities.

[0110] As shown in Table 1 (accuracy on the NTU RGB-D60 dataset), the experimental results indicate that, under standard test partitioning, the accuracy of single-video modality recognition is lower than that of dual-modality fusion, and single-skeleton modality also has certain limitations. The fixed-weight fusion method shows improvement compared to single-modality, but significant misclassification still exists in some samples. In contrast, our proposed method further improves the recognition accuracy on the overall test set and demonstrates a more significant advantage in high-conflict sample subsets.

[0111] Table 1

[0112] Further analysis revealed that when the prediction results of the two modalities are highly consistent, the output of this method remains largely consistent with the fixed fusion method, without introducing additional fluctuations. However, when there are significant discrepancies between the prediction results of the two modalities, this method can automatically strengthen the weight of the high-confidence modality, thereby reducing the probability of misjudgment caused by erroneous fusion. Therefore, in samples with complex conditions including occlusion, viewpoint changes, and skeleton detection errors, this method exhibits stronger stability and robustness.

[0113] The comprehensive experimental results show that this method achieves dynamic perception and adaptive weight adjustment of modal conflicts without significantly increasing model complexity. Compared with the traditional fixed fusion strategy, it achieves a stable improvement in both overall recognition accuracy and robustness in complex scenarios.

[0114] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction, characterized in that, Includes the following steps: S1: Acquire video and skeletal modal data of behavioral samples, complete time alignment and standardization preprocessing, and obtain time-synchronized bimodal input tensors; S2: Encode the features of the dual-modal input tensor separately to obtain the global semantic features and behavior category prediction probability distribution of each modality; S3: Calculate the credibility weights of video and skeletal modality based on the uncertainty of the predicted probability distribution; S4: Calculate the comprehensive conflict intensity of the bimodal prediction distribution and generate a continuous conflict adjustment factor; S5: Modulate the credibility weight using the conflict adjustment factor to complete the dual-modal probability adaptive fusion and output the final behavior recognition result.

2. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 1, characterized in that, S1 specifically includes: S1.1: Modeling video modal data, the original video sequence consists of T frames, represented as follows: ,in: For the RGB image of frame t, , The image height and width are represented by 3, and the number of color channels is 3; after uniform length processing: a fixed time length is set. , Equal-interval sampling is used. The normalized video input tensor is obtained by repeatedly interpolating frames. ; S1.2: Modeling the skeletal modal data. The original skeletal sequence consists of T frames of 3D keypoints, represented as follows: ,in: Let J be the 3D coordinates of the J-th skeletal keypoint in frame t; after uniform length processing aligned with the video modal time, a standardized skeletal input tensor is obtained. ; S1.3: Dual-mode time synchronization is achieved through a time index mapping function, wherein the mapping function is: ,in: Represents a standardized time index. To perform the floor function, a mapping function is used to map the original video frames and skeleton frames to a unified time scale, ensuring that the bimodal data correspond one-to-one in the time dimension.

3. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 1, characterized in that, S2 specifically includes: S2.1: Perform feature encoding on the video modalities, and define the video modal feature extraction function. ,in, The video feature vector dimension is used to extract global semantic features of the video modality. ; S2.2: Perform feature encoding on the skeletal modalities. Let the skeletal modal feature extraction function be... ,in, Using the skeletal feature vector dimension, global semantic features of the skeletal modality are obtained through feature extraction. ; S2.3: The classification score vector for the corresponding modality is obtained through linear mapping: , ,in, , For the corresponding modality classifier parameter matrix, For the corresponding bias vector The total number of behavior categories; S2.4: The classification score vector is mapped to a normalized predicted probability distribution using the Softmax function. The calculation formula is as follows: ,in, Modal identifier, corresponding to the video modality or skeletal modality , The corresponding modal prediction is the first The probabilities of class behaviors are ultimately used to obtain video modal probability vectors that satisfy normalization constraints. Skeletal modal probability vector ,and .

4. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 3, characterized in that, In S2, the video modality uses a three-dimensional convolutional network for spatiotemporal feature extraction and encoding, while the skeletal modality uses a temporal graph structure network for spatial topology and temporal dynamic modeling. The feature encoding networks of the two modalities are trained independently until convergence, and then connected to the credibility calculation and conflict adjustment fusion module for joint training.

5. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 1, characterized in that, S3 specifically includes: S3.1: Information entropy is used to characterize the uncertainty of the prediction probability distribution of each modality. The formula for calculating the entropy value is: ,in, For each modal prediction distribution, the entropy value is used to calculate the video modal uncertainty. Skeletal modal uncertainty The lower the entropy value, the higher the certainty of modal prediction; S3.2: Construct an inverse entropy index that is inversely proportional to the entropy value. The calculation formula is as follows: , where ε is a small positive constant to prevent the denominator from being zero; S3.3: Normalize the anti-entropy index to obtain the dual-modal confidence weight. ,in, , These are the credibility weights for video and skeletal modalities, respectively, satisfying... .

6. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 1, characterized in that, The calculation of the overall conflict intensity in S4 specifically includes: S4.1 Calculate the overall bias of the bimodal prediction distribution using the following formula: ,in, The overall deviation between the two modal prediction distributions, with a value range of [value missing]. ; S4.2: Construct a primary category consistency enhancement term to determine the primary prediction category for the bimodal model: ,in, , These are the category indices with the highest prediction probabilities for video and skeletal modalities, respectively; a principal category conflict indicator function is constructed as follows: ; S4.3 Combines the overall deviation and the main category conflict term to obtain the comprehensive conflict intensity: ,in, This is a comprehensive conflict intensity value. Primary category conflict enhancement coefficient; S4.4: Set the conflict triggering threshold τ and construct a binary conflict determination function: ,in, In a low-conflict state, This is a state of high conflict.

7. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 6, characterized in that, In step S4, the overall conflict intensity is mapped to a continuously differentiable conflict adjustment factor using the Sigmoid function, and the calculation formula is as follows: ,in, As a conflict moderating factor, For the Sigmoid function, This is the slope control parameter. The conflict trigger threshold must be met. ;when hour This corresponds to a low-conflict state; when hour This corresponds to a high-conflict state; the slope parameter Conflict trigger threshold Primary category conflict enhancement coefficient All were determined through validation set optimization.

8. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 7, characterized in that, In step S5, the credibility weight is modulated using a conflict adjustment factor to obtain the modulated effective weight, calculated using the following formula: , ,in, , The effective weights for the modulated video and the skeletal modality are respectively, satisfying... ;when When the effective weights approach the original credibility weights; when At that time, the exponential enhancement effect amplifies the weight of the high-confidence mode, strengthening its decision-making dominance.

9. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 8, characterized in that, In step S5, dual-modal probability adaptive fusion is performed based on the modulated effective weights, and the calculation formula is as follows: ,in, The prediction after fusion is the first The final probability of class behavior, the fusion probability satisfies the normalization constraint. The final behavior recognition result is the category corresponding to the maximum fusion probability, i.e.: ,in, The behavior prediction category output by the system.

10. The multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction according to claim 1, characterized in that, During the training phase, the method uses cross-entropy loss function for supervision, and the optimizer is stochastic gradient descent. The initial learning rate is set to 0.001, and a segmented decay strategy is used for updating. The batch size is set to 128, and the total number of training rounds is set to 80. During the inference phase, Top-1 accuracy and Top-5 accuracy are used as the core evaluation metrics.

Images