Small sample action recognition method and system based on federated active inference

Abstract

The application provides a small sample action recognition method and system based on federal active reasoning, relates to the field of computer vision, and comprises the following steps: a client selects a reasoning dominant model of the current client through free energy calculation and uploads the reasoning dominant model to an edge server; the edge server performs global model aggregation according to a modal specific free energy and a reasoning dominant model set, trains the global model using a public data set, obtains a trained global model, and broadcasts the trained global model to all clients; a client uses the replaced model to perform client support set sample prototype calculation and client query set variational posterior probability calculation, calculates the confidence score of each modal of a client query set sample, calculates the fusion weight of different modes in the classification inference of the current sample, dynamically weights and fuses multi-modal information according to the modal confidence, and obtains a classification prediction result of the query sample. The application effectively improves the recognition accuracy of the multi-modal action recognition task.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a few-sample action recognition method and system based on federated active reasoning. Background Technology

[0002] Federated learning, as a distributed training paradigm focused on data privacy protection, has been widely applied in action recognition tasks. By training models locally on distributed terminals and aggregating them on a central server, federated learning enables collaborative modeling across multiple devices without transmitting raw data, effectively improving user data privacy and model generalization capabilities. Meanwhile, few-shot learning, by performing feature modeling and category generalization under conditions of minimal labeled samples, has become an important means to address the challenges of data acquisition and high labeling costs in action recognition. The combination of federated learning and few-shot learning holds promise for improving category classification capabilities under conditions of scarce samples within a distributed framework, constructing a distributed action recognition system that combines privacy protection, adaptability, and generalization capabilities. Existing federated multimodal action recognition methods generally employ a centralized modeling process, where each client extracts features and trains models from local multimodal data, then uploads the model parameters or gradients to a central server, which aggregates the models before distributing them back to the clients. Modality fusion is typically performed on the client side, using a unified structure to concatenate, weight, or perform attention calculations on multimodal features to complete classification inference. During training, the modalities used by the client are fixed, the aggregation method is uniform, and the parameters are synchronized between each communication round according to a preset frequency, ultimately forming a unified global model for inference.

[0003] In developing this invention, the applicant discovered that traditional Federated Multimodal Action Recognition (FMAR) methods face three key challenges in practical deployment. First, due to differences in client hardware capabilities, sensor types, and acquisition environments, the data modalities and quality are often inconsistent, leading to significant differences in inference performance across different clients for the same task, making unified modeling difficult. Second, existing federated learning aggregation strategies easily introduce low-quality model updates, reducing overall training efficiency and model generalization ability. Finally, in few-shot learning (FSL) scenarios, the extreme scarcity of labeled data further amplifies these problems, causing the trained model to exhibit unstable performance when faced with new data.

[0004] Therefore, there is an urgent need for an action recognition method that can solve the problems of modal inconsistency, aggregation failure and data scarcity in federated multimodal action recognition. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art or related technologies. It discloses a small sample action recognition method and system based on federated active reasoning, which solves the problems of large reliability differences between different modalities during client local model inference in multimodal action recognition tasks, noise amplification by traditional federated averaging aggregation, and failure of multimodal fusion strategies, thereby improving recognition accuracy.

[0006] The first aspect of the present invention discloses a small sample action recognition method based on federated active reasoning, comprising: the client performing free energy calculation in the active reasoning framework according to the variational posterior probability of the public query set sample to obtain the mode-specific free energy of each mode of the public query set sample, taking the mode-specific model corresponding to the mode with the smallest free energy value as the current client's inference dominant model, and uploading the inference dominant model and the mode-specific free energy of the corresponding mode to the edge server;

[0007] The calculation process for modal-specific free energy includes:

[0008] Let the variational posterior probability of the query sample be the client inference probability of the sample class by the client model, and the column vector of the observation likelihood matrix describe the ideal probability distribution that the client model should output when the true class of the sample is k. The prior probability is uniform, that is, it is assumed that all action classes have the same probability of occurrence.

[0009] The client infers probabilities based on the observed likelihood matrix and the client's inference probability. It extracts the column vector corresponding to the observed likelihood matrix as the ideal distribution based on the true category of the sample, where each element represents the ideal probability. For each category, it calculates the ratio of the client's probability to the ideal probability, takes the negative logarithm to obtain the single-class bias, and uses the client's probability as the weight to sum the bias values ​​of all categories to obtain the expected negative log-likelihood value.

[0010] Calculate the Kullback-Leibler divergence between the client's inferred probability and the prior probability to measure the difference between the client's inferred probability and the prior probability.

[0011] The client sums the expected negative log-likelihood value with the Kullback-Leibler divergence value to obtain the mode-specific free energy of the sample; the mode-specific free energy represents the mode-specific inference uncertainty of the current client model;

[0012] The edge server divides and aggregates the inference-dominant models according to modality, so as to aggregate models of the same modality into a set of inference-dominant models; based on the modality-specific free energy and the set of inference-dominant models uploaded by all clients, the modality-specific models that participate in the current global model aggregation and training are selected to obtain the aggregated global model;

[0013] The edge server uses a public dataset to train a global model, obtains the trained global model, and broadcasts it to all clients so that the clients can replace the model.

[0014] The client receives motion video data to construct a client support set and a client query set, and uses the replaced model to perform client support set sample prototype calculation and client query set variational posterior probability calculation respectively.

[0015] The client calculates the confidence score of each modality of the client query set sample based on the client support set sample prototype and the variational posterior probability of the client query set; based on the confidence score of each sample in different modalities, it calculates the fusion weight of different modalities in the classification inference of the current sample; after obtaining the fusion weight of different modalities, it dynamically weights and fuses multimodal information according to the modal confidence to obtain the classification prediction result of the query sample.

[0016] According to the small-sample action recognition method based on federated active reasoning disclosed in this invention, preferably, the calculation process of the variational posterior probability of the public query set samples specifically includes:

[0017] The client performs prototype network computation on samples of the public support set of the edge server to obtain the prototype of the public support set samples;

[0018] The client extracts feature vectors from the multimodal data in the public query set of the edge server using various modality-specific models to obtain the public query set sample feature vectors. Based on the public query set sample feature vectors and the public support set sample prototypes, the client calculates the variational posterior probability of the public query set samples: the Manhattan distance between each query set sample feature vector and the support sample prototype is calculated; the calculated Manhattan distance is negatively evaluated to obtain the inverse distance metric; this inverse distance metric is input into an exponential function for non-linear amplification, i.e., the logarithm of the inverse distance metric is taken; the amplified result is divided by the sum of the exponential function values ​​corresponding to all categories to complete the normalization process, thus obtaining the variational posterior probability of the query sample.

[0019] According to the few-sample action recognition method based on federated active reasoning disclosed in this invention, preferably, the prototype network computation specifically includes:

[0020] The client extracts corresponding samples from the public support set to form a category support subset based on the category of the target action;

[0021] The client calls the local modality-specific model to extract modality-specific feature vectors for samples in the category support subset on a sample-by-sample basis;

[0022] After obtaining the modality-specific feature vectors of all samples in the subset, the vectors are summed, and the total value is the modality prototype of the target action category.

[0023] According to the few-sample action recognition method based on federated active reasoning disclosed in this invention, preferably, the specific process of global model aggregation includes:

[0024] Let the sum of the free energies of all clients under the target mode m be E^m, and the sum of the free energies of the modes m' that are complementary to mode m be E^{m'}. The maximum number of clients allowed to participate in the aggregation in a single round of communication is L, and the subset of clients participating in the aggregation is {G'}^m.

[0025] The edge server allocates the client models participating in the aggregation based on the reliability ratio of the two modalities:

[0026] Calculate the client quota l^m for the target mode m. The value of l^m is the proportion of E^m to E^m+E^{m'}, multiplied by L and rounded up to ensure that the more reliable mode gets more aggregated resources.

[0027] After obtaining the client quota l^m of the target mode m, the difference between the upper limit of the total number of clients L and the client quota l^m of the target mode m is the client quota of the complementary mode m'.

[0028] From the set of inference-dominant models G^m, select the top l^m clients with the lowest average free energy to form a subset {G'}^m of clients participating in the aggregation;

[0029] The selected client model parameters are summed with equal weights and then divided by the number of clients to output the aggregated global model.

[0030] The method for few-sample action recognition based on federated active reasoning disclosed in this invention, preferably, includes the step of dynamically weighting and fusing multimodal information according to modal confidence, specifically comprising:

[0031] Let the modal feature vector of sample x_k be s_i^m, the prototype of the support set D_c^{sup} be φ'_k^m, and the feature similarity function be ρ(), which is used to calculate the Euclidean distance between the two vectors. The prediction result is Pre_c(x_k).

[0032] The client calculates the RGB modal contribution g^r using the following method:

[0033] Calculate the similarity ρ(s_i^r, φ'_k^r) between the RGB feature vector s_i^r and the RGB prototype φ'_k^r, and multiply it by the RGB fusion weight y_k^r to obtain the RGB modal contribution g^r;

[0034] The client calculates the optical flow modal contribution g^f using the following method:

[0035] Calculate the similarity ρ(s_i^f, φ'_k^f) between the optical flow feature vector s_i^f and the optical flow prototype φ'_k^f, and multiply it by the optical flow fusion weight y_k^f to obtain the optical flow mode contribution g^f;

[0036] The combined difference value of the target category is obtained by adding the optical flow modal contribution g^f and the RGB modal contribution g^r.

[0037] The negative of the comprehensive difference value is then used to perform a non-linear transformation through the natural exponential function to obtain the probability Pre_c(k|x_k) that sample x_k belongs to class k.

[0038] Repeat the above process for all action categories to obtain the probability distribution of sample x_k.

[0039] According to the federated active reasoning-based few-sample action recognition method disclosed in this invention, preferably, each client maintains multiple modality-specific models, and the edge server maintains a public dataset, which includes a public support set and a public query set; the samples in the public support set and the public query set are multimodal data corresponding to multiple modality-specific models.

[0040] According to the federated active reasoning-based few-sample action recognition method disclosed in this invention, preferably, the multimodal data includes RGB data and optical flow data of action videos, and the modality-specific model includes an optical flow model for extracting optical flow features and an RGB model for extracting RGB features.

[0041] A second aspect of the present invention discloses a few-shot action recognition system based on federated active reasoning, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory to implement the few-shot action recognition method based on federated active reasoning as described in any of the above technical solutions.

[0042] The technical solution provided by this invention can be applied to industries that require real-time motion recognition, such as autonomous driving, low-altitude economy, urban security, and virtual reality. Based on the above technical solution, the motions appearing in motion videos or motion images are classified and recognized.

[0043] The beneficial effects of this invention include at least the following: First, by employing a client-side modality selection mechanism driven by minimizing free energy, this invention selects only high-confidence modal models for aggregation in each round of communication, controlling model quality from the source, improving the system's adaptability to modal heterogeneity and data distribution differences, and enhancing the personalization and stability of inference. Second, by introducing a free energy-based uncertainty measurement mechanism, this invention adaptively adjusts aggregation weights based on the confidence level of the client model during the aggregation phase, effectively avoiding the negative impact of low-quality models. Third, this invention proposes a dynamically weighted active multimodal inference mechanism, adjusting the fusion weights in real time based on the posterior confidence level of each modality, enabling the inference process to have stronger suppression capabilities against noisy modalities, and improving the accuracy and reliability of the final decision. Fourth, the proposed end-edge collaborative optimization strategy, based on the joint optimization objectives of communication latency, computational consumption, and model quality, can effectively reduce the number of models and data volume in each round of communication. Attached Figure Description

[0044] Figure 1 A schematic diagram of the overall architecture of a few-sample action recognition method based on federated active reasoning according to an embodiment of the present invention is shown.

[0045] Figure 2 A schematic diagram illustrating the computation process of a public support set sample prototype according to an embodiment of the present invention is shown.

[0046] Figure 3 A schematic diagram illustrating the process of calculating the variational posterior probability of a public query set sample according to an embodiment of the present invention is shown.

[0047] Figure 4 A schematic diagram of the modal-specific free energy calculation process according to an embodiment of the present invention is shown.

[0048] Figure 5 A schematic diagram of the reasoning-driven model grouping process according to an embodiment of the present invention is shown.

[0049] Figure 6 A schematic diagram of the aggregation process of a global model according to an embodiment of the present invention is shown.

[0050] Figure 7 A schematic diagram illustrating the client support set sample prototype calculation process according to an embodiment of the present invention is shown.

[0051] Figure 8 A schematic diagram illustrating the process of calculating the variational posterior probability of a client query set according to an embodiment of the present invention is shown.

[0052] Figure 9 A schematic diagram illustrating the confidence score calculation process according to an embodiment of the present invention is shown.

[0053] Figure 10 A schematic diagram of the fusion weight calculation process according to an embodiment of the present invention is shown.

[0054] Figure 11 A schematic diagram illustrating the calculation process of classification prediction results according to an embodiment of the present invention is shown.

[0055] Figure 12 A schematic diagram illustrating the calculation process of a modal-specific free energy calculation module according to an embodiment of the present invention is shown.

[0056] Figure 13 A schematic block diagram of a small-sample action recognition system based on federated active reasoning according to an embodiment of the present invention is shown. Detailed Implementation

[0057] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0058] Definitions:

[0059] Federated Learning (FL) is a distributed machine learning technique that allows multiple data owners to build machine learning models without sharing their data. The goal of federated learning is to achieve collaborative modeling and improve the performance of artificial intelligence models while ensuring data privacy, security, and legal compliance.

[0060] Few-Shot Learning (FSL) is a machine learning method designed to achieve effective learning and generalization with a very small number of training samples. This method typically relies on techniques such as transfer learning and metric learning to extract useful features from limited data.

[0061] Meta-learning (ML): Its core goal is to enable machine learning models to "learn how to learn." Instead of training for a single specific task, it trains and "practices the learning process" on a large number of different but related tasks, thereby acquiring "meta-knowledge" or "meta-capability" that can quickly adapt to and efficiently learn new tasks (even with limited data).

[0062] Active Inference (AIF): A theoretical framework derived from the Bayesian brain hypothesis, emphasizing that agents interpret external observations and guide behavior by minimizing free energy. This mechanism combines perception, learning, and decision-making, and is widely used in neuroscience and artificial intelligence research.

[0063] Free energy (FE): In probabilistic graphical models, free energy quantifies the uncertainty and prediction error of a system, typically representing the difference between the variational approximation and the true distribution. A smaller free energy value indicates a stronger fit or interpretative ability of the model to the data.

[0064] Multimodal Action Recognition (MAR) is a computer vision task that integrates information from different sensor modalities (such as images, optical flow, depth, etc.) to recognize human actions, aiming to improve recognition accuracy and adaptability to complex scenes.

[0065] It should be noted that the "multi-modality" mentioned in this invention specifically refers to "RGB mode and optical flow mode". The multi-modal data in this invention includes RGB data and optical flow data of motion video. The modality-specific models used in this invention include an optical flow model for extracting optical flow features and an RGB model for extracting RGB features.

[0066] like Figure 1 As shown, this invention addresses the problems of large reliability differences between different modalities during client-side local model inference in federated multimodal action recognition tasks, the amplification of noise by traditional federated averaging aggregation, and the failure of multimodal fusion strategies. It proposes a small-sample action recognition method for end-edge systems based on federated active inference. This method follows the meta-learning concept and specifically includes the following steps:

[0067] Step 101: Each client in the current network maintains a modality-specific model (such as an optical flow model and an RGB model). The edge server maintains a public dataset, which includes a public support set and a public query set. The samples are RGB data and optical flow data pairs. The optical flow model can be a dual-stream dilated 3D convolutional network (I3D), and the RGB model can be a ResNet50 model. However, other RGB models such as CNN and MobileNet, and optical flow models such as FlowNet, can also be used. Therefore, models used for RGB image feature extraction and optical flow image feature extraction can be used to replace ResNet50 and I3D.

[0068] In this step, each client maintains multiple modality-specific models, while the edge server maintains a public dataset, which includes a public support set and a public query set. The samples in the public support set and public query set are multimodal data corresponding to multiple modality-specific models.

[0069] like Figure 2 As shown, at the start of training, the client first performs prototype calculations on the samples in the public support set to extract the effective information from the samples. The specific calculation method for the prototype of the public support set samples is as follows:

[0070] Let the public support set be D_e^{sup}, containing N action categories, where the samples are denoted as x_i^m. The category support subset is D_{e,k}^{sup}, representing all samples in the public support set belonging to the k-th category, where k ranges from 1 to N (k=1, 2, ..., N), and the samples are denoted as x_{i,k}^m. The sample prototype is φ_k^m, representing the typical feature vector of the k-th action in modality m, where m ranges from m={m|RGB, optical flow}. The client-maintained local modality-specific model is ψ_c^m.

[0071] The client extracts corresponding samples from the public support set D_e^{sup} based on the target action category k to form a category support subset D_{e,k}^{sup}.

[0072] The client calls the local modality-specific model ψ_c^m to extract the feature vector ψ_c^m(x_{i,k}^m) of each sample in the category support subset D_{e,k}^{sup}. The essence of extracting the feature vector is that the machine cannot directly read the image data, so it needs to be converted into digital data (such as feature vectors) to facilitate machine reading.

[0073] After obtaining the modality-specific feature vector ψ_c^m(x_{i,k}^m) of all samples within the category support subset, the client performs vector summation, and the resulting total value is the modality prototype φ_k^m of category k.

[0074] Step 102: After obtaining all action category modal prototypes φ_k^m in the category support subset D_{e,k}^{sup}, the client calculates the variational posterior probability of each action category for the samples in the public query set.

[0075] like Figure 3 As shown, the method for calculating the variational posterior probability of the common query set samples is as follows:

[0076] Let the common query set be D_e^{qry}, and the number of action categories be k, with the number of samples in each category set to 1, denoted as x_k^m;

[0077] The client-side modality-specific model ψ_c^m extracts the feature vector ψ_c^m(x_k^m) of the query samples in the public query set.

[0078] Calculate the Manhattan distance between the feature vector ψ_c^m(x_k^m) of each query sample and the prototype φ_k^m of the k-th class support sample (i.e., calculate the sum of the straight-line distances between the two points on the coordinate axes);

[0079] The calculated Manhattan distance is negatively evaluated to obtain the inverse distance metric. This inverse distance metric is then non-linearly amplified by an exponential function, i.e., by taking the logarithm of the inverse distance metric. The amplified result is then divided by the sum of the exponential function values ​​corresponding to all categories to complete the normalization process.

[0080] The normalized output yields the variational posterior probability Q_k^m(k|x_k^m).

[0081] Step 103: After the client outputs the variational posterior probability of the query sample, it performs free energy calculation in the active inference framework to obtain the inference uncertainty of each mode of the query sample for the model. The lower the free energy value, the lower the inference uncertainty of the mode for the model and the higher the confidence.

[0082] like Figure 4 As shown, the method for calculating the modal-specific free energy of the sample is as follows:

[0083] Let Q_k^m(k|x_k^m) be the variational posterior probability of the query sample x_k^m, and q(s) be the inference probability of the sample class by the client model ψ_c^m. V^m is the observation likelihood matrix, whose column vectors describe the ideal probability distribution that the model should output when the true class of the sample is k. The prior probability p(s) is uniformly distributed, meaning that all action classes are assumed to have the same probability of occurrence.

[0084] The client extracts the column vector V^m[:,k] corresponding to the observation matrix as the ideal distribution based on the true category of the sample. The symbol : indicates that all rows of the matrix are selected, and [:,k] indicates that the k-th column element of all rows is selected. Each element v_k in the ideal distribution vector represents the ideal probability. For each category, the ratio of the client probability q_k(s) to the ideal probability v_k is calculated, and the negative logarithm is taken to obtain the single-class bias. The bias values ​​of all categories are weighted and summed with the client probability as the weight to obtain the expected negative log-likelihood value P_{q(s)} (which reflects the degree of deviation between the model prediction and the ideal distribution).

[0085] The deviation between the client's inferred probability q(s) and the uniform prior p(s) is calculated. Specifically, the Kullback-Leibler divergence between the client's inferred probability q(s) and the uniform prior p(s) is calculated to measure the difference between the client's inferred probability q(s) and the uniform prior p(s).

[0086] The client sums the expected negative log-likelihood value P_{q(s)} with the Kullback-Leibler divergence value, denoted as the mode-specific free energy E_k^m of the sample x_k^m.

[0087] Step 104: After obtaining the modality-specific free energy E_k^m corresponding to all samples x_k^m, the client sums the free energies of all samples to obtain the client's modality-specific free energy E_c^m, which represents the modality-specific inference uncertainty of the current client model.

[0088] Step 105: The client compares the specific free energies E_c^m of different modes, takes the exclusive model of the mode with the smaller free energy value as the current client's inference dominant model ψ_{c,d}^m, and uploads the inference dominant model and the corresponding client mode-specific free energy E_c^m to the edge server.

[0089] Step 106, as follows Figure 5 As shown, after receiving the modality-specific free energy E_c^m and the inference-dominant model ψ_{c,d}^m uploaded by the client, the edge server first divides and aggregates the inference-dominant models ψ_{c,d}^m according to the modality. The primary criterion is that models specific to the same modality are aggregated into an inference-dominant model set G^m. This grouping mechanism ensures that during global aggregation, each modality only merges the client model that performs best in that modality, thus avoiding cross-modal interference at the source.

[0090] Step 107, as follows Figure 6 As shown, the edge server processes the modality-specific free energy uploaded by all clients, selects client-specific models to participate in the current global model aggregation and training, and outputs the aggregated global model. The specific method is as follows:

[0091] Let the sum of the free energies of all clients in the target mode m be E^m, and the sum of the free energies of all clients in the complementary mode m' be E^{m'}. The maximum number of clients allowed to participate in aggregation in a single round of communication is L, and the subset of clients participating in aggregation is {G'}^m.

[0092] The edge server allocates the client models participating in the aggregation based on the reliability ratio of the two modalities:

[0093] Calculate the client quota l^m for the target mode m. Its value is the ratio of the modal free energy E^m to the sum of the total free energy values ​​E^m + E^{m'}, multiplied by the total upper limit L and rounded up to ensure that the more reliable mode obtains more aggregate resources.

[0094] After obtaining the client quota l^m of the target mode m, the difference between the upper limit of the total number of clients L and the client quota l^m of the target mode m is the client quota of the complementary mode m'.

[0095] From the set of inference-dominated models G^m, select the top l^m client-dominated models with the lowest average free energy to form a client subset {G'}^m that participates in the aggregation;

[0096] The selected client-dominant model parameters are summed with equal weights and then divided by the number of clients to output the aggregated global model ψ^m.

[0097] Step 108: The edge server trains the global model ψ^m using the public dataset to obtain the trained global model ψ'^m, and broadcasts the global model ψ'^m to all clients for model replacement.

[0098] Step 109: The client and edge server repeat the above steps until the global model achieves convergence.

[0099] Step 110, as follows Figure 7 , Figure 8 As shown, the client collects video data and processes it using tools such as Denseflow and OpenCV to obtain the client support set D_c^{sup} and the client query set D_c^{qry}. Then, the replaced model is used to calculate the variational posterior probabilities of the support set samples and the query set samples, respectively. The specific method is as follows:

[0100] Let the replaced client model be ψ'_c^m, the client support set sample prototype be φ'_k^m, and the query set variational posterior probability be Q'_k^m(k|x_k^m).

[0101] The client follows steps 101 and 102 to calculate the prototype of the support set sample and the variational posterior probability of the query set sample to obtain the prototype φ'_k^m of the client support set D_c^{sup} sample and the variational posterior probability Q'_k^m (k|x_k^m) of the client query set sample.

[0102] Step 111, as follows Figure 9As shown, the client operates on the obtained client support set D_c^{sup} sample prototype φ'_k^m and the client query set sample variational posterior probability Q'_k^m(k|x_k^m) to obtain the confidence scores of different modes m of the query set sample x_k. The specific calculation method is as follows:

[0103] Let o_k^m be the modality confidence score of different modalities in the query set sample;

[0104] The client filters the maximum value of the variational posterior probability Q'_k^m (k|x_k^m) of the query set sample x_k for mode m and uses it as the confidence score o_k^m of mode m of the current query set sample x_k, which represents the maximum classification probability value of the client model for the current sample in a single mode.

[0105] Step 112, as follows Figure 10 As shown, the client processes the confidence scores o_k^m of different modalities for each query set sample to obtain the fusion weights of different modalities in the classification inference of the current sample. The specific calculation method is as follows:

[0106] Let the weights for different modes be y_k^m;

[0107] The client sums the modal confidence scores o_k^m of sample x_k to obtain the full modal confidence score o_k;

[0108] The client normalizes the modal confidence scores o_k^m and the overall modal confidence scores o_k. That is, the fusion weight y_k^m of different modalities is obtained by dividing the modal confidence scores o_k^m by the overall modal confidence scores o_k, and y_k^m satisfies that the sum of y_k^r and y_k^f is 1.

[0109] Step 113, as follows Figure 11 As shown, after obtaining the fusion weights y_k^m for different modalities, the client dynamically weights and fuses the bimodal information based on the modality confidence to obtain the final classification prediction result for the query sample. The specific calculation method is as follows:

[0110] Let the modal feature vector of sample x_k be s_i^m, the prototype of the support set D_c^{sup} be φ'_k^m, and the feature similarity function be ρ(), used to calculate the Euclidean distance between the two vectors. The prediction result is Pre_c(x_k);

[0111] The client calculates the RGB modal contribution g^r using the following method:

[0112] Calculate the similarity ρ(s_i^r, φ'_k^r) between the RGB feature vector s_i^r and the RGB prototype φ'_k^r, and multiply it by the RGB fusion weight y_k^r to obtain the RGB modal contribution g^r;

[0113] The client calculates the optical flow modal contribution g^f using the following method:

[0114] Calculate the similarity ρ(s_i^f, φ'_k^f) between the optical flow feature vector s_i^f and the optical flow prototype φ'_k^f, and multiply it by the optical flow fusion weight y_k^f to obtain the optical flow mode contribution g^f;

[0115] The combined difference value of the target category is obtained by adding the optical flow modal contribution g^f and the RGB modal contribution g^r.

[0116] The negative of the comprehensive difference value is then used to perform a non-linear transformation through the natural exponential function to obtain the probability Pre_c(k|x_k) that sample x_k belongs to class k.

[0117] Repeat the above process for all action categories (k=1,2,…,N) to obtain the probability distribution of sample x_k.

[0118] like Figure 12 As shown, according to another embodiment of the present invention, a specific implementation method for a modal-specific free energy calculation module based on the above method is also disclosed:

[0119] The modality-specific free energy calculation module includes a data input module 1, a data processing module 2, a single-modal feature extraction module 3, a support sample prototype calculation module 4, a query set variational posterior probability calculation module 5, and a modality-specific free energy calculation module 6. Data input module 1 uses various user devices, such as mobile phones, tablets, and laptops. Data processing module 2 uses the Denseflow library and OpenCV models released by open-mmlab. Single-modal feature extraction module 3 uses ResNet and I3D series models. Support sample prototype calculation module 4 obtains the support sample prototype by summing and weighting the RGB and optical flow features of the support samples extracted by single-modal feature extraction module 3. Query set variational posterior probability calculation module 5 obtains the query set variational posterior probability by calculating the distance between the RGB and optical flow features of the query samples extracted by single-modal feature extraction module 3 and the support sample prototype, and then performing a negative logarithm operation.

[0120] The workflow of this modality-specific free energy calculation module includes:

[0121] S1, Data input module 1 collects motion video;

[0122] S2, the collected motion video is processed by data processing module 2 to obtain RGB image data and optical flow image data;

[0123] S3, the single-modal feature extraction module 3 extracts the features of the RGB image data and optical flow image data obtained in S2 respectively;

[0124] S4, the sample prototype calculation module 4 first performs a summation operation on the feature vectors of different modal data to obtain the total features of different modal data, and then divides the features of each sample by the total features to obtain the prototype of the sample.

[0125] S5, the query set variational posterior probability calculation module 5 receives the supporting sample prototype, RGB query sample features and optical flow query sample features, calculates the Manhattan distance between the RGB query sample features, optical flow query sample features and the supporting sample prototype, respectively, takes the negative value of the calculated Manhattan distance to obtain the inverse distance metric, inputs the value of the inverse distance metric into the exponential function for nonlinear amplification, divides the amplified result by the sum of the exponential function values ​​corresponding to all categories to complete the normalization process, and outputs the query set variational posterior probability.

[0126] S6, the modality-specific free energy calculation module 6 pre-sets a prior distribution probability and a modality-specific observation likelihood matrix. The modality-specific free energy calculation module 6 uses the query set variational posterior probability obtained in S5 as the model's inference probability of the sample class. The modality-specific free energy calculation module 6 processes the modality-specific observation likelihood matrix and the query set variational posterior probability to obtain the expected negative log-likelihood value; the KL divergence function is used to obtain the probability difference between the query set variational posterior probability and the prior distribution probability.

[0127] S7, the modality-specific free energy calculation module 6 sums the probability difference value and the expected negative log-likelihood value obtained in S6 to obtain the modality-specific free energy of the query sample.

[0128] like Figure 13 As shown, according to another embodiment of the present invention, a few-shot action recognition system 1300 based on federated active reasoning is also disclosed, including: a memory 1301 for storing program instructions; and a processor 1302 for calling the program instructions stored in the memory to implement the few-shot action recognition method based on federated active reasoning as described in the above embodiment.

[0129] All or part of the steps in the various methods of the above embodiments can be implemented by a program controlling the relevant hardware. The program can be stored in a readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A few-sample action recognition method based on federated active reasoning, characterized in that, include: The client performs free energy calculations in the active inference framework based on the variational posterior probability of the public query set samples to obtain the mode-specific free energy of each mode of the public query set samples. The mode-specific model corresponding to the mode with the smallest free energy value is taken as the current client's dominant inference model, and the dominant inference model and the mode-specific free energy of the corresponding mode are uploaded to the edge server. The calculation process for modal-specific free energy includes: Let the variational posterior probability of the query sample be the client inference probability of the sample class by the client model, and the column vector of the observation likelihood matrix describe the ideal probability distribution that the client model should output when the true class of the sample is k. The prior probability is uniform, that is, it is assumed that all action classes have the same probability of occurrence. The client infers probabilities based on the observed likelihood matrix and the client's inference probability. It extracts the column vector corresponding to the observed likelihood matrix as the ideal distribution based on the true category of the sample, where each element represents the ideal probability. For each category, it calculates the ratio of the client's probability to the ideal probability, takes the negative logarithm to obtain the single-category bias, and uses the client's probability as the weight to sum the bias values ​​of all categories to obtain the expected negative log-likelihood value. Calculate the Kullback-Leibler divergence between the client's inferred probability and the prior probability to measure the difference between the client's inferred probability and the prior probability. The client sums the expected negative log-likelihood value with the Kullback-Leibler divergence value to obtain the mode-specific free energy of the sample; the mode-specific free energy represents the mode-specific inference uncertainty of the current client model; The edge server divides and aggregates the inference-dominant models according to modality, so as to aggregate models of the same modality into an inference-dominant model set; based on the modality-specific free energy uploaded by all clients and the inference-dominant model set, the modality-specific models that participate in the current global model aggregation and training are selected to obtain the aggregated global model; The edge server trains the global model using a public dataset, obtains the trained global model, and broadcasts it to all clients so that the clients can replace the model. The client receives motion video data to construct a client support set and a client query set, and uses the replaced model to perform client support set sample prototype calculation and client query set variational posterior probability calculation respectively. The client calculates the confidence score of each modality of the client query set sample based on the client-supported set sample prototype and the variational posterior probability of the client query set; based on the confidence score of each sample in different modalities, it calculates the fusion weight of different modalities in the classification inference of the current sample; after obtaining the fusion weight of different modalities, it dynamically weights and fuses multimodal information based on modal confidence to obtain the classification prediction result of the query sample.

2. The few-sample action recognition method based on federated active reasoning according to claim 1, characterized in that, The calculation process of the variational posterior probability of the public query set samples specifically includes: The client performs prototype network computation on samples of the public support set of the edge server to obtain the prototype of the public support set samples; The client extracts feature vectors from the multimodal data in the public query set of the edge server using various modality-specific models to obtain the public query set sample feature vectors. Based on the public query set sample feature vectors and the public support set sample prototypes, the client calculates the variational posterior probability of the public query set samples: the Manhattan distance between each query set sample feature vector and the support sample prototype is calculated; the calculated Manhattan distance is negatively evaluated to obtain the inverse distance metric; the value of the inverse distance metric is input into an exponential function for nonlinear amplification, i.e., the logarithm of the inverse distance metric is taken; the amplified result is divided by the sum of the exponential function values ​​corresponding to all categories to complete the normalization process, thus obtaining the variational posterior probability of the query sample.

3. The few-sample action recognition method based on federated active reasoning according to claim 2, characterized in that, Prototype network computation specifically includes: The client extracts corresponding samples from the public support set to form a category support subset based on the category of the target action; The client calls the local modality-specific model to extract modality-specific feature vectors for samples in the category support subset on a sample-by-sample basis; After obtaining the modality-specific feature vectors of all samples in the subset, the vectors are summed, and the total value is the modality prototype of the target action category.

4. The few-sample action recognition method based on federated active reasoning according to claim 1, characterized in that, The specific process of global model aggregation includes: Let the sum of the free energies of all clients under the target mode m be E^m, the sum of the free energies of the mode m' which is complementary to mode m be E^{m'}, the upper limit of the total number of clients allowed to participate in aggregation in a single round of communication be L, the set of inference-dominant models be G^m, and the subset of clients participating in aggregation be {G'}^m. The edge server allocates the client models participating in the aggregation based on the reliability ratio of the two modalities: Calculate the client quota l^m for the target mode m. The value of l^m is the proportion of E^m to E^m+E^{m'}, multiplied by L and rounded up to ensure that the more reliable mode gets more aggregated resources. After obtaining the client quota l^m of the target mode m, the difference between the upper limit of the total number of clients L and the client quota l^m of the target mode m is the client quota of the complementary mode m'. From the set of inference-dominant models G^m, select the top l^m clients with the lowest average free energy to form a subset {G'}^m of clients participating in the aggregation; The selected client model parameters are summed with equal weights and then divided by the number of clients to output the aggregated global model.

5. The few-sample action recognition method based on federated active reasoning according to claim 1, characterized in that, The steps of dynamically weighting and fusing multimodal information based on modal confidence specifically include: Let the modal feature vector of sample x_k be s_i^m, the prototype of the support set D_c^{sup} be φ'_k^m, and the feature similarity function be ρ(), which is used to calculate the Euclidean distance between the two vectors. The prediction result is Pre_c(x_k). The client calculates the RGB modal contribution g^r using the following method: Calculate the similarity ρ(s_i^r, φ'_k^r) between the RGB feature vector s_i^r and the RGB prototype φ'_k^r, and then multiply ρ(s_i^r, φ'_k^r) by the RGB fusion weight y_k^r to obtain the RGB modal contribution g^r; The client calculates the optical flow modal contribution g^f using the following method: Calculate the similarity ρ(s_i^f, φ'_k^f) between the optical flow feature vector s_i^f and the optical flow prototype φ'_k^f, and then multiply ρ(s_i^f, φ'_k^f) by the optical flow fusion weight y_k^f to obtain the optical flow mode contribution g^f; The combined difference value of the target category is obtained by adding the optical flow modal contribution g^f and the RGB modal contribution g^r. The negative of the comprehensive difference value is then used to perform a non-linear transformation through the natural exponential function to obtain the probability Pre_c(k|x_k) that sample x_k belongs to class k. Repeat the above process for all action categories to obtain the probability distribution of sample x_k.

6. The few-sample action recognition method based on federated active reasoning according to any one of claims 1 to 5, characterized in that, Each client maintains multiple modality-specific models, and the edge server maintains a public dataset, which includes a public support set and a public query set; the samples in the public support set and the public query set are multimodal data corresponding to multiple modality-specific models.

7. The few-sample action recognition method based on federated active reasoning according to claim 6, characterized in that, The multimodal data includes RGB data and optical flow data from motion videos, and the modality-specific models include an optical flow model for extracting optical flow features and an RGB model for extracting RGB features.

8. A few-sample action recognition system based on federated active reasoning, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke the program instructions stored in the memory to implement the small-sample action recognition method based on federated active reasoning as described in any one of claims 1 to 7.

Images