A human behavior recognition method and system based on visual semantic text enhancement

Abstract

The application relates to a human body behavior recognition method and system based on visual semantic text enhancement, and the method comprises the following steps: S10, acquiring human body behavior WIFI sensing data and human body behavior video frame data; S20, respectively extracting WIFI amplitude energy images and high-dimensional semantic feature vectors from the human body behavior WIFI sensing data and the human body behavior video frame data; S30, performing deep correlation fusion on the WIFI amplitude energy images and the high-dimensional semantic feature vectors, and calculating the probability distribution of human body action categories. The human body behavior recognition method based on visual semantic text enhancement provided by the application realizes effective fusion of semantic information and WiFi features by cross-modal similarity retrieval, and calculates the probability distribution of human body action categories based on the global feature vector obtained after fusion, so that the recognition precision of similar actions is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, and in particular to a human behavior recognition method and system based on visual semantic text enhancement. Background Technology

[0002] In recent years, human action recognition has been widely applied and researched in fields such as smart healthcare, intelligent security, and human-computer interaction. Among these, Wi-Fi signal sensing and pure visual recognition, as non-contact sensing methods, have become two major research hotspots in the field of action recognition. Single Wi-Fi signal action recognition is susceptible to multipath interference, resulting in limited accuracy; single visual action recognition, on the other hand, carries the risk of privacy leaks. To overcome these shortcomings, researchers have gradually shifted towards multimodal fusion, combining Wi-Fi signals with visual data in modeling to compensate for the deficiencies of wireless sensing in feature representation.

[0003] However, existing multimodal fusion human recognition research typically uses visual modalities to compensate for the shortcomings of Wi-Fi perception in fine-grained semantic representation, and uses this as a basis for joint modeling of Wi-Fi and visual information. However, the modal interaction between Wi-Fi and vision remains at a shallow alignment of low-level signals and basic features, lacking deep interaction with high-level action semantics. This makes it impossible for existing multimodal human behavior recognition methods to finely distinguish similar actions. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a method and system for human behavior recognition based on visual semantic text enhancement.

[0005] A method for human behavior recognition based on visual semantic text enhancement includes the following steps:

[0006] S10. Acquire human behavior WIFI sensing data and human behavior video frame data; S20. Extract WIFI amplitude energy images from the human behavior WIFI sensing data and human behavior video frame data respectively. and high-dimensional semantic feature vectors ; S30, the WIFI amplitude energy image and the high-dimensional semantic feature vector Perform deep correlation fusion and calculate the probability distribution of human action categories. .

[0007] The human behavior recognition method based on visual semantic text enhancement described in this invention converts WiFi signals into image representations, generates corresponding behavioral semantic text using a large language model, and then effectively fuses semantic information with WiFi features through cross-modal similarity retrieval. Finally, it calculates the probability distribution of human action categories based on the fused global feature vector. During the calculation process, the language model is used to enhance the high-level semantic support for WiFi action recognition, thereby improving the recognition accuracy of similar actions.

[0008] Further, step S30 includes: S31. Extract the global feature vector of WIFI based on the WIFI amplitude energy image. ; S32, The WIFI global feature vector and the high-dimensional semantic feature vector Perform association aggregation to obtain semantically enhanced feature vectors. ; S33. The semantically enhanced feature vector Perform global feature aggregation to obtain global feature vectors. ; S34. Based on the global feature vector Obtain the probability distribution of human action categories. .

[0009] Furthermore, the process of step S31 includes: The WIFI amplitude energy image is segmented into non-overlapping patches using Patch Embedding:

[0010] Where N is the total number of segments; .

[0011] Then, each patch is mapped to a D-dimensional vector using linear projection:

[0012] In the formula, The projection matrix; This is a bias term.

[0013] Then, positional encoding is added to each vector to preserve its original spatial location information, thus obtaining the input sequence. :

[0014] In the formula, For vectors The location encoding of the corresponding patch.

[0015] Finally, the input sequence is encoded using self-attention (SA) to obtain the global feature vector of WIFI. :

[0016] Furthermore, the process of step S32 includes: Using the aforementioned WIFI global feature vector As the query vector, the high-dimensional semantic feature vector The global feature vector of WIFI is calculated by normalization, using the key vector and value vector as the key vector and value vector. and the high-dimensional semantic feature vector The similarity weights are calculated as follows:

[0017] In the formula , and This is the weight matrix; Semantic information is injected into the WIFI global feature vector through residual connections. In the process, semantically enhanced feature vectors are obtained. :

[0018] Furthermore, the process of step S33 includes: The frozen patch features are aggregated globally using the Class-Attention layer of the CaiT framework, as follows: In the semantically enhanced feature vector Prepend a learnable [CLS]Token item. This forms a complete sequence:

[0019] Attention-weighted calculation is performed on the complete sequence G:

[0020] right and [CLS]Token item Perform residual join and normalization calculations:

[0021] right Perform feedforward computation and then perform residual concatenation on the computation results to obtain the global feature vector. :

[0022] Further, step S20 includes: S21. Extract the WIFI amplitude energy image based on the human behavior WIFI sensing data. ; S22. Based on the human behavior video frame data, extract high-dimensional semantic feature vectors that are highly correlated with the behavior. .

[0023] Further, step S21 includes: S211. Preprocess the human behavior WIFI sensing data; S212, Process the pre-processed human behavior WIFI sensing data Stacked into a two-dimensional matrix M; S213. Normalize the two-dimensional matrix M to obtain the normalized matrix. ; S214. The normalized matrix Pseudo-color encoding is performed to obtain the WIFI amplitude energy image. .

[0024] Further, step S22 includes: S221. Construct a structured query instruction set; S222. Based on the structured query instruction set, generate action description text using the multimodal large language model paradigm; S223. Perform semantic vectorization encoding on the action description text to generate a high-dimensional semantic feature vector. .

[0025] Furthermore, the process of step S223 includes: Convert the action description text into a token ID sequence and add special markers [CLS] and [SEP]:

[0026] Then, it is transformed into an intermediate vector E through embedding:

[0027] Using the intermediate vector E as the initial input The output of layer L(L-1) is used as the input of layer L, and L layers of BERT encoding are stacked to obtain the context feature vector. : Layer 1 BERT output The calculation process is as follows: In multi-head attention computation, the single-head attention term Head can be represented as:

[0028] For the kth attention head Its output is calculated according to the following formula:

[0029] In the formula, and For the k-th attention head The corresponding weight matrix; Perform multi-head concatenation computation on the single-head attention term to obtain the multi-head concatenation vector. :

[0030] In the formula, For projection output matrix; For the input vector E and the multi-head concatenation vector The residuals are summed, and then layer normalization is applied to obtain the intermediate feature vector Z:

[0031] in, The normalization function is defined as follows:

[0032] In the formula, and These are the mean and variance of parameter x, respectively; and These are the scaling parameter and the offset parameter, respectively; It is a very small constant, usually taken as This is used to ensure numerical stability; The operator represents element-wise multiplication; The intermediate vector Z is fed forward, and the result is added to the residual of the intermediate feature vector Z. Then, layer normalization is performed again to obtain the BERT output of the first layer. :

[0033] in, The feedforward computation function is defined as follows:

[0034] In the formula, It is the ReLU activation function. and These are the first weight term and the first bias term, respectively; and These are the second weight term and the second bias term, respectively. Assuming a total of L layers of BERT stacked computation are set up, then the context-aware vector for:

[0035] Finally, the context-aware vector corresponding to the [CLS] position is taken as the high-dimensional semantic feature vector of the action description text. :

[0036] This invention also provides a human behavior recognition system based on visual semantic text enhancement, comprising: The signal acquisition module is used to acquire human behavior WIFI sensing data and human behavior video frame data; The signal processing module is used to extract WIFI amplitude energy images from the human behavior WIFI sensing data and the human behavior video frame data, respectively. and high-dimensional semantic feature vectors ; A multimodal fusion module is used to process the WIFI amplitude energy image. and the high-dimensional semantic feature vector Perform deep correlation fusion and calculate the probability distribution of human action categories. .

[0037] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0038] Figure 1 A schematic diagram of the modules of the human behavior recognition system based on visual semantic text enhancement provided by the present invention; Figure 2 The execution flowchart of the human behavior recognition method based on visual semantic text enhancement provided by the present invention; Figure 3 The execution flowchart of step S21 provided by the present invention; Figure 4 The execution flowchart of step S22 provided by the present invention; Figure 5 The execution flowchart of step S30 provided by the present invention; Figure 6 The execution flowchart of the training method for the lightweight WIFI identification model provided by the present invention; Figure 7 A bar chart showing the recognition accuracy of various actions for the lightweight WIFI signal recognition model provided by this invention; Figure 8 This is a comparison chart showing the accuracy of human action recognition provided by the present invention based on visual semantic text enhancement, and the accuracy of human action recognition based on a single WIFI signal and a single visual semantic text. Detailed Implementation

[0039] To address the limitation of existing multimodal human action recognition methods in accurately distinguishing similar actions, this application proposes a human action recognition method based on visual semantic text enhancement. This method converts WiFi signals into image representations and generates corresponding semantic text using a large language model. Cross-modal similarity retrieval is then used to effectively fuse semantic information with WiFi features. Finally, the probability distribution of human action categories is calculated based on the fused global feature vector. During the calculation process, a language model is used to enhance the high-level semantic support for WiFi action recognition, thereby improving the accuracy of similar action recognition. Furthermore, this application constructs a knowledge distillation framework to transfer the human action category recognition capability under multimodal fusion to a lightweight WiFi recognition model, achieving lightweight deployment of high-precision human action recognition and effectively expanding its application scope.

[0040] The following is a detailed description of the human behavior recognition method based on visual semantic text enhancement of the present invention: Please see Figure 1 and Figure 2 The present invention provides a human behavior recognition system based on visual semantic text enhancement, including a signal acquisition module 10, a signal processing module 20 and a multimodal fusion module 30.

[0041] The signal acquisition module 10 is used to perform step S10: acquiring human behavior WIFI sensing data and human behavior video frame data.

[0042] Specifically, the signal acquisition module 10 is communicatively connected to the WIFI device and the camera, respectively, reads the human behavior data collected by the WIFI device and the human behavior video frame data collected by the camera, and arranges them in a time sequence.

[0043] The signal processing module 20 is used to perform step S20: extracting WIFI amplitude energy images from the human behavior WIFI sensing data and the human behavior video frame data, respectively. and high-dimensional semantic feature vectors .

[0044] Specifically, step S20 includes: S21. Extract the WIFI amplitude energy image based on the human behavior WIFI sensing data. .

[0045] Please see Figure 3Step S21 includes: S211. Preprocess the human behavior WIFI sensing data.

[0046] The human behavior WIFI sensing data is the CSI (signal status information) data of the WIFI device, which is the superposition of signals from all propagation paths, and can be represented as:

[0047] In the formula, and These are the carrier center frequency and time indices, respectively; m is the multipath component index. For the complex amplitude attenuation of the m-th multipath component; Let m be the signal propagation distance of the m-th path; It is the set consisting of all multipath components; The wavelength is the signal carrier wavelength.

[0048] First, Hampel filtering is used to detect and correct outliers in the time series, and the median is calculated within a sliding window W of size 21. And the absolute median difference (MAD):

[0049]

[0050] when At that time, Replace with This helps to suppress abnormal disturbances.

[0051] Then, the signal is smoothed using Savitzky-Golay filtering to obtain preprocessed human behavior Wi-Fi sensing data. :

[0052] In the formula, These are the polynomial fitting coefficients.

[0053] S212, Process the pre-processed human behavior WIFI sensing data Stacked into a two-dimensional matrix M.

[0054] Specifically, the preprocessed human behavior WIFI sensing data is analyzed according to the subcarrier and time dimensions. The stacking of the two-dimensional matrix M is as follows:

[0055]

[0056] In the formula, This represents the total number of subcarriers. This represents the number of time sampling points; Let be the element in the i-th row and j-th column of the two-dimensional matrix M; Preprocessed human behavior WIFI sensing data The amplitude value of the i-th subcarrier at time j.

[0057] S213. Normalize the two-dimensional matrix M to obtain the normalized matrix. .

[0058] Specifically, through normalization, the values ​​of the two-dimensional matrix M are mapped to the [0,1] interval, thereby eliminating the dimensional differences caused by hardware / environment. The calculation process is as follows:

[0059] In the formula, The normalized matrix The element in the i-th row and j-th column; Let M be the minimum value of all elements in the two-dimensional matrix M; Let M be the maximum value of each element in the two-dimensional matrix M.

[0060] S214. The normalized matrix Pseudo-color encoding is performed to obtain the WIFI amplitude energy image. .

[0061] Specifically, the color lookup table function C is defined as follows:

[0062] Then the WIFI amplitude energy image It can be calculated using the following formula:

[0063] In the formula, The WIFI amplitude energy image 3 represents the spatial resolution; 3 represents the number of RGB channels.

[0064] For color lookup table functions C, the mathematical piecewise definition of Jet color spectrum is commonly used:

[0065] In the formula, .

[0066] S22. Based on the human behavior video frame data, extract high-dimensional semantic feature vectors that are highly correlated with the behavior. .

[0067] Specifically, by combining a multimodal large language model with structured queries, high-level semantic features strongly correlated with human behavior are extracted from the human behavior video frame data. Please refer to [link / reference]. Figure 4 Step S22 includes: S221. Construct a structured query instruction set.

[0068] Specifically, based on the semantic granularity requirements of the human behavior recognition task, three sets of structured query instructions are designed to guide the multimodal large language model to eliminate irrelevant information interference and focus on the action ontology. The three sets of structured query instructions are as follows: First instruction: Extract the core movements of the person in the image, excluding irrelevant descriptions such as the person's appearance and clothing; The second instruction describes the overall posture during the execution of the task, and distinguishes the specific movement states of the left hand, right hand, left leg, and right leg. The third instruction is to determine the image clarity and identify whether there are objects such as boxes on the table in the scene.

[0069] S222. Based on the structured query instruction set, generate action description text using the multimodal large language model paradigm.

[0070] Specifically, the structured query instruction set is concatenated with preset answer constraints to form input prompts. The input prompts and the human behavior video frame data are then input into a multimodal large language model such as LLaVA in the prior art. The combined architecture of the visual encoder and the large language model built into the multimodal large language model can encode the human behavior video frame data into visual features and perform cross-modal alignment with the structured query instruction set, outputting a standardized natural language description that corresponds one-to-one with the input prompts, thereby obtaining a multi-dimensional semantic action description text containing action categories, posture details, and environmental information.

[0071] S223. Perform semantic vectorization encoding on the action description text to generate a high-dimensional semantic feature vector. .

[0072] Specifically, the pre-trained language model BERT based on the Transformer architecture is used as the text encoding unit. The action description text is encoded by stacking multiple layers of BERT, transforming discrete natural language into continuous high-dimensional semantic feature vectors. This provides a stable and highly discriminative semantic foundation for subsequent cross-modal feature matching, semantic enhancement, and fusion modeling. The specific calculation process is as follows: Specifically, the action description text is first converted into a Token ID sequence, and special markers [CLS] and [SEP] are added:

[0073] Then, it is transformed into an intermediate vector E through embedding:

[0074] Using the intermediate vector E as the initial input The output of layer L(L-1) is used as the input of layer L, and L layers of BERT encoding are stacked to obtain the context feature vector. : Layer 1 BERT output The calculation process is as follows: In multi-head attention computation, the single-head attention term Head can be represented as:

[0075] For the kth attention head Its output is calculated according to the following formula:

[0076] In the formula, and For the k-th attention head The corresponding weight matrix.

[0077] Perform multi-head concatenation computation on the single-head attention term to obtain the multi-head concatenation vector. :

[0078] In the formula, This is the projection output matrix.

[0079] For the input vector E and the multi-head concatenation vector The residuals are summed, and then layer normalization is applied to obtain the intermediate feature vector Z:

[0080] in, The normalization function is defined as follows:

[0081] In the formula, and These are the mean and variance of parameter x, respectively; and These are the scaling parameter and the offset parameter, respectively; It is a very small constant, usually taken as This is used to ensure numerical stability; The operator represents element-wise multiplication.

[0082] The intermediate vector Z is fed forward, and the result is added to the residual of the intermediate feature vector Z. Then, layer normalization is performed again to obtain the BERT output of the first layer. :

[0083] in, The feedforward computation function is defined as follows:

[0084] In the formula, It is the ReLU activation function. and These are the first weight term and the first bias term, respectively; and These are the second weight term and the second bias term, respectively.

[0085] Assuming a total of L layers of BERT stacked computation are set up, then the context-aware vector for:

[0086] Finally, the context-aware vector corresponding to the [CLS] position is taken as the high-dimensional semantic feature vector of the action description text. :

[0087] The multimodal fusion module 30 is used to perform step S30: processing the WIFI amplitude energy image. and the high-dimensional semantic feature vector Perform deep correlation fusion and calculate the probability distribution of human action categories. .

[0088] In this embodiment, a cross-modal semantic enhancement module is inserted between the local feature modeling and global feature aggregation in the Cait (Class-Attention in Image Transformers) framework. This allows for the fusion of high-level action semantics without disrupting the local spatiotemporal structure of the Wi-Fi signal, thereby efficiently extracting action category information. Please refer to [link to Cait (Class-Attention in Image Transformers)]. Figure 5 Step S30 includes: S31. Extract the global feature vector of WIFI based on the WIFI amplitude energy image. .

[0089] First, the WIFI amplitude energy image is segmented into non-overlapping patches using patch embedding:

[0090] Where N is the total number of segments; .

[0091] Then, each patch is mapped to a D-dimensional vector using linear projection:

[0092] In the formula, The projection matrix; This is a bias term.

[0093] Then, positional encoding is added to each vector to preserve its original spatial location information, thus obtaining the input sequence. :

[0094] In the formula, For vectors The location encoding of the corresponding patch.

[0095] Finally, the input sequence is encoded using self-attention (SA) to obtain the global feature vector of WIFI. :

[0096] S32, The WIFI global feature vector and the high-dimensional semantic feature vector Perform association aggregation to obtain semantically enhanced feature vectors. .

[0097] Specifically, given that WIFI signals are susceptible to environmental interference and noise, while text can provide clear and stable category boundaries, this step uses the aforementioned WIFI global feature vector. As the query vector (Q, Query), the high-dimensional semantic feature vector The global feature vector of WIFI is calculated through normalization, serving as the key vector (K, Key) and value vector (V, Value). and the high-dimensional semantic feature vector The similarity weights enable WIFI representations to retrieve and align effective behavioral semantic information from text modalities. The calculation process is as follows:

[0098] In the formula , and This is the weight matrix.

[0099] Then, semantic information is injected into the WIFI global feature vector through residual connections. In the process, semantically enhanced feature vectors are obtained. :

[0100] S33. The semantically enhanced feature vector Perform global feature aggregation to obtain global feature vectors. .

[0101] Specifically, the frozen patch features are aggregated globally through the Class-Attention layer of the CaiT framework, and the process is as follows: In the semantically enhanced feature vector Prepend a learnable [CLS]Token item. This forms a complete sequence:

[0102] Attention-weighted calculation is performed on the complete sequence G:

[0103] right and [CLS]Token item Perform residual join and normalization calculations:

[0104] right Perform feedforward computation and then perform residual concatenation on the computation results to obtain the global feature vector. :

[0105] S34. Based on the global feature vector Obtain the probability distribution of human action categories. .

[0106] Specifically, the global feature vector Inputting data into a multilayer perceptron (MLP Head) yields the probability distribution of human action categories. :

[0107] In the formula, This is the classification weight matrix; This is for classification bias.

[0108] For further information, please refer to [link / reference]. Figure 6To improve the performance of WIFI-based action recognition under the constraints of practical deployment, this application also provides a training method for a lightweight WIFI recognition model, including the following steps: Using the multimodal features corresponding to human behavior as input, and through steps S10, S20 and S30, the probability distribution of human action categories under multimodal fusion is obtained as a reference set; a lightweight WIFI signal recognition model is selected to obtain the probability distribution of human action categories in a single WIFI mode as a training set. Step R1: Calculate the prediction layer distillation loss for the reference set and the training set.

[0109] This application constructs a knowledge distillation framework to transfer the human action category recognition capability under multimodal fusion to a lightweight WIFI recognition model. In this step, a soft-label distillation mechanism is introduced at the prediction layer to align the category distributions under multimodal fusion and single WIFI recognition. The prediction layer distillation loss is calculated using the following formula:

[0110] In the formula, KL is the divergence coefficient; T is the temperature parameter used to smooth the distribution. The probability distribution of human action categories corresponding to the reference set; This represents the probability distribution of human action categories corresponding to the training set.

[0111] Step R2: Calculate the feature layer distillation loss for the reference set and the training set.

[0112] In this step, the probability distribution of human action categories corresponding to the reference set is used as positive sample features. The probability distribution of the corresponding human action categories corresponding to the training set is mapped to the same feature space, and comparative learning is performed based on the InfoNCE loss function to obtain the feature layer distillation loss.

[0113] In the formula, The feature representation of the probability distribution of the human action categories corresponding to the training set after projection; This represents the probability distribution of human action categories corresponding to the reference set.

[0114] Step R3: Calculate the total loss function and use the Adam algorithm to optimize and update the parameters of the lightweight WIFI identification model.

[0115] The total loss function is calculated using the following formula:

[0116] In the formula, The cross-entropy loss is calculated between the training set and the actual results. and These are the weights for equilibrium distillation loss and contrast loss, respectively.

[0117] Based on the total loss function, backpropagation is performed using the Adam optimizer in the prior art to optimize and update the parameters of the lightweight WIFI identification model.

[0118] Step R4: Determine if the total loss function is less than the convergence threshold. If yes, training is complete; otherwise, repeat steps R1-R4.

[0119] Specifically, a convergence threshold is set based on experience. When the total loss function L is less than the convergence threshold, it indicates that the lightweight WIFI recognition model has learned the ability to recognize human action categories under multimodal fusion and the training is complete.

[0120] This application collected human behavior data in four different indoor environments, encompassing 55 human behaviors covering various scenarios such as daily activities, fitness, and human-computer interaction. Each action was repeated 20 times by the subject. The first 14 repetitions were used to generate training and reference sets to train the lightweight WIFI signal recognition model, while the last 6 repetitions served as test samples for the trained lightweight WIFI signal recognition model to evaluate its behavior recognition performance. To balance representativeness and experimental overhead, this paper selects the common ResNet1D+DML+BERT model as the baseline model for human action recognition and MobileNet V3_small and ShuffleNet V2_1.0x lightweight convolutional neural networks as lightweight WIFI signal recognition models to verify the applicability of the training methods. MobileNet V3_small represents a lightweight model based on neural architecture search, while ShuffleNet V2 represents a lightweight network built based on efficient design principles.

[0121] The accuracy rates of each model in recognizing human behavior are shown in the table below. It can be seen that the multimodal fusion model based on visual semantic text enhancement provided in this application achieves an accuracy of 94.65%, showing a significant improvement compared to the benchmark model. Furthermore, after training, the accuracies of MobileNet V3_small and ShuffleNet V2_1.0x reach 92.67% and 93.18% respectively, not only significantly outperforming their respective untrained models, but also surpassing the benchmark model in overall performance.

[0122]

[0123] In addition, to further analyze the model's recognition performance across different action categories, Figure 7The distribution of classification accuracy of the MobileNetV3_small model for various human actions is presented, and it can be observed that the model has relatively stable performance in various recognition tasks.

[0124] Furthermore, to verify the effectiveness of the multimodal fusion of visual semantic text and WIFI signals in the human behavior recognition system based on visual semantic text enhancement proposed in this application, ablation experiments were conducted under three different settings: multimodal fusion of visual semantic text and WIFI signals, recognition only through WIFI signals, and recognition only through visual semantic text. The experimental results are as follows: Figure 8 As shown: As can be seen, the complete model fusing visual semantic text and WIFI signals exhibits the best performance, achieving a recognition accuracy of 94.65%. When visual semantic enhancement is removed and only WIFI signal recognition is used, the model accuracy drops significantly to 92.12%, indicating that introducing semantic information effectively improves the discriminative power of features. Furthermore, when only BERT's visual semantic text is used to classify environmental action text, the model accuracy drops to 82.67%, indicating that single visual semantic text is insufficient to support high-precision human behavior recognition tasks. This shows that the main role of visual semantic text information is not to independently complete classification, but rather to provide high-level semantic supplementation and structural constraints for WIFI features. Based on the above results, this application achieves effective fusion of visual semantic text and WIFI information, bridging the gap between WIFI signals and high-level semantics through visual semantic text, thereby significantly improving the performance of human action recognition.

[0125] Compared with existing technologies, this application has the following advantages: (1) This application converts WiFi signals into image representations, generates behavioral semantic text corresponding to video frames using a large language model, and then uses cross-modal similarity retrieval to effectively fuse behavioral semantic information with WiFi features. Finally, it calculates the probability distribution of human action categories based on the global feature vector obtained after fusion. During the calculation process, the language model is used to strengthen the high-level semantic support of WiFi action recognition, thereby improving the recognition accuracy of similar actions.

[0126] (2) This application constructs a knowledge distillation framework to transfer the human action category recognition capability under multimodal fusion to the WIFI recognition lightweight model, thereby realizing the lightweight deployment of high-precision human action recognition and effectively improving the application scope.

[0127] Based on the same inventive concept described above, the present invention also provides an electronic device, which may be a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). This device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the aforementioned human behavior recognition method based on visual semantic text enhancement; and the memory is used to store computer programs executable by the processor.

[0128] Based on the same inventive concept, the present invention also provides a computer-readable storage medium corresponding to the aforementioned embodiments of the human behavior recognition method based on visual semantic text enhancement. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps described in any of the above embodiments.

[0129] This invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. The above embodiments are merely illustrative of several implementations of the invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention is also intended to include these modifications and variations.

Claims

1. A method for human behavior recognition based on visual semantic text enhancement, characterized in that, Including the following steps: S10. Acquire human behavior WIFI sensing data and human behavior video frame data; S20. Extract WIFI amplitude energy images from the human behavior WIFI sensing data and human behavior video frame data respectively. and high-dimensional semantic feature vectors ; S30, the WIFI amplitude energy image and the high-dimensional semantic feature vector Perform deep correlation fusion and calculate the probability distribution of human action categories. .

2. The human behavior recognition method based on visual semantic text enhancement according to claim 1, characterized in that, Step S30 includes: S31. Extract the global feature vector of WIFI based on the WIFI amplitude energy image. ; S32, The WIFI global feature vector and the high-dimensional semantic feature vector Perform association aggregation to obtain semantically enhanced feature vectors. ; S33. The semantically enhanced feature vector Perform global feature aggregation to obtain global feature vectors. ; S34. Based on the global feature vector Obtain the probability distribution of human action categories. .

3. The human behavior recognition method based on visual semantic text enhancement according to claim 2, characterized in that, The process of step S31 includes: The WIFI amplitude energy image is segmented into non-overlapping patches using Patch Embedding: Where N is the total number of segments; ; Then, each patch is mapped to a D-dimensional vector using linear projection: In the formula, The projection matrix; For bias terms; Then, positional encoding is added to each vector to preserve its original spatial location information, thus obtaining the input sequence. : In the formula, For vectors The location encoding of the corresponding patch; Finally, the input sequence is encoded using self-attention (SA) to obtain the global feature vector of WIFI. : 。 4. The human behavior recognition method based on visual semantic text enhancement according to claim 2, characterized in that, The process of step S32 includes: Using the aforementioned WIFI global feature vector As the query vector, the high-dimensional semantic feature vector The WIFI global feature vector is calculated using key and value vectors through normalization. and the high-dimensional semantic feature vector The similarity weights are calculated as follows: In the formula , and This is the weight matrix; Semantic information is injected into the WIFI global feature vector through residual connections. In the process, semantically enhanced feature vectors are obtained. : 。 5. The human behavior recognition method based on visual semantic text enhancement according to claim 2, characterized in that, The process of step S33 includes: The frozen patch features are aggregated globally using the Class-Attention layer of the CaiT framework, as follows: In the semantically enhanced feature vector Prepend a learnable [CLS]Token item. This forms a complete sequence: Attention-weighted calculation is performed on the complete sequence G: right and [CLS]Token item Perform residual join and normalization calculations: right Perform feedforward computation and then perform residual concatenation on the computation results to obtain the global feature vector. : 。 6. The human behavior recognition method based on visual semantic text enhancement according to claim 1, characterized in that, Step S20 includes: S21. Extract the WIFI amplitude energy image based on the human behavior WIFI sensing data. ; S22. Based on the human behavior video frame data, extract high-dimensional semantic feature vectors that are highly correlated with the behavior. .

7. The human behavior recognition method based on visual semantic text enhancement according to claim 6, characterized in that, Step S21 includes: S211. Preprocess the human behavior WIFI sensing data; S212, Process the pre-processed human behavior WIFI sensing data Stacked into a two-dimensional matrix M; S213. Normalize the two-dimensional matrix M to obtain the normalized matrix. ; S214. The normalized matrix Pseudo-color encoding is performed to obtain the WIFI amplitude energy image. .

8. The human behavior recognition method based on visual semantic text enhancement according to claim 6, characterized in that, Step S22 includes: S221. Construct a structured query instruction set; S222. Based on the structured query instruction set, generate action description text using the multimodal large language model paradigm; S223. Perform semantic vectorization encoding on the action description text to generate a high-dimensional semantic feature vector. .

9. The human behavior recognition method based on visual semantic text enhancement according to claim 8, characterized in that, The process of step S223 includes: Convert the action description text into a token ID sequence and add special markers [CLS] and [SEP]: Then, it is transformed into an intermediate vector E through embedding: Using the intermediate vector E as the initial input The output of layer L(L-1) is used as the input of layer L, and L layers of BERT encoding are stacked to obtain the context feature vector. : Layer 1 BERT output The calculation process is as follows: In multi-head attention computation, the single-head attention term Head can be represented as: For the kth attention head Its output is calculated according to the following formula: In the formula, and For the k-th attention head The corresponding weight matrix; Perform multi-head concatenation computation on the single-head attention term to obtain the multi-head concatenation vector. : In the formula, For projection output matrix; For the input vector E and the multi-head concatenation vector The residuals are summed, and then layer normalization is applied to obtain the intermediate feature vector Z: in, The normalization function is defined as follows: In the formula, and These are the mean and variance of parameter x, respectively; and These are the scaling parameter and the offset parameter, respectively; It is a very small constant, usually taken as This is used to ensure numerical stability; The operator represents element-wise multiplication; The intermediate vector Z is fed forward, and the result is added to the residual of the intermediate feature vector Z. Then, layer normalization is performed again to obtain the BERT output of the first layer. : in, The feedforward computation function is defined as follows: In the formula, It is the ReLU activation function. and These are the first weight term and the first bias term, respectively; and These are the second weight term and the second bias term, respectively. Assuming a total of L layers of BERT stacked computation are set up, then the context-aware vector for: Finally, the context-aware vector corresponding to the [CLS] position is taken as the high-dimensional semantic feature vector of the action description text. : 。 10. A human behavior recognition system based on visual semantic text enhancement, characterized in that, include: The signal acquisition module is used to acquire human behavior WIFI sensing data and human behavior video frame data; The signal processing module is used to extract WIFI amplitude energy images from the human behavior WIFI sensing data and the human behavior video frame data, respectively. and high-dimensional semantic feature vectors ; A multimodal fusion module is used to process the WIFI amplitude energy image. and the high-dimensional semantic feature vector Perform deep correlation fusion and calculate the probability distribution of human action categories. .

Images