Thermal discomfort action detection method and device based on fusion of time attention and multi-level features

Abstract

The invention provides a thermal discomfort action detection method and device based on time attention and multi-level feature fusion. The method comprises the following specific steps: acquiring a personnel action video; transmitting the acquired person action video to a space-time action detection network of convolution time attention and multi-scale information learning to obtain a thermal discomfort space-time action detection result; wherein the space-time action detection network based on convolutional time attention and multi-scale information learning comprises a multi-scale information learning 2D network, an enhanced space-time feature extraction 3D network and a feature fusion detection head, the multi-scale information learning 2D network is used for extracting apparent features, the enhanced space-time feature extraction 3D network is used for extracting space-time features, and the feature fusion detection head is used for fusing the features. And the feature fusion detection head is used for fusing the apparent features and the spatio-temporal features and obtaining a spatio-temporal motion detection result. The problem of inaccurate behavior recognition caused by slight changes of thermal discomfort actions and similar characterization of different actions is solved.

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of computer vision and thermal comfort, and specifically belongs to a method and device for detecting thermal discomfort movements by fusing temporal attention with multi-level features. Background Art

[0002] Thermal comfort is an individual's subjective feeling of the thermal environment, and there are significant individual differences. In order to optimize the thermal environment and reduce energy consumption, it is particularly important to obtain the thermal comfort status of indoor occupants in real time and accurately. However, traditional thermal comfort prediction methods, such as questionnaires, environmental parameter measurements, and contact physiological parameter measurements, have many shortcomings, such as strong invasiveness and non-intuitive data acquisition. In recent years, studies have pointed out that human movements can intuitively reflect their thermal comfort state, and these adaptive movements caused by temperature changes are defined as thermal discomfort movements. Detecting these movements through visual technology provides a new perspective for understanding the perception and adaptation mechanism of the human body in a thermal environment, and also opens up a new way for thermal comfort assessment.

[0003] Although existing literature has defined thermal discomfort movements based on questionnaire surveys and designed thermal comfort prediction models, the recognition of movements has not received enough attention. Yang Bin et al. used the Open Pose posture estimation method to preliminarily explore the recognition of thermal discomfort movements; Duan Wenjun et al. used a multi-stream adaptive graph convolutional network to improve the recognition effect under occlusion; Chen Mengwei proposed an end-to-end recognition algorithm based on ResNet-50 to alleviate the information loss problem of skeleton posture estimation; Wang et al. combined Open Pose with CNN to build a recognition model, but there is still room for improvement in distinguishing similar movements. Although these methods provide feasibility for using visual technology to identify thermal discomfort movements, they still face many challenges in practical applications.

[0004] Skeleton information mainly focuses on the depiction of joint positions and motion trajectories, and its recognition ability is limited for complex thermal discomfort actions involving object interactions or subtle changes, such as stomping, breathing, and putting on clothes. In addition, the behavior recognition method based on skeleton information mainly focuses on the overall classification of behavior, while accurately locating and identifying thermal discomfort actions in continuous video sequences to meet the needs of thermal comfort status assessment is still a major problem.

[0005] The temporal motion detection method can accurately identify the start and end of a specific action by deeply analyzing the dynamic changes between consecutive frames, providing technical support for the accurate identification of thermal discomfort actions. However, these methods perform well in capturing periodic and repetitive actions, but perform poorly in motion recognition involving multiple interacting objects or complex scenes. To make up for this shortcoming, researchers have proposed spatiotemporal motion detection methods that comprehensively consider temporal evolution and spatial information. However, these methods are not ideal when faced with complex and easily confused similar actions that often exist in thermal comfort scenes. For example, actions such as dressing and undressing, putting hands on the neck and lifting hair show high similarity in a specific time segment, which can easily lead to recognition confusion. These actions often represent diametrically opposite thermal comfort states, so accurately identifying them is crucial to optimizing the indoor environment.

[0006] In addition, the application of deep learning technology in thermal discomfort motion detection also relies on sufficient training samples to effectively train the model. However, there is still a lack of public thermal discomfort motion datasets in current research, which limits the widespread application of this technology in real scenarios. Therefore, future research needs to focus on how to build high-quality thermal discomfort motion datasets to promote the application and development of deep learning technology in this field. At the same time, it is necessary to continue to explore more effective spatiotemporal motion detection methods to improve the recognition accuracy and robustness of thermal discomfort motion. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present invention provides a method and device for thermal discomfort action detection by fusing temporal attention and multi-level features, establishes a convolutional temporal attention and multiscale information learning for thermal discomfort spatio-temporal action detection network (CTAMIL-Net), uses 2D CNN and 3D CNN to extract target appearance information and motion information respectively, and finally fuses the above two features through a feature fusion detection head to obtain the action detection result.

[0008] To achieve the above object, the present invention provides the following technical solution: a method for detecting thermal discomfort by integrating time attention with multi-level features, the specific steps of which are as follows: Get video of people in action; The acquired action video of the person is transmitted to the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning to obtain the spatiotemporal action detection result of thermal discomfort; Among them, the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning includes a multi-scale information learning 2D network, an enhanced spatiotemporal feature extraction 3D network and a feature fusion detection head. The multi-scale information learning 2D network is used to extract apparent features, the enhanced spatiotemporal feature extraction 3D network is used to extract spatiotemporal features, and the feature fusion detection head is used to fuse apparent features and spatiotemporal features to obtain spatiotemporal action detection results.

[0009] Furthermore, the input of the multi-scale information learning 2D network is the key frame of the personnel action video, and the output is the apparent feature. The multi-scale information learning 2D network includes a multi-level feature extractor and a multi-level feature fusion strategy, wherein the multi-level feature extractor includes 5 stages, and the multi-level feature fusion strategy includes 4 attention feature fusion modules. The low-level features output by Stage1 enter Stage2 to obtain high-level features. After the low-level features are subjected to a 3×3 convolution operation, they are input into the first attention feature fusion module with the high-level features to obtain new low-level features. The outputs of Stage3~Stage5 are high-level features. The low-level features output by the first to third attention feature fusion modules are respectively input into the corresponding next attention feature fusion module with the high-level features, and the output fusion features are the apparent features.

[0010] Furthermore, the number of convolutional layers and the specific convolution kernel settings of the multi-stage feature extractor including 5 Stage convolutional layers are as follows: Stage 1: Number of convolutional layers: 1; convolution kernel: 5×5; input channels: 3, output channels: 32; Stage 2: Number of convolutional layers: 1; convolution kernel: 3×3; input channels: 32, output channels: 64; Stage 3: Number of convolutional layers: 6; Convolution kernels: First: 3×3; input channels 64, output channels 128; Second: 1×1; input channels 128, output channels 64; Third: 3×3; input channels 64, output channels 128; Fourth: 3×3; input channels 128, output channels 256; Fifth: 1×1; input channels 256, output channels 128; Sixth: 3×3; input channels 128, output channels 256; Stage 4: Number of convolutional layers: 5; Convolution kernel: First: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; Second: 1×1; input channel 512, output channel 256; Third: 5×5; input channel 256, output channel 512; Fourth: 1×1; input channel 512, output channel 256; Fifth: 3×3; input channel 256, output channel 512; Stage 5: Number of convolutional layers: 5; Convolution kernel: First one: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; Second one: 1×1; input channel 512, output channel 256; Third one: 5×5; input channel 256, output channel 512; Fourth one: 1×1; input channel 512, output channel 256; Fifth one: 3×3; input channel 256, output channel 512.

[0011] Furthermore, it is noted that in the feature fusion module, the low-level features and the high-level features are first preliminarily fused by summing each element, and then the local context information of the high-level features and the global context information of the low-level features are obtained, and the local context information of the high-level features and the global context information of the low-level features are fused, and the fusion weight is obtained through the Sigmoid function; the fusion weight is used to realize the adaptive fusion of the high-level features and the low-level features to obtain the fusion feature, where: Local context information is passed through branches Get, expressed as follows:

[0012] in, , B For batch normalization, represents ReLu; Global context information through branches Get, expressed as follows:

[0013] Among them, global context information , B For batch normalization, represents ReLu, stands for global average pooling; The local context information is integrated into the global context information, and the fusion weight is obtained through the Sigmoid function. The fusion weight is used to process the high-level features and low-level features, as follows:

[0014] in, represents the fusion feature, represents the initial integration, Represents low-level features, represents high-level features, ⊗ is element-by-element multiplication, represents the fusion weight, .

[0015] Furthermore, the input of the enhanced spatiotemporal feature extraction 3D network is the action video frame of the person, and the output is the spatiotemporal feature; the backbone network of the enhanced spatiotemporal feature extraction 3D network is 3D-ResNeXt-101, and a 3D convolutional time attention module is inserted between Stage 2 and Stage 3 of 3D-ResNeXt-101. The 3D convolutional time attention module includes: an intra-frame static attention branch and an inter-frame dynamic attention branch. Specifically: The intra-frame static attention branch includes depth convolution, depth dilated convolution and point convolution, which are expressed as follows:

[0016] in, , It is the output feature after downsampling of Stage 2; The inter-frame dynamic attention branch includes maximum pooling, fully connected layer + ReLu and fully connected layer + Sigmoid function, which are expressed as follows:

[0017] in, , represents the Sigmoid function, FC is the fully connected layer, Represents the maximum pooling operation; Aggregate the static attention feature map and the dynamic attention feature map within the frame to obtain an attention map containing long-short time information, which is expressed as follows:

[0018] in, , , Indicates multiplication.

[0019] Furthermore, the feature fusion detection head inputs the appearance features and spatiotemporal features, and outputs the spatiotemporal action detection results; the feature fusion detection head splices the appearance features and spatiotemporal features in the channel dimension, and then uses Convolution and The convolutional layer performs semantic interaction to obtain fused spatiotemporal features; The fused spatiotemporal features use the Gram matrix to obtain the feature enhancement feature map, and the Pred layer is used to analyze and judge the feature enhancement feature map to obtain the spatiotemporal action detection result. The Gram matrix is ​​fused to obtain the spatiotemporal action detection results.

[0020] Furthermore, the spatiotemporal action detection network with convolutional temporal attention and multi-scale information learning is trained and tested using the thermal discomfort action dataset TDA. The construction method of the thermal discomfort action dataset TDA is as follows: 1) The thermal discomfort action dataset TDA contains 10 types of thermal discomfort actions, namely: hugging arms, rubbing hands, stamping feet, putting on clothes, breathing, fanning, shaking clothes in front of the chest, wiping sweat, taking off clothes and rolling up sleeves; 2) Recruit 7 subjects aged between 22 and 26, and use RGB cameras to collect 10 types of thermal discomfort action video data in real indoor scenes of office buildings: ① The subject arrives at the designated area; ② Select a type of action; ③ Take action videos from the front, side, and 45° angle at both far and near distances; ④ Change different action types; ⑤ Repeat ①~④; A total of 3528 unedited videos were collected, with a video length of 10~30s and a uniform resolution of 640 480; 3) The collected video is divided into frames using video frame cutting technology and saved in JPG format, with each type of action saved in a separate folder; after removing redundant images, the data set is annotated and the labels are saved as .txt files.

[0021] The present invention also provides a thermal discomfort action detection system combining time attention and multi-level feature fusion, comprising: A data acquisition module, used to acquire videos of people's actions; The motion detection module is used to transmit the acquired personnel motion video to the spatiotemporal motion detection network of convolutional temporal attention and multi-scale information learning to obtain the thermal discomfort spatiotemporal motion detection results; Among them, the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning includes a multi-scale information learning 2D network, an enhanced spatiotemporal feature extraction 3D network and a feature fusion detection head. The multi-scale information learning 2D network is used to extract apparent features, the enhanced spatiotemporal feature extraction 3D network is used to extract spatiotemporal features, and the feature fusion detection head is used to fuse apparent features and spatiotemporal features to obtain spatiotemporal action detection results.

[0022] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for detecting thermal discomfort movements by integrating temporal attention with multi-level features are implemented.

[0023] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned method for detecting thermal discomfort movements by integrating time attention with multi-level features.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention provides a method for detecting thermal discomfort motion by fusing temporal attention and multi-level features. Aiming at the problem of inaccurate behavior recognition caused by subtle changes in thermal discomfort motion and similar representations of different motions, the present invention proposes a convolutional temporal attention and multi-scale information learning thermal discomfort spatiotemporal motion detection network (CTAMIL-Net). A multi-scale information learning 2D network (MSILN) is designed in the network, which uses different-scale convolutional layers to extract feature information of different granularities from key frames, and adaptively aggregates high-level semantics and low-level detail features to capture target appearance features and perceive subtle changes in motion; at the same time, a 3D convolutional temporal attention (3D-CTA) is proposed to capture short-to-long-term temporal dependencies between frames and within frames, respectively, and mine more discriminative target motion features to jointly improve the discrimination of similar motions; then, a feature fusion detection head is used to semantically interact 2D appearance features and 3D motion features and obtain motion detection results. Experiments were conducted on the self-built dataset TDA and the public dataset UCF-24, with Frame-mAP values ​​of 78.28% and 85.41%, and FPS of 35 and 40. The experimental results show that the algorithm of the present invention can effectively extract the target appearance information and motion information, realize real-time personnel motion detection, and the objective evaluation index on the dataset is better than other advanced algorithms. The present invention uses computer vision technology to identify thermal discomfort movements, integrates the thermal discomfort movement recognition results into the smart home (office) system, and automatically adjusts indoor environmental parameters (such as temperature, humidity, wind speed, etc.) through intelligent algorithms to meet the thermal comfort needs of different members in different time periods and scenarios, thereby providing an innovative and efficient new way for contactless human thermal comfort prediction.

[0025] Preferably, the multi-scale information learning 2D network uses a different-scale convolution layer to capture feature information of multiple granularities from the input key frame, and then performs feature downsampling and channel dimension expansion operations to ensure that the spatial dimensions of high-level and low-level feature maps are consistent, and the adaptive aggregation of high-level semantic features and low-level detail features is realized through the attention feature fusion module, extracting fine-grained target appearance features, and deeply mining the change information of subtle movements; the enhanced spatiotemporal feature extraction 3D network takes a video clip including key frames as the input of the network, and on the basis of using the backbone network 3D-ResNeXt 101 to deeply mine the temporal and spatial information in the video data, 3D convolutional temporal attention (3D Convolutional Temporal Attention, 3D-CTA) captures the long-term-short-term temporal dependencies within and between frames, and enhances the ability to capture and pay attention to feature changes in key time periods, thereby more accurately identifying dynamic behaviors in the video; Preferably, in view of the scarcity of current data sets, the present invention constructs a thermal discomfort action data set (ThermalDiscomfort Action Dataset, TDA) to provide a data basis for the research of thermal discomfort spatiotemporal action detection methods, solving the problem of lack of public thermal discomfort action recognition data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a diagram of the spatiotemporal action detection network architecture of the present invention's convolutional temporal attention and multi-scale information learning; Figure 2 This is the structure diagram of the multi-scale information learning 2D network MSILN of the present invention; Figure 3 This is the structure diagram of the attention feature fusion module AFF of the present invention; Figure 4 3D-CTA structure diagram for 3D convolutional time attention; Figure 5 This is the structural diagram of the feature fusion detection head; Figure 6 This is an example of the thermal discomfort action dataset TDA; Figure 7 The spatiotemporal motion visualization detection results for the thermal discomfort action dataset TDA; Figure 8 Visualize spatiotemporal action detection results for the UCF-24 dataset; Fig. 9 This is a diagram showing the frame-level detection effect of the method of the present invention in complex and changing scenes. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0028] The present invention provides a method for detecting thermal discomfort motion by integrating temporal attention and multi-level features, and realizes real-time personnel motion detection by establishing a spatiotemporal motion detection network of convolutional temporal attention and multi-scale information learning (CTAMIL-Net). Figure 1As shown in the figure, the CTAMIL-Net network is mainly composed of three parts: Multi-Scale Information Learning 2D Networks (MSILN), Enhanced Spatio-Temporal Feature E×traction 3D Network (ESTFEN) and Feature Fusion Detection Head (FFDH). The MSILN network uses different-scale convolutional layers to extract feature information of different granularities from key frames, and adaptively aggregates high-level semantics and low-level detail features through the attention feature fusion module (AFF) to capture the target appearance features and perceive the subtle changes in actions; the ESTFEN network captures short-to-long-term temporal dependencies between frames and within frames respectively, and uses the 3D backbone network (3D-ResNeXt-101) to mine more discriminative target motion features to jointly improve the distinction between similar actions; then, the FFDH detection head is used to interact the 2D appearance features and 3D motion features semantically and obtain the action detection results, as follows: 1. Multi-Scale Information Learning 2D Networks (MSILN) The characteristics of heat discomfort actions such as breathing, rubbing hands, and stamping feet are that they change slightly and minutely. Therefore, it is easy to ignore or miss these subtle changes during the capture process of sensors or cameras, resulting in insufficient characterization of action features, which increases the difficulty of behavior recognition. To remedy this shortcoming, the present invention contemplates the use of a multi-level feature extraction method, using small convolution kernels to capture local detail features and large convolution kernels to extract global abstract features, so as to comprehensively extract multi-level feature information in video frames. However, feature extraction alone is not enough to improve its expressive power, so it is also necessary to combine feature fusion strategies to make full use of these features of different granularities. Based on the above analysis, the present invention designs a multi-scale information learning 2D network (MSILN), such as Figure 2 As shown in the figure, it consists of two parts: Multi-Level Feature E×traction (MLFE) and Multi-Level Feature Fusion (MLFF), where: 1.1. Multi-Level Feature Extraction (MLFE) In order to realize the real-time and accurate detection of thermal discomfort actions, the present invention designs a multi-level feature extractor, such as Figure 2As shown in the figure, it consists of 5 stages. Stage 1 contains a convolutional batch normalization layer (CBL layer) and a downsampling operation, which is mainly responsible for the current key frame image of the input video. Perform preliminary shallow feature extraction and scale transformation, where: C Indicates the number of channels of the input image, H and W Represent the height and width of the input image respectively. From Stage 2 to Stage 4, the feature extraction process gradually deepens, and the number of convolution layers in each Stage is different to ensure that the network can gradually go deep into the subtle details of the image during the feature extraction process, so as to understand the image content more comprehensively. At the same time, these stages set convolution kernels of different sizes (1×1, 3×3, 5×5) to improve the feature extraction capability. Specifically, large convolution kernels (such as 5×5) can cover a larger image area in the convolution operation, thereby capturing more extensive and global feature information, including spatial context information in the image, and providing strong support for extracting higher-level semantic features. In contrast, small convolution kernels (such as 1×1 or 3×3) can finely capture low-level information such as edges and textures in the image due to their smaller receptive fields. By combining the use of convolution kernels of different scales to obtain these two types of features, MLFE can capture global structural information while maintaining local details, realizing the input keyframe. Multi-level feature extraction and output of different levels of features , Respectively represent the number of channels, height and width of the feature map F; The number of convolutional layers and the specific convolution kernel settings of the five Stage convolutional layers are as follows: Stage 1: Number of convolutional layers: 1; convolution kernel: 5×5; input channels: 3, output channels: 32; Stage 2: Number of convolutional layers: 1; convolution kernel: 3×3; input channels: 32, output channels: 64; Stage 3: Number of convolutional layers: 6; Convolution kernels: First: 3×3; input channels 64, output channels 128; Second: 1×1; input channels 128, output channels 64; Third: 3×3; input channels 64, output channels 128; Fourth: 3×3; input channels 128, output channels 256; Fifth: 1×1; input channels 256, output channels 128; Sixth: 3×3; input channels 128, output channels 256; Stage 4: Number of convolutional layers: 5; Convolution kernel: First: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; Second: 1×1; input channel 512, output channel 256; Third: 5×5; input channel 256, output channel 512; Fourth: 1×1; input channel 512, output channel 256; Fifth: 3×3; input channel 256, output channel 512; Stage 5: Number of convolutional layers: 5; Convolution kernel: First: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; Second: 1×1; input channel 512, output channel 256; Third: 5×5; input channel 256, output channel 512; Fourth: 1×1; input channel 512, output channel 256; Fifth: 3×3; input channel 256, output channel 512; 1.2 Multi-level feature fusion strategy MLFF In view of the differences in scales of feature maps at different levels of MLFE output, the present invention first performs a 3×3 convolution operation on the extracted low-level features to achieve feature downsampling and channel dimension expansion, and then converts the feature maps From The branch scale is converted to The scale of the branch is adjusted to ensure that the feature maps of different levels are consistent in spatial dimensions. and high-level features Input to the attention feature fusion module (Attention Feature Fusion Module, AFF) of the multi-level feature fusion strategy MLFF. Figure 3 As shown, the low-level features are firstly summed element by element. and high-level features Perform preliminary fusion to ensure that information at different levels can complement and benefit each other. Then, the module uses two parallel branches and branches Perform feature refinement and calculate the contribution of each feature map to the fusion result to generate the corresponding weights Compared with simple feature fusion methods, AFF can dynamically adjust the fusion weights between features at different levels, thereby achieving more accurate and efficient feature integration. Realize adaptive fusion of features and output refined target appearance information feature map , specifically: 1.2.1 Branches Point-by-point convolution is used as a local channel context aggregator to capture subtle changes in low-level features through convolution operations, effectively retaining and highlighting detail information, as shown in formula (1): (1) in, , B is batch normalization (BN), Representation (ReLu); 1.2.2 Branches By adjusting the size of the spatial pool, a channel attention mechanism is constructed at multiple scales to evaluate the importance of each feature channel and calculate the weight coefficient, which is used to weightedly fuse the features of different channels and capture the global context information of high-level features, as shown in formula (2): (2) in, , B is batch normalization (BN), ReLu, Stands for Global Average Pooling (GAP).

[0029] 1.2.3. Local context information Global context information incorporated into the attention module In , the adaptive fusion of features is achieved by calculating the weighted sum, as shown in formula (3): (3) in, is the refined feature after processing, , Represents the Sigmoid function, ⊗ represents element-by-element multiplication, and ⊕ represents element-by-element addition.

[0030] 1.2.4. The fused features retain low-level details and integrate high-level global context information to capture subtle action changes, as shown in formula (4): (4) in, represents the fusion feature, represents the initial integration, Represents low-level features, represents high-level features, ⊗ is element-by-element multiplication, Represents the fusion weight, which consists of a real number between 0 and 1, which enables the network to perform soft selection or weighted averaging between high-level and low-level features, alleviating the problem of inconsistent feature scales.

[0031] 2. Enhanced Spatiotemporal Feature Extraction 3D Network (ESTFEN) By learning 2D networks with multi-scale information, it is possible to accurately capture the apparent features of key frames and thus identify the actions occurring at the current moment. However, relying solely on key frames may lead to ambiguity in the interpretation of certain actions, such as dressing and undressing, which may present similar action patterns in different time periods. To address the above problems, the present invention proposes an enhanced spatiotemporal feature extraction 3D network (ESTFEN), whose overall structure is as follows: Figure 1 As shown in the figure. The network takes a video clip including key frames as input, and deeply mines the temporal and spatial information in the video data through the efficient backbone network 3D-ResNeXt-101. At the same time, 3D convolutional temporal attention (3D-CTA) is designed to capture the long-term and short-term temporal dependencies within and between frames, and enhance the ability to capture and pay attention to the changes in the features of the key time period. When processing video sequences, the model can more accurately focus on and capture the changes in the features of the key time period, thereby realizing accurate recognition of dynamic behaviors in the video.

[0032] 2.1. Backbone Network 3D-ResNeXt-101 3D-ResNeXt-101 combines the high efficiency of ResNeXt-101 and the processing ability of 3D convolution on spatiotemporal information. It has potential advantages in application scenarios with complex spatiotemporal information such as video understanding. Its structure is as follows Figure 1 As shown in the figure. Similar to the standard ResNeXt-101, 3D-ResNeXt-101 also uses group convolution to reduce the number of parameters and computation. By splitting the input channels into multiple groups and performing three-dimensional convolution operations independently on each group, the network can reduce computational complexity while maintaining high performance. The 3D-ResNeXt-101 network structure diagram has 101 layers, the first layer is 7 7 7 convolutions for preliminary feature extraction and transformation. The subsequent 4 stages (Stage1~Stage4) gradually extract deep features. Each stage contains multiple 3D residual blocks, each of which consists of two 3D residual blocks. 3 3 convolutional layers with batch normalization and ReLU activation functions. The first residual block in each stage uses 1 1 The convolutional layer of 1 converts the number of input channels to the number of output channels for addition operation with the subsequent residual block.

[0033] Table 1 3D-ResNeXt-101 network structure

[0034] 2.2. 3D Convolutional Temporal Attention (3D-CTA) By introducing temporal attention, not only can different frames in the action sequence be weighted, making the model pay more attention to key frames, but also the timing information of the action sequence can be learned, and the evolution process of the action can be better understood. However, relying solely on temporal attention may still be insufficient to solve certain challenges in complex action recognition problems, such as confusion and ambiguity between actions. Considering that large kernel convolution can capture long-distance dependencies in the input data, help the model better understand the global structure of the action, and reduce confusion caused by local similarities, the present invention believes that combining large kernel convolution with temporal attention can help improve the model's ability to understand action sequences, thereby improving the distinction between similar actions. Based on this, a 3D convolutional temporal attention module (3D-CTA) is proposed, with a structure such as Figure 4 As shown in the figure, the depth of the 3D-ResNeXt-101 network increases with the increase of Stage, and the information contained becomes richer. The present invention selects the output feature X of Stage 2. As the input of 3D-CTA, N represents the input sample size, i.e., Batch size, C represents the number of channels, D represents the depth of the input video frame sequence, and H and W represent the height and width of the input image.

[0035] The input feature X of 3D-CTA is as shown in formula (5): 1 1 After convolution downsampling, the downsampled output features are obtained : (5) Among them, GELU is the activation function; Output features after downsampling Enter 3D-TAM to generate an attention map containing long-short time sequence information, and then apply the attention map to the original features to enhance the representation of time sequence related features, as shown in formula (6): (6) in, is the output feature after downsampling, and GELU is the activation function; The present invention divides 3D-TAM into an intra-frame static attention branch (Static Attention, SA) and an inter-frame dynamic attention branch (Dynamic Attention, DA). The principle is as follows Figure 4 As shown in (a), the structure is Figure 4 (b) as shown.

[0036] 2.2.1. The intra-frame static attention branch (SA) uses deep convolution (3D DW Conv), deep dilated convolution (3DDW-D Conv) and point convolution (1 1 1 Conv) to simulate large kernel convolution to obtain a larger receptive field, thereby capturing long-term temporal relationships, and then estimating the importance of feature points and generating feature maps with intra-frame static attention, as shown in formula (7): (7) in, .

[0037] 2.2.2, such as Figure 4 As shown in (a), the inter-frame dynamic attention branch (DA) is squeezed and incentives The attention weight of the channel is learned in a way to adaptively recalibrate the feature response of the channel direction, thereby learning the time change between frames. When applied in the present invention, it is embodied as follows Figure 4 (b) As shown in formula (8), a feature map with inter-frame dynamic attention branch is obtained.

[0038] (8) in, , represents the Sigmoid function, FC is the fully connected layer, Represents the maximum pooling operation; 2.2.3. Aggregate the static attention feature map and the dynamic attention feature map in the frame through formula (9) to obtain an attention map containing long-term and short-term temporal information: (9) in, , , Indicates multiplication.

[0039] 3. Feature fusion detection head Existing detection heads often use feature splicing or weighted summation to fuse features, and then obtain the results through the Pred layer. However, such detection heads often cannot effectively focus on key feature areas and channels when performing action detection, resulting in reduced detection performance. In order to solve this problem and improve accuracy, the present invention adopts a method of splicing first and then performing semantic interaction in the feature fusion stage, and introduces a Gram matrix that can learn spatial distribution correlation and inter-channel feature correlation, forming a feature fusion detection head with a "feature fusion + attention mechanism" strategy. Its structure is as follows: Figure 5 shown.

[0040] The feature fusion part first combines the apparent features extracted by the multi-scale information learning network Spatial-temporal features extracted by 3D network with enhanced spatial-temporal feature extraction Perform splicing on the channel dimension, and then use the spliced ​​features Input to Convolution and Semantic interaction is carried out in the convolution layer to obtain fully integrated spatiotemporal features. .

[0041] The spatial distribution of actions in videos is regular and structured, and different feature channels correspond to different action features, so accurately capturing these features is crucial for action detection. The Gram matrix calculates the inner product between feature points, which not only captures the spatial distribution correlation and helps the model understand the action layout; it also reveals the correlation between channels, indicating which channels are more critical for specific action detection, and thus guiding the model to focus more on these important feature channels. Therefore, the present invention combines the fused spatiotemporal features into a Learning is performed through the Gram matrix to enhance the feature representation ability in action detection.

[0042] Finally, the Pred layer is used to analyze and judge the processed features to obtain the spatiotemporal action detection results.

[0043] 4. Loss Function The total loss function of the present invention L It is composed of detection loss and classification loss, and aims to comprehensively and accurately measure the performance of the model in the action detection task. It is defined as follows: (10) in, and is the indicator function; It is classification prediction, regression prediction and confidence prediction. For basic facts.

[0044] The confidence loss and classification loss used in the present invention are the same as those of YOWO, and GIoU loss is used as Bo×Regression.

[0045] 6. Experiment and analysis 6.1 Data Acquisition In order to verify the effectiveness and generalization ability of the proposed algorithm, this paper builds the TDA dataset and conducts experiments. At the same time, comparative experiments are set up on UCF-24 and AVA. Through comprehensive verification of the two datasets, the algorithm performance is comprehensively evaluated.

[0046] 6.1.1 Thermal Discomfort Action Dataset (TDA) In order to promote the application and development of video detection technology in the field of thermal discomfort action recognition research, based on the previous laboratory research results, the present invention has identified the 10 types of thermal discomfort actions most related to thermal discomfort in daily life, namely hugging arms, rubbing hands, stamping feet, putting on clothes, breathing, fanning, shaking clothes in front of the chest, wiping sweat, taking off clothes and rolling up sleeves, and established a thermal discomfort action (TDA) dataset. The experiment recruited 7 subjects aged between 22 and 26 years old, and used RGB cameras to collect 10 types of thermal discomfort action video data in real indoor scenes of office buildings: ① The subjects arrived at the designated area; ② Select a type of action; ③ Shoot action videos from the front, side, and at a 45° angle at both far and near distances; ④ Change different action types; ⑤ Repeat ①~④. A total of 3528 unedited videos were collected, with a video length of 10~30s and a unified resolution of 640 480. The collected video is divided into frames using video frame cutting technology and saved in JPG format. Each type of action is saved in a separate folder. After removing redundant images, the dataset is annotated using the make-sense tool and the labels are saved as .txt files. The dataset example is as follows Figure 6 shown.

[0047] 6.1.2 Public Datasets As a public and widely used spatiotemporal action detection dataset, UCF-24 can provide a reference for the benchmark performance of the algorithm of the present invention. The dataset contains 24 different daily life action categories such as making phone calls, dancing, and playing basketball. Each action category contains multiple video clips with different resolutions, frame rates, shooting angles, and lighting conditions, so that it can be used for action recognition in complex situations. The dataset contains 3207 uncut videos and provides corresponding spatiotemporal annotations. Each frame may have multiple action instances.

[0048] 6.2 Experimental Details The proposed method is implemented based on the PyTorch deep learning framework, and the Intel ×eon E5 2650 processor and NVIDIA GeForce RT× 2080 Ti 11G graphics card are used for model training and testing. The optimizer of the experiment is AdamW, the initial learning rate is 0.0001, the weight decay is 0.0005, and 5 epochs are trained on the public dataset and the thermal discomfort action dataset, and the learning rate is decayed to 10 times on the 1st, 2nd, 3rd, and 4th epochs respectively; the batch size is 6, and the size of the input frame is randomly cropped to 224 224.

[0049] In the field of spatiotemporal action detection, Frame-mAP (F-mAP) and Video-mAP (V-mAP) are often used to evaluate action detection performance. Frame-mAP follows the PASCAL VOC 2012 metric standard. Its core purpose is to measure the area under the precision and recall curves of the detection results of each frame, thereby comprehensively reflecting the performance of the algorithm on each frame. Video-mAP is specifically used to measure the accuracy of action pipeline generation to ensure accurate recognition of action sequences. In addition, FPS (Frames Per Second, frames per second) is another important indicator, which is mainly used to measure the detection speed of the action detection algorithm. In order to better compare performance, the present invention also selects the above evaluation indicators.

[0050] 6.3 Experimental Results and Analysis 6.3.1 Analysis of TDA Experimental Results on Self-built Dataset In order to verify the effectiveness of the self-built dataset in the thermal discomfort motion detection task, the present invention carried out detection accuracy and efficiency comparison experiments on multiple algorithms. The experiment adopted the parameter settings disclosed by the original author to maintain comparability and eliminate the interference of parameter adjustment. The experimental results are summarized in Table 2, where the best results are highlighted in bold font.

[0051] Table 2. Experimental results of multi-algorithm comparison on self-built dataset

[0052] Note: K is the number of video frame inputs, which is the same as the source code setting; the FPS index of this invention is measured on an NVIDIA GeForce RT× 2080 Ti graphics card.

[0053] The results show that the model proposed in the present invention has significant advantages in detection accuracy. Compared with the previous advanced methods MOC, YOWO, YOWO-Plus, YOWOv2-T, YOWOv2-M, and YOWOv2-L, the detection accuracy of the model of the present invention is improved by 24.47%, 7.2%, 1.5%, 10.97%, 17.66%, and 7.44%, respectively. This fully verifies the ability of the method of the present invention in mining information on subtle movement changes. Specifically, for subtle movements such as breathing and rubbing hands, the small-scale convolution kernels in MSILN can more finely extract the tiny movement features of body parts, while the large-scale convolution kernels help to understand the position and relationship of these subtle movements in the overall movement. In addition, although the detection speed is slightly inferior to MOC and YOWOv2, which use a lightweight backbone network structure, the method of the present invention achieves relatively efficient detection while maintaining high accuracy through multi-scale information learning and enhanced spatiotemporal feature extraction. In summary, the algorithm of the present invention effectively captures and identifies subtle changes in thermal discomfort movements, achieving high-precision and high-efficiency movement detection.

[0054] 6.3.2 Analysis of Experimental Results on Public Datasets To prove the advancedness of the algorithm proposed in this paper, this paper compares the proposed algorithm with mainstream spatiotemporal action detection methods on the public dataset UCF-24, including MOC, STEP, YOWO, TubeR, T-CNN, HISAN, CFAD, SAMOC, ACAR-Net, YOWO-v2, etc. The input of the algorithm uses RGB image or optical flow mode. The experimental results are shown in Table 3. Table 3. Comparative experiments on multiple algorithms on the UCF-24 public dataset

[0055] Note: " / " in the table means that the source code of the original document is not given, so the indicator cannot be measured; the FPS indicator of this invention is measured on the NVIDIA GeForce RT× 2080 Ti graphics card; It can be seen that the F-mAP values ​​of the proposed algorithm are higher than those of other algorithms, and compared with the advanced algorithms based on the combination of 2D and 3D convolution (such as YOWO and YOWO-v2) in the table above, the spatiotemporal action detection performance of the proposed algorithm is also significantly improved (compared with YOWO-V2, under the same conditions, the proposed algorithm has an F-mAP improvement of 2.38% and a V-mAP improvement of 0.49%). This is because the proposed algorithm can not only capture static and dynamic temporal features through the temporal attention module, but also extract more discriminative behavior space features from different levels through the multi-level feature extraction and fusion network. The proposed algorithm has a competitive advantage over other single-modal input algorithms in terms of F-mAP, but in terms of V-mAP, the proposed algorithm is slightly inferior to the results of the algorithm using RGB and optical flow multimodal input in the literature. This shows that the introduction of multimodal information can enhance the performance of the algorithm. However, since it is difficult to obtain high-quality optical flow data in real time online in practical applications, this not only limits the practicality of the model, but also may lead to a decrease in detection speed and an increase in algorithm complexity. In terms of real-time performance, the algorithm of the present invention has F-mAP and V-mAP values ​​of 85.41% and 51.38%, and an FPS of 40 on a RT× 2080Ti GPU. Compared with other algorithms, it achieves a better balance between detection accuracy and speed. Even compared with the latest YOWO-V2 model, the real-time CTAMIL-Net of the present invention still shows competitive performance.

[0056] 7. Ablation experiment 7.1. 3D-CTA insertion position experiment In order to explore the optimal application position of 3D convolutional temporal attention (3D-CTA) in the model, this paper designs a series of comparative experiments and embeds 3D-CTA into different stages of the 3D-ResNeXt-101 network to evaluate its impact on the spatiotemporal feature extraction capability. The experimental results are shown in Table 5.

[0057] Table 5 Experiments on the best application position of 3D-CTA in 3D-ResNeXt-101 network

[0058] The results show that when only 3D-ResNeXt-101 is used without 3D-CTA, the network lacks effective attention to temporal information, and its ability to extract spatiotemporal features is relatively weak, resulting in low action detection accuracy. After the introduction of 3D-CTA, the ESTFEN network's ability to extract spatiotemporal features in video data is effectively enhanced, thereby achieving significant improvements in multiple evaluation indicators. Further analysis shows that when 3D-CTA is between Stage 2 and Stage 3, the action detection effect is optimal. This is because as the stage deepens, the feature extraction ability of 3D-ResNeXt-101 is enhanced, and the temporal attention can focus more accurately on the temporal information, improving the understanding of dynamic features. However, the effect decreases when placed between Stage 3 and Stage 4, probably because although the deep features have strong semantic information after multi-layer processing, their ability to describe details is weakened, affecting the accurate capture of temporal information by 3D-CTA.

[0059] 7.2. Validation Experiments of Different Modules

[0060] Table 5 Ablation experiments on the self-built dataset TDA and the public dataset UCF-24

[0061] From the experimental results, we can see that compared with the baseline model, after adding MLFF, the performance indicators of the model on both the self-built dataset and the public dataset have been improved, which fully proves the effectiveness of this strategy in aggregating high-level semantic features and low-level detail features. Specifically, the multi-level feature fusion strategy effectively aggregates high-level semantic features containing more global and abstract information and low-level detail features that retain the original details of the image, realizing the complementarity of the two types of information, so that the model can grasp both global information and detail information. After adding MSILN, on the UCF-24 dataset, F-mAP increased by 0.95% and V-mAP increased by 0.62%; on the self-built dataset TDA, F-mAP increased by 0.22%, which shows that MSILN improves the extraction and expression capabilities of apparent features. After adding the 3D convolutional temporal attention module, F-mAP increased by 0.59% and V-mAP increased by 2.23% on UCF-24; on TDA, F-mAP increased by 2.5%, indicating that 3D-CTA has good temporal modeling capabilities and improves the performance of the spatiotemporal action detection network. With the integration of the above two modules in the network, F-mAP continued to improve on UCF-24, and V-mAP also improved compared to the baseline model and after the introduction of MSILN; on TDA, F-mAP increased by 5.85%, indicating that the network structure designed by the present invention can make full use of the rich information from local details to global context, and can also capture the temporal information of the action sequence, verifying the rationality and effectiveness of the algorithm of the present invention.

[0062] 7.3 Visualization and qualitative analysis of experimental results In order to intuitively understand the effect of the text method, this section conducts visual experimental analysis on the UCF-24 dataset and the self-built dataset TDA. Figure 7 and Figure 8 As shown, in order to more clearly show the dynamic change process of the action, the video sequence result image is extracted every 2 frames. It can be seen that: 1) The algorithm of the present invention has achieved remarkable results in solving the problem of action ambiguity. Figure 7 The similarity of the dressing and undressing actions in (b) and (c) requires the algorithm to rely on time series information for accurate recognition. The algorithm of the present invention has no false detection when recognizing such actions, effectively alleviating the problem of action ambiguity, and demonstrating the advantages of the algorithm in terms of time series information utilization and recognition accuracy. 2) If Figure 7 (d)(e) and Figure 8 As shown in (a)(b)(d), the algorithm of the present invention can not only accurately identify subtle and complex movements such as rubbing hands and fanning, but also detect the movements of multiple individuals at the same time in a multi-person scene. 3) The algorithm of the present invention has good behavior recognition results in complex environments and can locate the target person. Figure 7 In (a) and (c), the background information color is similar to the color of the actor. Figure 8 In the scene (e) with complex environment and dense population, the algorithm of the present invention can still accurately identify the actions of the target person and mark the action location with a label box. 4) From Figure 7 and Figure 8 It can be seen that the linking algorithm of the present invention can effectively utilize the frame-level detection results, accurately locate the action according to the action category and the motion area, and ensure the accuracy and consistency of the detection results.

[0063] Fig. 9The frame-level detection effect of the method of the present invention in complex and changeable scenes is demonstrated. It can be seen that in the recognition of thermal discomfort actions in the first and second rows, the method of the present invention not only has the ability to accurately distinguish individual behaviors and actions, but also can infer the thermal sensation state of the human body based on these actions, that is, cold or hot, which lays the foundation for thermal environment regulation based on thermal comfort; in addition, the method of video detection effectively alleviates the problem of decreased behavior recognition performance caused by incomplete skeleton information. For example, in the actions of breathing, fanning, riding, etc. shown in the figure, although some joint point information is missing, the video-based method can still accurately capture the action features, avoiding the possible misjudgment of the skeleton-based method; and for actions such as dressing, undressing, and fencing that require interaction with objects, the method of the present invention realizes the correct detection of the action by integrating background information and time information; finally, for the actions of playing basketball, diving, riding, etc. with large scene range and small moving targets shown in the third row, the method of the present invention successfully realizes the accurate detection of small target actions by extracting and utilizing multi-scale features. In general, the method of the present invention shows a certain performance stability in a variety of complex environments.

[0064] The present invention proposes a method for detecting thermal discomfort movements by fusing temporal attention with multi-level features, and has achieved remarkable results in the study of thermal discomfort movement detection as an example. In terms of network architecture, a multi-scale information learning 2D network is designed to extract the appearance information features of the target person; 3D convolutional temporal attention is used to capture long-term and short-term temporal dependencies, and combined with a backbone network (3D-ResNeXt-101) to enhance spatiotemporal feature extraction; a feature fusion detection head is used to fuse the appearance features with the spatiotemporal features to obtain the movement detection results. In order to solve the problem of data scarcity, the present invention also constructs a thermal discomfort movement dataset in real scenarios.

[0065] Experimental results show that on the public dataset UCF-24, the Frame-mAP value is 85.41% and the video-mAP value is 51.38%. The objective evaluation index of the algorithm of the present invention is better than other advanced algorithms. On the self-built dataset, the Frame-mAP value is 78.28% and the FPS is 35, which fully proves the effectiveness of the algorithm of the present invention in real-time thermal discomfort motion detection. Compared with the mainstream spatiotemporal motion detection algorithm, the algorithm of the present invention shows higher detection accuracy. It can not only accurately identify thermal discomfort motion, but also show excellent adaptability and generalization ability in other human behavior detection tasks.

[0066] The following are device embodiments of the present invention, which can be used to perform method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0067] In yet another embodiment of the present invention, a thermal discomfort action detection system combining time attention with multi-level features is provided, which can implement the operation of the thermal discomfort action detection method combining time attention with multi-level features, and includes a data acquisition module and an action detection module. Specifically: A data acquisition module, used to acquire videos of people's actions; The motion detection module is used to transmit the acquired personnel motion video to the spatiotemporal motion detection network of convolutional temporal attention and multi-scale information learning to obtain the thermal discomfort spatiotemporal motion detection results; Among them, the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning includes a multi-scale information learning 2D network, an enhanced spatiotemporal feature extraction 3D network and a feature fusion detection head. The multi-scale information learning 2D network is used to extract apparent features, the enhanced spatiotemporal feature extraction 3D network is used to extract spatiotemporal features, and the feature fusion detection head is used to fuse apparent features and spatiotemporal features to obtain spatiotemporal action detection results.

[0068] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be a method for detecting thermal discomfort by integrating time attention and multi-level features.

[0069] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the thermal discomfort action detection method of a time attention and multi-level feature fusion in the above embodiment.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting thermal discomfort by combining temporal attention with multi-level features, characterized in that: The specific steps are as follows: Get video of people in action; The acquired action video of the person is transmitted to the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning to obtain the spatiotemporal action detection result of thermal discomfort; Among them, the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning includes a multi-scale information learning 2D network, an enhanced spatiotemporal feature extraction 3D network and a feature fusion detection head. The multi-scale information learning 2D network is used to extract apparent features, the enhanced spatiotemporal feature extraction 3D network is used to extract spatiotemporal features, and the feature fusion detection head is used to fuse apparent features and spatiotemporal features to obtain spatiotemporal action detection results.

2. According to claim 1, a method for detecting thermal discomfort by combining temporal attention with multi-level features is characterized in that: The input of the multi-scale information learning 2D network is the key frame of the personnel action video, and the output is the apparent feature. The multi-scale information learning 2D network includes a multi-level feature extractor and a multi-level feature fusion strategy, wherein the multi-level feature extractor includes 5 stages, and the multi-level feature fusion strategy includes 4 attention feature fusion modules. The low-level features output by Stage1 enter Stage2 to obtain high-level features. After the low-level features are subjected to a 3×3 convolution operation, they are input into the first attention feature fusion module with the high-level features to obtain new low-level features. The outputs of Stage3~Stage5 are high-level features. The low-level features output by the first to third attention feature fusion modules are respectively input into the corresponding next attention feature fusion module with the high-level features, and the output fusion features are the apparent features.

3. According to claim 2, a method for detecting thermal discomfort by combining time attention with multi-level features is characterized in that: The multi-stage feature extractor includes 5 Stage convolution layers. The number of convolution layers and the specific convolution kernel settings are as follows: Stage 1: Number of convolutional layers: 1; Convolution kernel: 5×5; input channels 3, output channels 32; Stage 2: Number of convolutional layers: 1; Convolution kernel: 3×3; input channels 32, output channels 64; Stage 3: Number of convolutional layers: 6; Convolution kernel: The first one: 3×3; input channels 64, output channels 128; The second one: 1×1; 128 input channels, 64 output channels; The third one: 3×3; 64 input channels, 128 output channels; The fourth one: 3×3; input channels 128, output channels 256; The fifth one: 1×1; input channels 256, output channels 128; The sixth one: 3×3; input channels 128, output channels 256; Stage 4: Number of convolutional layers: 5; Convolution kernel: The first one: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; The second one: 1×1; input channels 512, output channels 256; the third one: 5×5; input channels 256, output channels 512; The fourth one: 1×1; input channels 512, output channels 256; The fifth one: 3×3; input channels 256, output channels 512; Stage 5: Number of convolutional layers: 5; Convolution kernel: The first one: 3×3; input channel 256; after pooling, it is equivalent to 512, and the output channel is 512; The second one: 1×1; input channels 512, output channels 256; the third one: 5×5; input channels 256, output channels 512; The fourth one: 1×1; 512 input channels, 256 output channels; the fifth one: 3×3; 256 input channels, 512 output channels.

4. According to claim 2, the method for detecting thermal discomfort by combining time attention with multi-level features is characterized in that: Note that in the feature fusion module, the low-level features and high-level features are firstly fused by element-by-element summation, and then the local context information of the high-level features and the global context information of the low-level features are obtained, and the local context information of the high-level features and the global context information of the low-level features are fused, and the fusion weight is obtained by the Sigmoid function; The fusion weights are used to achieve adaptive fusion of high-level features and low-level features to obtain fusion features, where: Local context information is passed through branches Get, expressed as follows: in, , B For batch normalization, represents ReLu; Global context information through branches Get, expressed as follows: Among them, global context information , B For batch normalization, represents ReLu, stands for global average pooling; The local context information is integrated into the global context information, and the fusion weight is obtained through the Sigmoid function. The fusion weight is used to process the high-level features and low-level features, as follows: in, represents the fusion feature, represents the initial integration, Represents low-level features, represents high-level features, ⊗ is element-by-element multiplication, represents the fusion weight, .

5. According to claim 1, the method for detecting thermal discomfort by combining time attention with multi-level features is characterized in that: The input of the enhanced spatiotemporal feature extraction 3D network is the video frame of the person's action, and the output is the spatiotemporal features; The backbone network of the enhanced spatiotemporal feature extraction 3D network is 3D-ResNeXt-101. A 3D convolutional time attention module is inserted between Stage 2 and Stage 3 of 3D-ResNeXt-101. The 3D convolutional time attention module includes: intra-frame static attention branch and inter-frame dynamic attention branch. Specifically: The intra-frame static attention branch includes depth convolution, depth dilated convolution and point convolution, which are expressed as follows: in, , It is the output feature after downsampling of Stage 2; The inter-frame dynamic attention branch includes maximum pooling, fully connected layer + ReLu and fully connected layer + Sigmoid function, which are expressed as follows: in, , represents the Sigmoid function, FC is the fully connected layer, Represents the maximum pooling operation; Aggregate the static attention feature map and the dynamic attention feature map within the frame to obtain an attention map containing long-short time information, which is expressed as follows: in, , , Indicates multiplication.

6. The method for detecting thermal discomfort by combining temporal attention with multi-level features according to claim 1, characterized in that: The feature fusion detection head inputs the appearance features and spatiotemporal features, and outputs the spatiotemporal action detection results; the feature fusion detection head splices the appearance features and spatiotemporal features in the channel dimension, and then uses Convolution and The convolutional layer performs semantic interaction to obtain fused spatiotemporal features; The fused spatiotemporal features use the Gram matrix to obtain the feature enhanced feature map, and the Pred layer is used to analyze and judge the feature enhanced feature map to obtain the spatiotemporal action detection result.

7. The method for detecting thermal discomfort by combining temporal attention with multi-level features according to claim 1, characterized in that: The spatiotemporal action detection network with convolutional temporal attention and multi-scale information learning is trained and tested using the thermal discomfort action dataset TDA. The construction method of the thermal discomfort action dataset TDA is as follows: 1) The thermal discomfort action dataset TDA contains 10 types of thermal discomfort actions, namely: hugging arms, rubbing hands, stamping feet, putting on clothes, breathing, fanning, shaking clothes in front of the chest, wiping sweat, taking off clothes and rolling up sleeves; 2) Recruit 7 subjects aged between 22 and 26, and use RGB cameras to collect 10 types of thermal discomfort action video data in real indoor scenes of office buildings: ① The subject arrives at the designated area; ② Select a type of action; ③ Take action videos from the front, side, and 45° angle at both far and near distances; ④ Change different action types; ⑤ Repeat ①~④; A total of 3528 unedited videos were collected, with a video length of 10~30s and a uniform resolution of 640 480; 3) The collected video is divided into frames using video frame cutting technology and saved in JPG format, with each type of action saved in a separate folder; after removing redundant images, the data set is annotated and the labels are saved as .txt files.

8. A thermal discomfort action detection system combining temporal attention and multi-level feature fusion, characterized in that: include: A data acquisition module, used to acquire videos of people's actions; The motion detection module is used to transmit the acquired personnel motion video to the spatiotemporal motion detection network of convolutional temporal attention and multi-scale information learning to obtain the thermal discomfort spatiotemporal motion detection results; Among them, the spatiotemporal action detection network of convolutional temporal attention and multi-scale information learning includes a multi-scale information learning 2D network, an enhanced spatiotemporal feature extraction 3D network and a feature fusion detection head. The multi-scale information learning 2D network is used to extract apparent features, the enhanced spatiotemporal feature extraction 3D network is used to extract spatiotemporal features, and the feature fusion detection head is used to fuse apparent features and spatiotemporal features to obtain spatiotemporal action detection results.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a method for detecting thermal discomfort movements by integrating time attention and multi-level features as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting thermal discomfort movements by integrating time attention and multi-level features as described in any one of claims 1 to 8 are implemented.

Images