Human body small target detection and action recognition integrated method for building edge scene

Abstract

The invention provides a human body small target detection and action recognition integrated method for a building edge scene. The method comprises the steps that a human body small target detector is arranged and pre-trained; constructing a target data set special for the building edge scene, adjusting the target size distribution of the pre-training data set through an adaptive scale matching strategy, and optimizing the target data set by adopting a two-section fine tuning strategy; inserting a time sequence ROI alignment operator at the tail end of the detector backbone network; generating a human body candidate frame through a sequence-level semantic aggregation head; extracting spatio-temporal features of the video clips by adopting a dual-channel video backbone network, and extracting human body spatio-temporal features from the global feature map; a person-object relation feature map is generated through a person-object time coding module; and through a human-object-human space-time coding module, converting a human-object relation characteristic graph, capturing an interaction relation between different human bodies by using stacked non-local blocks, generating high-order space-time characteristics, synchronously inputting a bounding box regression branch and an action classification branch, and synchronously outputting a human body position and an action category.

Description

Technical Field

[0001] This invention relates to target detection and human motion recognition in the field of computer vision. It aims to address difficult issues such as tiny human targets, severe occlusion, and complex lighting conditions at the edge of high-rise buildings, and to achieve efficient detection of human positions and accurate recognition of human motions through an integrated technical solution. Background Art

[0002] With rapid economic development and accelerated urbanization, the number of buildings has increased dramatically. Simultaneously, accidents such as people falling from buildings have become frequent, posing a serious threat to personal and public safety. Automated surveillance systems, as a potential solution, have been widely deployed inside and outside buildings. Their goal is to use surveillance video to promptly detect people at building edges (such as balconies, windows, and roof railings) and analyze their movements to prevent accidents. However, this task faces numerous challenges. Complex and cluttered backgrounds, extreme lighting conditions (too bright or too dark), the small size of human subjects due to long-distance photography, and large areas of the human body obscured by architectural elements all pose significant challenges to person detection and motion recognition. Therefore, developing effective methods for person detection and motion recognition is crucial for preventing injuries and saving lives.

[0003] In the field of person detection, currently common methods are primarily based on convolutional neural networks (CNNs) and Transformers. CNN-based object detection methods can be categorized into two-stage and one-stage approaches. Two-stage detectors (such as Faster R-CNN, Cascade R-CNN, and Mask R-CNN) can accurately locate objects but are slow; one-stage detectors (such as the YOLO series, SSD, and RetinaNet) are fast but relatively inaccurate. Furthermore, common object detection methods have also achieved some success in handling scale variations, such as feature pyramid networks and scale-aware training strategies. Anchor-free detectors (such as CornerNet, CenterNet, and FCOS) have also emerged as an alternative. Since Dosovitskiy et al. demonstrated the applicability of the Transformer to computer vision, object detection algorithms based on attention mechanisms have continued to develop. The first category uses the Transformer as the core, such as DETR, which uses an encoder-decoder Transformer as the core and directly detects all objects by introducing a target query and collective prediction. The second category uses the Transformer as the backbone, such as the Swin Transformer, which introduces a hierarchical Transformer backbone to capture multi-scale features. In the field of pedestrian detection, researchers have proposed a variety of targeted methods to address challenges such as scale changes and occlusion. For example, Cao, Pang, and others are committed to improving the detection of small-sized pedestrians; Zhang, Zhou, and others use visible information to detect occluded pedestrians; Chi, Huang, Wang, and others explore combining highly overlapping target boxes to improve pedestrian detection in crowded scenes. However, these methods are mainly trained and evaluated based on traffic scene datasets, which are significantly different from building edge scenes and are difficult to directly apply to the latter.

[0004] The field of human action recognition has also undergone a long period of development and evolution. Early traditional methods, using handcrafted feature extractors like SIFT and HOG, could only capture local motion information and were sensitive to changes in lighting and background. This made it difficult to process large-scale, complex video data, limiting its application. Following the rise of deep learning, simple neural networks (such as multi-layer perceptrons) proved ineffective due to their inability to effectively process spatiotemporal information in videos. Subsequently, 2D convolutional neural networks achieved significant progress. First, 2D convolutional networks used for image classification were applied to video action recognition, extracting features from video frames and aggregating them into video-level feature classification using methods like average pooling. Subsequently, two-stream networks emerged, where spatial and temporal streams extracted spatial and temporal features, respectively, and then fused them for classification. However, this approach was computationally complex. To optimize spatiotemporal information processing, 3D convolutional neural networks were developed. C3D and 3D ResNet, among others, achieved excellent performance across multiple datasets using 3D convolution kernels. Subsequently, FSTCN and P3D improved 3D network efficiency through low-rank approximation, modified 2D convolutions, and hyperparameter adjustment. Meanwhile, recurrent neural network variants such as LSTM and GRU were used to capture temporal information about video actions. Furthermore, the introduction of the Transformer has invigorated this field. Models such as ActiViT treat video frames as sequences and utilize a self-attention mechanism to capture long-range dependencies and identify actions. In multi-view fusion, after 3D CNN spatiotemporal modeling, the Transformer's self-attention encoder and cross-view fusion module are used to improve deep video action recognition performance. Other approaches combine the Transformer with dynamic PSO-ConvNets to enhance accuracy through their combined advantages. Finally, methods that employ a high-precision Transformer backbone network incorporating temporal shifting and a lightweight attention mechanism have demonstrated excellent performance across multiple datasets.

[0005] However, existing technologies for both person detection and action recognition face numerous challenges when applied to building edge scenarios. Data-drivenly, commonly used person detection datasets (such as PASCAL VOC, MS COCO, and CityPersons) and action recognition datasets (such as UCF101, Kinetics400, and AVA) do not focus on building edge scenarios. In real-world building edge scenarios, people appear small, heavily occluded, and in diverse poses, with variable lighting and cluttered backgrounds. People in common datasets are often found in relatively simple, standardized environments. Their appearance and background characteristics differ significantly from those of people in building edge scenarios, making it difficult for algorithms trained on these datasets to perform well in this scenario. At the algorithmic design level, current detection and recognition algorithms are mostly optimized for general scenarios and fail to comprehensively consider the unique characteristics and challenges of building edge scenarios. Consequently, they often struggle to achieve optimal detection results when applied to building edge scenarios. Furthermore, simultaneously detecting object position and recognizing action categories in these scenarios further increases the complexity of the task. Existing technologies struggle to effectively balance these two aspects, failing to provide a reliable solution. Summary of the Invention

[0006] In view of the above-mentioned defects of the existing technology in the detection and motion recognition of small targets of people at the edge of buildings, the present invention aims to propose an efficient integrated detection and recognition method that is adapted thereto, so as to overcome the shortcomings of the existing technology and improve the accuracy and reliability of the detection and motion recognition of people at the edge of buildings.

[0007] The technical solution provided by the present invention is an integrated method for small human target detection and action recognition in building edge scenes, comprising: Set up a small human object detector and perform pre-training; Construct a target dataset dedicated to building edge scenes, adjust the target size distribution of the pre-training dataset through an adaptive scale matching strategy, and optimize the target dataset using a two-stage fine-tuning strategy; A temporal ROI alignment operator is inserted at the end of the detector backbone network. By calculating the spatial position similarity between the target frame and the support frame, weighted aggregation of the support frame features is performed, and the temporal attention mechanism is used to fuse the target frame features with the support frame features to generate enhanced temporal ROI features. The sequence-level semantic aggregation head performs multiple rounds of iterative integration of temporal ROI features to generate human candidate frames; A dual-channel video backbone network is used to extract the spatiotemporal features of video clips, and the spatiotemporal features of human bodies are extracted from the global feature map through ROI alignment. Through the human-object temporal coding module, the temporal and spatial features of the human body are copied and spliced ​​with the global features, and then convolutional coding is performed to generate a human-object relationship feature map; Through the person-object-person spatiotemporal encoding module, the person-object relationship feature map is converted into query, key and value vector embeddings, and the stacked non-local blocks are used to capture the interaction relationship between different human bodies and generate high-order spatiotemporal features; The high-order spatiotemporal features are synchronously input into the bounding box regression branch and the action classification branch, and the human body position and action category are synchronously output.

[0008] Moreover, the adaptive scale matching strategy includes analyzing the target size probability distribution of the target dataset and the pre-training dataset, mapping the average video size of the pre-training dataset to be consistent with the distribution of the target dataset through a monotonic function, and calculating and adjusting the video resolution scaling ratio based on the mapping result; the two-stage fine-tuning strategy is used to optimize the target dataset, including performing the first fine-tuning on the pre-training dataset after scale matching and the second fine-tuning on the target dataset.

[0009] Moreover, the temporal attention mechanism includes grouping the target frame features and the supporting frame features by channel, calculating the normalized attention weight for each group of features, and performing weighted fusion of the supporting frame features according to the weight.

[0010] Moreover, the sequence-level semantic aggregation head includes mapping all frame features through a shared fully connected layer, calculating the semantic similarity weights between the target frame and other frames, dynamically aggregating multi-frame features according to the weights, and repeatedly performing multiple optimizations.

[0011] Moreover, in the dual-channel video backbone network, the slow channel adopts a low-frequency sampling strategy to extract spatial features, and the fast channel adopts a high-frequency sampling strategy to extract motion features. After the dual-channel features are fused, a global spatiotemporal feature map is generated.

[0012] Moreover, the implementation of the human-object interaction coding module includes the following operations: The spatiotemporal features of the human body are replicated and expanded in the spatial dimension to make them consistent with the spatial size of the global spatiotemporal feature map; The copied human features are concatenated with the global spatiotemporal feature map along the channel dimension to generate a joint feature map. Performing a convolutional coding operation on the joint feature map to extract a person-object relationship feature map; The generated human-object relationship feature graph is stored in the relationship feature library for subsequent human-object-human relationship modeling.

[0013] Moreover, two non-local blocks are stacked in the person-object-person spatiotemporal encoding module to calculate the spatial position attention weights and capture high-order interaction relationship features.

[0014] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for integrating small human target detection and action recognition in the building edge scene as described above is implemented.

[0015] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the integrated method of small human target detection and action recognition in the building edge scene as described above.

[0016] On the other hand, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the integrated method of small human target detection and action recognition in the building edge scene as described above.

[0017] The present invention solves the problem of small human target size by constructing a dedicated dataset, designing an adaptive scale matching and two-stage fine-tuning strategy; embedding a temporal ROI alignment operator in the network to fuse the target frame and adjacent frame information, and using a sequence-level semantic aggregation head for deep semantic integration to improve the model's perception of difficult targets; introducing a "human-object" spatiotemporal coding module and a "human-object-human" spatiotemporal coding module to fully encode the multi-order complex relationship between the human body and the background, and improving the model's understanding of spatiotemporal semantics. Compared with the existing technology, the present invention has many advantages: constructing a dataset adapted to the scene and introducing innovative modules, significantly improving the detection and recognition accuracy; designing algorithms for specific scene challenges, enhancing the robustness of the model; the integrated detection and recognition output and the versatility of the technical modules expand the scope of application. The present invention effectively overcomes the shortcomings of the existing technology, improves the accuracy and reliability of personnel detection and action recognition in building edge scenes, and provides strong technical support for safety monitoring of high-rise building edge scenes.

[0018] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art: (1) High Precision: By constructing a dataset specifically for high-rise building edge scenes, the model is highly compatible with actual application scenarios, enhancing scene adaptability. In addition, the introduction of temporal ROI alignment operators, sequence-level semantic aggregation heads, and multi-order spatiotemporal encoding modules further enhances the model's ability to perceive dynamic information and understand spatiotemporal semantics, thereby significantly improving the accuracy of detection and recognition.

[0019] (2) Robustness: The adaptive scale matching strategy enables the model to adapt to the distribution of objects of different scales in the pre-training dataset, effectively solving the problem of small human targets. In addition, the design of each module of the model fully considers the complexity of the building edge scene. Even when factors such as lighting, background and occlusion change, it can maintain performance stability and achieve accurate detection and recognition.

[0020] (3) Wide range of applications: The present invention integrates small human target detection and action recognition, and can simultaneously output bounding boxes and action categories, providing comprehensive information for fields such as security monitoring, improving work efficiency, and preventing accidents. In addition, the technology and modules in the invention are universal and transferable. After appropriate adjustment and optimization, they can be applied to other complex scenarios, providing an effective solution to the problems of target detection and action recognition, and have a wide range of potential applications.

[0021] Therefore, the present invention provides an integrated technical solution for small human target detection and motion recognition in high-rise building edge scenes, which effectively solves the problem of insufficient accuracy and reliability of personnel detection and motion recognition in the existing technology in this scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is an overall flow chart of an embodiment of the present invention; Figure 2 is a flow chart of two-stage fine-tuning using scale matching in an embodiment of the present invention; Figure 3 is a flow chart of a temporal ROI alignment operator in an embodiment of the present invention; Figure 4 4 is a flowchart of multi-stage space-time coding in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application.

[0024] like Figure 1 As shown, the embodiment provides an integrated method for small human target detection and action recognition in building edge scenes, and the specific steps are as follows: S1. Setting up and pre-training a small human object detector: This embodiment preferably constructs a small human object detector based on the Faster R-CNN framework and selects the large-scale open dataset ImageNet VID as a pre-training dataset to pre-train the small human object detector.

[0025] This step lays the foundation for the subsequent application of the model in high-rise building edge scenarios. By leveraging the rich data resources of ImageNet VID, the model can initially learn general target feature representations.

[0026] The specific implementation of the Faster R-CNN framework can be found in the literature: Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149. S2. Construct a scene-specific dataset and adopt a scale matching strategy for two-stage fine-tuning training to address the challenge of small human targets in high-rise building edge scenes.

[0027] like Figure 2 As shown, the preferred implementation method of the embodiment includes the following specific sub-steps: S201. Obtain target data sets separately The probability density function approximates the histogram of the average size of objects in the (newly constructed high-rise building edge scene people spatiotemporal action detection dataset) and the pre-training dataset (ImageNet VID) to gain a deeper understanding of the scale distribution characteristics of the two.

[0028] In specific implementation, the spatiotemporal motion detection dataset of people in high-rise building edge scenes can be constructed using a dual-track strategy, which is collected through offline shooting and public video platforms. The scene-specific dataset constructed in the embodiment covers more than 300 building scenes and contains 1,314 video clips.

[0029] The method for extracting the approximate histogram of the probability density function of the average object size can be found in the literature, which is not described in detail in this article: Yu X, Gong Y, Jiang N, et al. Scale match for tiny person detection[C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2020: 1257-1265. S202. From the target dataset The target average size of each video is sampled from the probability distribution of , the monotonicity of the target size is maintained during the sampling process. The specific method is to use the monotonic function The pre-training dataset Average target size of videos in Map to , so that the size after mapping The distribution of the target dataset Same. For any size , a monotonic function () can be solved by the following integral equation:

[0030] in and are the minimum and maximum average object sizes of videos in the pre-training dataset, respectively; and The probability distribution of target sizes under different datasets, e.g. Represents the pre-training dataset Medium target size The probability distribution of Indicates that the integral is about the target size carried out.

[0031] S203. Calculate the scaling ratio ,in The first Videos, Its resolution.

[0032] S204. Adjust the video resolution of the pre-training dataset according to the scaling ratio, denoted as ,in 、 Indicates the The scale-matched pre-training dataset is used for the first round of fine-tuning, followed by a second round of fine-tuning on the target dataset. This improves the model's performance in small object detection, enabling it to better address the challenge of small human targets in scenes with edges of high-rise buildings.

[0033] In specific implementation, the first round of fine-tuning loads the open source pre-trained weights and trains on the scale-matched pre-training dataset; the second round of fine-tuning loads the weights just after the first round of fine-tuning (that is, the model weights obtained after the first round of training) and trains on the target dataset.

[0034] S3. Insert a temporal ROI alignment operator at the end of the backbone network of the human small object detector set in S1 to incorporate the temporal information of adjacent frames into the target frame ROI.

[0035] like Figure 3 As shown, the preferred implementation method of the embodiment includes calculating the spatial position similarity between the target frame and the support frame, weighted aggregation of the support frame features, and using the temporal attention mechanism to fuse the target frame features and the support frame features to generate enhanced temporal ROI features. The specific implementation process includes the following sub-steps: S301. After the backbone network outputs, target frame , first use standard ROI alignment (ROI Align) to extract ROI features from the target frame. Then, for each spatial position in the ROI feature , calculate its feature map with the support frame The similarity of the corresponding positions in :

[0036] in, Indicates that the target frame is the first frame of the video frame, Indicates the difference between the target frame and the support frame Frames, e.g. Figure 3 Medium support frame 、 ; Indicates the position in the target frame ROI Before calculating the similarity, the features need to be L2 normalized to enhance the cosine similarity effect. Support frame feature map represented by Transpose, Represents the cosine similarity operator.

[0037] S302. Based on the calculated similarity , select from each support frame Most similar locations ( It is a preset parameter. It can be set as an empirical value during implementation. It is recommended to select ),in is the height direction index of the feature map, is the width index of the feature map; and based on the normalized similarity weight Perform weighted aggregation on the support frames to generate support frame feature maps .in, is the normalized exponential function.

[0038] This process can be expressed as:

[0039] S303. Using temporal attention mechanism to fuse target frame ROI features and supporting features . Group ROI features and support features along the channel dimension. Group, use attention block to calculate attention weight :

[0040] in To embed the network in a lightweight way, let the size of the target frame ROI be w h, where a certain point is recorded For spatial location.

[0041] After normalization, we get:

[0042] Where T is the number of frames and exp is the exponential function. Based on this normalized attention weight , the weighted sum of the support features of each group is obtained :

[0043] in, Enhance features for grouped timing.

[0044] Finally, the results of all groups are spliced ​​together to obtain the final temporal ROI features of the target frame This feature fuses the target frame and support frame information, thereby enhancing the model's ability to cope with complex situations such as occlusion and lighting changes.

[0045] S4. For the temporal ROI of each frame in the video, a sequence-level semantic aggregation head is introduced to perform deep semantic integration. This embodiment proposes to perform multiple rounds of iterative integration of temporal ROI features through a sequence-level semantic aggregation head to generate a human candidate frame. The specific sub-steps preferably adopted are as follows: S401. Introduce sequence-level semantic aggregation head and treat the entire video sequence as a set of unordered frames. First, the temporal ROI features of the target frame are and single frame ROI features of other frames By sharing the fully connected layer Then, the semantic similarity between the target frame and other frame features is calculated. :

[0046] This similarity can measure the semantic relevance of features between different frames and provide a basis for subsequent feature aggregation.

[0047] S402. Perform softmax operation on semantic similarity along the first dimension Normalize it and convert it into attention weights. Then, project the features of other frames according to these attention weights. Perform weighted summation to obtain the preliminary aggregated features of the target frame :

[0048] This aggregation process can dynamically adjust the contribution of features from different frames according to their importance, so that the feature representation of the target frame can fuse the semantic information of multiple frames.

[0049] S403. The above polymerization process is repeated Second-rate( It is a preset parameter, the default is three times, verified by experiments The best balance of performance and efficiency) enables deep semantic integration. With each iteration, the feature representation is continuously optimized, more comprehensively capturing the spatiotemporal semantic information in the video sequence. After multiple iterations, detection candidate boxes containing rich spatiotemporal semantic information are obtained, providing strong support for subsequent accurate detection of person positions and recognition of actions.

[0050] S5. Extracting global spatiotemporal feature maps of video clips using a video backbone network , and use the human candidate frame obtained in S4 to extract the human spatiotemporal features ( Indicates the first personal).

[0051] The preferred implementation method proposed in the embodiment includes the following specific sub-steps: S501. Use the SlowFast-based video backbone network to process the video clip. The network can effectively capture the spatiotemporal information in the video and model the features of different time scales and spatial resolutions. The input is a 64-frame video clip, where the slow channel is Frame is the sampling interval, sampling step ; Fast track is based on ( The fast channel sampling ratio is verified by experiments and is recommended to be set ) to extract spatiotemporal feature blocks.

[0052] The specific implementation of SlowFast can be found in the literature: Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6202-6211. S502. Generate a global spatiotemporal feature map by fusing the dual-channel features obtained in S501: Perform average pooling on the spatiotemporal feature blocks to integrate the feature information in the spatial dimension. ,in 、 、 Represent the number of channels, height and width respectively, The field of real numbers.

[0053] S503. For each candidate box of the human body, use ROI Align and maximum pooling operations to extract the global feature map Accurately extract human spatiotemporal features This operation can be performed based on the position of the candidate box in the feature map. The features of the corresponding area are extracted from the human body, so that the extracted temporal and spatial features of the human body can accurately represent the temporal and spatial state of the human body in the video, laying the foundation for the subsequent analysis of the relationship between the human body and the surrounding environment.

[0054] S6. Introduce the “human-object” spatiotemporal encoding module to encode the interaction features between the human body and the background, such as Figure 4 The specific sub-steps are as follows: S601. First, the temporal and spatial characteristics of the human body Copy in space and global features Each position of is spliced ​​along the channel, so that the human features and background features are spatially correlated, the interactive information between the features is enhanced, and a joint feature map is generated.

[0055] S602. Use convolution operation to further encode the fused joint feature map to obtain the "person-object" relationship feature map The convolutional layer can automatically learn the interaction features between the human body and the background, enhance the model's understanding of the environment in which human behavior occurs, and help to more accurately identify human actions. Then, by drawing on LFB, a "human-object" relationship feature library is created to store the "human-object" relationship feature graph. .

[0056] In specific implementations, LFB stands for long-term video feature banks. For implementation details, please refer to the relevant literature: Wu CY, Feichtenhofer C, Fan H, et al. Long-term feature banks for detailed video understanding[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 284-293. S7. Introduce a “person-object-person” spatiotemporal encoding module to capture high-order relational features between the human body and other “person-object” relationships.

[0057] like Figure 4 As shown, the preferred implementation method proposed in the embodiment includes the following specific sub-steps: S701. “People-Object” Relationship Feature Map As input, the input feature map is converted into dimension through convolution operation Query ,key Sum Embeddings have the same spatial size as the input feature maps. This operation helps aggregate local information and better capture the relationship between features.

[0058] S702. By stacking non-local blocks, the model can capture complex high-order relationships between the human body and other "human-object" relationships on a global scale. Non-Local blocks are a type of attention mechanism module. The present invention preferably stacks non-local blocks to capture long-range dependencies by calculating the correlation between any positions in the feature map. In each non-local block, the attention vector at each spatial position According to S701, for Individuals can get query embedding , key embedding Sum value embedding Similarly, for other Individuals also have corresponding query embeddings , key embedding Sum value embedding . Personal query embedding Iterate over everyone else's key embeddings , the attention weight can be calculated :

[0059] in Represents the normalization operation after traversing all other people, For the Individual queries are embedded in spatial locations The value at For the Individual bonds are embedded in spatial locations The value at The dimension of the vector for key embedding is then embedded with the values ​​of others Perform linear combination to obtain new feature representation :

[0060] in, For the Individual values ​​are embedded in spatial locations The value at .

[0061] Then, after normalization, ReLU activation, convolution and Dropout operations, the input features The residuals are added and fused to obtain the updated features, which are further input into the next Non-Local block for processing, thereby achieving in-depth capture of high-order relationships.

[0062] The present invention obtains the "people-object-people" spatiotemporal features by stacking multiple Non-Local blocks (preferably stacking two by default). This feature fully considers the mutual influence between different people and objects in the scene, and improves the accuracy of the model in recognizing people's actions in complex scenes.

[0063] S8. Process the “person-object-person” spatiotemporal features obtained through the above steps The data is then fed into the bounding box regression and action classification modules, respectively. In practice, both modules can be simple fully connected layers that, through nonlinear mapping, output the bounding box of a person and the action category. This enables position detection and action recognition of people in high-rise building edge scenarios, providing accurate decision-making for security monitoring and intelligent management.

[0064] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0065] The following describes the integrated electronic device for small human target detection and motion recognition in building edge scenes provided by the present invention. The integrated electronic device for small human target detection and motion recognition in building edge scenes described below and the integrated method for small human target detection and motion recognition in building edge scenes described above can be referenced to each other.

[0066] The electronic device may include a processor, a communications interface, memory, and a communications bus, wherein the processor, communications interface, and memory communicate with each other via the communications bus. The processor may invoke logic instructions in the memory to execute an integrated method for detecting and recognizing small human targets and actions in building edge scenes, primarily including the software processing portion of the aforementioned steps.

[0067] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the software processing part of the integrated method of small human target detection and action recognition in building edge scenes provided by the above-mentioned methods.

[0069] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the software processing part of the integrated method for small human target detection and action recognition in building edge scenes provided by the above-mentioned methods.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0071] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for integrating small human target detection and action recognition in building edge scenes, characterized in that: include: Set up a small human object detector and perform pre-training; Construct a target dataset dedicated to building edge scenes, adjust the target size distribution of the pre-training dataset through an adaptive scale matching strategy, and optimize the target dataset using a two-stage fine-tuning strategy; A temporal ROI alignment operator is inserted at the end of the detector backbone network. By calculating the spatial position similarity between the target frame and the support frame, weighted aggregation of the support frame features is performed, and the temporal attention mechanism is used to fuse the target frame features with the support frame features to generate enhanced temporal ROI features. The sequence-level semantic aggregation head performs multiple rounds of iterative integration of temporal ROI features to generate human candidate frames; A dual-channel video backbone network is used to extract the spatiotemporal features of video clips, and the spatiotemporal features of human bodies are extracted from the global feature map through ROI alignment. Through the human-object temporal coding module, the temporal and spatial features of the human body are copied and spliced ​​with the global features, and then convolutional coding is performed to generate a human-object relationship feature map; Through the person-object-person spatiotemporal encoding module, the person-object relationship feature map is converted into query, key and value vector embeddings, and the stacked non-local blocks are used to capture the interaction relationship between different human bodies and generate high-order spatiotemporal features; The high-order spatiotemporal features are synchronously input into the bounding box regression branch and the action classification branch, and the human body position and action category are synchronously output.

2. The integrated method for small human target detection and action recognition in building edge scenes according to claim 1 is characterized by: The adaptive scale matching strategy includes analyzing the target size probability distribution of the target dataset and the pre-training dataset, mapping the average video size of the pre-training dataset to be consistent with the target dataset distribution through a monotonic function, and calculating and adjusting the video resolution scaling ratio based on the mapping result; The two-stage fine-tuning strategy is used to optimize the target dataset, including first fine-tuning on the pre-training dataset after scale matching and second fine-tuning on the target dataset.

3. The integrated method for small human target detection and action recognition in building edge scenes according to claim 1 is characterized by: The temporal attention mechanism includes grouping the target frame features and the supporting frame features by channel, calculating the normalized attention weight for each group of features, and performing weighted fusion of the supporting frame features according to the weight.

4. The integrated method for small human target detection and action recognition in building edge scenes according to claim 1 is characterized by: The sequence-level semantic aggregation head includes mapping all frame features through a shared fully connected layer, calculating the semantic similarity weights between the target frame and other frames, dynamically aggregating multi-frame features according to the weights, and repeatedly performing multiple optimizations.

5. The integrated method for human small target detection and action recognition in building edge scenes according to claim 1 is characterized by: In the dual-channel video backbone network, the slow channel uses a low-frequency sampling strategy to extract spatial features, and the fast channel uses a high-frequency sampling strategy to extract motion features. The dual-channel features are fused to generate a global spatiotemporal feature map.

6. The integrated method for small human target detection and action recognition in building edge scenes according to claim 1 is characterized by: The implementation of the human-object interaction coding module includes the following operations: The spatiotemporal features of the human body are replicated and expanded in the spatial dimension to make them consistent with the spatial size of the global spatiotemporal feature map; The copied human features are concatenated with the global spatiotemporal feature map along the channel dimension to generate a joint feature map. Performing a convolutional coding operation on the joint feature map to extract a person-object relationship feature map; The generated human-object relationship feature graph is stored in the relationship feature library for subsequent human-object-human relationship modeling.

7. The integrated method for small human target detection and action recognition in building edge scenes according to claim 1 is characterized by: Two non-local blocks are stacked in the person-object-person spatiotemporal encoding module to calculate the spatial position attention weights and capture high-order interaction relationship features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the integrated method for small human target detection and action recognition in a building edge scene as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for integrating small human target detection and action recognition in a building edge scene as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the method for integrating small human target detection and action recognition in a building edge scene as described in any one of claims 1 to 7 is implemented.

Images