Human interaction behavior recognition method and device for emergency and rescue

Abstract

The invention discloses a human interaction behavior identification method and device for emergency and rescue, and the method comprises the steps: obtaining video data captured by an unmanned plane, obtaining a plurality of video frames, extracting key frames in the plurality of video frames, and determining a plurality of sliding windows of the video frames; carrying out human behavior recognition on the key frame by adopting a convolutional neural network to generate a feature vector, constructing a probability vector to obtain a feature set, carrying out prediction analysis on the feature set by adopting a support vector machine algorithm to obtain a corresponding action label, and recognizing human behaviors and marking corresponding actions according to the action label. The efficiency of rescue and emergency actions is improved by using the video clips of the unmanned aerial vehicle, the man-machine interaction recognition in a disaster scene is improved, human-free detection, tracking or multiple image instances are improved, the efficiency in the aspects of rescue and emergency management response accuracy, processing time and calculation complexity is improved, and the rescue and emergency management efficiency is improved by analyzing the first image and the last image. The image quality is improved, and disaster management and rescue actions are accelerated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and a device for identifying human interaction behavior for emergency and rescue. Background Art

[0002] There is a critical research gap in exploring how advanced technologies and machines can improve the efficiency of disaster management efforts. Despite advances, even the smartest machines lack sufficient visual techniques to fully understand and manage the content of video data from disaster scenes. Deep learning models, especially neural networks, have achieved great success in addressing the challenge of human behavior recognition.

[0003] To address this problem, most people turn to computer vision techniques to enhance the capabilities of these devices: (1) Convolutional neural networks (CNNs) are one of the most effective deep learning structures and have been widely used in fields such as recognition and segmentation. However, CNNs have their limitations. For example, when an image contains objects of interest at different spatial locations, it is necessary to select many regions, resulting in a time-consuming detection and classification process. (2) When applying CNNs to efficient and automatic object recognition in aerial images, the performance of CNNs is highly dependent on the choice of operating parameters and network training configuration. (3) Region proposal methods and region-based CNNs (R-CNNs) have limited application in object detection tasks. (4) R-CNN is designed for object detection in a specific area, but R-CNN is limited to processing 2000 region proposals for each classification, which is a fixed constraint; (5) Inferring the interaction forces between objects based on video data without relying on traditional sensors; (6) Using three-dimensional (3D) CNN to predict the physical interaction forces of video frames, a bottleneck-based 3D depth-separable CNN architecture is introduced, which can disentangle spatial and temporal information, but the target detection algorithm and image classification tasks are significantly different, and CNNs face limitations in processing a fixed number of region proposals.

[0004] In summary, there is an urgent need to provide a method and device for identifying human interaction behavior for emergency response and rescue. Summary of the invention

[0005] In view of this, the present invention provides a method and device for human interaction behavior recognition for emergency and rescue, which provides behavior recognition performance and human body detection efficiency in emergency rescue operations, and is specifically implemented by the following technical solutions.

[0006] In a first aspect, the present invention provides a method for identifying human interaction behaviors for emergency and rescue, comprising the following steps: Obtaining video data captured by the drone, and obtaining a plurality of video frames from the video data; Extracting key frames from the multiple video frames, and determining multiple sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; Using a convolutional neural network to perform human behavior recognition on the key frame according to the sliding window to generate a feature vector corresponding to the key frame, and constructing a probability vector according to the feature vector to obtain a feature set; A support vector machine algorithm is used to perform predictive analysis on the feature set to obtain corresponding action labels, and human behaviors are identified and corresponding actions are marked based on the action labels.

[0007] As a preferred embodiment of the above technical solution, extracting key frames from the multiple video frames includes: Divide a video frame into M Pixel images; Extracting key frames from each processed video frame; According to pixel size Calculating the length and width of the key frame, and dividing the key frame into image segments of a preset size, wherein the image segment includes a plurality of pixel sizes; The image segment is taken as a sliding window on the image, wherein the size of the sliding window covers the entire image.

[0008] As a preferred embodiment of the above technical solution, a convolutional neural network is used to perform human behavior recognition on the video frame according to the sliding window to generate a feature vector corresponding to the key frame, including: Using a convolutional neural network to perform human behavior recognition on the key frame according to the sliding window to generate a feature vector corresponding to the key frame; Calculating the maximum likelihood value of the vector of each sliding window, and constructing a probability vector corresponding to the feature vector according to the maximum likelihood value of the vector; A feature set is obtained based on the probability vector and the feature vector.

[0009] As a preferred embodiment of the above technical solution, extracting a key frame from each processed video frame includes: Multiple video frames are segmented into high-resolution images and focus on the first and last images to identify all objects in the scene, where a super-resolution algorithm is used to map the low-resolution images to high-resolution images.

[0010] As a preferred embodiment of the above technical solution, the convolutional neural network includes a pre-trained CNN based on ImageNet and drone images, and the key frames include objects of interest in different spatial positions and aspect ratios.

[0011] As a preferred embodiment of the above technical solution, the convolutional neural network generates a probability vector for each image to indicate the existence of different objects; If a vector vec with an element r_i corresponds to the value n of the hurricane image, it is interpreted as the probability of object x appearing in the image being n; Support vector machine (SVM) is used to predict the labels of human interactions based on CNN output, where each CNN output of a sliding window corresponds to a probability vector.

[0012] As a preferred embodiment of the above technical solution, each image component is given a length of The probability vector of the image position is With length units, the vector Defined as , and All have length units. The unit is ; The video frames corresponding to the two probability vectors of the first and last images are converted into vectors through a union function; When extracting the feature vector, a classification framework is used to identify the types of human interactions and actions depicted in the video, wherein the classification framework includes a support vector machine (SVM) algorithm.

[0013] As a preferred embodiment of the above technical solution, the operation mode of the support vector machine SVM algorithm includes generating two parallel lines for each set of data and using a hyperplane to effectively separate the space, wherein the support vector machine SVM constructs a hyperplane with the largest boundary in a high-dimensional space to classify the given data into different classes, and the expression of the optimal hyperplane is: (1) in, yes and scalar product of ; The nonlinear support vector machine (SVM) algorithm is used to calculate clusters and identify human interactions in videos. The support vector machine (SVM) algorithm uses a mathematical function and defines it as a kernel. The function of the kernel is to take data as input and transform it into the required form. The polynomial kernel equation (2) is used for image processing: (2) in, Represents the kernel function, for the vector and Perform transformation operation, k represents the kernel, and d represents the degree of the polynomial.

[0014] As a preferred embodiment of the above technical solution, human behavior recognition includes identifying human behavior in video data, and its process includes video acquisition, defining region of interest ROI, region of interest ROI processing, feature extraction and representation, tracking, and action understanding, wherein defining region of interest ROI includes image segmentation, and action understanding includes classification and assignment of action labels.

[0015] In a second aspect, the present invention further provides a human interaction behavior recognition device for emergency and rescue, which is applied to the human interaction behavior recognition method for emergency and rescue as described above, comprising: A video data acquisition unit, used to acquire video data captured by the drone, and to acquire a plurality of video frames from the video data; A key frame extraction unit, configured to extract key frames from the plurality of video frames, and determine a plurality of sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; A human behavior recognition unit, configured to perform human behavior recognition on the key frame using a convolutional neural network according to the sliding window to generate a feature vector corresponding to the key frame, and construct a probability vector according to the feature vector to obtain a feature set; The action prediction unit is used to use a support vector machine algorithm to perform prediction analysis on the feature set to obtain corresponding action labels, and to identify human behaviors and mark corresponding actions based on the action labels.

[0016] The present invention provides a method and device for identifying human interaction behavior for emergency and rescue. The method obtains video data captured by a drone, obtains multiple video frames from the video data, extracts key frames from the multiple video frames, and determines multiple sliding windows of the video frames according to the key frames. A convolutional neural network is used to identify human behavior on the key frames according to the sliding windows to generate feature vectors corresponding to the key frames, and a probability vector is constructed according to the feature vectors to obtain a feature set. A support vector machine algorithm is used to predict and analyze the feature set to obtain corresponding action labels, and human behavior and corresponding actions are identified according to the action labels. The efficiency of rescue and emergency operations is improved by using drone video clips, and human-computer interaction recognition in disaster scenes is improved. CNN is used to extract high-level features from key frame videos, and a support vector machine (SVM) is used to classify high-level features based on sliding, without the need for human detection, tracking or multiple image instances, thereby improving the accuracy of rescue and emergency management responses, processing time and efficiency in terms of computational complexity. By analyzing the first and last images (FLIs), image quality is improved, and disaster management and rescue operations are accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flow chart of the human interaction behavior recognition method for emergency and rescue provided by the present invention; Figure 2 A schematic diagram of the human identification system provided by the present invention; Figure 3 A schematic diagram of selecting a sliding window from a video frame provided by the present invention; Figure 4 This is a schematic diagram of the average recognition speed of the drone aerial video provided by the present invention; Figure 5 A schematic diagram of the accuracy of using CNN and CNN-FLI methods to identify three data sets provided by the present invention; Figure 6 This is a structural block diagram of the human interaction behavior recognition device for emergency and rescue provided by the present invention. DETAILED DESCRIPTION

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0020] This paper introduces a new method integrating convolutional neural networks (CNNs) to address key issues such as motion blur, poor video quality, occlusions, object size, and high computational or memory requirements, especially in the context of disaster management. This method uses drone video footage to improve the efficiency of rescue and emergency operations and enhances human-machine interaction recognition in disaster scenarios. CNN is used to extract high-level features from keyframe videos, while support vector machines (SVM) classify these stationary features based on window proposals. This method captures the context of a scene, which plays a vital role in understanding human behavior in videos. Unlike traditional methods, no person detection, tracking, or multiple image instances are required, which makes it more effective.

[0021] See also Figure 1 ,The present invention provides a method for identifying human interaction behaviors for emergency and rescue, comprising the following steps; S1: Obtain video data captured by a drone, and obtain multiple video frames from the video data; S2: extracting key frames from the multiple video frames, and determining multiple sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; S3: using a convolutional neural network to perform human behavior recognition on the key frame according to the sliding window to generate a feature vector corresponding to the key frame, and constructing a probability vector according to the feature vector to obtain a feature set; S4: using a support vector machine algorithm to perform predictive analysis on the feature set to obtain corresponding action labels, and identifying human behaviors and marking corresponding actions based on the action labels.

[0022] In this embodiment, extracting key frames from the plurality of video frames includes: dividing a video frame into M pixel image; extract key frames from each processed video frame; based on pixel size The length and width of the key frame are calculated, and the key frame is divided into image segments of a preset size, wherein the image segment includes a plurality of pixel sizes; the image segment is used as a sliding window on the image, wherein the size of the sliding window covers the entire image.

[0023] According to the sliding window, a convolutional neural network is used to perform human behavior recognition on the video frame to generate a feature vector corresponding to the key frame, including: according to the sliding window, a convolutional neural network is used to perform human behavior recognition on the key frame to generate a feature vector corresponding to the key frame; calculating the maximum likelihood value of the vector of each sliding window, and constructing a probability vector corresponding to the feature vector according to the maximum likelihood value of the vector; and obtaining a feature set based on the probability vector and the feature vector. Human behavior recognition includes recognizing human behavior in video data, and the process includes video acquisition, defining a region of interest ROI, processing a region of interest ROI, feature extraction and representation, tracking, and action understanding, wherein defining a region of interest ROI includes image segmentation, and action understanding includes classification and assignment of action labels.

[0024] It should be noted that the convolutional neural network (CNN) algorithm in the present invention is to identify human interactions from videos captured by drones, which plays an important role in improving disaster response and rescue operations. One of the key goals of computer vision and remote sensing is to develop autonomous systems that can analyze and understand their interactions with humans in their environment. For example, in order to effectively interpret a scene, such a system must be able to detect and identify human bodies, the geospatial location of objects, background elements, and overall human behavior. Among these tasks, human recognition, especially for the purpose of monitoring, control, and security, has attracted considerable attention since the beginning of computer vision research. This algorithm focuses on enhancing the detection of humans and objects, solving a fundamental challenge in this field. Many algorithms have been developed to detect and recognize human behavior in different environments. Human behavior recognition refers to the recognition of human behavior in video data. This process usually includes six key steps: (1) video acquisition; (2) defining regions of interest (ROIs), including segmentation; (3) ROI preprocessing; (4) feature extraction and representation; (5) tracking; and (6) action understanding, which includes classification and assignment of action labels. An overview of a simple human behavior recognition system is shown in Figure 1. Figure 2 As shown in the figure. Video frames are captured by drone cameras, and each frame is then segmented to separate key areas, usually human bodies. Preprocessing techniques are used to minimize the impact of acquisition inconsistencies, ensuring that the extracted features still represent essential information. Human bodies are tracked across all frames, allowing the identification of motion trajectories. Finally, by applying classification techniques, each trajectory is mapped to the appropriate action class to obtain a specified action label. The above process involves analyzing and tracking human bodies in all video frames, generating a large amount of data and requiring a lot of computing resources and memory to extract features. This method aims to optimize this process and improve the efficiency of real-time applications in disaster management.

[0025] Among them, feature extraction plays a vital role in understanding behavior and is also a key factor in the method of identifying human behavior. The feature extraction models for human recognition are generally divided into classical models and deep learning models. Classical models rely on local, global or a combination of multiple features; global features provide a high-level description of the entire object, capturing aspects such as shape, contour representation and texture, while local features focus on small patches in the image; global features are commonly used for tasks such as image retrieval, object detection, classification and human recognition, but they are extremely susceptible to noise (such as Gaussian noise, salt and pepper noise); although local features are effective, they include segmenting the image into smaller parts, resulting in high computational costs, and most classical models face limitations in video processing, such as handling background clutter, occlusion and changes in human structure. Therefore, the use of deep learning models has made great breakthroughs in solving the challenges of human behavior recognition. In this embodiment, a new method is introduced, which is called the first and last image (FLIs) algorithm, which includes extracting frames from the video, selecting the first and last images, and then applying CNN for object recognition and video classification within a predefined window of interest.

[0026] Specifically, Table 1 outlines several popular human action recognition models and algorithms, which represent well-defined computations and are the result of advanced algorithmic designs.

[0027] Table 1 Common methods

[0028] An advanced algorithm for human action recognition in video clips was developed using a convolutional neural network (CNN) trained on the ImageNet dataset. The algorithm significantly improves on previous methods. A key improvement is the use of super-resolution as a pre-processing technique to increase the image resolution, allowing for better information extraction. Various challenges such as motion blur, noise, the height of the drone, and varying brightness levels make it difficult to achieve accurate recognition. To overcome these issues, a pre-trained CNN was used to extract feature vectors with conceptual representations of image objects, and then a support vector machine (SVM) was used to determine the relationships between these objects and assign labels to video sequences. The algorithm provides the following key benefits: 1. A new method and algorithm specifically designed to identify human-object interactions; 2. Improved action recognition performance, with an accuracy improvement of 3.82% compared to the above methods, making it particularly suitable for potential disaster management and rescue applications; 3. Improved efficiency of human detection in emergency rescue operations, providing further progress for future research.

[0029] Specifically, this method was tested on datasets such as FLLDZ University (UCF Sports Action) and sports videos of a certain athletic event, as well as drone-captured videos from the Southwest. Although the accuracy of the method of the present invention is comparable to that of the standard CNN method, it has achieved improvements in accuracy, computational efficiency, and processing time. The accuracy of the method is 90.42%, which is 3.82% higher than the existing method. This method is named the "First and Last Images" (FLIs) algorithm because it focuses on key frames at the beginning and end of the video sequence to optimize recognition and reduce computational complexity. Drone videos were also combined to further evaluate the performance of the method of the present invention. Specifically, traffic footage of the SWJTU University Sports Center was recorded using a DJI drone in the Southwest. In order to simulate real-world disaster scenarios, drone videos depicting earthquakes, hurricanes, and floods were also downloaded from the YT website. The drone used in the experiment is equipped with a camera that can transmit real-time video to the ground, which is critical for real-time disaster management.

[0030] In disaster scenarios, drone footage can help identify injured or missing people and those who need help. Through target detection and analysis of human interactions, the proposed method shows the potential to locate these individuals. The method combines CNN and the first and last image (FLIs) method to identify human behavior through CNN. The CNN-FLIs method works by processing a video frame and dividing it into M smaller images of m×n pixels. From each video frame, the first and last images are extracted for further analysis. For each complete image, the length of the image ( l ) and width (w) and divide it into smaller segments with specified size (e.g., m / 3×n / 3, m / 4×n / 4, or m / 5×n / 5), which act as "sliding windows" on the image. The size of the movable window is m / 3×n / 3 / m / 4×n / 4 or m / 5×n / 5, with 50% overlap, covering the entire frame. For each movable window, CNN is applied to detect objects, generate feature vectors, and calculate the maximum likelihood value of the vector for each window, and then the same process is applied to the last image of the video frame.

[0031] Specifically, the algorithm below outlines the process of creating windows from an image to ensure comprehensive target detection across frames. The combination of CNN and FLIs methods provides an effective method for target detection, which is particularly useful in complex environments such as disaster areas. Rapid identification and classification of individuals can significantly improve rescue efforts. The dataset used can be used as a reference here: The process of segmenting the sliding window is as follows: Input the image of the video frame , from the image Extracting window sets , and initialize and , according to the entries in the collection calculate Size, create and ;according to calculate , calculate ,get and , the expression corresponding to the output window is ,in, and They represent the initial scaled size of the window set, m and n represent the size of the movable window, Round up.

[0032] Maximum likelihood supervised classification is applied to map each feature vector to appropriate human action recognition and labeling. The process consists of three main steps: (1) selection of key frames from the video; (2) creation of object representation; and (3) supervised classification and action labeling. In step (1), key frames are extracted from the video as representative snapshots; in step (2), adaptive windows are applied and CNN and CNN-FLIs features are calculated for both images to create a probability vector containing 1000 elements; finally, in step (3), a support vector machine (SVM) is trained to predict action labels based on these features to recognize human actions and label the corresponding actions.

[0033] Optionally, extracting key frames from each processed video frame includes: segmenting multiple video frames into high-resolution images and focusing on the first and last images to identify all objects in the scene, wherein a super-resolution algorithm is used to map the low-resolution image to the high-resolution image.

[0034] In this embodiment, the convolutional neural network includes a pre-trained CNN based on ImageNet and drone images, and the keyframes include objects of interest at different spatial positions and aspect ratios. The convolutional neural network generates a probability vector for each image to represent the existence of different objects; if a vector vec of an element r_i corresponds to a value n of a hurricane image, it is interpreted as the probability of object x appearing in the image is n; a support vector machine SVM is used to predict the label of human interaction based on the CNN output, wherein each CNN output of the sliding window corresponds to a probability vector. Construct a probability vector of length for each image The probability vector of the image position is With length units, the vector Defined as , and All have length units. The unit is ; The video frames corresponding to the two probability vectors of the first and last images are converted into vectors through a union function; when extracting the feature vectors, a classification framework is used to identify the types of human interactions and actions described in the video, wherein the classification framework includes a support vector machine SVM algorithm.

[0035] It should be noted that the raw video clips from the camera are used as input, which inherently contain temporal and spatial features. Although many advanced methods use temporal features to track human movement and capture motion, the method of the present invention eliminates the need for human tracking by discarding temporal features. Instead, the video is segmented into images with a high frame rate and focuses on the first and last images to identify all objects in the scene, which helps to reduce occlusion and ensure that objects are preserved throughout the process. Another advantage of segmenting the video into images is that it greatly reduces the computational complexity. In some cases, the video clips may be of low resolution. To solve this problem, super-resolution methods are applied to reconstruct low-quality images into high-quality images, effectively eliminating problems such as motion blur and noise, and improving the overall clarity of the image. A preprocessing step is added to increase the image resolution to extract more accurate information. Then, the CNN method is applied to the images extracted from the video. The super-resolution method learns to map the low-resolution image to the high-resolution corresponding image, and uses a hierarchical structure to build this mapping, which improves the quality of the input image and allows more effective target detection and recognition in subsequent stages.

[0036] Specifically, in order to capture spatial features, a pre-trained CNN based on ImageNet and drone images was used. Considering that the objects of interest in the selected frames have different spatial positions and aspect ratios, it is necessary to implement a movable window to calculate the images obtained from the frames. The CNN generates a probability vector for each image, indicating the presence of different objects. For example, if a vector vec with an element r_i has a value n for a hurricane image, it can be interpreted as the probability of object x appearing in the image is n. The CNN in this embodiment consists of seven fully connected layers, and the classification layer is eliminated from the end. Instead, the output of the previous layer, called the fc7 layer, is used as the image descriptor for recognition. A support vector machine (SVM) is then used to predict the label of the human interaction based on this output. In addition, during the CNN processing, principal component analysis (PCA) is applied to each window of the image (such as Figure 3) to reduce the dimensionality of the probability vector to 1. Images often contain objects of different sizes, such as buildings, ships, trees, cars, and animals, which can show an infinite variety of structures and shapes. The appearance of objects in the video is also affected by the distance from the camera and the angle at which the drone captures the scene. To account for this variability, an adaptive filter is used to segment 9 different movable window sizes. This method facilitates the extraction of people and objects, where the movable window is a sliding window.

[0037] Among them, different overlapping coverage ratios are applied to windows and experiments to test the optimal overlap conditions. From the complete image, a window size is selected with , and (like Figure 3 ) moves the window in the x- and y-directions, taking into account a 50% overlap in both axes; in the next stage, a window size is selected , and (like Figure 3 ). Similarly, the window was moved and a 50% image overlap was applied; in the last processing stage, the window size was chosen , and (like Figure 3 ), performs a similar process as described above and applies a CNN to each video. This process of window partitioning continues throughout the image, in this case large objects are detected in small windows and small objects are detected in large windows. The output of the CNN for each window is a probability vector, and the goal is to identify the maximum probability vector therein. To achieve this, each probability vector is compared to the previous one to determine the maximum value. For example, in detecting environmental trees in a video captured by a DJI camera on a drone, if the probability vector from the first window indicates a probability of n% existence and the second window shows m% (where m>n), the desired result would be to select m% as the maximum probability. This process ensures that the strongest signs of the presence of an object in a video frame are accurately identified.

[0038] Optionally, the operation mode of the support vector machine SVM algorithm includes generating two parallel lines for each set of data and using a hyperplane to effectively separate the space, wherein the support vector machine SVM constructs a hyperplane with the largest boundary in the high-dimensional space to classify the given data into different classes, and the expression of the optimal hyperplane is: (1) in, yes and scalar product of ; The nonlinear support vector machine (SVM) algorithm is used to calculate clusters and identify human interactions in videos. The support vector machine (SVM) algorithm uses a mathematical function and defines it as a kernel. The function of the kernel is to take data as input and transform it into the required form. The polynomial kernel equation (2) is used for image processing: (2) in, Represents the kernel function, for the vector and Perform transformation operation, k represents the kernel, and d represents the degree of the polynomial.

[0039] In this embodiment, the SVM operates by generating two parallel lines for each set of data, effectively separating the space. This results in a linear partition of the plane, creating a clear gap between the two categories, maximizing the width of this separation. This partition is achieved using a plane called a hyperplane. The SVM constructs a hyperplane with the largest margin in a high-dimensional space to accurately classify the given data into different classes. An original support vector machine (SVM) was trained using standard quadratic programming techniques. The dataset was divided into three subsets: 70% of the videos were assigned to the training set, 10% to the validation set, and 20% for testing purposes; the training data is used to learn the mapping function of the SVM, allowing the model to adjust its parameters based on the output provided. The validation set is crucial for minimizing overfitting and evaluating the performance of the model on unseen data. Finally, the test set is used to evaluate the overall performance of the SVM model.

[0040] As mentioned above, the purpose of this invention is to provide a method for extracting human interactions from videos, especially footage from camera drones, to enhance disaster management and rescue operations. To achieve this, the images extracted from the videos are divided into different sizes, and once the feature vector is built for each video, the corresponding labels can be identified using support vector machine (SVM) classification. The accuracy of the results is evaluated and compared with other existing technologies. The algorithm is applied to the sports event and UCF sports action datasets. In addition, transfer learning experiments are conducted using drone videos from the SWJTU Sports Center and the YT website, focusing on events such as earthquakes, hurricanes, and floods. This is done to evaluate the potential effectiveness of the proposed algorithm in disaster management and rescue applications. Among them, the sports event and UCF sports action datasets include 150 annotated videos classified into more than 20 different categories, which include intra-class and inter-class variations, including categories such as team sports, winter sports, and ball sports. For example, these datasets include 14 different types of diving, 6 types of weightlifting, and 12 types of horse riding. The drone videos are taken with a DJ camera drone, which is also used to collect videos of natural disasters, including earthquakes, hurricanes, and floods. To facilitate analysis, the resolution of these videos was changed, converting them from full HD (1080p) to 240p and 320p.

[0041] Above, we designed the evaluation procedure for three cases, (1) p segment length, (2) overlapping windows, and (3) many image settings. For each example, the details are as follows: The first experimental case is the length of the window: it can recognize all objects of different sizes. Nine windows of different sizes were used for each image. According to the obtained results, the best recognition of objects in length was achieved in the following way: , , and windows , , and windows , , and windows The second experimental case is based on the overlap ratio of each window: three experiments are applied to find the most suitable overlap size of 20%, 50% and 80% overlap. According to the calculation results and computational complexity, the best overlap window is 50% pixels.

[0042] The third experimental case mainly studies the impact of the number of keyframes on human action recognition. The effectiveness of this recognition process is affected by the number of images selected from the video frames. Although increasing the number of images usually improves the performance of the method, it also leads to higher memory requirements, longer processing time and higher computational complexity (e.g. Figure 4 ). Therefore, the proposed method only utilizes the first and last frames of each video for the recognition process, while existing methods randomly select three images. By relying only on the first and last images, the proposed method has the potential to improve the efficiency of disaster management and rescue operations. Using only these two frames can significantly speed up the computational process without compromising performance or quality. In disaster and rescue applications, timely response, affordable computing needs, rapid analysis, and speed of rescue teams are crucial to effectively address immediate risks and make informed decisions.

[0043] To further evaluate the proposed method, the accuracy is evaluated by comparing the performance of using the first and last images with the additional five frames. The experimental results are shown in Table 3. Although the results show that using more images can produce the best performance, it is important to note that the accuracy difference between using 2, 3, and 7 images is minimal. Therefore, it can be used as a feasible method for disaster management and rescue planning. All experiments were conducted using MATLAB R2018b on a Windows 10 system with an ASUS Core i7 processor (2.0 GHz) and 8 GB of memory. Table 2 provides the results of the method applied to drone videos, and Table 3 summarizes the results of the compared methods using UCF and a sports game dataset.

[0044] Table 2 Results of using the proposed algorithm to identify drone aerial videos (video contents include: XNJT University Sports Center, flood, hurricane, earthquake)

[0045] Table 3 Comparison of experimental results of using the algorithm of the present invention and other algorithms to identify the FLLD university sports behavior dataset and a sports meeting dataset

[0046] Three verification methods are used to evaluate the performance of the method of the present invention. The first verification is Recall, which is the ratio of the retrieved relevant instances to the total number of relevant instances. Recall is also called sensitivity and is calculated by the following equation: (3) Among them, TP is true positive and FN is false negative; The second verification is Precision, which is the proportion of relevant instances in the retrieved instances. Precision is also called positive prediction value and is calculated as follows: (4) The third verification method is Accuracy, which is calculated as follows: Accuracy=(TP+TN) / (TP+TN+FP+FN) (5) Among them, TP is the number of positive cases correctly identified, FP is the number of negative cases misclassified as positive, and FN is the number of positive cases misclassified as negative. Using the same videos from UCF sports operations and a sports dataset of a sports meeting, as well as drone footage video data, the performance of the algorithm was evaluated compared with the existing technology. The comparison aims to evaluate the effectiveness of the CNN-FLIs method in disaster management and rescue operations, focusing on time, speed, and accuracy (such as Figure 5 ), the proposed method outperforms the traditional CNN in both training and testing stages, showing faster processing capabilities.

[0047] The above demonstrates the effectiveness of utilizing the first and last images extracted from video frames in the emergency response process, using these two images to maintain quality and performance. Figure 5 As shown, Figure 5 The comparison of recognition rates (%) for different videos is shown, and increasing the number of images leads to significant differences in performance, which supports that the proposed method is well suited for emergency situations where time and speed are key factors in rescue operations. To validate the method, experiments were conducted using drone videos from XN Aviation, as well as footage of earthquakes, hurricanes, and floods, comparing the effectiveness of using the first and last images versus using more images in the video frames. The proposed method outperforms existing methods on two benchmark datasets and drone footage for recognizing human actions in videos, using a hierarchy of pre-trained convolutional neural networks (CNNs), omitting the classification layer and using the output of the previous layer as image descriptors. Inspired by object recognition, the proposed CNN-FLIs method focuses on recognizing human actions based on object and scene recognition without the need to track all video frames. In addition, incorporating learning methods in the feature extraction process can improve the accuracy of action recognition by more than 3.82%, which can significantly advance research on developing infrastructure to improve the efficiency of emergency response and human-computer interaction recognition in future research using drone footage.

[0048] See also Figure 6 The present invention also provides a human interaction behavior recognition device for emergency and rescue, which is applied to the human interaction behavior recognition method for emergency and rescue as described above, comprising: A video data acquisition unit, used to acquire video data captured by the drone, and to acquire a plurality of video frames from the video data; A key frame extraction unit, configured to extract key frames from the plurality of video frames, and determine a plurality of sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; A human behavior recognition unit, configured to perform human behavior recognition on the key frame using a convolutional neural network according to the sliding window to generate a feature vector corresponding to the key frame, and construct a probability vector according to the feature vector to obtain a feature set; The action prediction unit is used to use a support vector machine algorithm to perform prediction analysis on the feature set to obtain corresponding action labels, and to identify human behaviors and mark corresponding actions based on the action labels.

[0049] In this embodiment, convolutional neural networks (CNNs) are used to address challenges related to video quality and computational efficiency. By evaluating the performance of UCF sports actions and sports videos of a certain athletic event, human-computer interaction recognition is enhanced, and the efficiency in terms of accuracy, processing time, and computational complexity of rescue and emergency management responses is improved. By analyzing the first and last images (FLIs), image quality is improved, and disaster management and rescue operations are accelerated. The proposed CNN integration method achieves an accuracy rate of 90.42%.

[0050] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0052] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A human interaction behavior recognition method for emergency and rescue, characterized in that: The steps include: Obtaining video data captured by the drone, and obtaining a plurality of video frames from the video data; Extracting key frames from the multiple video frames, and determining multiple sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; Using a convolutional neural network to perform human behavior recognition on the key frame according to the sliding window to generate a feature vector corresponding to the key frame, and constructing a probability vector according to the feature vector to obtain a feature set; A support vector machine algorithm is used to perform predictive analysis on the feature set to obtain corresponding action labels, and human behaviors are identified and corresponding actions are marked based on the action labels.

2. The human interaction behavior recognition method for emergency and rescue according to claim 1, characterized in that: Extracting key frames from the multiple video frames includes: Divide a video frame into M Pixel images; Extracting key frames from each processed video frame; According to pixel size Calculating the length and width of the key frame, and dividing the key frame into image segments of a preset size, wherein the image segment includes a plurality of pixel sizes; The image segment is taken as a sliding window on the image, wherein the size of the sliding window covers the entire image.

3. The human interaction behavior recognition method for emergency and rescue according to claim 2, characterized in that: Using a convolutional neural network to perform human behavior recognition on the video frame according to the sliding window to generate a feature vector corresponding to the key frame, including: Using a convolutional neural network to perform human behavior recognition on the key frame according to the sliding window to generate a feature vector corresponding to the key frame; Calculating the maximum likelihood value of the vector of each sliding window, and constructing a probability vector corresponding to the feature vector according to the maximum likelihood value of the vector; A feature set is obtained based on the probability vector and the feature vector.

4. The human interaction behavior recognition method for emergency and rescue according to claim 2, characterized in that: Extract key frames from each processed video frame, including: Multiple video frames are segmented into high-resolution images and focus on the first and last images to identify all objects in the scene, where a super-resolution algorithm is used to map the low-resolution images to high-resolution images.

5. The human interaction behavior recognition method for emergency and rescue according to claim 1, characterized in that: The convolutional neural network includes a pre-trained CNN based on ImageNet and drone images, and the key frames include objects of interest at different spatial positions and aspect ratios.

6. The human interaction behavior recognition method for emergency and rescue according to claim 5, characterized in that: The convolutional neural network generates a probability vector for each image to represent the presence of different objects; If a vector vec with an element r_i corresponds to the value n of the hurricane image, it is interpreted as the probability of object x appearing in the image being n; Support vector machine (SVM) is used to predict the labels of human interactions based on CNN output, where each CNN output of a sliding window corresponds to a probability vector.

7. The human interaction behavior recognition method for emergency and rescue according to claim 6, characterized in that: Also includes: For each image, construct a The probability vector of the image position is With length units, the vector Defined as , and All have length units. The unit is ; The video frames corresponding to the two probability vectors of the first and last images are converted into vectors through a union function; When extracting the feature vector, a classification framework is used to identify the types of human interactions and actions depicted in the video, wherein the classification framework includes a support vector machine (SVM) algorithm.

8. The human interaction behavior recognition method for emergency and rescue according to claim 7, characterized in that: The operation mode of the support vector machine (SVM) algorithm includes generating two parallel lines for each set of data and using a hyperplane to effectively separate the space. The support vector machine (SVM) constructs a hyperplane with the largest boundary in the high-dimensional space to classify the given data into different classes. The expression of the optimal hyperplane is: （1） in, yes and scalar product of ; The nonlinear support vector machine (SVM) algorithm is used to calculate clusters and identify human interactions in videos. The support vector machine (SVM) algorithm uses a mathematical function and defines it as a kernel. The function of the kernel is to take data as input and transform it into the required form. The polynomial kernel equation (2) is used for image processing: （2） in, Represents the kernel function, for the vector and Perform transformation operation, k represents the kernel, and d represents the degree of the polynomial.

9. The human interaction behavior recognition method for emergency and rescue according to claim 1, characterized in that: Human action recognition includes identifying human actions in video data, and its process includes video acquisition, defining region of interest ROI, ROI processing, feature extraction and representation, tracking, and action understanding, wherein defining region of interest ROI includes image segmentation, and action understanding includes classification and assignment of action labels.

10. A human interaction behavior recognition device for emergency and rescue, characterized in that: The human interaction behavior recognition method for emergency and rescue as claimed in any one of claims 1 to 9 comprises: A video data acquisition unit, used to acquire video data captured by the drone, and to acquire a plurality of video frames from the video data; A key frame extraction unit, configured to extract key frames from the plurality of video frames, and determine a plurality of sliding windows of the video frames according to the key frames, wherein the key frames include the first image and the last image in the image sequence corresponding to each video frame, and the sliding windows include calculating the length and width of the image and dividing the image into preset sizes; A human behavior recognition unit, configured to perform human behavior recognition on the key frame using a convolutional neural network according to the sliding window to generate a feature vector corresponding to the key frame, and construct a probability vector according to the feature vector to obtain a feature set; The action prediction unit is used to use a support vector machine algorithm to perform prediction analysis on the feature set to obtain corresponding action labels, and to identify human behaviors and mark corresponding actions based on the action labels.

Images