Method, device and equipment for action recognition based on images of unmanned aerial vehicle

Abstract

The application relates to a motion recognition method, device and equipment based on a UAV image, and relates to the technical field of image processing. The method comprises the following steps: obtaining a bounding box of an active object in a UAV image through an object recognition model and extracting a plurality of key points of the active object based on the bounding box to construct a skeleton model of the active object according to a UAV image obtained by the UAV shooting the active object, then extracting key point features and global features of the active object to capture multi-dimensional information of a motion performed by the active object, and finally inputting the key point features and the global features into a motion recognition model to obtain a motion recognition result, which can accurately indicate the motion performed by the active object, thereby improving the accuracy of motion recognition in a UAV shooting scene.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and device for motion recognition based on UAV images. Background Technology

[0002] Currently, unmanned aerial vehicles (UAVs) are being used more and more widely in various functions, including sports, entertainment, security, search and rescue, and monitoring of human activities in the air.

[0003] However, the movement of drones is quite complex, and the images obtained by drones of moving objects (such as humans) often change in size, angle, and presentation, making it difficult to accurately identify the movements of moving objects from drone images. Summary of the Invention

[0004] This application provides a method, apparatus, and device for motion recognition based on UAV images. The technical solution of this application is as follows: In a first aspect, this application provides a motion recognition method based on UAV images, the method comprising: Acquire the first drone image obtained by the drone capturing images of the active object; The first drone image is preprocessed to remove background noise other than the active object, resulting in a second drone image. The second UAV image is input into the object recognition model to obtain the object recognition result, which indicates the bounding box of the active object in the second UAV image. Based on the object recognition results, multiple key points of the active object in the second UAV image are extracted, and a skeletal model of the active object is constructed based on the multiple key points; Based on the multiple key points and the skeletal model, key point features of the active object in the second UAV image are extracted, and the key point features are used to indicate the joint angles and geometric structure of the active object; and, based on the second UAV image and the object recognition result, global features of the active object in the second UAV image are extracted, and the global features are used to indicate the three-dimensional shape and regional stability of the active object. The key point features and the global features are input into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone photographs the active object.

[0005] In some embodiments, the plurality of key points includes a plurality of first key points and a plurality of second key points; Based on the object recognition result, extracting multiple key points of the active object in the second UAV image, and constructing a skeletal model of the active object based on the multiple key points, includes: Based on the object recognition result and the reference skeleton model, the plurality of first key points are extracted. The reference skeleton model indicates the reference key points required to construct the skeleton model of the reference object. The first key points are used to define the key positions of the active object. Based on the aforementioned first key points and formula (1), the aforementioned second key points are determined, which are used to supplement the definition of the key locations of the active object: (1); in, i Represents any first key point. j Indicates the relationship with the first i The first key point is different from the first key point. n To represent any second key point, x i Indicates the first i The horizontal position of the first key point y i Indicates the first i The vertical position of the first key point x j Indicates the first j The horizontal position of the first key point y j Indicates the first j The vertical position of the first key point; Based on the plurality of first key points and the plurality of second key points, a skeletal model of the active object is constructed.

[0006] In some embodiments, extracting key point features of the active object in the second UAV image based on the plurality of key points and the skeletal model includes: Based on the multiple key points and formula (2), the first key point feature of the active object is extracted, which indicates the joint angle of the active object: (2); in, q To represent any key point, p Indicates the relationship with the first q Different key points x q Indicates the first q The horizontal position of each key point y q Indicates the first q Vertical position of key pointsy p Indicates the first p The horizontal position of each key point y p Indicates the first p Vertical position of each key point; Based on the multiple key points and the skeletal model, the Euclidean distance between each pair of adjacent key points is determined, and based on the Euclidean distance between each pair of adjacent key points, the geodesic distance between each pair of adjacent key points is determined. The geodesic distance between each pair of adjacent key points is used as the second key point feature of the active object, and the second key point feature indicates the geometric structure of the active object.

[0007] In some embodiments, the step of extracting global features of the active object in the second UAV image based on the second UAV image and the object recognition result, wherein the global features are used to indicate the three-dimensional morphology and regional stability of the active object, includes: The outline of the active object in the second UAV image is located, a three-dimensional point cloud of the active object is generated based on the outline, and features are extracted from the three-dimensional point cloud to obtain the first global feature of the active object, which indicates the three-dimensional shape of the active object. Based on the bounding box indicated by the object recognition result, the maximum stable extreme value region of the active object in the second UAV image is determined, and the maximum stable extreme value region is used as the second global feature of the active object, which indicates the regional stability of the active object.

[0008] In some embodiments, inputting the key point features and the global features into the action recognition model to obtain the action recognition result of the active object includes: The keypoint features and the global features are input into the feature optimizer to obtain optimized keypoint features and global features. The feature optimizer is used to capture the complexity and non-correlation between features. The optimized key point features and global features are input into the action recognition model. The action recognition model is used to classify the actions of the active object to obtain the action recognition result.

[0009] In some embodiments, the method further includes at least one of the following: adjusting the flight trajectory of the drone according to the action indicated by the action recognition result; and feeding back a notification message related to the action according to the action indicated by the action recognition result.

[0010] Secondly, this application provides a motion recognition device based on UAV images, the device comprising: The acquisition module is used to acquire the first drone image obtained by the drone taking pictures of the active object; The preprocessing module is used to preprocess the first UAV image to remove background noise other than the active object in the first UAV image, so as to obtain the second UAV image. An object recognition module is used to input the second UAV image into an object recognition model to obtain an object recognition result, wherein the object recognition result indicates the bounding box of the active object in the second UAV image; The first extraction module is used to extract multiple key points of the active object in the second UAV image based on the object recognition result, and to construct a skeletal model of the active object based on the multiple key points; The second extraction module is used to extract key point features of the active object in the second UAV image based on the multiple key points and the skeletal model, wherein the key point features are used to indicate the joint angles and geometric structure of the active object; and to extract global features of the active object in the second UAV image based on the second UAV image and the object recognition result, wherein the global features are used to indicate the three-dimensional shape and regional stability of the active object. The action recognition module is used to input the key point features and the global features into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone photographs the active object.

[0011] In some embodiments, the plurality of key points includes a plurality of first key points and a plurality of second key points; The first extraction module is used for: Based on the object recognition result and the reference skeleton model, the plurality of first key points are extracted. The reference skeleton model indicates the reference key points required to construct the skeleton model of the reference object. The first key points are used to define the key positions of the active object. Based on the aforementioned first key points and formula (1), the aforementioned second key points are determined, which are used to supplement the definition of the key locations of the active object: (1); in, i Represents any first key point. j Indicates the relationship with the first i The first key point is different from the first key point. n To represent any second key point, x i Indicates the first i The horizontal position of the first key point yi Indicates the first i The vertical position of the first key point x j Indicates the first j The horizontal position of the first key point y j Indicates the first j The vertical position of the first key point; Based on the plurality of first key points and the plurality of second key points, a skeletal model of the active object is constructed.

[0012] In some embodiments, the second extraction module is configured to: Based on the multiple key points and formula (2), the first key point feature of the active object is extracted, which indicates the joint angle of the active object: (2); in, q To represent any key point, p Indicates the relationship with the first q Different key points x q Indicates the first q The horizontal position of each key point y q Indicates the first q Vertical position of key points y p Indicates the first p The horizontal position of each key point y p Indicates the first p Vertical position of each key point; Based on the multiple key points and the skeletal model, the Euclidean distance between each pair of adjacent key points is determined, and based on the Euclidean distance between each pair of adjacent key points, the geodesic distance between each pair of adjacent key points is determined. The geodesic distance between each pair of adjacent key points is used as the second key point feature of the active object, and the second key point feature indicates the geometric structure of the active object.

[0013] In some embodiments, the second extraction module is configured to: The outline of the active object in the second UAV image is located, a three-dimensional point cloud of the active object is generated based on the outline, and features are extracted from the three-dimensional point cloud to obtain the first global feature of the active object, which indicates the three-dimensional shape of the active object. Based on the bounding box indicated by the object recognition result, the maximum stable extreme value region of the active object in the second UAV image is determined, and the maximum stable extreme value region is used as the second global feature of the active object, which indicates the regional stability of the active object.

[0014] In some embodiments, the action recognition module is configured to: The keypoint features and the global features are input into the feature optimizer to obtain optimized keypoint features and global features. The feature optimizer is used to capture the complexity and non-correlation between features. The optimized key point features and global features are input into the action recognition model. The action recognition model is used to classify the actions of the active object to obtain the action recognition result.

[0015] In some embodiments, the apparatus further includes at least one of the following: an adjustment module, configured to adjust the flight trajectory of the UAV according to the action indicated by the action recognition result; and a feedback module, configured to provide a notification message related to the action according to the action indicated by the action recognition result.

[0016] Thirdly, this application provides an electronic device including a processor and a memory, the memory storing program code, and the processor executing the program code to implement the method provided in the first aspect above.

[0017] Fourthly, this application provides a computer-readable storage medium comprising: when program code in the computer-readable storage medium is executed by a processor of an electronic device, enabling the electronic device to perform the method provided in the first aspect.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the implementation environment for a motion recognition method based on UAV images; Figure 2 A flowchart of a motion recognition method based on drone images; Figure 3 This is a schematic diagram of a motion recognition device based on drone images. Figure 4 This is a schematic diagram of the structure of an electronic device. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0021] The data involved in this application may be data authorized by the user or fully authorized by all parties.

[0022] Figure 1 This is a schematic diagram illustrating the implementation environment of a motion recognition method based on UAV images. (See attached image) Figure 1 The implementation environment includes: terminal 101 and server 102.

[0023] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. Terminal 101 has communication capabilities and can access wired or wireless networks. Terminal 101 can refer to one of multiple terminals; this embodiment uses terminal 101 as an example only. Those skilled in the art will understand that the number of terminals can be more or less. Indicatively, terminal 101 can install and run an application program, and terminal 101 is communicatively connected to a drone, capable of receiving drone images captured by the drone and transmitting the drone images to server 102, so that server 102 can execute a motion recognition method based on the drone images.

[0024] Server 102 can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed file system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, server 102 and terminal 101 are directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment. Optionally, the number of servers 102 can be more or less, which is not limited in this embodiment. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0025] In this application, server 102 performs the main computational tasks, while terminal 101 performs secondary computational tasks. In some embodiments, the action recognition method based on UAV images can be executed independently by terminal 101 or server 102, and this application does not limit this.

[0026] Figure 2 This is a flowchart of a motion recognition method based on drone images. (Example) Figure 2 As shown, this method is performed by an electronic device, which can be... Figure 1 Terminal 101 in the implementation environment shown can also be Figure 1 Server 102 in the implementation environment shown. The motion recognition method based on UAV images includes the following steps 201 to 206.

[0027] 201. Electronic equipment acquires the first drone image obtained by the drone taking pictures of the active object.

[0028] The object of the activity can be, for example, a human, an animal, or any other object capable of movement. The number of objects can be one or more, and this application does not limit this. Indicatively, the electronic device acquires video footage taken by the drone of the object, extracts frames from the video, and obtains a first drone image.

[0029] 202. The electronic device preprocesses the first UAV image to remove background noise other than moving objects from the first UAV image, thereby obtaining the second UAV image.

[0030] An electronic device employs at least one preprocessing method to preprocess a first drone image to remove background noise (excluding moving objects) from the first drone image, thereby obtaining a second drone image. The preprocessing method includes one or more of Gaussian blur, grayscale conversion, and background removal. It should be understood that drone images typically contain elements such as trees, buildings, and vehicles, which can interfere with the recognition of moving objects. Removing background noise can improve the recognizability of moving objects in the drone image, facilitating the capture of the actions performed by the moving objects.

[0031] 203. The electronic device inputs the second UAV image into the object recognition model to obtain the object recognition result, which indicates the bounding box of the active object in the second UAV image.

[0032] The object recognition model is used to identify moving objects in an image and output their bounding boxes. The object recognition result includes the object's category label and bounding box coordinates. For example, the object recognition model is the YOLO (You Only Look Once) model. YOLO is a real-time object detection system based on deep learning that directly predicts the bounding box position and category of moving objects in an image using a single neural network, enabling rapid identification of moving objects. It should be understood that drone footage is often dynamic; therefore, utilizing the real-time image processing capabilities of the YOLO model can more accurately identify moving objects in drone images.

[0033] 204. Based on the object recognition results, the electronic device extracts multiple key points of the moving object in the second UAV image and constructs a skeletal model of the moving object based on the multiple key points.

[0034] In this application, the multiple key points of the active object include multiple first key points and multiple second key points. The first key points are used to define the key locations of the active object, and the second key points are used to supplement the definition of the key locations of the active object. Taking the active object as a human as an example, key locations refer to points that are anatomically significant and used to identify specific points on the human body, such as the nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, right knee, left knee, left hip, right hip, left ankle, and right ankle. The extraction process of multiple key points is described in detail below. Schematically, step 204 includes steps 2041 to 2043 as follows: 2041. Based on the object recognition results and the reference skeleton model, extract multiple first key points.

[0035] The reference skeleton model indicates the key reference points needed to construct the skeleton model of the reference object. Since the object recognition result includes the bounding box coordinates of the active object in the second UAV image, the electronic device can determine the coordinates of the key positions of the active object based on the bounding box coordinates and the reference skeleton model, that is, extract multiple first key points.

[0036] 2042. Based on multiple first key points and formula (1), multiple second key points are determined.

[0037] Formula (1) is shown below: (1); in, i Represents any first key point. j Indicates the relationship with the first i The first key point is different from the first key point. n To represent any second key point, x i Indicates the first i The horizontal position of the first key point y i Indicates the first i The vertical position of the first key point x j Indicates the first j The horizontal position of the first key point y j Indicates the first jThe vertical position of the first keypoint. That is, the midpoint between any two first keypoints is taken as the new keypoint, i.e., the second keypoint. For example, taking the human subject as an example, the second keypoint can be interpreted as the midpoint between key positions such as the left and right shoulders, elbows, wrists, knees, or ankles. These second keypoints further enhance the effect of skeletal representation by supplementing the key positions that define more precise spatial relationships between body parts.

[0038] 2043. Based on multiple first key points and multiple second key points, construct the skeletal model of the active object.

[0039] After obtaining multiple first keypoints and multiple second keypoints, these keypoints are connected to construct the skeletal model of the active object.

[0040] 205. The electronic device extracts key point features of active objects in the second UAV image based on multiple key points and a skeleton model; and extracts global features of active objects in the second UAV image based on the second UAV image and object recognition results.

[0041] Among them, key point features are used to indicate the joint angles and geometry of the moving object; global features are used to indicate the three-dimensional shape and regional stability of the moving object.

[0042] The extraction processes for key point features and global features are described below.

[0043] In some embodiments, the key point features include a first key point feature and a second key point feature, and the key point feature extraction process includes the following steps A1 and A2: Step A1: Based on multiple key points and formula (2), extract the first key point feature of the active object. The first key point feature indicates the joint angle of the active object. (2); in, q To represent any key point, p Indicates the relationship with the first q Different key points x q Indicates the first q The horizontal position of each key point y q Indicates the first q Vertical position of key points y p Indicates the first p The horizontal position of each key point y p Indicates the first pThe vertical position of the key points. That is, formula (2) is used to calculate the angle between the vector formed by the two key points and the horizontal axis. This angle is used to measure the orientation of the limb or joint relative to the horizontal plane. It should be understood that the angle between the key points is crucial for understanding the relative orientation of the limb in different movements. By calculating the joint angle, key details are provided for subsequent accurate movement recognition, thereby improving the accuracy of the movement recognition results.

[0044] Step A2: Based on multiple keypoints and the skeleton model, determine the Euclidean distance between each pair of adjacent keypoints, and based on the Euclidean distance between each pair of adjacent keypoints, determine the geodesic distance between each pair of adjacent keypoints. Use the geodesic distance between each pair of adjacent keypoints as the second keypoint feature of the active object. The second keypoint feature indicates the geometric structure of the active object.

[0045] In this application, geodesic distance, also known as geodesic or geodesic line, refers to the local shortest path between two points in space. It should be understood that the structure and shape of a moving object typically deform continuously due to movement. Euclidean distance alone cannot capture the inherent structure of key locations on the moving object. By calculating the geodesic distance between each pair of adjacent key points, the inherent geometry of the moving object can be accurately captured, thereby precisely measuring the spatial interactions between different parts of the moving object and providing technical support for the subsequent accurate output of motion recognition results.

[0046] Schematic, the electronic device calculates the Euclidean distance between each pair of adjacent key points based on the coordinates of each pair of adjacent key points in the skeletal model. Then, it calculates the path with the minimum sum of Euclidean distances based on Dijkstra's Algorithm to obtain the geodesic distance between each pair of adjacent points.

[0047] In some embodiments, the global features include a first global feature and a second global feature, and the extraction process of the global features includes the following steps B1 and B2: Step B1: Locate the outline of the moving object in the second UAV image, generate a 3D point cloud of the moving object based on the outline, extract features from the 3D point cloud to obtain the first global feature of the moving object, which indicates the 3D shape of the moving object.

[0048] In this system, a 3D point cloud is an arrangement of different points in a 3D coordinate system, with each point representing a different spatial location. The electronic device locates the outline of the moving object from the second UAV image using methods such as threshold segmentation or background subtraction, calculates depth values ​​based on pixel intensity, and projects the outline of the moving object into 3D space based on the depth values ​​to generate a 3D point cloud of the moving object. Subsequently, feature extraction, such as geometric features and curvature, is performed on the 3D point cloud to obtain the first global features of the moving object. In some embodiments, voxel mesh filtering can be used to simplify the 3D point cloud to address the redundancy problem of the original 3D point cloud, and feature extraction is performed based on the simplified 3D point cloud. The parameters involved in voxel mesh filtering can be configured according to requirements, and this application does not limit them.

[0049] By generating a 3D point cloud of the active object, the 2D contour of the active object is transformed into a 3D structure. The first global feature extracted based on the 3D point cloud can accurately represent the 3D shape of the active object, providing technical support for the subsequent accurate output of action recognition results.

[0050] Step B2: Based on the bounding box indicated by the object recognition result, determine the maximum stable extreme value region of the active object in the second UAV image, and use the maximum stable extreme value region as the second global feature of the active object. The second global feature indicates the regional stability of the active object.

[0051] The Maximum Stable Extremal Regions (MSER) ​​algorithm is a feature extraction method in computer vision used to detect stable regions in images and solve the problem of image correspondence across different viewpoints. This method detects key structural regions with affine invariance by analyzing the stability of connected regions during continuous changes in image grayscale thresholds. In this application, the electronic device analyzes the bounding boxes indicated by the object recognition results based on the MSER algorithm, determines which regions remain stable within a certain intensity range, and thus identifies the maximum stable extremal regions of the active object in the second UAV image to indicate the region stability of the active object. By identifying stable regions in the second UAV image, it provides technical support for subsequent differentiation of different action categories. Furthermore, the parameters involved in the MSER algorithm can be configured according to actual needs, and this application does not impose any limitations on this.

[0052] 206. The electronic device inputs key point features and global features into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone is shooting the active object.

[0053] The action recognition model is an artificial intelligence model obtained by training a deep neural network based on a drone image training set. This application does not limit the number of network layers in the action recognition model. Any deep neural network that can output action recognition results based on key point features and global features is applicable to the embodiments of this application.

[0054] In some embodiments, the electronic device inputs keypoint features and global features into a feature optimizer to obtain optimized keypoint features and global features. The feature optimizer is used to capture the complexity and non-correlation between features. The optimized keypoint features and global features are then input into an action recognition model. The action recognition model classifies the actions of the active object to obtain the action recognition result. For example, the feature optimizer is an optimizer based on Quadratic Discriminant Analysis (QDA), which can capture more complex relationships in the data. In this application, since the pose and motion differences between different action categories of the active object are large, the features optimized by the feature optimizer can enhance the discrimination ability between categories, thereby improving the accuracy of the action recognition result. In addition, the parameters involved in the feature optimizer can be configured according to actual needs, and this application does not limit them.

[0055] In some embodiments, the above method further includes at least one of the following: (1) Adjust the flight trajectory of the UAV according to the action indicated by the action recognition result. For example, in a mountain search and rescue scenario, the UAV captures the action of a moving object (such as a trapped person) making a "waving for help" action. After the action recognition result is output based on the action recognition method of this application, the electronic device can control the UAV to automatically adjust its flight trajectory, such as lowering its altitude or moving towards the location of the trapped person, so as to capture the location details of the moving object more clearly and assist rescuers in accurate positioning.

[0056] (2) Based on the action indicated by the action recognition result, a notification message related to the action is fed back. For example, in an event security scenario, if a drone captures an object in the event area making a "pushing" action, after the action recognition result is output based on the action recognition method of this application, the electronic device can send a notification to the terminal of the on-site security personnel to inform them of the action of the object in the event area, so that the security personnel can quickly arrive at the relevant area and take corresponding measures.

[0057] After obtaining the action recognition result of the active object through the action recognition model, since the action recognition result of this application can accurately indicate the action performed by the active object, the electronic device can adjust the flight trajectory in a timely manner according to the action recognition result, or feed back notification messages related to the action, which is applicable to multiple fields such as search and rescue, disaster response, and abnormal behavior early warning.

[0058] In summary, the action recognition method based on UAV images provided in this application, after acquiring UAV images of an active object, firstly, preprocesses the UAV images to remove background noise, then uses an object recognition model to determine the bounding box of the active object, focusing the effective area for subsequent processing. Next, a skeletal model is constructed by extracting key points of the active object, accurately representing its structure. Furthermore, joint angles, geodesic distances, 3D point cloud features, and regional stability features are extracted to achieve a multi-dimensional representation of the active object's actions. Finally, the extracted features are processed by the action recognition model to output accurate action recognition results. This method fully considers the complex environment, complex actions of active objects, and variable angles in UAV aerial photography scenarios, significantly improving the accuracy of action recognition in UAV images and providing precise technical support for search and rescue, security, and disaster response scenarios.

[0059] Figure 3 This is a schematic diagram of a motion recognition device based on drone images. (Refer to...) Figure 3 The device includes: The acquisition module 301 is used to acquire a first drone image obtained by the drone taking pictures of the active object; Preprocessing module 302 is used to preprocess the first UAV image to remove background noise other than moving objects from the first UAV image to obtain the second UAV image; The object recognition module 303 is used to input the second UAV image into the object recognition model to obtain the object recognition result, which indicates the bounding box of the active object in the second UAV image. The first extraction module 304 is used to extract multiple key points of the active object in the second UAV image based on the object recognition result, and to construct a skeletal model of the active object based on the multiple key points. The second extraction module 305 is used to extract key point features of the active object in the second UAV image based on multiple key points and a skeleton model. The key point features are used to indicate the joint angles and geometric structure of the active object. It is also used to extract global features of the active object in the second UAV image based on the second UAV image and the object recognition results. The global features are used to indicate the three-dimensional shape and regional stability of the active object. The action recognition module 306 is used to input key point features and global features into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone is shooting the active object.

[0060] In some embodiments, the plurality of key points includes a plurality of first key points and a plurality of second key points; The first extraction module 304 is used for: Based on the object recognition results and the reference skeleton model, multiple first key points are extracted. The reference skeleton model indicates the reference key points required to construct the skeleton model of the reference object. The first key points are used to define the key positions of the active object. Based on multiple first key points and formula (1), multiple second key points are determined. These second key points are used to supplement the definition of key locations of the active object. (1); in, i Represents any first key point. j Indicates the relationship with the first i The first key point is different from the first key point. n To represent any second key point, x i Indicates the first i The horizontal position of the first key point y i Indicates the first i The vertical position of the first key point x j Indicates the first j The horizontal position of the first key point y j Indicates the first j The vertical position of the first key point; A skeletal model of the active object is constructed based on multiple first key points and multiple second key points.

[0061] In some embodiments, the second extraction module 305 is used for: Based on multiple key points and formula (2), the first key point feature of the active object is extracted. The first key point feature indicates the joint angle of the active object. (2) in, q To represent any key point, p Indicates the relationship with the first q Different key points x q Indicates the first q The horizontal position of each key point y q Indicates the first q Vertical position of key points y p Indicates the first p The horizontal position of each key point y p Indicates the first p Vertical position of each key point; Based on the skeletal model of multiple keypoints and active objects, the Euclidean distance between each pair of adjacent keypoints is determined. Based on the Euclidean distance between each pair of adjacent keypoints, the geodesic distance between each pair of adjacent keypoints is determined. The geodesic distance between each pair of adjacent keypoints is used as the second keypoint feature of the active object. The second keypoint feature indicates the geometric structure of the active object.

[0062] In some embodiments, the second extraction module 305 is used for: The outline of the moving object in the second UAV image is located, a three-dimensional point cloud of the moving object is generated based on the outline, and the features of the three-dimensional point cloud are extracted to obtain the first global feature of the moving object. The first global feature indicates the three-dimensional shape of the moving object. Based on the bounding box indicated by the object recognition results, the maximum stable extreme value region of the active object in the second UAV image is determined, and the maximum stable extreme value region is used as the second global feature of the active object. The second global feature indicates the regional stability of the active object.

[0063] In some embodiments, the action recognition module 306 is used for: The keypoint features and global features are input into the feature optimizer to obtain optimized keypoint features and global features. The feature optimizer is used to capture the complexity and non-correlation between features. The optimized key point features and global features are input into the action recognition model. The action recognition model then classifies the actions of the active object to obtain the action recognition result.

[0064] In some embodiments, the device further includes at least one of the following: an adjustment module for adjusting the flight trajectory of the drone according to the action indicated by the action recognition result; and a feedback module for providing a notification message related to the action according to the action indicated by the action recognition result.

[0065] Using the aforementioned device, after acquiring drone images of moving objects, the images are first preprocessed to remove background noise. Then, an object recognition model is used to determine the bounding box of the moving object, focusing the effective area for subsequent processing. Next, key points of the moving object are extracted to construct a skeletal model, accurately representing the object's structure. Furthermore, joint angles, geodesic distances, 3D point cloud features, and regional stability features are extracted to achieve a multi-dimensional representation of the object's movements. Finally, a motion recognition model processes the extracted features to output accurate motion recognition results. This method fully considers the complex environment, complex movements, and variable angles of moving objects in drone aerial photography scenarios, significantly improving the accuracy of motion recognition in drone images and providing precise technical support for search and rescue, security, and disaster response scenarios.

[0066] Figure 4 This is a schematic diagram of an electronic device. This electronic device can be configured as a server or a terminal. The device can vary significantly depending on its configuration or performance. It may include one or more Central Processing Units (CPUs) 401 and one or more memories 402. Each memory 402 stores at least one line of program code, which is loaded and executed by the processors 401 to implement the process performed by the electronic device in the UAV image-based action recognition method provided in the above-described method embodiments. Of course, the electronic device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The electronic device 400 may also include other components for implementing device functions, which will not be elaborated here.

[0067] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0068] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A motion recognition method based on UAV images, characterized in that, The method includes: Acquire the first drone image obtained by the drone capturing images of the active object; The first drone image is preprocessed to remove background noise other than the active object, resulting in a second drone image. The second UAV image is input into the object recognition model to obtain the object recognition result, which indicates the bounding box of the active object in the second UAV image. Based on the object recognition results, multiple key points of the active object in the second UAV image are extracted, and a skeletal model of the active object is constructed based on the multiple key points; Based on the multiple key points and the skeletal model, key point features of the active object in the second UAV image are extracted, and the key point features are used to indicate the joint angles and geometric structure of the active object; and, based on the second UAV image and the object recognition result, global features of the active object in the second UAV image are extracted, and the global features are used to indicate the three-dimensional shape and regional stability of the active object. The key point features and the global features are input into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone photographs the active object.

2. The method according to claim 1, characterized in that, The multiple key points include multiple first key points and multiple second key points; Based on the object recognition result, extracting multiple key points of the active object in the second UAV image, and constructing a skeletal model of the active object based on the multiple key points, includes: Based on the object recognition result and the reference skeleton model, the plurality of first key points are extracted. The reference skeleton model indicates the reference key points required to construct the skeleton model of the reference object. The first key points are used to define the key positions of the active object. Based on the aforementioned first key points and formula (1), the aforementioned second key points are determined, which are used to supplement the definition of the key locations of the active object: （1）； （1）； in, i Represents any first key point. j Indicates the relationship with the first i The first key point is different from the first key point. n To represent any second key point, x i Indicates the first i The horizontal position of the first key point y i Indicates the first i The vertical position of the first key point. x j Indicates the first j The horizontal position of the first key point y j Indicates the first j The vertical position of the first key point; Based on the plurality of first key points and the plurality of second key points, a skeletal model of the active object is constructed.

3. The method according to claim 1, characterized in that, The step of extracting key point features of the active object in the second UAV image based on the multiple key points and the skeletal model includes: Based on the multiple key points and formula (2), the first key point feature of the active object is extracted, which indicates the joint angle of the active object: （2）； in, q To represent any key point, p Indicates the relationship with the first q Different key points x q Indicates the first q The horizontal position of each key point y q Indicates the first q Vertical position of key points y p Indicates the first p The horizontal position of each key point y p Indicates the first p Vertical position of each key point; Based on the plurality of key points and the skeleton model, the Euclidean distance between each pair of adjacent key points is determined, and based on the Euclidean distance between each pair of adjacent key points, the geodesic distance between each pair of adjacent key points is determined. The geodesic distance between each pair of adjacent key points is used as the second key point feature of the active object, and the second key point feature indicates the geometric structure of the active object.

4. The method according to claim 1, characterized in that, Based on the second UAV image and the object recognition result, global features of the active object in the second UAV image are extracted. These global features are used to indicate the three-dimensional morphology and regional stability of the active object, including: The outline of the active object in the second UAV image is located, a three-dimensional point cloud of the active object is generated based on the outline, and features are extracted from the three-dimensional point cloud to obtain the first global feature of the active object, which indicates the three-dimensional shape of the active object. Based on the bounding box indicated by the object recognition result, the maximum stable extreme value region of the active object in the second UAV image is determined, and the maximum stable extreme value region is used as the second global feature of the active object, which indicates the regional stability of the active object.

5. The method according to claim 1, characterized in that, The step of inputting the key point features and the global features into the action recognition model to obtain the action recognition result of the active object includes: The keypoint features and the global features are input into the feature optimizer to obtain optimized keypoint features and global features. The feature optimizer is used to capture the complexity and non-correlation between features. The optimized key point features and global features are input into the action recognition model. The action recognition model is used to classify the actions of the active object to obtain the action recognition result.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes at least one of the following: The flight trajectory of the drone is adjusted according to the action indicated by the action recognition result; Based on the action indicated by the action recognition result, a notification message related to the action is fed back.

7. A motion recognition device based on UAV images, characterized in that, The device includes: The acquisition module is used to acquire the first drone image obtained by the drone taking pictures of the active object; The preprocessing module is used to preprocess the first UAV image to remove background noise other than the active object from the first UAV image, so as to obtain the second UAV image. An object recognition module is used to input the second UAV image into an object recognition model to obtain an object recognition result, wherein the object recognition result indicates the bounding box of the active object in the second UAV image; The first extraction module is used to extract multiple key points of the active object in the second UAV image based on the object recognition result, and to construct a skeletal model of the active object based on the multiple key points; The second extraction module is used to extract key point features of the active object in the second UAV image based on the multiple key points and the skeletal model, wherein the key point features are used to indicate the joint angles and geometric structure of the active object; and to extract global features of the active object in the second UAV image based on the second UAV image and the object recognition result, wherein the global features are used to indicate the three-dimensional shape and regional stability of the active object. The action recognition module is used to input the key point features and the global features into the action recognition model to obtain the action recognition result of the active object. The action recognition result indicates the action performed by the active object when the drone photographs the active object.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing program code, and the processor executing the program code to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the program code in the computer-readable storage medium is executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the method as described in any one of claims 1 to 6.

Images