Object behavior recognition method, apparatus, device, and medium

Abstract

The application provides an object behavior recognition method and device, equipment and medium, relates to the technical field of artificial intelligence, and is used for improving the accuracy of recognizing object behavior, the method comprises the following steps: extracting the behavior characteristics of a target object based on the image characteristics of the target object in a video segment, which is equivalent to obtaining relatively coarse-grained feature information of the target object; extracting the posture characteristics of the target object based on the object key points of the target object in the video segment, which is equivalent to obtaining relatively fine-grained feature information of the target object; and recognizing the behavior of the target object in the video segment based on the behavior characteristics and the posture characteristics. Since the method combines the feature information of the target object in the video segment in multiple granularities and recognizes the behavior of the target object, the accuracy of the behavior recognition result of the target object is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence technology, providing a method, apparatus, device, and medium for object behavior recognition. Background Technology

[0002] With the continuous development of artificial intelligence technology, it has been widely applied in various fields. For example, artificial intelligence technology can be used to identify the behavior of people in videos.

[0003] Currently, one method for identifying human behavior in videos involves: detecting key points on the human body in each video frame; obtaining the person's posture information based on the detected key points; and then identifying the person's behavior based on this posture information. This method can detect behaviors with large ranges of motion in a video, but it cannot accurately identify behaviors with small ranges of motion. Furthermore, when there are many people in the video, it cannot accurately identify the behavior of each individual person. Therefore, the accuracy of this method for identifying behavior needs improvement. Summary of the Invention

[0004] This application provides an object behavior recognition method, apparatus, device, and medium to improve the accuracy of object behavior recognition.

[0005] On the one hand, an object behavior recognition method is provided, including:

[0006] Extract video segments from the video to be identified, wherein the video segments include at least one video frame;

[0007] From the video clip, at least one behavioral feature of each target object is extracted, wherein the behavioral feature is determined based on the image features of the corresponding target object in the video clip;

[0008] From the video segment, the pose features of each of the at least one target object are extracted, wherein the pose features are determined based on the set of object key points of the corresponding target object in the video segment;

[0009] Based on the behavioral and posture characteristics of each of the at least one target object, the target behavior recognition result of the at least one target object in the video segment is determined.

[0010] This application provides an object behavior recognition device, including:

[0011] A video segment extraction module is used to extract video segments from a video to be identified, wherein the video segment includes at least one video frame;

[0012] A behavior feature extraction module is used to extract behavior features of at least one target object from the video segment, wherein the behavior features are determined based on the image features of the corresponding target object in the video segment;

[0013] A pose feature extraction module is used to extract pose features of each of the at least one target object from the video segment, wherein the pose features are determined based on a set of object key points of the corresponding target object in the video segment;

[0014] The result determination module is used to determine the target behavior recognition result of the at least one target object in the video segment based on the respective behavioral and posture characteristics of the at least one target object.

[0015] In one possible embodiment, the behavioral feature extraction module is specifically used for:

[0016] From each video frame in the video segment, at least one image feature of a target object is extracted to obtain a video feature map corresponding to the video segment;

[0017] For each of the at least one target objects, perform the following operations:

[0018] From any video frame in the video segment, detect the relative position of at least one target object in any video frame;

[0019] Based on the relative position of the target object in any video frame, the behavioral characteristics of the target object are determined from the video feature map.

[0020] In one possible embodiment, the behavior feature extraction module is specifically used for:

[0021] From the video feature map, determine the feature region corresponding to the relative position of the target object;

[0022] The feature region is divided into multiple candidate units, and for the multiple candidate units, the following operations are performed: using an interpolation method, the coordinate value corresponding to a preset position in one of the multiple candidate units is determined, and the determined coordinate value is subjected to max pooling to obtain the pooling result;

[0023] Based on the pooling results of the multiple candidate units, the behavioral characteristics of the target object are obtained.

[0024] In one possible embodiment, the pose feature extraction module is specifically used for:

[0025] Extract the set of object key points for each of the at least one target object from each video frame in the video segment;

[0026] For each of the at least one target object, the following operations are performed: based on the set of object key points of one of the at least one target object, the pose features of the target object are obtained.

[0027] In one possible embodiment, the pose feature extraction module is specifically used for:

[0028] For each video frame in the video segment, perform one of the following operations:

[0029] Object detection is performed on one video frame from each video frame to obtain a detection box for at least one target object in the video frame, and at least one object key point of each target object is identified from each detected detection box.

[0030] Perform object keypoint detection on a video frame to obtain at least one object keypoint in the video frame, and determine the target object to which the at least one object keypoint belongs, so as to obtain at least one object keypoint for each of the at least one target object.

[0031] In one possible embodiment, the pose feature extraction module is specifically used for:

[0032] Based on the set of object key points of the target object, a first three-dimensional vector is obtained, wherein the first dimension of the first three-dimensional vector represents the duration of the video segment, the second dimension represents the number of object key points in the set of object key points of the target object, and the third dimension represents the number of channels.

[0033] Perform a convolution operation on the first three-dimensional vector to obtain a second three-dimensional vector;

[0034] By swapping the relative positions of the second and third dimensions in the second three-dimensional vector, a third three-dimensional vector is obtained.

[0035] Perform a convolution operation on the third 3D vector to obtain the pose features of the target object.

[0036] In one possible embodiment, the result determination module is specifically used for:

[0037] For each of the at least one target objects, perform the following operations:

[0038] The behavioral features of one of the at least one target objects are input into a pre-trained first behavior classification network to obtain the first probability value of the target object belonging to each single object behavior label.

[0039] The pose features of the target object are input into a pre-trained second behavior classification network to obtain the second probability value of the target object belonging to each single object behavior label.

[0040] The first probability value and the second probability value of the target object under the same single object behavior label are weighted and summed to obtain the third probability value of the target object belonging to each of the single object behavior labels.

[0041] Based on the obtained third probability values, the single-object behavior label to which the target object belongs is determined;

[0042] The single-object behavior labels of the at least one target object are determined and used as the target behavior recognition results corresponding to the at least one target object.

[0043] In one possible embodiment, the at least one target object includes multiple target objects, and the result determination module is specifically used for:

[0044] The behavioral features of each of the at least one target object are combined, and the combined behavioral features are input into a pre-trained third behavior classification network to obtain the fourth probability value of the at least one target object belonging to each multi-object behavior label.

[0045] The pose features of each of the at least one target object are combined, and the combined pose features are input into a pre-trained fourth behavior classification network to obtain the fifth probability value of the at least one target object belonging to each of the multi-object behavior labels.

[0046] The fourth and fifth probability values ​​of the at least one target object under the same multi-object behavior label are weighted and summed to obtain the sixth probability value of the target object belonging to each of the multi-object behavior labels.

[0047] Based on the obtained sixth probability values, the multi-object behavior label to which the at least one target object belongs is determined, and the determined multi-object behavior label is used as the target behavior recognition result of the at least one target object.

[0048] In one possible embodiment, the result determination module is specifically used for:

[0049] For the at least one target object, perform the following operations respectively: fuse the behavioral features of one of the at least one target objects with the pose features of the target object to obtain the fused features of the target object;

[0050] Based on the fusion features of each of the at least one target object, the target behavior recognition result of the at least one target object in the video segment is obtained.

[0051] In one possible embodiment, the result determining module is specifically configured to perform at least one of the following:

[0052] The fused features of each of the at least one target object are input into a pre-trained fifth behavior classification network to obtain the single-object behavior recognition results of each of the at least one target object in the video segment.

[0053] The at least one target object includes multiple target objects. The fusion features of the at least one target object are combined, and the combined fusion features are input into a pre-trained sixth behavior classification network to obtain the multi-object behavior recognition result of the at least one target object in the video segment.

[0054] In one possible embodiment, the video segments include multiple segments, and the device further includes an audio / video acquisition module, which is specifically used for:

[0055] After determining the target behavior recognition result of the at least one target object in the video segment based on the respective behavioral and posture characteristics of the at least one target object, the target video segment that meets the preset behavioral conditions is selected from multiple video segments.

[0056] Based on a set of object key points of a preset target object in the target video segment, the motion amplitude information of the target part of the preset target object in the target video segment is determined, wherein the preset target object is one of the at least one target object;

[0057] From a plurality of pre-stored background music tracks, a target background music track is selected whose music rhythm information matches the motion amplitude information.

[0058] The target video segment and the target background music are merged to obtain the target audio-visual content.

[0059] This application provides a computer device, including:

[0060] At least one processor, and

[0061] A memory that is communicatively connected to the at least one processor;

[0062] The memory stores instructions that can be executed by the at least one processor, which implements any of the object behavior recognition methods discussed above by executing the instructions stored in the memory.

[0063] This application provides a computer storage medium storing computer instructions. When the computer instructions are executed on a computer device, the computer device performs any of the object behavior recognition methods discussed above.

[0064] Since the embodiments of this application adopt the above-described technical solution, they have at least the following technical effects:

[0065] In this embodiment, the behavioral features of the target object can be determined based on its image features in a video clip, and its pose features can be determined based on the set of human key points in the video clip. Based on both the behavioral and pose features, the target behavior recognition result is determined. The behavioral features describe the overall image features of the target object in the video clip, representing coarse-grained features, while the pose features describe the features of key parts of the target object in the video clip, representing fine-grained features. Because behavior recognition combines features of different granularities, the behavior recognition result of the target object in the video clip can be obtained more accurately. Furthermore, since the behavioral features of the target object are integrated, the overall motion features of the target object in the video clip are considered, thus avoiding inaccurate behavior recognition due to inaccurate identification of object key points. This allows the object behavior recognition method to accurately identify the behavior of the target object in various scenarios. Attached Figure Description

[0066] Figure 1 This is a schematic diagram illustrating an application scenario of the object behavior recognition method provided in the embodiments of this application;

[0067] Figure 2 A flowchart illustrating the object behavior recognition method provided in this application embodiment. Figure 1 ;

[0068] Figure 3 An example diagram illustrating an object behavior recognition process provided in this application embodiment;

[0069] Figure 4 This is an example diagram illustrating a process for extracting video segments, provided in an embodiment of this application.

[0070] Figure 5 A flowchart illustrating a method for extracting behavioral features of a target object, provided in an embodiment of this application.

[0071] Figure 6 Example diagrams illustrating the process of extracting video feature maps provided in embodiments of this application;

[0072] Figure 7 Example diagram of the process for determining the behavioral characteristics of a target object provided in the embodiments of this application;

[0073] Figure 8 A schematic diagram illustrating the process of swapping the second and third dimensions of a second two-dimensional vector, as provided in an embodiment of this application;

[0074] Figure 9 An example diagram illustrating the process of synthesizing target audio and video provided in this application embodiment;

[0075] Figure 10 Flowchart of the object behavior recognition method provided in the embodiments of this application Figure 2 ;

[0076] Figure 11 Example diagrams illustrating the object behavior recognition process provided in this application embodiment;

[0077] Figure 12 This is a schematic diagram illustrating the interaction between the server and the terminal provided in an embodiment of this application;

[0078] Figure 13 Example diagrams illustrating interface changes of the terminal provided in this application embodiment;

[0079] Figure 14 This is a schematic diagram of the structure of the object behavior recognition device provided in the embodiments of this application;

[0080] Figure 15 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0081] To better understand the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0082] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, the terms involved in this application are introduced below.

[0083] 1. Target Object: In this application, "target object" refers to an object identified from an image. The target object can be a moving target object or a non-moving target object. Moving and non-moving target objects are relative concepts. A moving target object is a target object that can move on its own, such as pedestrians, vehicles, and animals. A non-moving target object is a target object that cannot move on its own, such as a volleyball. Target objects can belong to multiple broad categories, for example, target objects can be people and animals. Target objects can also belong to multiple subcategories within a broad category, for example, target objects can include people A and people B under the category of people.

[0084] 2. Behavioral Features and Image Features: Behavioral features are used to describe the behavioral changes of a target object in continuous video clips. The behavioral features in this application can be determined based on the image features of the target object in the video clips. The image features of the target object are used to describe the image features presented by the target object in the corresponding target area in the image, including one or more of the image's color features, contour features, or shape features.

[0085] 3. Posture Features and Object Keypoints: These describe the posture changes of the target object in the video clip. The posture features in this application can be determined based on the set of object keypoints of the target object in the video clip. Object keypoints are used to indicate the positions of key parts of the target object. For example, when the target object is a human body, the object keypoints are specifically human body keypoints. Human body keypoints refer to the positional markings of the core parts of the human body, including 17 keypoint information such as the head, left shoulder, right shoulder, left elbow, and right elbow.

[0086] 4. Target Behavior Recognition Result: This describes the behavior of a target object in a video clip. The target behavior recognition result can be one or both of the following: single-object behavior recognition result, multi-object behavior recognition result, and multiple-object behavior recognition result. For example, the target behavior recognition result can be represented by a behavior recognition tag. Behavior recognition tags can be divided into single-person behavior recognition tags and multi-person behavior recognition tags based on the number of target objects involved. A single-person behavior recognition tag refers to the type of behavior performed by a single person, while a multi-person behavior recognition result refers to the type of behavior performed by multiple people. Single-person and multi-person behavior recognition tags can contain the same behavior type. Examples of single-person behavior recognition tags include running, walking, and dancing. Examples of multi-person behavior recognition tags include dancing, hugging, and shaking hands.

[0087] 5. Top-down: A method for detecting key points of objects. Its detection idea is: first, detect the target objects in the image. After detecting each target object, based on the location information of the single target object, the key points of the target object are detected.

[0088] 6. Bottom-up keypoint detection: A method for detecting keypoints of objects. The detection idea is to detect all keypoints of objects in the image and then determine the target object corresponding to the keypoint.

[0089] 7. ROI Align: Since the size of each target object in an image is different, when sampling each target object on the feature map corresponding to the image, the feature lengths of the sampled target objects are different. Therefore, ROI Align can perform interpolation on the feature map to ensure that the feature lengths of each target object sampled from the feature map are the same.

[0090] 8. Max-pooling: Selects the maximum value of an image region as the pooled value for that region.

[0091] 9. Beat-synced video: A type of audio-visual video where the video footage matches the rhythm of the music, thereby enhancing the emotions the video needs to express. For example, when a person's body movement reaches its highest point, the music rhythm is also at its highest point.

[0092] 10. Artificial Intelligence (AI): This refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0093] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0094] 11. Computer Vision (CV): Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0095] 12. Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0096] 13. Cloud technology: A general term encompassing network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0097] 14. Cloud computing: A computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0098] It should be noted that in the embodiments of this application, "multiple" means two or more, and "at least one" means one or more.

[0099] Currently, behavior recognition is based on key points of people, which generally can only recognize strongly limb-related actions and cannot recognize some scene-related actions, such as playing beach volleyball, playing volleyball, playing the piano or electronic keyboard. In addition, when there are many people in the video frame, or when people in the video frame are severely occluded, the accuracy of the determined human key points may be poor, resulting in low accuracy of behavior recognition.

[0100] In view of the above, embodiments of this application provide an object behavior recognition method, apparatus, device and medium. The object behavior recognition method involved in the embodiments of this application will be described below.

[0101] In this object behavior recognition method, the behavioral features of the target object in the video clip are extracted based on the image features of the target object in the video clip, which is equivalent to extracting relatively coarse-grained features of the target object. The pose features of the target object in the video clip are extracted based on the set of object key points in the video clip, which is equivalent to extracting relatively fine-grained features of the target object. Combining behavioral and pose features, the behavior of the target object in the video clip is recognized to obtain the behavior recognition result. This object behavior recognition method improves the accuracy of the recognized behavior results by combining features of the target object at different granularity levels, i.e., by performing behavior recognition based on more comprehensive features of the target object.

[0102] The application scenarios of the object behavior recognition method according to the embodiments of this application are described below.

[0103] Please refer to Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application. The schematic diagram includes: a terminal 110, a software program 111 running on the terminal 110, a server 120, and a database 130.

[0104] Server 120 is used to provide corresponding services for software program 111. Software program 111 generally refers to various video applications. Software program 111 includes, for example, software applications pre-installed on terminal 110, web-based software programs, or subprograms (such as applets) pre-installed in third-party applications. This application does not limit this.

[0105] For example, a user can request recommended videos from server 120 via software program 111 in terminal 110. Based on this request, server 120 retrieves the corresponding videos from the videos stored in database 130, extracts video segments from the videos, performs behavior recognition on the video segments, and obtains the target behavior recognition result of the target object. The object behavior recognition process involved will be described below.

[0106] Furthermore, server 120 can select video segments to recommend to the user based on the target behavior recognition results. Server 120 can also select corresponding video segments based on the target behavior recognition results of each video segment, synthesize audio and video, and recommend them to the user.

[0107] In addition, server 120 can perform read and write operations on database 130. For example, server 120 can retrieve the video to be identified or background music from database 130. Database 130 can be set up within server 120 or in a device relatively independent of server 120. For example, database 130 can be implemented using a separate server cluster, which is equivalent to forming a storage system.

[0108] In one possible application scenario, the aforementioned server 120 can be applied to various specific application scenarios. For example, server 120 can be applied to video recommendation scenarios. Server 120 can also be applied to virtual reality, augmented reality, autonomous driving, smart furniture, smart office, smart wearables, smart transportation, smart cities, drones, robots, and other related application scenarios. This application does not limit the specific use cases of server 120.

[0109] In one possible application scenario, server 120 can be implemented using multiple servers deployed in various locations. These servers can achieve load balancing and reduce communication latency across different regions. The multiple servers share data via blockchain, essentially forming a data-sharing system. For example, terminal 110 located at location a communicates with the server, while terminal 110 located at location b communicates with other servers.

[0110] Each server in the data sharing system has a corresponding node identifier. Each server can also store node identifiers of other servers in the system, enabling subsequent broadcasting of generated blocks to other servers based on their node identifiers. Each server can maintain a node identifier list as shown in the table below, storing the server name and node identifier accordingly. The node identifier can be an Internet Protocol (IP) address or any other information that can be used to identify the node; Table 1 uses IP addresses as an example.

[0111] Table 1

[0112] Server Name Node identifier Node 1 119.115.151.174 Node 2 118.116.189.145 … … Node N 119.124.789.258

[0113] The terminal 110 mentioned above refers to an electronic device used by the user. This electronic device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, gaming device, smart TV, smart in-vehicle device, or smart wearable device, etc., possessing a certain computing power and running instant messaging software and websites or social networking software and websites. The server 120 mentioned above can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Server 120 can be implemented using cloud technology.

[0114] In another possible application scenario, as illustrated in this scenario, the object behavior recognition method involved in the embodiments of this application can be executed by a terminal.

[0115] For example, the terminal can obtain video clips based on user input, or extract video clips from pre-stored videos, and perform behavior recognition on target objects in the video clips.

[0116] Alternatively, for example, a small tool can be installed on the terminal to perform behavior recognition on target objects in video clips. The way the tool acquires video clips can be the same as how the terminal acquires video clips, which will not be elaborated here.

[0117] It should be noted that this tool is different from the software program 111 mentioned earlier. This tool can run in the terminal and does not require server assistance to achieve the corresponding functions.

[0118] The object behavior recognition method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0119] based on Figure 1 The application scenarios discussed in this application demonstrate that the object behavior recognition method involved in the embodiments can be executed by a server, a terminal, or a combination of both. The following examples illustrate this. Figure 1 Taking the method of object behavior recognition executed by the server as an example, combined with Figure 2 The flowchart shown illustrates the object behavior recognition method, which introduces the object behavior recognition method involved in the embodiments of this application.

[0120] S21, extract video segments from the video to be identified, the video segments including at least one video frame.

[0121] The server can retrieve relevant videos from a database or network resources, and each retrieved video can be considered as a video to be identified. Alternatively, the server can use videos previously watched by the user as videos to be identified. Or, the server can use videos with a similarity greater than or equal to a similarity threshold with videos previously watched by the user as videos to be identified. Video similarity can be obtained by calculating the similarity between video features or by calculating the similarity between video tags. The types of videos to be identified include, for example, TV series, movies, variety shows, or short videos, etc., and this application does not impose any restrictions on this.

[0122] The server can segment the video to be recognized, thereby obtaining video segments. A video segment is actually a part of the video to be recognized and may include at least one video frame from that video. To ensure better subsequent recognition of the target object's behavior, the video segment may include multiple consecutive video frames from the video to be recognized. This facilitates the server's subsequent capture of the target object's continuous behavior within the video segment, improving the accuracy of target object recognition. Furthermore, the server can segment the video to be recognized into one or more video segments. When there are multiple video segments, adjacent video segments may overlap, and the server can perform behavior recognition (described below) on each of the multiple video segments separately.

[0123] For example, please refer to Figure 3 This is an example diagram illustrating the process of object behavior recognition. The server obtains, for example, Figure 3 The video to be identified, shown in figure a, includes the video to be identified 300. Figure 3 The server can extract video frames 1, 2, 3, 4, and 5 from the video to be identified (300) shown in image a. Figure 3 The video clip shown in b includes video frame 1 and video frame 2 from the video to be identified.

[0124] S22, extract at least one behavioral feature of each target object from the video clip, wherein the behavioral feature is determined based on the image features of the corresponding target object in the video clip.

[0125] The server can extract image features of at least one target object from each video frame in the video segment, and obtain behavioral features of at least one target object based on the image features of the target object. For example, convolutional pooling operation can be performed on the image features of a target object to obtain the behavioral features of the target object.

[0126] It should be noted that if at least one target object includes one target object, then the behavioral characteristics of each of the at least one target object are the behavioral characteristics of that target object. For example, if at least one target object includes person A, then the behavioral characteristics of each of the at least one target object are the behavioral characteristics of person A. If at least one target object includes at least two target objects, then the behavioral characteristics of each of the at least two target objects represent the behavioral characteristics of each of the at least two target objects. For example, if at least one target object includes person A and person C, then the behavioral characteristics of each of the at least one target object are the behavioral characteristics of person A and the behavioral characteristics of person C.

[0127] As one embodiment, when at least one target object is multiple target objects, the at least one target object may include multiple subcategories under the same major category. For example, at least one target object may include Person A, Person B, and Person C. At least one target object may also include different major categories. For example, at least one target object may include Animals and People. At least one target object may also include subcategories under different major categories. For example, at least one target object may include Person A, Cat, and Volleyball, etc.

[0128] For example, continue to use Figure 3 For example, the server can detect various target objects in a video clip and obtain, such as Figure 3 The example diagram shown in c is... Figure 3 The c in the video frame includes the image region 310 of the target object in video frame 1 and the image region 320 of the target object in video frame 2. The server can extract the behavioral features of the target object based on the image regions of the target object in each video frame of the video segment.

[0129] S23, extract the pose features of at least one target object from the video clip, wherein the pose features are determined based on the set of object key points of the corresponding object in the video clip.

[0130] Similarly, the server can extract the key points of the target object in each video frame from the video clip, combine the key points of the target object in each video frame, and thus obtain the set of key points of the target object. By analogy, the server can obtain the set of key points of at least one target object.

[0131] The server can perform one or both of convolution and pooling operations on the set of object key points of a target object to obtain the pose features of the target object. The server can also determine the corresponding pose features of the target object by matching the set of object key points of the target object with the pre-stored set of object key points associated with each pose. In this way, the server can determine the pose features of at least one target object.

[0132] It should be noted that if at least one target object includes one target object, then the pose features of each of the at least one target object are the pose features of that target object. For example, if at least one target object includes person A, then the pose features of each of the at least one target object are the pose features of person A. If at least one target object includes at least two target objects, then the pose features of each of the at least two target objects represent the pose features of each of the at least two target objects. For example, if at least one target object includes animal D and animal F, then the pose features of each of the at least one target object are the pose features of animal D and the pose features of animal F.

[0133] For example, continue to refer to Figure 3 For example, the server can detect key points of the target object in each video frame and obtain, such as Figure 3 The example diagram shown in Figure d includes: object keypoints 330 of the target object in video frame 1, and object keypoints 340 of the target object in video frame 2. The server can obtain the pose features of the target object based on the object keypoints 330 in video frame 1 and the object keypoints 340 in video frame 2.

[0134] It should be noted that the order of steps S22 and S23 can be arbitrary, and this application does not impose any restrictions on this.

[0135] S24, based on the behavioral and posture features of at least one target object, determine the target behavior recognition result of at least one target object in the video segment.

[0136] After obtaining the behavioral and pose characteristics of at least one target object, the server can determine the single-object behavior recognition result based on these characteristics. The meaning of the single-object behavior recognition result is explained above and will not be repeated here. The server can also determine the multi-object behavior recognition result for at least one target object based on its behavioral and pose characteristics. The meaning of the multi-object behavior recognition result is explained above and will not be repeated here. The server can also determine the single-object behavior recognition result for a single target object, as well as the multi-object behavior recognition result for at least one target object.

[0137] For example, continue to use Figure 3 For example, the server can determine the target object's behavior in a video clip as walking based on the target object's behavioral and posture characteristics.

[0138] In this embodiment, the server can extract the behavioral features of the target object in the video clip based on its image features, which is equivalent to extracting the overall motion features of the target object in the video clip. The server can also extract the pose features of the target object in the video clip based on the set of key points of the target object, which is equivalent to extracting the location points of the target object in the video clip. By combining the behavioral and pose features, the server can identify the behavior of the target object in the video clip. Because it combines the overall motion features and location points of the target object in the video clip for behavior recognition, it can more accurately identify the behavior of the target object. After accurately identifying the behavior type of the target object in the video clip, it is equivalent to more accurately determining the tag of the video clip, so that subsequent editing processing of the video clip can be performed based on the tag to enrich the video material.

[0139] As one embodiment, S21 is implemented as follows:

[0140] Implementation Method 1:

[0141] Randomly extract video segments of a preset duration from the video to be identified.

[0142] Implementation Method Two:

[0143] The video to be identified is decoded to obtain multiple video frames;

[0144] Extract video segments from multiple video frames along a time window.

[0145] The server can decode the video to be recognized, obtaining multiple video frames, which can also be called a video frame sequence. The server can randomly sample at least one video frame from the video frame sequence as a video segment. Alternatively, the server can slide a time window along the sequentially arranged video frame sequence according to a preset step size. Each slide of the time window corresponds to obtaining a subsequence of video frames, which can be regarded as a video segment. The length of the time window can be represented by duration or frame count, and the length of the time window can be less than or equal to the total number of frames in the video. The preset step size is less than the total number of frames in the video to be recognized.

[0146] As one embodiment, the preset step size can be smaller than the length of the time window. Since the preset step size is smaller than the length of the time window, two adjacent slides of the time window can extract video segments with overlapping video frames, resulting in a certain degree of redundancy in the extracted video segments. This facilitates a more complete and accurate identification of the behavior of a target object in each video segment.

[0147] For example, please refer to Figure 4 This is an example diagram illustrating a process for extracting video segments according to an embodiment of this application. The server decodes the video to be identified to obtain... Figure 4 The multiple video frames 410 shown are specifically as follows: Figure 4 The system contains multiple video frames from video frame 1 to video frame 16. The time window 420 has a length of 6 video frames and a preset step size s of 2 video frames. The server slides the time window 420 along the multiple video frames 410 according to the preset step size s. For example, the server divides the first video segment from the multiple video frames 410 according to the time window 420, which includes each video frame from video frame 1 to video frame 6. The server slides the time window 420 along the time axis of the video to be identified for two video frames to obtain the second video segment, which includes each video frame from video frame 3 to video frame 8.

[0148] After obtaining the video clip, the server can execute S22, which is described below. Figure 5 The flowchart shown illustrates the method for extracting behavioral features of at least one target object, and provides an example of how S22 is implemented.

[0149] S51, extract at least one image feature of a target object from each video frame in the video segment to obtain the video feature map corresponding to the video segment.

[0150] The server can extract features from each video frame to obtain a video feature map of the video segment. Feature extraction can be performed using one or more operations such as convolution and pooling. This video feature map is essentially a fusion of image information from each video frame in the video segment.

[0151] The server can extract video feature maps from video clips using a pre-trained feature extraction network. This pre-trained network is trained based on sample video feature maps of the video clips. During training, the network is emphasized for learning the image features of the target object. Therefore, when a video clip is input into the pre-trained network, it focuses on the image features of the target object within the clip, thus extracting the video feature map of the target object. For example, the server can input the color information of each video frame into the pre-trained network to extract the video feature map from the color information. Color information could be, for example, Red, Green, and Blue (RGB) information.

[0152] Please refer to Figure 6 This is an example diagram illustrating the process of extracting video feature maps according to an embodiment of this application. The video segment includes, for example,... Figure 6 As shown in video frame 610 (a), the server obtains the RGB information corresponding to each pixel in the video frame. The specific details of each pixel in the video frame are as follows: Figure 6 As shown in b, the server can focus on extracting from the RGB information. Figure 6 Image features in image region 620 where the target object is located, as shown in b, are obtained. Figure 6 The video feature map shown in C is 630. Figure 6 The example uses a video clip consisting of one video frame, but there is no actual limit to the number of video frames a video clip can contain.

[0153] S52, detect the relative position of at least one target object in any video frame of the video segment.

[0154] The server can randomly select a video frame from a video segment, or it can select the video frame containing the most target objects. This selected frame is then used as any video frame, and target object detection is performed on that frame to obtain the relative position of at least one target object within that video frame. The relative position can be represented, for example, by a detection bounding box containing the target object within that video frame.

[0155] It should be noted that if at least one target object includes one target object, then the relative position of each of the at least one target object in any video frame is the relative position of that target object in any video frame; if at least one target object includes at least two target objects, then the relative position of each of the at least one target object in any video frame is the relative position of each of the at least two target objects in any video frame.

[0156] For example, continue to use Figure 6 For example, the server can perform object detection on video frame 610 to obtain, for instance, the following: Figure 6 The detection results of d include the detection boxes of each target object in the video frame, specifically including the first detection box 641 of person A in the video frame, the second detection box 642 of person B in the video frame, and the third detection box 643 of person C in the video frame.

[0157] S53, based on the relative position of each of the at least one target object in any video frame, determine the behavioral features of each of the at least one target object from the video feature map.

[0158] Regardless of the feature extraction method used, there is a certain correspondence between the size of video frames and the size of the video feature map. Therefore, after determining the relative position of the target object in the video frame, the server can identify the feature region matching that relative position from the video feature map. The identified feature region is the feature region of the target object in the video feature map. The server can then obtain the behavioral characteristics of the target object based on its corresponding feature region. Similarly, the server can obtain the behavioral characteristics of at least one target object.

[0159] For example, continue to use Figure 6 For example, after the server detects the bounding boxes of at least one target object in a video frame, it can determine the feature regions of each target object in the video feature map 630 based on each bounding box, specifically including... Figure 6 The feature regions of characters A, B, and C shown in f are represented by the rectangles containing A, B, and C, respectively.

[0160] In this embodiment, the server can extract the overall image features of a video segment to obtain a video feature map. Based on the relative position of each target object in a video frame, the server can determine the corresponding behavioral features of each target object from the video feature map. On the one hand, the extracted behavioral features are actually fused with the image information of each video frame in the video segment, so as to obtain more comprehensive and accurate behavioral features of the target objects. On the other hand, it is not necessary to perform feature extraction processing on each target object separately based on the video segment, so as to obtain the behavioral features of each target object at one time, which can improve the efficiency of obtaining the behavioral features of each target object.

[0161] In S53, when determining the corresponding feature region of the target object from the video feature map based on the relative position of the target object in the video frame, a decimal-type feature region may be determined. If the decimal-type feature region is directly quantized to obtain an integer-type feature region, the actual behavioral features corresponding to the target object will deviate significantly from the quantized feature region, thereby reducing the accuracy of the behavioral features of the target object. Therefore, in this embodiment, the server can use ROI align to determine the corresponding behavioral features of the target object from the video feature map based on the relative position of the target object in the video frame.

[0162] Specifically, after determining the feature region corresponding to the preset position of the target object, the server divides the feature region into multiple candidate units. An interpolation method is then used to determine the coordinate values ​​at the preset positions within each candidate unit. Max pooling is performed on the coordinate values ​​in the candidate units to obtain the pooling result. This process is repeated to obtain the pooling results for each candidate unit, and the pooling results of all candidate units are combined to obtain a combined pooling result. Based on the combined pooling result, the behavioral characteristics of the target object are obtained; for example, the combined pooling result can be used as the behavioral characteristics of the target object. The interpolation method, for example, uses bilinear interpolation. Each candidate unit can have one or more preset positions.

[0163] For example, please refer to Figure 7 This is an example diagram illustrating the process of determining the behavioral characteristics of a target object according to an embodiment of this application. From the video feature map 710, the corresponding feature region 720 of the target object is determined. The server can divide the feature region 720 into four candidate units and determine the coordinate values ​​at preset positions in each candidate unit. For example, the server determines the coordinate values ​​of each candidate unit, specifically as follows: Figure 7 S1, S2, S3, and S4 represent the four coordinate values. The server performs max pooling on each coordinate value (the four coordinate values ​​corresponding to S1, S2, S3, and S4) in the first candidate unit, and combines the pooling results of the four candidate units to obtain the following... Figure 7 The behavioral characteristic shown is 730.

[0164] In this embodiment, an interpolation method is used to obtain the integer type feature region of the target object. This avoids the deviation caused by directly quantizing the decimal type feature region, thereby improving the accuracy of the determined feature region of the target object and thus improving the accuracy of the behavioral features of the target object. Furthermore, the feature region extracted from the video feature map is processed into a fixed-size feature region so as to obtain behavioral features of the same size for each target object.

[0165] In addition to obtaining the behavioral characteristics of the target object, the server also needs to obtain the pose characteristics of the target object. The following is an example of how to extract the pose characteristics of the target object in S23.

[0166] S1.1 Extract at least one set of object key points for each target object from each video frame in the video segment.

[0167] There are several ways to detect the set of key points of an object in S1.1. Examples are given below:

[0168] Method 1: The server can use a top-down approach to extract key points of the target object from video frames.

[0169] The server can perform object detection on video frames, obtain the detection box of the target object in the video frame, and identify the key points of the target object from the obtained detection box.

[0170] In this method, key points of the object are extracted from the detection box. This method of extracting key points of the object is more intuitive and has relatively higher accuracy.

[0171] Method 2: The server can use bottom-up to extract key points of each object from the video frames.

[0172] Perform object keypoint detection on video frames to obtain at least one object keypoint in the video frame, and determine the target object to which the at least one object keypoint belongs, thereby obtaining at least one object keypoint for each of the at least one target object.

[0173] In this second method, the server can extract the key points of each object in the video frame at once, and then determine the target object to which the key points belong. This method is more efficient and has better real-time performance in determining the corresponding key points of the target object.

[0174] S1.2, based on the object key point set of at least one target object, obtain the pose features of at least one target object.

[0175] The server can extract keypoints of a target object in each video frame. These keypoints can be understood as the coordinates of key parts of the target object within the video frame. By combining these keypoints from different video frames, the server obtains a set of keypoints for the target object. Based on this set of keypoints, the server can then obtain the pose features of the target object. For example, the server can perform multiple convolution operations on the keypoint set to obtain its pose features, or it can sequentially perform multiple convolution and max-pooling operations on the keypoint set to obtain its pose features. This process can be repeated to obtain the pose features corresponding to each target object.

[0176] The keypoint set of the target object actually contains information in three dimensions: one dimension is the temporal sequence of video frames in the video clip, and the other two dimensions are the positional information of the target object's keypoints within the video frames. When performing a convolution operation on the target object's keypoint set, due to the limited scope of the convolution kernel, the obtained pose features may only learn the local common features between two adjacent dimensions, that is, they do not learn the global common features between the three dimensions. Therefore, in this embodiment, the keypoint set is transformed into a three-dimensional vector, and then a convolution operation is performed on the three-dimensional vector. During the convolution operation, the positions of the second and third dimensions of the three-dimensional vector can be swapped. This allows the convolution operation to learn the correlation between any two dimensions of the three-dimensional vector, which is equivalent to learning the global common features in the target object's keypoint set, thereby obtaining more accurate pose features.

[0177] Specifically, the server can convert the set of key points of the target object into a three-dimensional vector, which is referred to as the first three-dimensional vector for ease of description. The first dimension of the first three-dimensional vector is used to represent the duration of the video segment, the second dimension is used to represent the number of key points in the set of key points of the target object, and the third dimension is used to represent the number of channels.

[0178] The server can perform one or more convolution operations on the first 3D vector to obtain the second 2D vector. During this process, the server learns the correlation between the first and second dimensions, and the correlation between the second and third dimensions. The server then swaps the relative positions of the second and third dimensions within the second 3D vector to obtain the third 3D vector. Performing one or more convolution operations on this third 3D vector further learns the correlation between the first and original 3D vectors, and the correlation between the original 2D vectors, thus fusing more comprehensive information and obtaining more accurate pose features of the target object. By analogy, the server can obtain the pose features of each individual target object within at least one target object.

[0179] For example, please refer to Figure 8 This is a schematic diagram illustrating a process of swapping the second and third dimensions of a second two-dimensional vector, provided in an embodiment of this application. In this schematic diagram, the server obtains as follows: Figure 8 The second three-dimensional vector shown in Figure 'a' can be represented as: (T, number of object keypoints, number of channels), where T represents the duration of the video segment. The server swaps the positions of the second and third dimensions in the second three-dimensional vector to obtain, as shown in Figure 'a'. Figure 8 The third three-dimensional vector shown in b can be represented as: (T, number of channels, number of object key points).

[0180] After obtaining the behavioral and posture features of at least one target object, the server can execute S24, that is, determine the target behavior recognition result of at least one target object based on the behavioral and posture features of at least one target object.

[0181] Since a video clip contains one or more target objects, the server can identify one or both of the following when determining the target behavior recognition results: single-object behavior recognition result or multi-object behavior recognition result. The methods by which the server determines single-object behavior recognition results and multi-object behavior recognition results are described below:

[0182] I. Methods for determining the results of single-object behavior recognition.

[0183] Example 1.

[0184] S2.1 Input the behavioral features of a single target object into the pre-trained first behavior classification network to obtain the first probability value of the target object belonging to each single object behavior label;

[0185] S2.2, the server inputs the pose features of the target object into the pre-trained second behavior classification network to obtain the second probability value of the target object belonging to each single object behavior label;

[0186] S2.3, Based on each first probability value and each second probability value, obtain the third probability value of the target object belonging to each single object behavior label;

[0187] S2.4, Based on the obtained third probability values, determine the single-object behavior label to which the target object belongs. The single-object behavior label to which the target object belongs is the target behavior identification result of the target object.

[0188] In this context, the pre-trained first behavior classification network and the second behavior classification network refer to classification networks trained based on corresponding training data. The server can obtain pre-trained first behavior classification networks and second behavior classification networks from other devices, or it can train them itself; this application does not impose any restrictions on this. For example, the pre-trained first behavior classification network can be trained based on multiple first training data sets, each of which includes sample behavior features of the sample object and the single-object behavior label to which the sample object belongs; similarly, the pre-trained second behavior classification network can be trained based on multiple second training data sets, each of which includes sample behavior features of the sample object and the single-object behavior label to which the sample object belongs. It should be noted that the single-object behavior labels that the first behavior classification network can classify are the same as those that the second behavior classification network can classify.

[0189] In practical implementation, the server can use a first-behavior classification network to classify the behavioral features of the target object, thereby obtaining the first probability value of the target object belonging to each individual object behavior label. For example, the first probability values ​​of person A belonging to each individual object behavior label obtained by the server are shown in Table 2 below.

[0190] Table 2

[0191] Single object behavior label First probability value walk 0.3 run 0.4 Dance 0.3

[0192] As can be seen from Table 2 above, based on the posture characteristics of person A, the first probability values ​​for determining whether person A is walking, running, or dancing in the video clip are 0.3, 0.4, and 0.3, respectively.

[0193] Similarly, the server can classify the pose features of the target object to obtain the second probability value of the target object belonging to each single object behavior label. For example, the first probability value of the target object belonging to each single object behavior label obtained by the server is shown in Table 3 below.

[0194] Table 3

[0195] Single object behavior label Second probability value walk 0.1 run 0.6 Dance 0.3

[0196] As can be seen from Table 3 above, based on the posture characteristics of person A, the second probability values ​​for determining whether person A belongs to walking, running, and dancing in the video clip are 0.1, 0.6, and 0.3, respectively.

[0197] After obtaining the first and second probability values, the server can perform a weighted summation of the first and second probability values ​​belonging to the same single-object behavior tag to obtain a third probability value indicating that the target object belongs to that single-object behavior tag. The single-object behavior tag with the highest third probability value can be determined as the single-object behavior tag to which the target object belongs. When performing a weighted summation of the first and second probability values ​​under the same single-object behavior tag, the weights of the first and second probability values ​​can be the same, for example, both 0.5.

[0198] Continuing with the examples shown in Tables 2 and 3, the server determines the third probability value of each single object behavior tag as shown in Table 4 below.

[0199] Table 4

[0200] Single object behavior label Third probability value walk (0.1+0.3)*0.5＝0.2 run (0.6+0.4)*0.5＝0.5 Dance (0.3+0.3)*0.5＝0.3

[0201] As can be seen from Table 4 above, the third probability values ​​for person A belonging to walking, running, and dancing are 0.2, 0.5, and 0.3, respectively. The server determines that person A belongs to running with the highest third probability value. Therefore, it can be determined that the single object behavior label of person A in the video clip is running.

[0202] Similarly, when there are multiple targets, the server can determine the target behavior recognition result for each target target by following the above process.

[0203] In the above-described manner, this application provides a scheme for determining the target behavior recognition result of a target object. Furthermore, when the server determines the behavior recognition result of the target object, it combines the behavior recognition result determined based on the pose features of the target object with the behavior recognition result determined based on the behavior characteristics of the target object. This is equivalent to integrating the behavior characteristics and pose features of the target object, thus accurately determining the target behavior recognition result. Moreover, since the target behavior recognition result of the target object can be flexibly determined from its possible behavior recognition results, the flexibility in determining the target behavior recognition result is improved.

[0204] Example 2.

[0205] S3.1, The server fuses the behavioral features and pose features of the target object to obtain the fused features of the target object;

[0206] S3.2 The server inputs the fused features of the target object into the pre-trained fifth behavior classification network to obtain the single-object behavior recognition result of the target object in the video segment.

[0207] The method by which the server obtains the pre-trained fifth-row classification network can be referred to the content discussed above, and will not be repeated here. For example, the fifth-row classification network is trained based on multiple third training data. Each third training data includes the sample fusion features of the sample object and the single object behavior label to which the sample object belongs.

[0208] In practical implementation, the server can fuse the behavioral and pose features of a target object to obtain a fused feature. This fused feature essentially describes the target object's characteristics in multiple aspects of behavior and pose. The fusion method can be, for example, a weighted sum of behavioral and pose features, or a combination of behavioral and pose features. This application does not impose specific limitations on the fusion method. The server can then use a pre-trained fifth behavior classification network to classify this fused feature, thereby obtaining the single-object behavior recognition result corresponding to the target object in the video segment.

[0209] This method provides a way to determine the target behavior recognition result of a target object. In this method, the server can first fuse the features of the target object in various aspects, and then determine the target behavior recognition result of the target object based on the fused features. Since the pose features and behavior features of the target object are fused, the target behavior recognition result of the target object can be more accurately identified.

[0210] II. Examples of methods for determining the results of multi-object behavior recognition are as follows.

[0211] The first type.

[0212] S4.1 The server combines the behavioral features of multiple target objects and inputs the combined behavioral features into a pre-trained third behavior classification network to obtain the fourth probability value of the multiple target objects belonging to each multi-object behavior label.

[0213] S4.2, the server combines the pose features of multiple target objects and inputs the combined pose features into the pre-trained fourth behavior classification network to obtain the fifth probability value of the multiple target objects belonging to each multi-object behavior label;

[0214] S4.3 The server performs a weighted summation of the fourth and fifth probability values ​​of multiple target objects under the same multi-object behavior label to obtain the sixth probability value of each target object belonging to each multi-object behavior label.

[0215] S4.4, the server determines the multi-object behavior labels to which multiple target objects belong based on the obtained sixth probability values.

[0216] In practical implementation, the methods for obtaining the pre-trained third-behavior classification network and the pre-trained fourth-behavior classification network can refer to the method for obtaining the pre-trained first-behavior classification network described above, and will not be repeated here. For example, the third-behavior classification network can be obtained through training on multiple third-behavior training datasets. Each third-behavior training dataset includes combined sample behavior features and corresponding multi-object behavior labels. The combined behavior features are the result of combining the sample behavior features of multiple sample objects in the video clip, and the corresponding multi-object behavior labels are the multi-object behavior labels to which the multiple sample objects belong. The method for obtaining the sample behavior features can refer to the method for obtaining behavior features discussed above, and will not be repeated here.

[0217] Similarly, a fourth-order classification network can be obtained by training on multiple fourth training datasets. Each fourth training dataset includes combined sample pose features and corresponding multi-object behavior labels. The combined pose features are the result of combining the pose features of multiple sample objects in the video clip, and the corresponding multi-object behavior labels are the multi-object behavior labels to which the multiple sample objects belong. The method for obtaining sample pose features can refer to the method for obtaining pose features discussed above, and will not be repeated here.

[0218] After obtaining the behavioral features of at least one target object, the server can combine these features. For example, the server can use the behavioral features of each target object as rows in a matrix, and so on, to obtain a behavioral feature matrix for each target object. The server then uses a third behavioral classification network to output a fourth probability value indicating whether the multiple target objects belong to various multi-object behavioral labels, based on the combined behavioral features. For example, the specific fourth probability values ​​of the multiple target objects belonging to various multi-object behavioral labels obtained by the server are shown in Table 5 below.

[0219] Table 5

[0220] Multi-object behavior tags Fourth probability value Dance 0.3 Embrace 0.2 chorus 0.5

[0221] As can be seen from Table 5 above, the probabilities of multiple target objects belonging to the fourth category of dancing, hugging, and singing together are 0.3, 0.2, and 0.5, respectively.

[0222] Similarly, the server can combine the pose features of at least one target object to obtain the combined pose features. The combination method can be referred to the previous discussion and will not be repeated here. After obtaining the combined pose features, the server inputs the combined pose features into the fourth-order classification network to obtain the fifth probability value of at least one target object belonging to each multi-object behavior label. For example, the fifth probability values ​​of multiple target objects belonging to each multi-object behavior label obtained by the server are shown in Table 6 below.

[0223] Table 6

[0224] Multi-object behavior tags Fourth probability value Dance 0.2 Embrace 0.7 chorus 0.1

[0225] As can be seen from Table 6 above, the probability values ​​for the fifth category of dancing, hugging, and singing together for multiple target objects are 0.2, 0.7, and 0.1, respectively.

[0226] After obtaining the fourth and fifth probability values, the server can perform a weighted summation of the fourth and fifth probability values ​​under the same multi-object behavior label to obtain the sixth probability value under that multi-object behavior label. By analogy, the server can obtain the seventh probability value of multiple target objects belonging to each multi-object behavior label. The server can determine the multi-object behavior label with the largest seventh probability value corresponding to multiple target objects as the target behavior recognition result of multiple target objects.

[0227] For example, following the examples shown in Tables 5 and 6, the server can perform a weighted summation of the fourth and fifth probability values ​​under the same multi-object behavior label to obtain the various sixth probability values ​​shown in Table 7 below.

[0228] Table 7

[0229] Multi-object behavior tags Fourth probability value Dance (0.3+0.2)*0.5＝0.25 Embrace (0.5+0.7)*0.5＝0.6 chorus (0.2+0.1)*0.5＝0.15

[0230] As can be seen from Table 7 above, the probability values ​​of the multi-target objects belonging to the sixth category of dancing, hugging, and singing in a chorus are 0.25, 0.6, and 0.15, respectively.

[0231] The second type.

[0232] S5.1 The server fuses the behavioral features of each target object among multiple target objects with the pose features of the corresponding target objects to obtain the fused features of each of the multiple target objects.

[0233] S5.2 The server combines the fusion features of multiple target objects and inputs the combined fusion features into the pre-trained sixth-line classification network to obtain the multi-object behavior recognition results of multiple target objects in the video segment.

[0234] The server can fuse the behavioral features and pose features of a target object. The fusion method can be referred to the previous discussion and will not be repeated here. Similarly, the server can obtain the fused features of multiple target objects. After obtaining the fused features of multiple target objects, the server can combine the fused features of multiple target objects. The combination method can be referred to the previous discussion and will not be repeated here, thus obtaining the combined fused features. The server inputs the combined fused features into a pre-trained sixth-order behavior classification network to obtain the eighth probability value of multiple target objects belonging to each multi-object behavior label. Based on each eighth probability value, the server determines the multi-object behavior label with the highest eighth probability value as the target behavior recognition result of the multiple target objects.

[0235] The server can extract multiple video segments from the video to be identified and perform the above process on each video segment to determine the target behavior recognition result of at least one target object in each video segment. The server can then select video segments that meet preset behavior conditions from these multiple video segments as target video segments. The preset behavior conditions can be flexibly set according to needs. For example, video segments with preset behaviors, such as chorus, or video segments with preset target objects. The preset target objects are any one or more target objects from at least one target object.

[0236] The server can push target video clips to users, or edit various target video clips and push the edited results to users. To improve the playback effect of target video clips, the server can also match appropriate target background music to the target video clips, thereby obtaining target audio and video.

[0237] One way to obtain the target audio and video is:

[0238] The server can match a target video clip with a corresponding first target background music, and synthesize the video clip and the first target background music to obtain the first target audio and video.

[0239] For example, the server can determine the first motion amplitude information corresponding to the target part of the preset target object based on a set of object key points in the target video segment. The first motion amplitude information is used to represent the motion amplitude of the target part in the video segment. For example, the first motion amplitude information includes the motion amplitude value of the target part between two adjacent frames. The server can match the music rhythm information with the first motion amplitude information from multiple pre-stored background music tracks. The server synthesizes the target video segment and the first target background music to obtain the first target audio-visual video, which can be regarded as a beat-synced video. This method can be applied to process a single video segment to synthesize a rhythmic first target audio-visual video.

[0240] For example, please refer to Figure 9 This is an example diagram illustrating a process for synthesizing target audio and video according to an embodiment of this application. The target video segment determined by the server specifically includes, for example: Figure 9 The first video frame shown in Figure a and Figure 9 In the second video frame shown in b, the server can determine the first motion amplitude information 920 of the preset target object 910 based on the object key point s1 of the palm of the preset target object 910 in the first video frame and the object key point s2 of the palm of the preset target object 910 in the second video frame, specifically as follows: Figure 9 The straight line between s1 and s2 in c, and another Figure 9 The first video frame shown in Figure a is... Figure 9 The time interval between the second video frames shown in b can be represented as t1. The server can filter out the first target background music based on this first motion amplitude information 920. The music rhythm information of this target background music can be specifically as follows: Figure 9 As shown in Figure d, it can be seen that the rhythm information of the background music of the first target matches the first motion amplitude information 920, and the duration of the background music of the first target, as well as Figure 9 The first video frame shown in Figure a is... Figure 9 The time interval between the second video frames shown in b is t1. Therefore, the server can directly synthesize the target video segment and the first target background music to obtain the first target audio and video.

[0241] One way to obtain the target audio and video is:

[0242] The server can filter out target video frames with preset target objects from various target video segments. Based on the preset target objects, it determines the second motion amplitude information of the preset target objects in the set of object key points of each of the filtered target video frames, and matches the second target background music corresponding to the second motion amplitude information to obtain the second target audio and video.

[0243] In practical implementation, the server may filter out multiple target video segments, and each target video segment may contain a preset target object. Therefore, in this embodiment, the server can filter out each target video frame containing the preset target object from multiple target video segments, which is equivalent to obtaining a more complete activity trajectory of the preset target object. The server determines the second motion amplitude information of the target part of the preset target object in each target video frame. The meaning of the second motion amplitude information can be referred to the previous discussion and will not be repeated here. The server can match the music rhythm information with the second motion amplitude information from multiple pre-stored background music tracks to obtain a second target background music. The server synthesizes each target video frame with the second target background music to obtain a second target audio-visual video, which can be regarded as a beat-matched video. This method can obtain a more complete audio-visual video of the preset target object.

[0244] The object behavior recognition method involved in this application embodiment can be executed by a server, a terminal, or a combination of both. The following example uses a terminal executing the object behavior recognition method, with a person as the target object. Figure 10 The flowchart of the object behavior recognition method is provided to specifically describe the object behavior recognition method in the embodiments of this application:

[0245] S101 decodes the video to be identified, obtaining multiple video frames.

[0246] The method by which the terminal decodes the video to be identified can be referred to the previous discussion, and will not be repeated here. The video to be identified can be, for example, a video in the terminal's photo album, a video input by the user into the terminal, or a video that the user has previously watched on the terminal, etc.

[0247] S102 divides multiple video frames into time windows to obtain video segments.

[0248] The meaning of time windows and the methods for acquiring video clips can be found in the previous discussion, and will not be repeated here.

[0249] S103, Based on the RGB information of the video clip, extract the behavioral features of at least one person in the video clip.

[0250] The meaning of the behavioral characteristics of at least one individual can be referred to the meaning of the behavioral characteristics of at least one target object in the previous section, and will not be repeated here. The methods for extracting behavioral characteristics based on RGB information can be referred to the content discussed earlier, and will not be repeated here. Other methods for obtaining behavioral characteristics can be referred to the content discussed earlier, and will not be repeated here.

[0251] S104, Based on the human body key points of the video clip, extract the posture features of at least one person in the video clip.

[0252] The meaning of the posture features of at least one individual can be referred to the meaning of the posture features of at least one target object mentioned earlier, and will not be repeated here. The method by which the terminal determines the posture features can be referred to the content discussed earlier, and will not be repeated here.

[0253] S105, based on the behavioral characteristics of at least one person and the posture characteristics of at least one person, obtain the target behavior recognition result of at least one person in the video segment.

[0254] The method by which the terminal determines the target behavior recognition result can be referred to the content discussed above, and will not be repeated here.

[0255] S106, determine the target video segment that meets the preset behavioral conditions from multiple video segments in the video to be identified.

[0256] The meaning of pre-set behavioral conditions can be referred to the content discussed above, and will not be repeated here.

[0257] S107, Based on the set of human body key points of the preset target object in the target video segment, determine the target part of the preset target object and the motion amplitude information in the target video segment.

[0258] The meaning of motion amplitude information and the methods for determining motion amplitude information can be found in the previous discussion, and will not be repeated here.

[0259] S108 matches the target background music from multiple background music tracks, ensuring that the music rhythm information matches the motion amplitude information.

[0260] The method for determining the target background music can be referred to the previous discussion, and will not be repeated here.

[0261] S109, synthesize the target background music and motion amplitude information to obtain the target audio and video.

[0262] The methods for synthesizing target audio and video can be referred to the previous discussion, and will not be repeated here.

[0263] As an example, S106 to S109 are optional parts.

[0264] For example, please refer to Figure 11 This is an example diagram illustrating the process of object behavior recognition, where the terminal obtains... Figure 11 The video to be identified, shown in figure a, is decoded to obtain the following: Figure 11 As shown in b, multiple video frames are divided to obtain, as shown in the figure. Figure 11The first video segment shown in c1, and as shown in c1 Figure 11 The second video clip shown in c2.

[0265] The terminal performs object behavior recognition on the first video clip and the second video clip respectively, and determines that the target behavior of each person in the first video clip belongs to chorus, and the target behavior of each person in the second video clip belongs to dance.

[0266] Based on the target behavior corresponding to each video segment, the terminal determines target video segments where both the first and second video segments meet preset behavioral conditions. The terminal can then select two video frames of a preset character from each of the two target video segments, specifically as follows: Figure 11 The two video frames shown in d1 and d2 are used to filter background music. Based on these two video frames and the filtered target background music, the target audio and video are obtained, as detailed below. Figure 11 As shown in e.

[0267] exist Figure 10 In the illustrated embodiment, the terminal can combine the behavioral and posture features of a person in a video clip to identify their behavior. Because it combines multiple features of the person, it can obtain the person's behavior more accurately. Furthermore, after identifying the behavior of a person in a video clip, the terminal can flexibly perform video synthesis based on the behavior recognition results to obtain more rhythmic audio and video, thus enriching the video material.

[0268] The object behavior recognition method involved in the embodiments of this application can be executed by a server, a terminal, or by a server and a terminal working together. The following is based on... Figure 1 The application scenarios shown, combined with Figure 12 The diagram illustrating the interaction between the server and the terminal serves as an example of the collaborative execution of an object behavior recognition method by the terminal and the server. This paper describes the object behavior recognition method involved in the embodiments of this application.

[0269] S121, the terminal responds to the video acquisition operation by generating a video acquisition request.

[0270] Video acquisition operations are used to acquire videos, such as user clicks on a video acquisition control or swipes on a video playback interface. When the terminal determines that the user needs to acquire a video, it can generate a video acquisition request, which is used to request the acquisition of the corresponding video.

[0271] S122, the terminal sends a video acquisition request to the server.

[0272] S123, the server extracts video segments from the video to be identified, and the video segments include at least one video frame.

[0273] The methods for extracting video clips can be referred to the previous discussion, and will not be repeated here.

[0274] S124, the server extracts behavioral features of at least one target object from the video clip, wherein the behavioral features are determined based on the image features of the corresponding target object in the video clip.

[0275] The methods for extracting behavioral features can be referred to the previous discussion, and will not be repeated here.

[0276] S125, the server extracts the pose features of at least one target object from the video clip, wherein the pose features are determined based on the set of object key points of the corresponding target object in the video clip.

[0277] The methods for extracting pose features can be referred to the previous discussion, and will not be repeated here.

[0278] S126, the server determines the target behavior recognition result of at least one target object in the video segment based on the behavioral and posture characteristics of at least one target object.

[0279] The method for determining the target behavior recognition result can refer to the content discussed above, and will not be repeated here.

[0280] S127, the server selects target video segments that meet the preset behaviors based on the target behavior recognition results of each video segment.

[0281] The methods for selecting target video clips can be referred to the previous discussion, and will not be repeated here.

[0282] S128, the server synthesizes target audio and video based on the target video clip.

[0283] The methods for synthesizing target audio and video can be referred to the previous discussion, and will not be repeated here.

[0284] S129, the server sends the target audio and video to the terminal.

[0285] S130, the terminal plays the target audio and video.

[0286] As one example, Figure 12 S121 and S127-S130 are optional parts.

[0287] For example, please refer to Figure 13 This is an example diagram showing changes to the terminal interface. The terminal displays as follows: Figure 13 The interface shown in Figure a includes beat-synced video 1, beat-synced video 2, and beat-synced video 3. When the user... Figure 13In the interface shown in Figure a, sliding along the direction of the dotted arrow is equivalent to performing a video acquisition operation. Based on this operation, the terminal generates a video acquisition request and sends it to the server. After receiving the request, the server generates the corresponding beat-sync video (4) and sends it back to the terminal. The terminal then displays the video as shown in Figure a. Figure 13 The interface shown in b includes beat-sync video 2, beat-sync video 3, and beat-sync video 4.

[0288] exist Figure 12 In the illustrated embodiment, the server can synthesize corresponding audio and video for the terminal based on the terminal's real-time needs, thereby meeting the user's personalized audio and video viewing requirements. Furthermore, the server can identify the target object's behavior based on its behavioral and posture characteristics, enabling accurate identification of the target object's behavior and facilitating the provision of more accurate and tailored video clips to the user's needs.

[0289] Based on the same inventive concept, embodiments of this application provide an object behavior recognition device. This device can realize the functions of the aforementioned server or terminal. Please refer to [link / reference]. Figure 14 This is a schematic diagram of the structure of an object behavior recognition device, which includes:

[0290] The video segment extraction module 1401 is used to extract video segments from the video to be identified, wherein the video segment includes at least one video frame;

[0291] The behavior feature extraction module 1402 is used to extract the behavior features of at least one target object from a video clip, wherein the behavior features are determined based on the image features of the corresponding target object in the video clip;

[0292] The pose feature extraction module 1403 is used to extract pose features of at least one target object from a video clip, wherein the pose features are determined based on the set of object key points of the corresponding target object in the video clip;

[0293] The result determination module 1404 is used to determine the target behavior recognition result of at least one target object in a video segment based on the behavioral and pose features of at least one target object.

[0294] In one possible embodiment, the behavior feature extraction module 1402 is specifically used for:

[0295] Extract image features of at least one target object from each video frame in the video segment to obtain the video feature map corresponding to the video segment;

[0296] For at least one target object, perform the following operations respectively:

[0297] From any video frame in a video segment, detect the relative position of at least one target object in any video frame.

[0298] Based on the relative position of a target object in any video frame, the behavioral characteristics of the target object are determined from the video feature map.

[0299] In one possible embodiment, the behavior feature extraction module 1402 is specifically used for:

[0300] From the video feature map, identify the feature region corresponding to the relative position of a target object;

[0301] The feature region is divided into multiple candidate units, and for multiple candidate units, the following operations are performed: using interpolation methods, the coordinate values ​​corresponding to a preset position in one of the multiple candidate units are determined, and max pooling is performed on the determined coordinate values ​​to obtain the pooling result;

[0302] Based on the pooling results of multiple candidate units, the behavioral characteristics of a target object are obtained.

[0303] In one possible embodiment, the pose feature extraction module 1403 is specifically used for:

[0304] Extract at least one set of object key points for each target object from each video frame in the video clip;

[0305] For at least one target object, perform the following operations respectively: Based on the set of object key points of one of the target objects, obtain the pose features of the target object.

[0306] In one possible embodiment, the pose feature extraction module 1403 is specifically used for:

[0307] For each video frame in the video clip, perform one of the following operations:

[0308] Perform object detection on one video frame from each video frame to obtain the detection box of at least one target object in the video frame, and identify at least one object key point of at least one target object from each of the detected detection boxes.

[0309] Perform object keypoint detection on a video frame to obtain at least one object keypoint in the video frame, and determine the target object to which the at least one object keypoint belongs, so as to obtain at least one object keypoint for each of the at least one target object.

[0310] In one possible embodiment, the pose feature extraction module 1403 is specifically used for:

[0311] Based on a set of object key points of a target object, a first three-dimensional vector is obtained, where the first dimension of the first three-dimensional vector represents the duration of the video segment, the second dimension represents the number of object key points in the set of object key points of a target object, and the third dimension represents the number of channels.

[0312] Perform a convolution operation on the first three-dimensional vector to obtain the second three-dimensional vector;

[0313] By swapping the relative positions of the second and third dimensions in the second three-dimensional vector, the third three-dimensional vector is obtained.

[0314] Perform a convolution operation on the third 3D vector to obtain the pose features of a target object.

[0315] In one possible embodiment, the result determination module 1404 is specifically used for:

[0316] For at least one target object, perform the following operations respectively:

[0317] Input the behavioral features of one of the target objects into a pre-trained first behavior classification network to obtain the first probability value of a target object belonging to each single object behavior label.

[0318] Inputting the pose features of a target object into a pre-trained second behavior classification network, we obtain the second probability value of a target object belonging to each single object behavior label.

[0319] The first and second probability values ​​of a target object under the same single object behavior label are weighted and summed to obtain the third probability value of the target object belonging to each single object behavior label.

[0320] Based on the obtained third probability values, determine the single-object behavior label to which a target object belongs;

[0321] The single-object behavior labels of at least one target object are used as the target behavior recognition results corresponding to each of the at least one target object.

[0322] In one possible embodiment, at least one target object includes multiple target objects, and the result determination module 1404 is specifically used for:

[0323] Combine the behavioral features of at least one target object, and input the combined behavioral features into a pre-trained third behavior classification network to obtain a fourth probability value corresponding to at least one target object belonging to the various multi-object behavior labels;

[0324] Combine the pose features of at least one target object and input the combined pose features into a pre-trained fourth behavior classification network to obtain the fifth probability value of at least one target object belonging to each multi-object behavior label.

[0325] The fourth and fifth probability values ​​of at least one target object under the same multi-object behavior label are weighted and summed to obtain the sixth probability value of a target object belonging to each of the various multi-object behavior labels.

[0326] Based on the obtained sixth probability values, determine the multi-object behavior label to which at least one target object belongs, and use the determined multi-object behavior label as the target behavior recognition result of at least one target object.

[0327] In one possible embodiment, the result determination module 1404 is specifically used for:

[0328] For at least one target object, perform the following operations respectively: fuse the behavioral features of one target object with the pose features of another target object to obtain the fused features of the target object;

[0329] Based on the fusion features of at least one target object, the target behavior recognition result of at least one target object in the video segment is obtained.

[0330] In one possible embodiment, the result determination module 1404 is specifically configured to perform at least one of the following:

[0331] The fused features of at least one target object are input into a pre-trained fifth behavior classification network to obtain the single-object behavior recognition results of at least one target object in the video segment.

[0332] At least one target object includes multiple target objects. The fusion features of each of the at least one target object are combined, and the combined fusion features are input into a pre-trained sixth behavior classification network to obtain the multi-object behavior recognition result of at least one target object in the video segment.

[0333] In one possible embodiment, the video segments include multiple segments, and the apparatus further includes an audio / video acquisition module 1405, which is specifically used for:

[0334] Based on the behavioral and posture characteristics of at least one target object, after determining the target behavior recognition result of at least one target object in a video segment, target video segments that meet the preset behavioral conditions are selected from multiple video segments.

[0335] Based on the set of object key points of the preset target object in the target video segment, determine the motion amplitude information of the target part of the preset target object in the target video segment, wherein the preset target object is one of at least one target object;

[0336] From multiple pre-stored background music tracks, a target background music track is selected based on the matching of music rhythm information and motion amplitude information.

[0337] Merge the target video clip with the target background music to obtain the target audio and video.

[0338] As one embodiment, the audio and video acquisition module 1405 is an optional module.

[0339] It should be noted that, Figure 14 The device shown can also implement any of the object behavior recognition methods discussed above, which will not be elaborated here.

[0340] exist Figure 14 In the embodiment shown, the object behavior recognition device can combine the behavioral and posture features of the target object to more accurately determine the behavior of the target object, providing a more accurate video segment classification for subsequent video editing.

[0341] Based on the same inventive concept, embodiments of this application provide a computer device that can realize the functions of the aforementioned server or terminal. Please refer to... Figure 15 The computer device includes a processor 1501 and a memory 1502.

[0342] The processor 1501 can be a central processing unit (CPU), or a digital processing unit, etc. This application embodiment does not limit the specific connection medium between the memory 1502 and the processor 1501. This application embodiment... Figure 15 The memory 1502 and the processor 1501 are connected via a bus 1503, and the bus 1503 is in Figure 15 The connections between other components are shown in thick lines only and are not intended to be limiting. The 1503 bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0343] Memory 1502 may be volatile memory, such as random-access memory (RAM); memory 1502 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1502 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1502 may be a combination of the above-described memories.

[0344] Processor 1501 is used to execute the object behavior recognition method described above when calling the computer program stored in memory 1502, and can also be used to implement the aforementioned... Figure 14 The function of the device shown.

[0345] Based on the same inventive concept, embodiments of this application provide a computer storage medium storing computer instructions. When these computer instructions are executed on a computer device, the computer device performs any of the object behavior recognition methods discussed above. The computer device may be... Figure 15 The computer equipment shown.

[0346] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0347] Based on the same inventive concept, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned object behavior recognition method.

[0348] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0349] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0350] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for object behavior recognition, characterized in that, include: Extract video segments from the video to be identified, wherein the video segments include at least one video frame; From the video clip, at least one behavioral feature of each target object is extracted, wherein the behavioral feature is determined based on the image features of the corresponding target object in the video clip; From the video segment, the pose features of each of the at least one target object are extracted, wherein the pose features are determined based on the set of object key points of the corresponding target object in the video segment; Based on the behavioral and posture characteristics of each of the at least one target object, the target behavior recognition result of the at least one target object in the video segment is determined; The step of extracting the pose features of each of the at least one target object from the video segment includes: extracting the set of object key points for each of the at least one target object from each video frame in the video segment; and performing the following operations on each of the at least one target object to obtain the pose features of the at least one target object: Based on the set of object key points of one of the at least one target objects, a first three-dimensional vector is obtained, wherein the first dimension of the first three-dimensional vector represents the duration of the video segment, the second dimension represents the number of object key points in the set of object key points of the target object, and the third dimension represents the number of channels. A convolution operation is performed on the first three-dimensional vector to obtain a second three-dimensional vector. The relative positions of the second and third dimensions of the second three-dimensional vector are swapped to obtain a third three-dimensional vector. A convolution operation is then performed on the third three-dimensional vector to obtain the pose features of the target object.

2. The method as described in claim 1, characterized in that, Extracting behavioral features of at least one target object from the video segment includes: From each video frame in the video segment, at least one image feature of a target object is extracted to obtain a video feature map corresponding to the video segment; For each of the at least one target objects, perform the following operations: From any video frame in the video segment, detect the relative position of at least one target object in any video frame; Based on the relative position of the target object in any video frame, the behavioral characteristics of the target object are determined from the video feature map.

3. The method as described in claim 2, characterized in that, The step of determining the behavioral characteristics of a target object from the video feature map based on the relative position of the target object in any video frame includes: From the video feature map, determine the feature region corresponding to the relative position of the target object; The feature region is divided into multiple candidate units, and for the multiple candidate units, the following operations are performed: using an interpolation method, the coordinate value corresponding to a preset position in one of the multiple candidate units is determined, and the determined coordinate value is subjected to max pooling to obtain the pooling result; Based on the pooling results of the multiple candidate units, the behavioral characteristics of the target object are obtained.

4. The method as described in claim 1, characterized in that, Extracting the set of object key points for each of the at least one target object from each video frame in the video segment includes: For each video frame in the video segment, perform one of the following operations: Object detection is performed on one video frame from each video frame to obtain a detection box for at least one target object in the video frame, and at least one object key point of each target object is identified from each detected detection box. Perform object keypoint detection on a video frame to obtain at least one object keypoint in the video frame, and determine the target object to which the at least one object keypoint belongs, so as to obtain at least one object keypoint for each of the at least one target object.

5. The method as described in claim 1, characterized in that, The step of determining the target behavior recognition result of the at least one target object in the video segment based on the respective behavioral and pose features of the at least one target object includes: For each of the at least one target objects, perform the following operations: The behavioral features of one of the at least one target objects are input into a pre-trained first behavior classification network to obtain the first probability value of the target object belonging to each single object behavior label. The pose features of the target object are input into a pre-trained second behavior classification network to obtain the second probability value of the target object belonging to each single object behavior label. The first probability value and the second probability value of the target object under the same single object behavior label are weighted and summed to obtain the third probability value of the target object belonging to each of the single object behavior labels. Based on the obtained third probability values, the single-object behavior label to which the target object belongs is determined; The single-object behavior labels of the at least one target object are determined and used as the target behavior recognition results corresponding to the at least one target object.

6. The method as described in claim 1, characterized in that, The at least one target object includes multiple target objects; determining the target behavior recognition result of the at least one target object in the video segment based on the behavioral and pose features of each of the at least one target object includes: The behavioral features of each of the at least one target object are combined, and the combined behavioral features are input into a pre-trained third behavior classification network to obtain the fourth probability value of the at least one target object belonging to each multi-object behavior label. The pose features of each of the at least one target object are combined, and the combined pose features are input into a pre-trained fourth behavior classification network to obtain the fifth probability value of the at least one target object belonging to each of the multi-object behavior labels. The fourth and fifth probability values ​​of the at least one target object under the same multi-object behavior label are weighted and summed to obtain the sixth probability value of the target object belonging to each of the multi-object behavior labels. Based on the obtained sixth probability values, the multi-object behavior label to which the at least one target object belongs is determined, and the determined multi-object behavior label is used as the target behavior recognition result of the at least one target object.

7. The method as described in claim 1, characterized in that, The step of determining the target behavior recognition result of the at least one target object in the video segment based on the respective behavioral and pose features of the at least one target object includes: For the at least one target object, perform the following operations respectively: fuse the behavioral features of one of the at least one target objects with the pose features of the target object to obtain the fused features of the target object; Based on the fusion features of each of the at least one target object, the target behavior recognition result of the at least one target object in the video segment is obtained.

8. The method as described in claim 7, characterized in that, The step of obtaining the target behavior recognition result of the at least one target object in the video segment based on the fusion features of each of the at least one target object includes at least one of the following: The fused features of each of the at least one target object are input into a pre-trained fifth behavior classification network to obtain the single-object behavior recognition results of each of the at least one target object in the video segment. The at least one target object includes multiple target objects. The fusion features of the at least one target object are combined, and the combined fusion features are input into a pre-trained sixth behavior classification network to obtain the multi-object behavior recognition result of the at least one target object in the video segment.

9. The method according to any one of claims 1 to 7, characterized in that, The video segments include multiple segments; after determining the target behavior recognition result of the at least one target object in the video segment based on the respective behavioral and pose features of the at least one target object, the method further includes: Select target video clips that meet preset behavioral conditions from multiple video clips; Based on a set of object key points of a preset target object in the target video segment, the motion amplitude information of the target part of the preset target object in the target video segment is determined, wherein the preset target object is one of the at least one target object; From a plurality of pre-stored background music tracks, a target background music track is selected whose music rhythm information matches the motion amplitude information. The target video segment and the target background music are merged to obtain the target audio-visual content.

10. An object behavior recognition device, characterized in that, include: A video segment extraction module is used to extract video segments from a video to be identified, wherein the video segment includes at least one video frame; A behavior feature extraction module is used to extract behavior features of at least one target object from the video segment, wherein the behavior features are determined based on the image features of the corresponding target object in the video segment; A pose feature extraction module is used to extract pose features of each of the at least one target object from the video segment, wherein the pose features are determined based on the set of object key points of the corresponding target object in the video segment; The result determination module is used to determine the target behavior recognition result of the at least one target object in the video segment based on the behavioral and posture characteristics of the at least one target object. The step of extracting the pose features of each of the at least one target object from the video segment includes: extracting the set of object key points for each of the at least one target object from each video frame in the video segment; and performing the following operations on each of the at least one target object to obtain the pose features of the at least one target object: Based on the set of object key points of one of the at least one target objects, a first three-dimensional vector is obtained, wherein the first dimension of the first three-dimensional vector represents the duration of the video segment, the second dimension represents the number of object key points in the set of object key points of the target object, and the third dimension represents the number of channels. A convolution operation is performed on the first three-dimensional vector to obtain a second three-dimensional vector. The relative positions of the second and third dimensions of the second three-dimensional vector are swapped to obtain a third three-dimensional vector. A convolution operation is then performed on the third three-dimensional vector to obtain the pose features of the target object.

11. The apparatus as claimed in claim 10, characterized in that, The behavioral feature extraction module is specifically used for: From each video frame in the video segment, at least one image feature of a target object is extracted to obtain a video feature map corresponding to the video segment; For each of the at least one target objects, perform the following operations: From any video frame in the video segment, detect the relative position of at least one target object in any video frame; Based on the relative position of the target object in any video frame, the behavioral characteristics of the target object are determined from the video feature map.

12. The apparatus as claimed in claim 11, characterized in that, The behavioral feature extraction module is specifically used for: From the video feature map, determine the feature region corresponding to the relative position of the target object; The feature region is divided into multiple candidate units, and for the multiple candidate units, the following operations are performed: using an interpolation method, the coordinate value corresponding to a preset position in one of the multiple candidate units is determined, and the determined coordinate value is subjected to max pooling to obtain the pooling result; Based on the pooling results of the multiple candidate units, the behavioral characteristics of the target object are obtained.

13. A computer device, characterized in that, include: At least one processor, and A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of claims 1 to 9 by executing the instructions stored in the memory.

14. A computer storage medium, characterized in that, The computer storage medium stores computer instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 9.

Images