Video analysis method, electronic equipment and storage medium

By determining multiple objects and object actions in multi-frame images in video analysis, combining object and action tags, and using generalized models to identify the behavior of target objects, it solves the difficulty in expanding new behaviors and recognition accuracy in the prior art. Low problems achieve efficient behavior recognition and flexibility.

CN119028008APending Publication Date: 2024-11-26HONOR DEVICE CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310589539.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing video behavior recognition technology is difficult to expand new behaviors, and the recognition accuracy is low in complex scenarios, data acquisition and training are difficult to adapt to diverse video behaviors.

Method used

By acquiring multi-frame images, determining multiple objects and object actions, combining object and action tags, using generalized models to identify the behavior of the target object, integrating visual relationship information and bone point information, improving the flexibility and accuracy of behavior recognition, Reduce data and training costs.

Benefits of technology

It realizes the more convenient expansion of new behavior recognition capabilities at low cost, improves the flexibility and accuracy of complex behavior recognition, and reduces data and training costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028008A_ABST
    Figure CN119028008A_ABST
Patent Text Reader

Abstract

The invention discloses a video analysis method, electronic equipment and a storage medium, and the method can obtain a to-be-recognized video which comprises multiple frames of images; multiple objects in the multi-frame image and the action of at least one object in the multiple objects are determined, the at least one object comprises a target object, and the action of the target object acts on other objects except the target object in the multiple objects; and determining behaviors of the target object in the multi-frame image according to the action of the at least one object and the plurality of objects. Wherein the actions or the objects have generalization, and new behaviors can be obtained by combining the same action with different objects or combining the same object with different actions, so that the behaviors in the video are analyzed through a splitting method, the flexibility of complex behavior recognition is enhanced, and the new behaviors are easier to expand.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video analysis, and in particular to a video analysis method, an electronic device and a storage medium. Background Art

[0002] With the widespread application of videos, video behavior recognition technology has become a current research hotspot. Video behavior recognition, as the name suggests, refers to the recognition of the behavior of the target in the video. For example, in a video with people as the subject, it is necessary to recognize and understand the movements and behaviors of people. Video behavior recognition has been widely used in video retrieval, video editing, intelligent monitoring and other fields.

[0003] The main method of video behavior recognition is to input the video into the model for recognition. The current models are trained for specific behaviors in specific scenarios and are suitable for the recognition of fixed behaviors. For example, the model trained for playing badminton can only recognize the behavior of playing badminton, but not the behavior of playing volleyball. Since the behaviors in the video may be complex and diverse, it is difficult to expand new behaviors with existing methods. Summary of the invention

[0004] The present application provides a video analysis method, an electronic device and a storage medium, which can expand new behaviors more conveniently at a lower cost. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a video analysis method, the method comprising:

[0006] Obtain a video to be identified, where the video to be identified includes multiple frames of images;

[0007] Determine multiple objects in multiple frames of images and an action of at least one of the multiple objects, the at least one object includes a target object, and the action of the target object acts on other objects of the multiple objects except the target object;

[0008] According to the action of at least one object and the multiple objects, the behavior of the target object in the multiple frames of images is determined.

[0009] Based on the above technical solution, when performing video analysis, after obtaining the video to be identified, multiple objects in multiple frames of images and the actions of at least one of the multiple objects can be determined separately, and then the actions of at least one object and multiple objects are combined to determine the behavior of the target object in the multiple frames of images. Among them, the action or object has generalization, and the same action combined with different objects, or the same object combined with different actions, can obtain new behaviors. Therefore, the present application analyzes the behavior in the video by splitting the method, which enhances the flexibility of complex behavior recognition and makes it easier to expand new behaviors.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the behavior of the target object in multiple frames of images is determined based on the action of at least one object and multiple objects, including: inputting a label corresponding to the action of at least one object and the labels corresponding to the multiple objects into a first model to determine the behavior of the target object in multiple frames of images.

[0011] The first model is trained based on many object labels and many action labels, and the training difficulty is relatively low. When expanding new compound action categories, only a small amount of data is used to train the first model to effectively identify the behavior. Therefore, the present application can expand new behaviors with lower data costs and training costs.

[0012] In combination with the first aspect and the above implementations, in some implementations of the first aspect, after determining the multiple objects in the multiple frames of images, the method further includes: determining a relative position relationship between the multiple objects;

[0013] Determining the behavior of the target object in multiple frames of images according to the action of at least one object and multiple objects includes: determining the behavior of the target object in multiple frames of images according to the relative position relationship, the action of at least one object and multiple objects.

[0014] The above relative position relationship is the position relationship between objects perceived visually. By integrating visual relationship information, the semantic information required for behavior recognition can be more effectively mined, thereby improving the accuracy of behavior recognition.

[0015] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, the behavior of the target object in multiple frames of images is determined based on the relative position relationship, the action of at least one object and multiple objects, including: inputting the labels corresponding to the relative position relationship, the action of at least one object and the labels corresponding to the multiple objects into the second model to determine the behavior of the target object in the multiple frames of images.

[0016] The second model is trained based on relative position relationships, numerous object labels, and numerous action labels. It is relatively easy to train. When expanding new compound action categories, only a small amount of data is needed to train the second model to effectively identify the behavior.

[0017] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining the action of at least one of the multiple objects includes: determining the action of the at least one object according to the position of the skeleton point corresponding to the at least one object in the multiple frames of images. In this calculation method, the fusion of the skeleton point information allows the calculation process to pay more attention to the limb action itself, and the subtle changes in the action can be captured by the position changes of the skeleton points, which can improve the accuracy of action recognition.

[0018] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining the action of at least one of the multiple objects includes: determining the action of the at least one object according to the change in displacement of a pixel point corresponding to the at least one object in two adjacent frames of the multiple frames over time. In this calculation method, the direction and intensity of the action are inferred by detecting the change in pixels at the same position in the video over time, which can enhance the attention to the moving object during the calculation process and improve the accuracy of action recognition.

[0019] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, determining the action of at least one object among multiple objects includes: determining at least one first action based on the position of the skeleton points corresponding to the at least one object in multiple frames of images, and the at least one first action is a prediction of the action of the at least one object; determining at least one second action based on the change over time of the displacement of the pixel points corresponding to the at least one object in two adjacent frames of the multiple frames of images, and the at least one second action is a prediction of the action of the at least one object; determining the action of the at least one object based on the at least one first action and the at least one second action.

[0020] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; one or more memories; the memory stores one or more programs, and when one or more programs are executed by the processor, the electronic device executes any possible method of the first aspect above.

[0021] In a third aspect, an embodiment of the present application provides a device, which is included in an electronic device, and the device has the function of realizing the above-mentioned aspects and the behavior of the electronic device in the possible implementation methods of the above-mentioned aspects. The function can be implemented by hardware, or the corresponding software can be implemented by hardware. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the method described in the first aspect above.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect above.

[0024] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application is shown;

[0026] Figure 2 An example of a software structure block diagram provided in an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram of a process of a video analysis method provided in an embodiment of the present application is shown;

[0028] Figure 4 A schematic diagram of a scenario provided by an embodiment of the present application is shown;

[0029] Figure 5 A schematic diagram showing a process of another video analysis method provided in an embodiment of the present application is shown;

[0030] Figure 6 A schematic diagram showing a process of another video analysis method provided in an embodiment of the present application is shown;

[0031] Figure 7 Another schematic diagram of a scenario provided by an embodiment of the present application is shown;

[0032] Figure 8 A schematic diagram showing a process of another video analysis method provided in an embodiment of the present application is shown;

[0033] Fig. 9 Another schematic diagram of a scenario provided by an embodiment of the present application is shown;

[0034] Fig.10 Another schematic diagram of a scenario provided by an embodiment of the present application is shown;

[0035] Fig.11 A schematic diagram of a process for determining an atomic action provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings. Hereinafter, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0037] With the widespread application of videos, video behavior recognition technology has become a current research hotspot. As the name suggests, video behavior recognition refers to the recognition of the behavior of the target object in the video. For example, in a video with a person as the main subject, it is necessary to recognize and understand the person's actions and behaviors. Video behavior recognition has been widely used in video content retrieval, video editing, intelligent monitoring, sports analysis, virtual reality and other fields. For example, in the field of intelligent monitoring, by identifying human behavior in the video, it is possible to warn of human violations and make process judgments on a series of human actions. For example, in the field of sports analysis, identifying the behavior of athletes in sports videos can retrieve wonderful actions and edit personalized content. In life scenarios, with the increase in mobile phone storage capacity and the increase in the popularity of online video sharing, users are more and more likely to record their lives through mobile phone videos. Users can use video behavior recognition to retrieve videos, edit, generate wonderful moments and memories, etc.

[0038] When performing video behavior recognition, it can be based on single-frame image recognition or fixed category recognition. Among them, single-frame image recognition refers to performing image recognition on a single frame image to determine the behavior included in the image. This method does not consider the time dimension and it is difficult to obtain information on the behavior in time sequence; fixed category recognition refers to pre-training the behavior recognition model and inputting the video into the behavior recognition model for recognition. In this method, the models are trained for specific behaviors in specific scenarios, which is suitable for the recognition of fixed behaviors and difficult to expand. For example, the model trained for the badminton scene can only recognize the behavior of playing badminton, but not the behavior of playing volleyball. When the behavior of playing volleyball needs to be recognized, a large amount of new video data including volleyball behavior needs to be collected for model training. Since the behavior in the video may be complex and diverse, when new recognition behaviors need to be added, a large amount of new behavior video data needs to be collected for model incremental training. Due to the high difficulty of video data collection and training, it is difficult to expand new behaviors in this method.

[0039] In view of this, an embodiment of the present application provides a video analysis method, which can determine multiple objects in the multiple frames of images and the actions of at least one of the multiple objects based on the multiple frames of images in the video to be identified, and then combine the objects and the actions to determine the behavior of the target object in the multiple frames of images. Since the multiple objects and the actions of at least one object are determined separately, the same action combined with different objects, or the same object combined with different actions, can obtain new behaviors. Therefore, for many actions or many objects that have been trained, they can be applied in many scenes, and new behaviors can be expanded by combining them with each other. Therefore, for different scenes in the video, the present application can effectively identify the behavior of the target object. Therefore, the present application analyzes the behavior in the video by the splitting method, which can enhance the flexibility of complex behavior recognition and make it easier to expand new behaviors.

[0040] The method provided in the embodiment of the present application can be applied to mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices and other electronic devices. It can be applied to mobile phone gallery video search, video intelligent editing, and image and video processing software. This application does not limit this.

[0041] The following is a schematic diagram of the hardware structure of the electronic device 100 that can implement the above method. Figure 1 A hardware structure diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0042] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0045] Figure 2 It is a software structure block diagram of the electronic device 100 provided in an embodiment of the present application.

[0046] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0047] The application layer can include a series of application packages.

[0048] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0049] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0050] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0051] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0052] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0053] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0054] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0055] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0056] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0057] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0058] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0059] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0060] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0061] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0062] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0063] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0064] A 2D graphics engine is a drawing engine for 2D drawings.

[0065] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0066] The video analysis method provided in the embodiments of the present application is described in detail below. Figure 3 The flowchart of a video analysis method provided by an embodiment of the present application is shown, including the following S301-S303. It can be understood that the electronic device hereinafter refers to the electronic device 100 in the above content.

[0067] S301, obtaining a video to be identified, where the video to be identified includes multiple frames of images.

[0068] In some video editing scenarios, users can use the one-key automatic editing function, or users can use the search function to input the behavior they want to edit, and the electronic device can receive the user's instructions. After that, the electronic device can identify the behavior in the video based on the video analysis method provided in the embodiment of the present application, and then automatically edit the part of the video that includes the behavior. In some video retrieval scenarios, users can search for videos based on the behavior included in the video. The electronic device can receive the behavior input by the user, and then identify the behavior included in each video based on the video analysis method provided in the embodiment of the present application, and filter out the videos that include the behavior input by the user.

[0069] The above scenarios all require obtaining the video to be identified first. The video to be identified can be obtained locally by the electronic device. For example, a user uses an electronic device to shoot a video and saves it locally on the electronic device. Then, in some scenarios, the electronic device can obtain the video to be identified locally. The video to be identified can also be obtained by the electronic device from a cloud server. For example, the user uploads the video to the cloud server of some software or the cloud server of the electronic device. When editing is required, the electronic device can obtain the video to be identified from the cloud server. The video to be identified can also be obtained by the electronic device from the Internet. For example, if a user watches a sports game and wants to edit some clips of the sports game, he can download the video of the sports game (i.e., the video to be identified) from the Internet.

[0070] In one implementation, after the electronic device acquires the video to be identified, it processes all frame images in the video to be identified.

[0071] In another implementation, after the electronic device obtains the video to be identified, it first performs frame extraction, and then processes the multiple frames of images obtained by frame extraction. Frame extraction refers to a method of extracting a frame of image every certain number of frames in a video, and finally obtaining several frames of images. Exemplarily, the video to be identified includes 20 frames, and one frame is extracted every 5 frames. Then the multiple frames of images obtained are the first frame, the sixth frame, the twelfth frame, and the eighteenth frame. Through frame extraction, the amount of data processing can be reduced and the efficiency of video analysis can be improved.

[0072] S302, determining multiple objects in multiple frames of images and an action of at least one of the multiple objects, wherein the at least one object includes a target object, and the action of the target object acts on other objects of the multiple objects except the target object.

[0073] The action in the embodiments of the present application refers to the physical movement of the object performing the action in the image itself, and has nothing to do with the scene and the interactive object. For example, the object's behavior is riding a bicycle outdoors. In this behavior, the person riding the bicycle is the object performing the behavior, the outdoors is the scene, and the bicycle is the person's interactive object. The above action refers to the action of "riding".

[0074] For another example, the behavior of the object is to pick up a cup. In this behavior, the person picking up the cup is the object of the behavior, and the cup is the person's interactive object. The above action refers to the action of "picking up".

[0075] In the embodiment of the present application, the above actions can be called "atomic actions". Compared with atomic actions, the above "picking up a cup" and "riding a bicycle" can be called "compound actions", and "compound actions" can also be called "behaviors".

[0076] Atomic actions are universal and a coarse-grained division that meets the following two characteristics: 1) They are applicable to a variety of scenarios in daily life; 2) They are independent of the scenario and the interactive object and focus on the visual action itself.

[0077] Atomic actions can be divided into many categories. For example, based on the actions of the human body, atomic actions can include walking, running, jumping, standing up, sitting down, squatting, clapping, waving, and climbing. Based on the possible behaviors when people interact with each other, atomic actions can include hugging, kissing, and patting. Based on the possible behaviors when people interact with objects, atomic actions can include taking, throwing, riding, eating, drinking, and pushing.

[0078] The object refers to a person, animal, or object that appears in multiple frames of images. In the embodiment of the present application, multiple objects in multiple frames of images can be identified based on methods such as image classification models or image detection algorithms. Figure 4 In the multi-person scene shown in (a), the objects that can be identified are two people. Figure 4In the cycling scene shown in (b) of Figure 1, the objects that can be identified are people and cars. Figure 4 In the indoor scene shown in (c) in FIG. 1 , the objects that can be identified are a person, a table, a sofa, a television, etc. It should be understood that multiple objects can be dispersedly distributed in multiple frames of images.

[0079] Among the multiple objects identified, one object may have made an action, or two or more objects may have made an action, and the actions made by the two or more objects may be different. In the embodiment of the present application, multiple actions included in multiple frames of images can be identified. For example, in a basketball game scene with spectators watching, the spectators stand up and cheer after the player scores a goal. The objects that can be identified are people (including basketball players and spectators), baskets, basketballs, etc., and the actions that can be identified are standing up, raising hands, clapping, and throwing.

[0080] S303: Determine the behavior of the target object in the multiple frames of images according to the action of the at least one object and the multiple objects.

[0081] Behaviors combine atomic actions and objects, and include scene and interactive object information. Behaviors include more content than atomic actions and are fine-grained. For example, behaviors including scenes can be "running indoors" or "running outdoors", behaviors including interactive objects can be "skipping rope", "hugging", "clapping", and behaviors including scenes and interactive objects can be "skipping rope outdoors".

[0082] After determining the action of at least one object and multiple objects, the electronic device can arbitrarily combine the at least one object, the action of at least one object and other objects to determine the behavior of the target object from multiple combinations. The target object is the executor of the finally determined behavior. In the embodiment of the present application, multiple target objects can be determined.

[0083] Atomic actions can be determined by a pre-trained action recognition model. During model training, multiple videos including the above-mentioned multiple atomic actions are obtained, labels are annotated for each atomic action, the videos are used as input, and the corresponding labels annotated for the atomic actions in the videos are used as results, and the action recognition model is obtained through training. The action recognition model can effectively identify multiple types of atomic actions. During video analysis, the video to be identified is input into the action recognition model, and the labels corresponding to the atomic actions in the video output by the action recognition model can be obtained.

[0084] Multiple objects can be identified by a pre-trained image classification model. During model training, multiple videos including different objects are obtained, labels are annotated for each object, the video is used as input, and the label corresponding to each object in the video is used as the result, and the image classification model is obtained through training. During video analysis, the video to be identified is input into the image classification model, and the label corresponding to each object in the video output by the image classification model can be obtained. In one implementation, the image classification model can be an image label classification model based on a convnext network, which can recognize 1k+ rich label categories, can be used to label images and video frames in mobile phone galleries, can effectively identify many different objects, and the results output by the model can be used to identify behaviors in combination with atomic actions.

[0085] It should be noted that the video collected during the training of the action recognition model does not necessarily include interactive objects. The video collected during the training of the image classification model does not necessarily include atomic actions. In other words, for a behavior (e.g., riding a carousel) in this application, when the model used is trained, the collected data is not necessarily required to include the data of the behavior (riding a carousel). It is only necessary to collect data including the atomic actions (riding) in the behavior when training the action recognition model, and to collect data including the interactive objects (carousel) in the behavior when training the image classification model. When it is necessary to expand another behavior (e.g., riding a bike), on the basis that the action recognition model can already recognize the action of "riding", it is only necessary to collect data related to "car" to train the image classification model, and another behavior (riding a bike) can be combined. The categories of atomic actions and the types of objects in the image are far less than behaviors. This application can reduce the difficulty of collecting data and the difficulty of training models, and expand new behaviors with less data cost and training cost.

[0086] Based on the process of the above model training, during video analysis, the present application is applicable to many scenes. For example, for a video of riding a carousel, the above method can identify the behavior (riding a carousel), and for a video of riding a bicycle, the above method can also identify the behavior (riding a bicycle). The action recognition model can also identify other atomic actions, such as pushing, so for a video of a cart, the above method can also identify the behavior (pushing a cart). In short, the method provided in the embodiment of the present application, the action or object has generalization, the same action combined with different objects, or the same object combined with different actions, can obtain new behaviors. Therefore, the present application analyzes the behavior in the video by the splitting method, enhances the flexibility of complex behavior recognition, and makes it easier to expand new behaviors.

[0087] Labels are used for training models. When analyzing videos, the above models output atomic action labels and labels corresponding to objects. When determining behaviors, the label keywords may not be unique or fixed, and an effective combination method is required. The following describes the combination method of atomic action labels and labels corresponding to objects.

[0088] In one implementation, the electronic device inputs a label corresponding to an action of at least one object and labels corresponding to multiple objects into a first model to determine the behavior of the target object in multiple frames of images.

[0089] Figure 5 FIG. 4 shows a flow chart of determining behavior using the first model. Figure 5 As shown, multiple frames of images are input into an action recognition model for atomic action recognition to obtain an atomic action label corresponding to at least one object; in addition, multiple frames of images are input into an image classification model for image label recognition to obtain labels corresponding to multiple objects; the atomic action labels and labels corresponding to multiple objects are input into the first model to obtain the behavior of the target object.

[0090] Among them, the first model is trained based on a large number of atomic action labels and a large number of object labels. When training the first model, for each possible scenario, a set of labels (including at least one atomic action label and multiple object labels) is determined, and the behavior to be output is set for this set of labels. The multiple sets of labels are used as training samples, and the behavior set for each set of labels is used as the target result to train the first model. The network structure of the first model is not limited in the embodiment of the present application.

[0091] Exemplarily, the first model may be a fully connected sub-network. Figure 6 As shown, the fully connected subnetwork includes an input layer, a hidden layer, and an output layer. In the embodiment of the present application, the image label and the atomic action label are encoded as text features and input into the fully connected subnetwork for automatic learning. After receiving the text features, the input layer sends them to each neuron in the hidden layer. After receiving the text features, each neuron activates to generate an output signal and summarizes it according to its own judgment, and outputs a compound action. The fully connected subnetwork can recognize and classify the input, find the most reasonable result between categories, and can effectively combine the image label and the atomic action label to determine the compound action. The hidden layer is set to a multi-layer neural network structure, which can realize a multi-layer fully connected subnetwork. The embodiment of the present application does not limit the number of hidden layers.

[0092] The first model is trained based on labels, and the training difficulty is relatively low. When expanding new behaviors, a behavior can be set for a new set of labels, or a new behavior can be set for an existing set of labels. Therefore, only a small amount of data is needed to train the first model to obtain effective behavior recognition results, which can reduce data costs and training costs.

[0093] In another implementation, the electronic device can determine the behavior of the target object in multiple frames of images by manually set matching rules. Exemplarily, the matching rules can be implemented by a decision tree. By combining the object label and the action label through the manually set matching rules, zero-sample learning can be achieved, which is more scalable.

[0094] In some scenarios, objects included in different behaviors with the same atomic action, or atomic actions included in different behaviors with the same object, may appear simultaneously in multiple frames, which may cause confusion in behavior discrimination. Figure 7 A schematic diagram of a scenario is shown, such as Figure 7 As shown, in multiple frames of images, the atomic action "open" of an object can be identified, and doors and windows appear simultaneously in multiple frames of images, and the object labels are determined to be "door" and "window". Therefore, two behaviors can be determined by two label combinations: "open" and "door" constitute "opening the door", and "open" and "window" constitute "opening the window", then it is difficult to identify whether the behavior is "opening the door" or "opening the window".

[0095] To this end, the video analysis method provided in the embodiment of the present application also includes: determining the relative position relationship between multiple objects.

[0096] like Figure 8 As shown, the electronic device not only inputs multiple frames of images into the action recognition model for atomic action recognition to obtain an atomic action label corresponding to at least one object; and inputs multiple frames of images into the image classification model for image label recognition to obtain labels corresponding to multiple objects; it also performs visual relationship detection on the multiple frames of images to determine the relative position relationship between the multiple objects; then, the relative position relationship, the label corresponding to the action of at least one object, and the labels corresponding to the multiple objects are input into the second model to determine the behavior of the target object in the multiple frames of images.

[0097] Relative position relationship is a visual spatial relationship, which can also be called visual relationship, such as above, below, beside, etc. Figure 7 In the scene shown, through visual relationship detection, the relative position relationship between multiple objects can be determined as "person by the window" and "hand on the window", thereby determining that the behavior of the target object is "opening the window".

[0098] The relative position relationship can be determined by an image feature extraction algorithm. For example, each frame of the image is automatically segmented to divide the region of each object included in each frame of the image. After determining what the multiple objects are, the relative position relationship between the multiple objects can be determined based on the regional position. It should be understood that the relevant position relationship is determined between two objects that are closer, but it is not necessary to determine the relevant position relationship between two objects that are farther away. In multiple frames of images, there must be a relative position relationship between the target object and its interactive object.

[0099] The relative position relationship can also be determined by a multimodal large model. "Multimodal" refers to the unification of modalities such as text, image, audio, and video to achieve cross-modal understanding, conversion, and generation. "Large model" refers to the large parameter scale of the model and the strong model computing power. The multimodal large model can complete tasks such as text-to-image search, image-to-sound generation, intelligent question and answer, image generation, and video understanding. In an embodiment of the present application, a multimodal large model can be generated based on images and texts, and then the multimodal large model can be used to predict visual relationships and output text information of the visual relationships.

[0100] It should be noted that Figure 8 The process of visual relationship detection alone is only exemplified. In other implementations, image feature extraction algorithms, multimodal large models, etc. can also be combined in the image classification model. The image classification model can output not only multiple objects, but also the relative position relationship between multiple objects.

[0101] In the embodiment of the present application, the second model is trained based on the relative position relationship and a plurality of atomic action labels and a plurality of object labels. When training the second model, for each possible scenario, a set of labels (including at least one atomic action label, a plurality of object labels, and the relative position relationship between a plurality of objects) is determined, and the behavior to be output is set for this set of labels, and the plurality of sets of labels are used as training samples, and the behavior corresponding to each set of labels is used as the target result, and the second model is trained. In the embodiment of the present application, the network structure of the second model is not limited.

[0102] It should be noted that Figure 8 It is only exemplarily shown that the second model can be used to determine the behavior of the target object in the multiple frames of images. In other implementations, a matching rule can be preset, and the electronic device can determine the behavior of the target object in the multiple frames of images based on the relative position relationship, the label corresponding to the action of at least one object, the labels corresponding to multiple objects, and the preset matching rule.

[0103] The embodiment of the present application integrates the information of relative position relationship in the label, which can more effectively mine the semantics required for behavior recognition and improve the accuracy of behavior recognition.

[0104] The target object is explained below. If there is a single object in the video, or there is only one person among multiple objects, the single object or the person can be determined as the target object. However, in some scenes, two or more objects may perform atomic actions. If the interactive objects corresponding to the actions do not conflict, the behaviors of multiple objects can be determined.

[0105] After the behaviors of multiple objects are determined, when editing the video, the multiple objects can be used as target objects, and the behaviors of the multiple objects can be edited. Alternatively, multiple objects can be screened first, and the target object can be determined, and the behaviors of the target object can be edited.

[0106] In one implementation, the electronic device can determine the scene where multiple objects in the video are located through semantic recognition, divide the objects into primary objects and secondary objects according to the scene, and determine the primary object as the target object in the video. For example, taking a basketball game scene as an example, after identifying objects such as people (including basketball players and spectators), baskets, basketballs, and actions such as standing up, raising hands, patting, and throwing, it can be determined that the scene in the video is a basketball game scene, and the object that makes the throwing and patting actions is the target object (player), and the behavior of the target object in multiple frames of images is pitching, patting, etc.

[0107] In another implementation, the electronic device can receive action keywords input by the user, or receive behaviors input by the user. After determining the behaviors of multiple objects, the behaviors of the multiple objects are matched with the behaviors and / or action keywords input by the user, and the successfully matched behaviors are determined as the behaviors of the target objects, thereby editing the video. This method can realize video content retrieval and positioning.

[0108] In addition to the above-mentioned video analysis method, the embodiment of the present application also provides a video analysis method in combination with time information, which is described below.

[0109] After acquiring the video to be identified, the electronic device can determine the atomic action of at least one object and the first time period in which each atomic action occurs based on the multiple frames of images in the video to be identified, and determine the multiple objects in the multiple frames of images and the second time period in which each object appears. Further, for the first atomic action, based on the first time period in which the first atomic action occurs and the second time period in which each object appears, determine at least one object that appears in the time period in which the first atomic action occurs, and then determine the behavior of the target object based on the first atomic action and the at least one object that appears in the time period in which the first atomic action occurs.

[0110] In an embodiment of the present application, when training the above-mentioned action recognition model, the time labels of the atomic actions can also be marked to indicate in which video frames the atomic action occurs (first time period), and the action recognition model is trained based on the video, the labels of the atomic actions in the video, and the time labels. Thus, during video analysis, the video to be recognized is input into the action recognition model, and the action recognition model can not only output the label corresponding to at least one atomic action in the video, but also output the time label corresponding to each atomic action.

[0111] Similarly, when training the above-mentioned image classification model, the time label of the object is annotated to indicate in which video frames the object appears (the second time period), and the image classification model is trained based on the video, the label corresponding to each object in the video, and the time label of each object. Thus, when analyzing the video, the video to be identified is input into the image classification model, and the image classification model can output not only the label corresponding to each object in the video, but also the time label corresponding to each object.

[0112] The first atomic action is one of at least one atomic action. When a second time period with an object overlaps with the first time period in which the first atomic action occurs, at least one object appearing in the time period in which the first atomic action occurs can be determined. Furthermore, the electronic device determines the behavior of the target object based on the first atomic action, at least one object appearing in the time period in which the first atomic action occurs, and the first model / matching rule.

[0113] Alternatively, the electronic device may determine at least one object that appears within the time period when the first atomic action occurs, and the relative position relationship between these objects. Further, the electronic device determines the behavior of the target object based on the first atomic action, at least one object that appears within the time period when the first atomic action occurs, the relative position relationship between at least one object, and the second model / matching rule. The above scheme combined with time information can improve the accuracy of behavior recognition.

[0114] The following is a detailed description of atomic action recognition. Atomic action recognition has the following pain points:

[0115] 1) Inter-class similarity. The differences between different atomic action categories under the same background are local or even subtle. RGB-based methods are difficult to accurately capture action information in complex scenes. For example, in Fig. 9 (a) and Fig. 9 In the scene shown in (b), it is difficult to identify whether the behavior occurring in the illustrated background is a hug or clapping.

[0116] 2) Intra-class diversity. The same atomic action can generate different semantic information in different contexts. Fig.10 (a) and Fig.10 In the scene shown in (b), it is difficult to identify whether the behavior is riding a bicycle or riding a horse.

[0117] In response to the above pain point 1), a calculation method 1 is proposed in the embodiment of the present application: the electronic device determines the action of at least one object based on the position of the skeleton point corresponding to at least one object in multiple frames of images. This calculation method integrates the skeleton point information, pays more attention to the limb action itself, and can capture the subtle changes of the action through the change of the skeleton point position, thereby perceiving the differences between different action categories, and can improve the accuracy of action recognition.

[0118] In response to the above pain point 2), a second calculation method is proposed in the embodiment of the present application: the electronic device determines the action of at least one object based on the change in displacement of the pixel points corresponding to the at least one object in two adjacent frames of multiple frames over time. In this calculation method, the direction and intensity of the action are inferred by detecting the change in pixels at the same position in the video over time, which can identify the key features of the action, enhance the model's attention to the moving object, and improve the accuracy of action recognition.

[0119] The above two calculation methods can be used separately to determine the action of at least one object, or can be combined to jointly determine the action of at least one object.

[0120] The following introduces a solution for combining the above two calculation methods to jointly determine the action of at least one object. Fig.11 FIG. 1 is a flow chart showing a method of combining two calculation methods to determine atomic actions provided by an embodiment of the present application. Fig.11 As shown, the method includes: the electronic device determines at least one first action according to the position of the skeleton point corresponding to at least one object in multiple frames of images; determines at least one second action according to the change over time of the displacement of the pixel point corresponding to at least one object in two adjacent frames of images in the multiple frames of images; determines the action of at least one object according to the at least one first action and the at least one second action.

[0121] Specifically, for calculation method one, the electronic device converts multiple frames of images into a bone point heat map, and inputs it into the first convolutional neural network to determine the first action. The bone points are taken from the key joints in the human body. First, the bone points of each character in the multiple frames of images are extracted to obtain a bone point heat map. For example, a posture estimator can be used to perform 2D human posture extraction and extract bone points. The bone point heat map is then input into the first convolutional neural network. The first convolutional neural network stacks all the bone point heat maps along the time dimension according to a certain time step to form a 3D heat map body, and then uses a 3D convolution to process the 3D heat map body to identify at least one first action.

[0122] For calculation method two, the electronic device inputs multiple frames of images into the second convolutional neural network to determine the second action. In the embodiment of the present application, the second convolutional neural network uses Resnet-50 as the backbone network, which can be divided into five parts (not all of which are shown in the figure). The first part does not include a residual block, and mainly performs convolution, regularization, activation function, and maximum pooling calculations on the input. The second, third, fourth, and fifth structures all contain residual blocks, and a temporal shift module (TSM) is inserted into each residual block of Resnet-50. Each residual block includes multiple layers of convolution. TSM moves a part of the input feature channel on the time axis at the front of the convolution operation, so that the two-dimensional convolution obtains the effect of three-dimensional convolution, for example, in each TSM, 1 / 8 of the input feature channel is moved forward and another 1 / 8 of the channel is moved backward. A motion feature extraction module is inserted after the third stage of Resnet-50. The convolutional layer before the motion feature extraction module converts the input frame into a frame-related appearance feature F(t). The motion feature extraction module uses the paired adjacent appearance features to generate frame-by-frame motion features M(t); the obtained motion features are fused into the appearance features. After merging the two features, the combined features are passed through the next downstream layer to finally obtain at least one second action.

[0123] At least one first action and at least one second action are predictions made for the actions of at least one object using two calculation methods, respectively, wherein for each object, at least one first action and at least one second action are predicted. A first weight is assigned to calculation method one in advance, and a second weight is assigned to calculation method two. After obtaining the predicted probability of the first action and the predicted probability of the second action, the final atomic action is determined by weighted calculation. For example, for an object, the predicted at least one first action and its probabilities are respectively running (95%), walking (80%), and jumping (70%), and the predicted at least one second action and its probabilities are respectively walking (85%), running (80%), and jumping (70%). The first weight is 0.4, and the second weight is 0.6. Then the probabilities of the three actions are respectively: running (0.38+0.48=0.86), walking (0.32+0.51=0.83), and jumping (0.28+0.42=0.7), and finally the atomic action of the object is determined to be running. The embodiments of the present application utilize skeleton point heat maps and motion feature extraction to more effectively identify atomic actions that are unrelated to scenes and interactive objects.

[0124] The following is an explanation of the motion feature extraction module. The motion feature extraction module obtains the correlation tensor S(t) by calculating the correlation based on the two feature maps (F(t), F(t+1)) of the given two adjacent frames, and then uses the correlation tensor to estimate the displacement tensor D(t). Through the transformation process of the convolution layer, the final motion feature M(t) is obtained to estimate the motion between the two frame-by-frame feature maps of the two adjacent frames.

[0125] Correlation calculation: Two adjacent input feature maps are represented by F(t) and F(t+1), each of which is a three-dimensional tensor of size H×W×C. H×W×C is the height, width, and number of channels of the input. The spatial resolution is H×W, and the C-dimensional feature at the spatial position x is Fx. The correlation score of the pixel position x relative to the displacement p is defined as:

[0126]

[0127] Where · represents the dot product and t represents time. p∈[-k,k] 2 , the correlation score of position x is calculated only in the neighborhood of size P = 2k + 1, that is, for each pixel at position x, a fixed-size area is divided with the pixel as the center, and the maximum displacement of the pixel does not exceed the area. The correlation between the pixel and other pixels is calculated in the area. For each pixel, the number of pixel pairs to be calculated is (2k + 1) 2 For t th frame, the size of the obtained tensor S(t) is H×W×P 2 The correlation calculation is similar to performing convolution between two adjacent frames of images. th Frame features and t+1 th The features of the frame are mapped.

[0128] Displacement estimation: In the embodiment of the present application, the displacement of position x is calculated by the Softmax function and the argmax function, and the calculation method is as follows: Formula (2) and Formula (3).

[0129]

[0130] in,

[0131]

[0132] g(x,p,t) is the Gaussian kernel function, τ is the temperature factor that adjusts the softmax distribution. As τ decreases, softmax approaches argmax.

[0133] In addition, a correlation confidence map is used as auxiliary motion information, which is obtained by pooling the highest correlation at each position x, as shown in Equation (4).

[0134]

[0135] The displacement map and confidence map are concatenated into a displacement tensor D(t) of size H×W×3 for the next step of motion feature conversion.

[0136] Feature transformation: The displacement tensor D(t) is input into four depth-wise separable convolutional layers conv, a 1×7×7 layer, followed by three 1×3×3 layers, and converted into motion features M(t) with the same number of channels C as the original input F(t). This feature transformation process aims to interpret the semantics of displacement and confidence through the convolutional layer and learn motion features. In summary, the motion feature extraction module generates motion features M(t) by calculating the displacement of pixels using the appearance features F(t) and F(t+1) of two adjacent frames.

[0137] The above-mentioned first convolutional neural network and second convolutional neural network are merely exemplary descriptions, and the present application may also use other neural network structures without limitation.

[0138] In summary, when performing video analysis, the embodiments of the present application can respectively determine multiple objects in multiple frames of images and the actions of at least one of the multiple objects after obtaining the video to be identified, and then combine the actions of at least one object with multiple objects to determine the behavior of the target object in the multiple frames of images. Among them, the action or object has generalization, and the same action combined with different objects, or the same object combined with different actions, can obtain new behaviors. Therefore, the present application analyzes the behavior in the video by splitting the method, which enhances the flexibility of complex behavior recognition and makes it easier to expand new behaviors.

[0139] The electronic device provided in this embodiment is used to execute the above method, so the same effect as the above implementation method can be achieved. In the case of an integrated unit, the electronic device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the electronic device, for example, it can be used to support the electronic device to execute the steps performed by the processing unit. The storage module can be used to support the electronic device to execute stored program codes and data, etc. The communication module can be used to support the communication between the electronic device and other devices.

[0140] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0141] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0142] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0143] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In the above description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0148] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0149] It should also be understood that the "multiple" mentioned in the specification and the appended claims of this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, which means any combination of one or more of the items listed in the association and all possible combinations, and includes these combinations, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0150] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0151] In addition, in order to clearly describe the technical solution of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and cannot be understood as indicating or implying relative importance, and words such as "first" and "second" do not necessarily limit differences.

[0152] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0153] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A video analysis method, It is characterized in that The method comprises: Acquire a video to be identified, where the video to be identified includes multiple frames of images; Determine a plurality of objects in the plurality of frames of images and an action of at least one of the plurality of objects, wherein the at least one object includes a target object, and the action of the target object acts on other objects of the plurality of objects except the target object; According to the action of the at least one object and the multiple objects, a behavior of the target object in the multiple frames of images is determined.

2. The method according to claim 1, It is characterized in that The determining, according to the action of the at least one object and the multiple objects, the behavior of the target object in the multiple frames of images comprises: The label corresponding to the action of the at least one object and the labels corresponding to the multiple objects are input into the first model to determine the behavior of the target object in the multiple frames of images.

3. The method according to claim 1 or 2, It is characterized in that After determining the multiple objects in the multiple frames of images, the method further includes: Determining the relative positional relationship between the plurality of objects; The determining, according to the action of the at least one object and the multiple objects, the behavior of the target object in the multiple frames of images comprises: The behavior of the target object in the multiple frames of images is determined according to the relative position relationship, the action of the at least one object and the multiple objects.

4. The method according to claim 3, It is characterized in that The determining, according to the relative position relationship, the action of the at least one object and the multiple objects, the behavior of the target object in the multiple frames of images comprises: The relative position relationship, the label corresponding to the action of the at least one object, and the labels corresponding to the multiple objects are input into a second model to determine the behavior of the target object in the multiple frames of images.

5. The method according to any one of claims 1 to 4, It is characterized in that Determining an action of at least one of the plurality of objects includes: The action of the at least one object is determined according to the positions of the skeleton points corresponding to the at least one object in the multiple frames of images.

6. The method according to any one of claims 1 to 5, It is characterized in that Determining an action of at least one of the plurality of objects includes: The action of the at least one object is determined according to the change over time of the displacement of the pixel points corresponding to the at least one object in two adjacent frames of the multiple frames of images.

7. The method according to any one of claims 1 to 6, It is characterized in that Determining an action of at least one of the plurality of objects includes: Determine at least one first action according to positions of the skeleton points corresponding to the at least one object in the multiple frames of images, where the at least one first action is a prediction of an action of the at least one object; Determine at least one second action according to a change in displacement of a pixel point corresponding to the at least one object in two adjacent frames of the multiple frames of images over time, where the at least one second action is a prediction of an action of the at least one object; A motion of the at least one object is determined according to the at least one first motion and the at least one second motion.

8. An electronic device, It is characterized in that include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Video editing method and device, electronic equipment and storage medium

    CN121357351A

  • A video editing method, apparatus, electronic device, and storage medium

    CN121357351B