Action recognition method and device, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application provide a kind of action recognition method and device, electronic equipment and computer readable storage medium.The method comprises: constructing preset feature library;Obtain the video data to be identified;The first feature vector of the identification space action relationship of each video frame in the video data to be identified is matched with the second feature vector in the preset feature library, to obtain the interactive action category in the video data to be identified, wherein the preset feature library includes multiple first feature vectors and the interactive action category corresponding to the second feature vector.This embodiment of the present application uses the preset feature library to identify the action category of the space action relationship in video frame based on the space action relationship of frame level, so as to reduce the dependence of the annotation amount and the consumed computing resources, improve the accuracy of the capture of interactive action.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an action recognition method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] In the field of video or image acquisition, while it's possible to identify target objects by recognizing their features within captured images or video frames, this technology only identifies their identity. With the widespread application of video or image acquisition technology in people's lives and work, there's a growing need to identify the actions of target objects, especially their interactive actions. For example, the detection of public emergencies and abnormal behavior relies heavily on the accurate identification of object interactions. Furthermore, the recognition of object interactions plays a significant role in video conferencing, medical clinical diagnosis, and other fields, showing broad application prospects.

[0003] Specifically, existing technologies have proposed determining the action category of each video frame based on the node sequence information corresponding to N consecutive video frames in the acquired video data, thereby determining the interaction action performed by the interactive object based on the action category. However, such action recognition methods based on consecutive video frames require acquiring the node information of consecutive video frames and performing interaction action recognition on each video frame, resulting in a large amount of annotation, high hardware requirements, and limited applicability. Therefore, a technical solution that can efficiently and accurately identify interactive actions is needed. Summary of the Invention

[0004] This application provides an action recognition method and apparatus, an electronic device, and a computer-readable storage medium to solve the problems of high hardware requirements and large annotation volume in the prior art.

[0005] According to a first aspect of the embodiments of this application, an action recognition method is provided, including: Build a pre-defined feature library; Obtain the video data to be identified; The first feature vector of the spatial action relationship of each video frame in the video data to be identified is matched with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified. The preset feature library includes multiple second feature vectors and interactive action categories corresponding to the second feature vectors.

[0006] Furthermore, in the action recognition method according to the embodiments of this application, the step of matching the first feature vector with the second feature vector in the preset feature library to obtain the object interaction action category in the video data to be recognized includes: The first feature vector is compared with the second feature vector in the preset feature library to obtain at least one interactive action category; The at least one type of interactive action is filtered based on the first structured attribute and the corresponding first confidence level, the second structured attribute and the corresponding second confidence level, and a preset common sense base; Based on the filtering results, the interactive action category with the most selections is determined as the interactive action category in the video data to be identified.

[0007] Furthermore, in the action recognition method according to the embodiments of this application, the step of selecting video frames containing a first object and a second object from the video data to be recognized to form a spatiotemporal scene group based on the temporal and spatial information of objects in each video frame of the video data to be recognized includes: Video frames of the first object and the second object within a common time period are obtained from the video data to be identified, and these frames are used as the first video frame sequence and the second video frame sequence; wherein, the common time period is the time period during which the first object and the second object interact.

[0008] Furthermore, the action recognition method according to the embodiments of this application may further include: Based on the interactive action category, search a preset video library for video frames containing interactive actions corresponding to the interactive action category; The found video frames are output together with the corresponding interactive actions of the interactive action category through a preset interactive interface.

[0009] According to a second aspect of the embodiments of this application, an action recognition device is provided, comprising: The building module is used to build a pre-defined feature library; The acquisition module is used to acquire the video data to be identified; The matching module is used to match the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified. The preset feature library includes multiple second feature vectors and interactive action categories corresponding to the second feature vectors.

[0010] This application also provides an electronic device, including: Memory, used to store programs; A processor is used to run the program stored in the memory, and the program executes the action recognition method provided in the embodiments of this application when it runs.

[0011] This application also provides a computer-readable storage medium storing a computer program executable by a processor, wherein the program, when executed by the processor, implements the action recognition method provided in this application.

[0012] The action recognition method, apparatus, electronic device, and computer-readable storage medium provided in this application identify interactive action categories by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Therefore, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between feature vectors of multiple spatial action relationships and interactive action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interactive actions.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0014] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1a A schematic diagram illustrating the principle of the action recognition scheme provided in this application embodiment; Figure 1b This diagram illustrates a motion recognition scheme according to an embodiment of the present application, where a target frame is located based on an object frame. Figure 1c A schematic diagram illustrating examples of spatial action relationships in an action recognition scheme according to an embodiment of this application is shown. Figure 2 A flowchart of an embodiment of the action recognition method provided in this application; Figure 3 A flowchart of another embodiment of the action recognition method provided in this application; Figure 4 A flowchart of yet another embodiment of the action recognition method provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the motion recognition device provided in this application; Figure 6 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0015] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0016] Example 1 The solution provided in this application can be applied to any system with image recognition capabilities, such as a computing system that includes a video or image recognition module, etc. Figure 1a This is a schematic diagram illustrating the principle of the action recognition scheme provided in the embodiments of this application. Figure 1a The schematic diagram shown is merely one example of the principle of the technical solution of this application.

[0017] With the development of video / image recognition technology, users are no longer satisfied with simply identifying the identities of objects in captured images; they want to obtain more information about the target objects. For example, in service scenarios that provide various services to users, the standardization of employee actions directly affects the quality of the services provided. Therefore, current technologies typically rely on on-site supervisors to monitor and guide employees, which is not only labor-intensive but also heavily dependent on the supervisors' technical skills, experience, and diligence. Therefore, existing technologies have proposed applying video / image recognition technology to such on-site supervision and guidance scenarios. For instance, one approach involves extracting information about each joint of the target object from N consecutive video frames in the captured video data as node sequence information. Based on this node sequence information, the action category corresponding to each video frame is determined, thereby identifying the interactive action performed by the target object on-site. However, existing motion recognition methods based on continuous video frames require first identifying each joint of the object in the video frame as a node, and then obtaining the node information. Furthermore, they require interactive motion recognition for each video frame based on the already labeled actions. Since human actions are diverse and varied, the amount of labeling required is very large.

[0018] Therefore, in the action recognition scheme of this application embodiment, pre-collected video data containing various object interaction actions can be labeled, and the labeled data can be used to form a training dataset. By training on this training dataset, the correspondence between feature vectors of the video data and interaction action categories can be obtained. Finally, a feature library is constructed based on this correspondence. Thus, when identifying the category of object interaction actions from the video data to be identified, it is only necessary to obtain the feature vectors from the video data to be identified and compare them with the feature vectors in the preset feature library to obtain the category of the interaction action in the video data to be identified. For example, when constructing the feature library, one can first... Figure 1a The diagram illustrates how to extract two frames from pre-captured motion video frames in which objects have an interactive relationship to construct an action genome. Figure 1a A schematic diagram of the action genome used by the action recognition scheme provided in this application embodiment is shown.

[0019] Specifically, when building a feature library, you can first obtain video data containing object interaction actions in different scenarios, such as residential community scenarios, public transportation scenarios, and public office area scenarios, and then annotate the interaction actions between target objects in the collected video data, or you can directly obtain the annotation information of video frames.

[0020] For example, such as Figure 1a As shown, n video frames can be acquired for the first object o1 and the second object o2 respectively. For example, the first object o11 is identified in the first video frame of the first object, and the second object o21 is identified in the first video frame of the second object. The spatial action relationship r11 between these two objects is obtained accordingly, that is, the spatial action relationship between these two objects is that they are slightly in contact with each other. Similarly, the first object o12 is identified in the second video frame of the first object, and the second object o22 is identified in the second video frame of the second object. The spatial action relationship r12 between these two objects o12 and o22 is obtained accordingly, that is, the interaction between these two objects is that they almost overlap, for example, they are embracing each other. And so on, the first object o1n is identified in the nth video frame of the first object, and the second object o2n is identified in the nth video frame of the second object. The spatial action relationship r1n between these two objects o1n and o2n is obtained accordingly, that is, the interaction between these two objects is that they are moving away from each other. Therefore, these n frames can constitute the action genome of the first and second objects, and can be labeled, for example, as hugging actions, or the annotation information of these frames that have been pre-annotated with the interaction action categories of the objects in the video frames can be directly obtained.

[0021] In addition, to make annotation more efficient, interactive actions can be pre-categorized, and annotators can select from the pre-set categories during the actual annotation process. If the actual interactive action in the video data does not belong to the pre-set category, the annotator can manually enter the specific category of the interactive action.

[0022] For example, taking a residential community scenario, five categories of interactive actions can be pre-defined as follows: Conversation: Two people are talking to each other.

[0023] Hugging, embracing: holding a child, carrying a child on one's back, hugging each other, one person riding a vehicle with another person on the back, etc.

[0024] Holding hands or shaking hands: Two people hold hands or shake hands.

[0025] Pushing: pushing a stroller, a wheelchair, a children's vehicle, etc.

[0026] Play: Various forms of "kids-out" activities, such as adults playing with children together.

[0027] After annotators identify the interactive actions, they use machine learning methods to learn the features of these actions and then use the model to determine the interactive actions in unknown video data. Therefore, after completing the category labeling of interactive actions in video data, a video data and its corresponding interactive action category constitute a training sample, and multiple training samples constitute the training dataset.

[0028] After obtaining the training dataset, the following operations can be performed on each training sample in the training dataset: The process involves acquiring a first video frame sequence of a first object and a second video frame sequence of a second object from the training samples. M frames are then sampled from each of the first and second video frame sequences. For each frame in the first video frame sequence, the spatiotemporal information of the first object, its first structured attribute, and the corresponding first confidence score are obtained. Similarly, for each frame in the second video frame sequence, the spatiotemporal information of the second object, its second structured attribute, and the corresponding second confidence score are obtained. The spatiotemporal information of the first object includes the first shooting time and the detection bounding box of the first object. The spatiotemporal information of the second object includes the second shooting time and the detection bounding box of the second object. The first structured attribute includes the clothing color, hat color, etc., of the first object, and the second structured attribute includes the clothing color, hat color, etc., of the second object. The confidence score is a probability value, typically between 0 and 1. Each structured attribute corresponds to a confidence score, which characterizes the credibility of that attribute. A higher confidence score indicates higher credibility of the attribute.

[0029] Therefore, in this embodiment, if a frame in the video frame sequence containing the first object is taken as the object frame, then the frame found in the video frame sequence containing the second object that corresponds to the video frame containing the first object is called the neighboring target frame. If a frame in the video frame sequence containing the second object is taken as the object frame, then the frame found in the video frame sequence containing the first object that corresponds to the video frame containing the first object is called the neighboring target frame.

[0030] For example, such as Figure 1a As shown, the video frame sequence containing the first object is a1,...,an, and the video frame sequence containing the second object is b1,...,bn. By combining the video frames of these two objects in the manner described above, an action genome is constructed. For example, the number of frames containing the first and second objects within a common time period (i.e., the time period in which the interactive action occurs) can be calculated first. The video frame sequence containing the object with fewer frames is taken as the object frame. For each of these frames, a corresponding video frame is found in another video frame sequence as its neighboring target frame. This frame and its neighboring target frame form a pair.

[0031] Furthermore, if the number of frames within the time period in which object interaction occurs is the same, the video frame sequence containing the object with the shortest time among the selected frames can be used as the object frame sequence.

[0032] Furthermore, in this embodiment, two conditions must be met simultaneously to form a neighboring target frame: first, the time difference between the object frame and the target frame must be less than or equal to 0.1 seconds, which is an adjustable parameter but generally cannot exceed 1 second; second, the target frame can only be combined with a maximum of one object frame to form a pair. If either of the above conditions is not met, the target frame of the object frame will be empty.

[0033] like Figure 1b As shown, Figure 1b This diagram illustrates the process of finding a target frame based on an object frame in an action recognition scheme according to an embodiment of this application. Figure 1b In the diagram, small squares represent a frame, and the horizontal axis represents the time axis. If the coordinates are the same, it means the frames are at the same time; the difference in the horizontal coordinates represents the time difference between the frames. For example, ... Figure 1b As shown, the video frame sequence containing the first object is a1, a2, a3, a4, a5, a(M-1), aM, and the video frame sequence containing the second object is b1, b2, b3, b4, b5, b(M-1), bM. Within a common time period, the number of frames containing the first object is 5, and the number of frames containing the second object is 6. Therefore, the video frames containing the first object constitute the object frame sequence, and the video frames containing the second object constitute the target frame sequence.

[0034] Furthermore, according to the embodiments of this application, for object frame a3 within a common time period, a corresponding video frame b1 can be found in the target frame sequence as a neighboring target frame of object frame a3. For object frame a4 within a common time period, a corresponding video frame b2 can be found in the target frame sequence as a neighboring target frame of object frame a4. For object frame a5 within a common time period, a corresponding video frame b5 can be found in the target frame sequence as a neighboring target frame of object frame a5. For object frame a(M-1) within a common time period, a corresponding video frame b(M-1) can be found in the target frame sequence as a neighboring target frame of object frame a(M-1). However, for object frame aM within a common time period, if no target frame meeting the above conditions is found in the target frame sequence, the target frame of object frame aM can be empty.

[0035] Therefore, in the embodiments of this application, object frame a3 and neighboring target frame b1 can constitute the spatiotemporal scene group of the frame, object frame a4 and neighboring target frame b2 can constitute the spatiotemporal scene group of the frame, object frame a5 and neighboring target frame b5 can constitute the spatiotemporal scene group of the frame, and object frame a(M-1) and neighboring target frame b(M-1) can constitute the spatiotemporal scene group of the frame. Thus, as shown... Figure 1b The action genomes of the first and second objects shown are: {'a1': NaN, 'a2': NaN, 'a3': 'b1', 'a4': 'b2', 'a5': 'b5', 'a(M-1)': 'b(M-1)', 'aM': NaN}.

[0036] Furthermore, in this embodiment, for each spatiotemporal scene group, the positional relationship r between the detection bounding boxes of the first object and the second object can be calculated, such as... Figure 1c As shown, Figure 1c A schematic diagram illustrating examples of spatial action relationships in an action recognition scheme according to an embodiment of this application is shown. For example... Figure 1c In the example categories shown, the spatial action relationship r between the first object and the second object can include: up-down (intersecting), up-down (not intersecting), bottom left-top right (intersecting), bottom left-top right (not intersecting), left-right (intersecting), left-right (not intersecting), top left-bottom right (intersecting), top left-bottom right (not intersecting), and containing-being contained.

[0037] Since some M-frame video frames may contain empty target frames, retaining this portion of video frame data would cause interference. Therefore, in this embodiment, after calculating the spatial action relationship r, the spatial action relationship r corresponding to N video frames can be filtered from the M-frame spatiotemporal scene group. For example, if r has M elements, and the target frame of a certain element rK is empty, it is removed. If the number of remaining elements is greater than or equal to N, then N more elements are selected from them, for example, elements whose bounding boxes intersect. If the number of remaining elements is less than N, the sample is discarded as an invalid sample. After obtaining the sample, the spatial action relationship r corresponding to the N video frames filtered from the M-frame spatiotemporal scene graph can be aggregated into an N-dimensional feature vector corresponding to the video data of the training sample. In this way, a feature vector corresponding to a training sample is obtained.

[0038] Furthermore, the action recognition scheme according to the embodiments of this application, because it uses a preset feature library to identify action types based on spatial action relationships, can be widely applied in various fields. For example, in the gaming field, multiple user / player action video data, especially action interaction data between users, such as arm crossing, can be captured by image acquisition devices such as cameras. The feature vectors identifying spatial action relationships in the video frames of such interactive video data can be extracted and matched with feature vectors corresponding to the interactive action categories in a pre-established feature library. Thus, the interactive actions between users can be determined based on the matching results, and the determined interactive actions can then be fed back into the game. This scheme is particularly suitable for multiplayer interactive games, such as multiplayer sports games. The action recognition scheme of the embodiments of this application can be used to capture interactive action videos between multiple users, and the interactive actions between users can be fed back to the corresponding characters in the game through matching with the feature library. In particular, because the action recognition scheme of the embodiments of this application reduces the amount of annotation and the demand for computing resources by utilizing a preset feature library, it can greatly improve the efficiency of action type recognition and greatly improve the user experience in real-time feedback in action games.

[0039] Furthermore, the action recognition scheme according to the embodiments of this application can also be applied to sports teaching and training. It can acquire practice videos of students receiving teaching and training or collect real-time practice videos of students through cameras, etc., and collect the actions of each video frame in the video, especially the action interactions between multiple students. By extracting the feature vectors that identify spatial action relationships in the video frames obtained in this way, it matches them with the feature vectors corresponding to the action categories in a pre-established teaching action feature library. Based on the matching results, it can determine the similarity between the user's spatial actions and the standard actions in the teaching action feature library in terms of spatial relationship. Based on the similarity or difference points, it can issue prompts to the user. For example, it can mark the difference parts in the user's action video frames, or it can further provide correction suggestions. Since the action recognition scheme of this application requires less annotation and less computing resources, it is not only highly efficient in recognition but also has low application costs, which is conducive to its promotion in sports training.

[0040] Furthermore, the action recognition scheme according to the embodiments of this application can also be applied to film shooting scenarios. For example, a film shooting action feature library can be established in advance based on the film script. This feature library can store the correspondence between action feature vectors and feature vectors in video frames. Therefore, during shooting or after shooting for a certain period of time, the feature vectors that identify spatial actions, especially the action feature relationships between multiple actors, in the video frames of the video data captured in the real-time shooting are matched with the feature vectors in the aforementioned pre-established film shooting action feature library. Based on the matching results, the action matching results are output to the director and / or actors. For example, the video frames can be scored based on the similarity or difference between the spatial actions in the video frames and the features in the shooting action feature library, or the differences can be further marked in the video frames for the director or actors to refer to. Therefore, since the action recognition scheme of the embodiments of this application requires less annotation and less computing resources, and can also ensure the accuracy of action type recognition, it can efficiently and accurately assist film shooting and save the director or actors the workload of supervising the shooting quality.

[0041] Furthermore, in this embodiment, for each training sample in the training dataset, the above operation can be repeated to generate a feature vector corresponding to each training sample, thereby obtaining a preset feature library that can be used by the action recognition scheme of this application. This preset feature library contains the correspondence between feature vectors and object interaction action categories. In this way, after obtaining the video data to be recognized, the feature vectors in the video data to be recognized can be obtained, and then compared with the feature vectors in the preset feature library to obtain the recognition result of the object interaction action in the video data to be recognized.

[0042] The object interaction action recognition scheme provided in this application identifies interaction action categories by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Therefore, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between feature vectors of multiple spatial action relationships and interaction action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interaction actions.

[0043] The above embodiments illustrate the technical principles and exemplary application framework of the embodiments of this application. The specific technical solutions of the embodiments of this application will be further described in detail below through multiple embodiments.

[0044] Example 2 Figure 2 This is a flowchart of one embodiment of the action recognition method provided in this application. The subject executing this method can be various terminal or server devices with image recognition capabilities, or it can be a device or chip integrated into these devices. Figure 2 As shown, the action recognition method includes the following steps: S210, Construct a preset feature library.

[0045] In this embodiment, a corresponding feature library can be constructed first based on the video to be identified or the type of video. In particular, the feature library can pre-store the correspondence between feature vectors identifying spatial action relationships in video frames of video data and feature vectors of predetermined action categories. For example, video frames with annotation information can be obtained in advance through manual annotation, and the feature library can be established in step S210 based on the video frames and annotation information.

[0046] S220, acquire the video data to be identified.

[0047] In this embodiment, video data to be identified can be obtained from various video data sources in step S220. This video data can be, for example, video captured by video / image acquisition devices installed in various areas such as shopping malls, buses, and public office areas. The video data to be identified can be real-time video data or video data acquired at preset time intervals. For example, video data can be acquired from a camera every 24 hours as the video data to be identified. Alternatively, video data can be acquired at any other time interval as needed, for example, requesting video data to be identified from a camera under user control. Furthermore, the video data to be identified in this embodiment can be video data containing target objects, particularly video data containing multiple target objects. For example, video data collected regarding the service status of a parcel sorting area.

[0048] S230, the first feature vector of the spatial action relationship of each video frame in the video data to be identified is matched with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified.

[0049] In step S230, the feature vectors of spatial action relationships of each video frame obtained in step S220 can be matched with the feature vectors in the preset feature library established in step S210. For example, the similarity between features can be calculated, and the interaction action category corresponding to the second feature vector with a similarity greater than a preset threshold can be taken as the action category of the video data to be identified.

[0050] Therefore, the object interaction action recognition method provided in this application embodiment identifies the interaction action category by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Thus, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between multiple spatial action relationships and feature vectors of interaction action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interaction actions.

[0051] Example 3 Figure 3 This is a flowchart of one embodiment of the action recognition method provided in this application. The subject executing this method can be various terminal or server devices with image recognition capabilities, or it can be a device or chip integrated into these devices. Figure 3 As shown, the action recognition method includes the following steps: S301, process the pre-acquired motion video frames to obtain the second feature vector.

[0052] In this embodiment, a corresponding feature library can be constructed first based on the video or video type to be identified. Specifically, this feature library can pre-store the correspondence between feature vectors identifying spatial action relationships in video frames and feature vectors representing predetermined action categories. For example, in step S301, the action video frames in the pre-acquired video data can be processed to obtain second feature vectors identifying the spatial action relationships in these action video frames. Specifically, these action video frames can be pre-selected or automatically filtered according to predetermined criteria, and these video frames contain action information that has been automatically or manually annotated. Therefore, the feature vectors reflecting spatial action relationships in such video frames can be used as reference feature vectors.

[0053] S302, Obtain annotation information for action video frames.

[0054] After obtaining the second feature vector of the action video frame in step S301, the annotation information of these video frames can be further obtained in step S302. In particular, the annotation information can be used to identify the interactive action category of the object in the action video frame.

[0055] S303, the second feature vector is matched with the interactive action category to construct a preset feature library.

[0056] Therefore, in step S303, a feature library can be established based on the annotation information for video frames obtained in step S302, especially the annotation information that identifies the interactive action category of the object in the action video frame obtained in step S301, and the second feature vector that identifies the spatial action relationship in the action video frame obtained by processing the action video frame in step S301. Thus, the feature library can include the action vectors of multiple video frames and their correspondence with the interactive action category.

[0057] S304, Obtain the video data to be identified.

[0058] In this embodiment, after establishing a preset feature library, video data to be identified can be obtained from various video data sources in step S304. This video data can be, for example, video captured by video / image acquisition devices installed in various areas such as shopping malls, buses, and public office areas. The video data to be identified can be real-time video data or video data acquired at preset time intervals. For example, video data can be acquired from a camera every 24 hours as the video data to be identified. Alternatively, video data can be acquired at any other time interval as needed, for example, requesting video data from a camera under user control. Furthermore, the video data to be identified in this embodiment can be video data containing target objects, particularly video data containing multiple target objects. For example, video data collected regarding the service status of a parcel sorting area.

[0059] S305, based on the temporal and spatial information of the objects in each video frame of the video data to be identified, select video frames containing the first object and the second object from the video data to be identified to form a spatiotemporal scene group.

[0060] After obtaining the video data to be identified in step S304, object recognition can be performed on each video frame obtained by dividing the video data. In this embodiment, various commonly used object recognition schemes can be used to identify objects in each video frame. Thus, the time and spatial information of the objects identified in the video frame can be used in step S305. For example, in the above-mentioned express delivery sorting service scenario, the time information of each courier in the acquired image in the video frame, such as 1:12:24 pm, and spatial information, such as the position coordinates of the object in the frame, can be used.

[0061] Furthermore, in step S305, video frames containing both a first object and a second object are selected based on the temporal and spatial information of the objects identified in each frame, thus forming a spatiotemporal scene group. In this embodiment, a spatiotemporal scene group is an abstract group used to describe the spatial relationships between objects in an image. For example, objects with the same temporal information can be selected based on the temporal information of the objects contained in the frame, and video frames that are spatially related can be selected based on the spatial information, thereby forming a spatiotemporal scene group containing these two objects.

[0062] S306, Based on each spatiotemporal scene group, determine the spatial action relationship between the first object and the second object.

[0063] After obtaining the spatiotemporal scene group containing the first object and the second object in step S305, the spatial action relationship between the first object and the second object can be further determined based on these spatiotemporal scene groups from step S305. For example, the spatial information of the objects in each frame is used when constructing the spatiotemporal scene group in step S305, so the spatial action relationship between the two objects can be determined based on this spatial information in step S306. For example, the spatial action relationship between the first object and the second object can include: up-down (intersecting), up-down (non-intersecting), lower left-upper right (intersecting), lower left-upper right (non-intersecting), left-right (intersecting), left-right (non-intersecting), upper left-lower right (intersecting), upper left-lower right (non-intersecting), and containing-being contained. As can be seen from the above example, these spatial action relationships can be determined based on the spatial position of the objects.

[0064] S307 aggregates the spatial motion relationships of each video frame into a first feature vector.

[0065] In step S307, video frames with such spatial action relationships can be aggregated into multi-dimensional feature vectors based on the spatial action relationship between the first object and the second object obtained in step S306, so as to match them with various feature vectors in a pre-set feature library.

[0066] S308, Match the first feature vector with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified.

[0067] In step S308, the feature vectors obtained in step S307 by aggregating the spatial action relationships of each frame in step S306 can be matched with feature vectors in a preset feature library. For example, the similarity between features can be calculated, and the interaction action category corresponding to the second feature vector with a similarity greater than a preset threshold can be used as the action category of the video data to be identified.

[0068] Therefore, the object interaction action recognition method provided in this application embodiment identifies the interaction action category by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Thus, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between multiple spatial action relationships and feature vectors of interaction action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interaction actions.

[0069] Example 4 Figure 4 This is a flowchart of another embodiment of the action recognition method provided in this application. The execution subject of this method can be various terminals or server devices with image recognition capabilities, or it can be a device or chip integrated into these devices. Figure 4 As shown, the action recognition method includes the following steps: S400, build a preset feature library.

[0070] In this embodiment, a corresponding feature library can be constructed first based on the video to be identified or the type of video. In particular, the feature library can pre-store the correspondence between feature vectors identifying spatial action relationships in video frames of video data and feature vectors of predetermined action categories. For example, video frames with annotation information can be obtained in advance through manual annotation, and the feature library can be established in step S400 based on the video frames and annotation information.

[0071] S410, acquire the video data to be identified.

[0072] In this embodiment, the video data to be identified can first be obtained from various video data sources in step S310. This video data can be, for example, video captured by video / image acquisition devices installed in various areas such as shopping malls, buses, and public office areas. The video data to be identified can be real-time video data or video data acquired at preset time intervals. For example, video data can be acquired from a camera every 24 hours as the video data to be identified. Alternatively, video data can be acquired at any other time interval as needed, for example, requesting video data to be identified from a camera under user control. Furthermore, the video data to be identified in this embodiment can be video data containing target objects, particularly video data containing multiple target objects. For example, video data collected regarding the service status of a parcel sorting area.

[0073] S420: Obtain a first video frame sequence containing a first object and a second video frame sequence containing a second object from the video data to be identified.

[0074] After obtaining the video data to be identified in step S410, object recognition can be performed on each video frame obtained by dividing the video data. In this embodiment, various commonly used object recognition schemes can be used to identify objects in each video frame, and multiple video frames containing a first object are selected to form a first video frame sequence, and multiple video frames containing a second object are selected to form a second video sequence based on the identified objects. For example, in a catering service scenario, video frames containing a service worker can be selected from the acquired catering service video frames to form a first video frame sequence, and video frames containing a customer can be selected to form a second video frame sequence. In this embodiment, each video frame in the first video frame sequence only needs to contain at least the service worker, not just the service worker. For example, video frames containing both the service worker and the store manager can be selected as video frames in the first video frame sequence, or video frames containing both the service worker and the customer forming the second video frame sequence can be selected as video frames in the first video frame sequence. Similarly, when selecting each video frame constituting the second video frame sequence, each video frame in the second video frame sequence only needs to contain at least the customer. Specifically, in this embodiment, video frames with a common time period can be selected first, and then video frames that at least contain a first object can be selected from these video frames as a first video frame sequence and video frames that at least contain a second object can be selected as a second video frame sequence. Alternatively, video frames that at least contain a first object and video frames that at least contain a second object can be selected first, and then video frames with a common time period can be further selected from these video frames as the first video frame sequence and the second video frame sequence.

[0075] S430, based on the first video frame sequence and the second video frame sequence, obtain the spatiotemporal scene group of the first object and the second object in each video frame.

[0076] Therefore, after selecting a first video frame sequence that includes at least a first object, such as a service worker, and a second video frame sequence that includes at least a second object, such as a customer, in step S420, the action recognition method according to this application embodiment can use the temporal and spatial information of the objects identified in the video frames in step S430. For example, in the above-mentioned catering service scenario, the temporal information of the service worker in each video frame of the first video frame sequence, such as 1:12:24 PM, and spatial information, such as the position coordinates of the object in that frame, can be used. Based on this temporal and spatial information, video frames containing both the first and second objects can be selected in step S330 to form a spatiotemporal scene group, that is, a scene group in which the service worker provides services to the customer. In particular, in this application embodiment, a spatiotemporal scene group is an abstract group used to describe the spatial action relationship between objects in an image. For example, in the above-mentioned catering service scenario, a spatiotemporal scene group containing the corresponding video frames of the service worker and the customer can represent the spatial action relationship of the service worker providing services to the customer, such as serving food to the customer, the customer asking the service worker about the menu, and other interactive action scenarios. Therefore, based on the temporal information of the two objects contained in each frame, objects with the same temporal information can be selected, and video frames that are spatially related can be selected based on spatial information, thereby forming a spatiotemporal scene group containing the two objects.

[0077] Specifically, in this embodiment, one of the two video frame sequences determined in step S420 can be selected as the object frame sequence. For example, in the above-mentioned catering service scenario, the first video frame sequence containing service personnel can be selected as the object frame sequence. Then, for each video frame in the object frame sequence, the object involved in the object frame sequence, such as the video frame of the service personnel, is searched in the other video frame sequence, i.e., the second video frame sequence containing customers, as the neighboring target frame. The object frame and the corresponding neighboring target frame form a spatiotemporal scene group of the first object and the second object in that frame. Therefore, in this way, video frames containing the customer and the service personnel that overlap or are close in time can be found. Because these video frames overlap or are close in time and also contain customers and service personnel, they can represent a video frame group in which the service personnel interact with the customer.

[0078] S440, obtain the first object spatiotemporal information, the first structured attribute and the corresponding first confidence level for each frame in the first video frame sequence.

[0079] S450, obtain the second object spatiotemporal information, second structured attribute and corresponding second confidence level for each frame in the second video frame sequence.

[0080] In this embodiment, spatiotemporal information, structured attributes, and confidence levels can be further obtained for each frame in the two video frame sequences. Specifically, the object spatiotemporal information can include the capture time of the object identified in the video frame and the object's detection bounding box. In this embodiment, the detection bounding box can be obtained based on the difference between the object's pixels and the pixels of its surrounding environment, or it can be determined based on the object's position coordinates in the video frame.

[0081] S460 calculates the spatial motion relationship between the detection bounding boxes of the first object and the second object for each spatiotemporal scene group.

[0082] After obtaining the spatiotemporal scene group containing the first object and the second object in step S430, the spatial action relationship between the first object and the second object can be further determined based on these spatiotemporal scene groups obtained in step S430. For example, the spatial action relationship between the two objects can be determined based on the spatiotemporal information of the first object and the second object obtained in steps S440 and S450. For example, the spatial action relationship between the first object and the second object can include: up-down (intersecting), up-down (non-intersecting), lower left-upper right (intersecting), lower left-upper right (non-intersecting), left-right (intersecting), left-right (non-intersecting), upper left-lower right (intersecting), upper left-lower right (non-intersecting), and containing-being contained. As can be seen from the above examples, these spatial action relationships can be determined based on the spatial position of the objects.

[0083] S470 aggregates the spatial motion relationships of each video frame into a first feature vector.

[0084] In step S470, video frames with such spatial action relationships can be aggregated into multi-dimensional feature vectors based on the spatial action relationship between the first object and the second object obtained in step S460, so as to match them with various feature vectors in a pre-set feature library.

[0085] S481, compare the first feature vector with the second feature vector in the preset feature library to obtain at least one interactive action category.

[0086] In step 481, the feature vectors obtained in step S470 by aggregating the spatial action relationships of each frame in step S460 can be compared with the feature vectors in a preset feature library. For example, the similarity between features can be calculated, and the interaction action category corresponding to the second feature vector with a similarity greater than a preset threshold can be used as the candidate action category of the video data to be identified.

[0087] S482, filter at least one type of interactive action based on a first structured attribute and its corresponding first confidence level, a second structured attribute and its corresponding second confidence level, and a preset common sense base.

[0088] S483. Based on the filtering results, the interactive action category with the most selections is determined as the interactive action category in the video data to be identified.

[0089] In step S482, the action categories obtained in step S481 can be further filtered, and in step S483, the action category with the highest number of selections in step S482 can be taken as the action category of the video frame to be identified.

[0090] Furthermore, after determining the action category of the video frame to be identified based on the screening results, the action recognition method according to the embodiments of this application may further include outputting the determined action category to the user through an interactive interface. For example, the interactive interface may be provided in the form of a visual interface, and the action recognition method according to the embodiments of this application may display the action category determined in step S483 on the visual interface, and may receive user feedback input on the displayed action category. For example, the user may provide feedback on the displayed action category recognition result through a keyboard, mouse, or touch screen. For example, the user may input a confirmation command by clicking on the action category, or may modify the result output in step S483 by directly inputting the name of another action category. Therefore, the action recognition method according to the embodiments of this application may further process the result based on the interactive feedback given by the user through the interactive interface. For example, when a user confirms the result of step S483 through the interactive interface, the action recognition method according to the embodiments of this application can input the recognition result and the first feature vector of the corresponding video frame into a preset feature library to update the feature library, thereby supplementing new known matching relationships. This allows the correspondence between reference feature vectors and action categories in the preset feature library to be automatically increased as the action recognition method of the embodiments of this application is used. Furthermore, the action recognition method according to the embodiments of this application can also output or display the posture corresponding to the action category identified in step S483 through the interactive interface, and can further output the video frames containing the posture obtained in step S410 to the user through the interactive interface for reference or confirmation. Additionally, when displaying the posture of the action category identified in step S483 through the interactive interface, videos from, for example, a specific video library other than the video obtained in step S410 can be used to output the video frames containing the posture to the user for reference.

[0091] Furthermore, the interactive action recognition method according to the embodiments of this application can also output the process of constructing a feature database in step S400 through the above-mentioned interactive interface. For example, a predetermined video source or multiple video frames can be displayed through the interactive interface, and the user's selection of the video source or video frame can be received. Thus, the interactive method according to the embodiments of this application can process the video frame of the video source selected by the user or the video frame directly selected by the user in step S400 to obtain a second feature vector. In addition, the annotation information of the obtained video frame can also be displayed to the user through the interactive interface, and similarly, the user's feedback on the annotation information can be received through the interactive interface, or even the user's annotation input for the displayed video frame can be directly received, and the annotation input by the user is used as the annotation information of the video frame.

[0092] Therefore, the action recognition method provided in this application embodiment identifies interactive action categories by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Thus, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between feature vectors of multiple spatial action relationships and interactive action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interactive actions.

[0093] Example 5 Figure 5 This is a schematic diagram of the structure of an embodiment of the action recognition device provided in this application, which can be used to perform actions such as... Figures 2 to 4 The method steps are shown. (As shown) Figure 5 As shown, the action recognition device may include: an acquisition module 41, a selection module 42, a determination module 43, an aggregation module 44, a matching module 45, and a construction module 46.

[0094] The acquisition module 41 can be used to acquire the video data to be identified.

[0095] In this embodiment, the acquisition module 41 can acquire video data to be identified from various video data sources. This video data can be, for example, video captured by video / image acquisition devices installed in various areas such as shopping malls, buses, and public office areas. The video data to be identified can be real-time acquired video data or video data acquired at preset time intervals. For example, video data can be acquired from a camera every 24 hours as the video data to be identified. Alternatively, video data can be acquired at any other time interval as needed, for example, requesting video data to be identified from a camera under user control. Furthermore, the video data to be identified in this embodiment can be video data containing target objects, particularly video data containing multiple target objects. For example, video data collected regarding the service status of a parcel sorting area.

[0096] The selection module 42 can be used to select video frames containing the first object and the second object from the video data to be identified, based on the temporal and spatial information of the objects in each video frame of the video data to be identified, to form a spatiotemporal scene group.

[0097] After the acquisition module 41 acquires the video data to be identified, it can perform object recognition on each video frame obtained by dividing the video data. In this embodiment, various commonly used object recognition schemes can be used to identify objects in each video frame. Thus, the selection module 42 can use the time information and spatial information of the objects identified in the video frame. For example, in the above-mentioned service scenario of sorting express delivery, the time information of each courier in the acquired image in the video frame, such as 1:12:24 pm, and spatial information, such as the position coordinates of the object in the frame, can be used.

[0098] Furthermore, the selection module 42 selects video frames containing both a first object and a second object based on the temporal and spatial information of the objects identified in each frame, thus forming a spatiotemporal scene group. In this embodiment, a spatiotemporal scene group is an abstract group used to describe the spatial relationships between objects in an image. For example, objects with the same temporal information can be selected based on the temporal information of the objects contained in the frame, and video frames that are spatially related can be selected based on the spatial information, thereby forming a spatiotemporal scene group containing these two objects.

[0099] Furthermore, in the context of providing catering services, video frames containing a service worker can be selected from the captured catering service video frames to form a first video frame sequence, and video frames containing a customer can be selected to form a second video frame sequence. In this embodiment, each video frame in the first video frame sequence only needs to contain at least the service worker, not just the service worker. For example, video frames containing both the service worker and the store manager can be selected as video frames in the first video frame sequence, or video frames containing both the service worker and the customer forming the second video frame sequence can be selected as video frames in the first video frame sequence. Similarly, when selecting each video frame constituting the second video frame sequence, it is only necessary that each video frame in the second video frame sequence contains at least the customer. In particular, in this embodiment, video frames with public time periods can be selected first, and then video frames containing at least a first object can be selected from these video frames as the first video frame sequence and video frames containing at least a second object can be selected as the second video frame sequence. Of course, you can first select video frames that contain at least the first object and video frames that contain at least the second object, and then further select video frames with a common time period from these video frames as the first video frame sequence and the second video frame sequence.

[0100] The determination module 43 can be used to determine the spatial action relationship between the first object and the second object based on each spatiotemporal scene group.

[0101] After the selection module 42 obtains the spatiotemporal scene group containing the first object and the second object, the determination module 43 can further determine the spatial action relationship between the first object and the second object based on these spatiotemporal scene groups constructed by the selection module 42. For example, when the selection module 42 constructs the spatiotemporal scene group, it uses the spatial information of the objects in each frame. Therefore, the determination module 43 can determine the spatial action relationship between the two objects based on this spatial information. For example, the spatial action relationship between the first object and the second object can include: up-down (intersecting), up-down (non-intersecting), lower left-upper right (intersecting), lower left-upper right (non-intersecting), left-right (intersecting), left-right (non-intersecting), upper left-lower right (intersecting), upper left-lower right (non-intersecting), and containing-being contained. As can be seen from the above example, these spatial action relationships can be determined based on the spatial position of the objects.

[0102] Furthermore, after selection module 42 selects a first video frame sequence containing at least a first object, such as a service worker, and a second video frame sequence containing at least a second object, such as a customer, determination module 43 can use the time information of the first object, such as the service worker, in each video frame of the first video frame sequence, for example, 1:12:24 PM, and spatial information, such as the object's position coordinates in that frame. Based on this time and spatial information, video frames containing both the first and second objects can be selected to form a spatiotemporal scene group, that is, a scene group in which the service worker provides services to the customer. In the above-mentioned catering service scenario, the spatiotemporal scene group consisting of each corresponding video frame of the service worker and the customer can represent the spatial action relationship of the service worker providing services to the customer, such as serving food to the customer, the customer asking the service worker for the menu, and other interactive action scenarios. Therefore, based on the time information of the two objects contained in each frame, objects with the same time information can be selected, and based on the spatial information, video frames that are spatially related can be selected, thereby forming a spatiotemporal scene group containing the two objects.

[0103] Specifically, in this embodiment, the determining module 43 can select one of the first video frame sequence and the second video frame sequence constructed by the selecting module 42 as the object frame sequence. For example, in the above-mentioned catering service scenario, the first video frame sequence containing service personnel can be selected as the object frame sequence. Then, for each video frame in the object frame sequence, the video frame containing the object involved in the object frame sequence, such as the service personnel, is searched in the other video frame sequence, i.e., the second video frame sequence containing customers, as the neighboring target frame. The object frame and the corresponding neighboring target frame are combined to form the spatiotemporal scene group of the first object and the second object in that frame. Therefore, in this way, video frames containing the customer and the service personnel that overlap or are close in time can be found. Because these video frames overlap or are close in time and also contain customers and service personnel, they can represent the video frame group in which the service personnel interact with the customer.

[0104] In this embodiment, the determining module 43 can further acquire spatiotemporal information, structured attributes, and confidence levels for each frame in the two video frame sequences. Specifically, the object spatiotemporal information can include the capture time of the object identified in the video frame and the object's detection bounding box. In this embodiment, the detection bounding box can be obtained based on the difference between the object's pixels and the pixels of its surrounding environment, or it can be determined based on the object's position coordinates in the video frame. Therefore, the determining module 43 can calculate the spatial action relationship between the detection bounding boxes of the first object and the second object for each spatiotemporal scene group.

[0105] The aggregation module 44 can be used to aggregate the spatial action relationships of each video frame into a first feature vector.

[0106] The aggregation module 44 can aggregate video frames with such spatial action relationships into multi-dimensional feature vectors based on the spatial action relationship between the first object and the second object obtained by the determination module 43, so as to facilitate matching with various feature vectors in the pre-set feature library.

[0107] The construction module 46 can be used to construct a preset feature library. For example, the construction module 46 can process pre-acquired action video frames to obtain a second feature vector. Specifically, the construction module 46 can first construct a corresponding feature library based on the video to be identified or the type of video. In particular, the feature library can pre-store the correspondence between feature vectors identifying spatial action relationships in video frames of video data and feature vectors of predetermined action categories. For example, in step S301, the action video frames in the pre-acquired video data can be processed to obtain second feature vectors identifying spatial action relationships in these action video frames. In particular, these action video frames can be pre-selected or automatically filtered according to predetermined criteria, and these video frames have action information that has been automatically or manually annotated. Therefore, the feature vectors reflecting spatial action relationships in such video frames can be used as reference feature vectors. In addition, the construction module 46 can obtain annotation information for the action video frames. After the construction module 46 obtains the second feature vectors of the action video frames, it can further obtain the annotation information of these video frames. In particular, this annotation information can be used to identify the interactive action categories of objects in the action video frames. The construction module 46 can also correlate the second feature vector with the interactive action category to construct a preset feature library. Therefore, the construction module 46 can build a feature library based on the obtained annotation information for video frames, especially the annotation information that identifies the interactive action category of the object in the obtained action video frame, and the second feature vector that identifies the spatial action relationship in the action video frame obtained by processing the action video frame. Thus, the preset feature library can include the action vectors of multiple video frames and their correspondence with the interactive action category.

[0108] The matching module 45 can be used to match the first feature vector with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified.

[0109] The matching module 45 can match the feature vectors obtained by the aggregation module 44 through the aggregation determination module 43 of the spatial action relationships of each frame with the feature vectors in the preset feature library. For example, it can calculate the similarity between features and take the interaction action category corresponding to the second feature vector with a similarity greater than a preset threshold as the action category of the video data to be identified.

[0110] For example, the matching module 45 can compare the first feature vector with the second feature vector in the preset feature library to obtain at least one interactive action category.

[0111] Specifically, the matching module 45 can compare the feature vectors obtained by the aggregation module 44 through aggregating the spatial action relationships of each frame with the feature vectors in a preset feature library. For example, it can calculate the similarity between features and use the interaction action category corresponding to the second feature vector with a similarity greater than a preset threshold as the candidate action category of the video data to be identified.

[0112] Furthermore, the matching module 45 can filter at least one type of interactive action based on a first structured attribute and its corresponding first confidence level, a second structured attribute and its corresponding second confidence level, and a preset common sense base. Based on the filtering results, the interactive action category with the most selections is determined as the interactive action category in the video data to be identified.

[0113] Therefore, the action recognition device provided in this application embodiment identifies interactive action categories by constructing a preset feature library and matching the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the constructed feature library. Thus, in the recognition scheme of this application embodiment, by first constructing a preset feature library, which stores the correspondence between feature vectors of multiple spatial action relationships and interactive action categories, the action category of the spatial action relationship in the video frame can be identified using the preset feature library based on the spatial action relationship at the frame level. This reduces the amount of annotation required and the computational resources consumed, and improves the accuracy of capturing interactive actions.

[0114] Example 6 The above describes the internal functions and structure of the motion recognition device, which can be implemented as an electronic device. Figure 6 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. Figure 6 As shown, the electronic device includes a memory 51 and a processor 52.

[0115] Memory 51 is used to store programs. In addition to the programs described above, memory 51 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0116] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0117] The processor 52 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. The processor 52 is coupled to the memory 51 and executes the program stored in the memory 51. When the program runs, it executes the action recognition methods described in embodiments two to four above.

[0118] Furthermore, such as Figure 6 As shown, the electronic device may also include other components such as a communication component 53, a power supply component 54, an audio component 55, and a display 56. Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown.

[0119] Communication component 53 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 53 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 53 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0120] Power supply component 54 provides power to various components of the electronic device. Power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0121] Audio component 55 is configured to output and / or input audio signals. For example, audio component 55 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 51 or transmitted via communication component 53. In some embodiments, audio component 55 also includes a speaker for outputting audio signals.

[0122] Display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0123] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An action recognition method, comprising: Build a pre-defined feature library; Obtain the video data to be identified; The first feature vector of the video data to be identified is matched with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified. The method for obtaining the first feature vector includes: obtaining a first video frame sequence containing a first object and a second video frame sequence containing a second object from the video data to be identified; determining one of the first video frame sequence and the second video frame sequence as an object frame sequence; for each video frame in the object frame sequence, finding a video frame in the other video frame sequence that contains the object involved in the object frame sequence as a neighboring target frame, and forming a spatiotemporal scene group of the first object and the second object in that frame with each video frame in the object frame sequence and the corresponding neighboring target frame; Based on each of the spatiotemporal scene groups, the spatial action relationship between the first object and the second object is determined according to the positional relationship between the detection bounding boxes of the first object and the second object; the spatial action relationships of each video frame are aggregated into a first feature vector; the preset feature library includes multiple second feature vectors and the interaction action categories corresponding to the second feature vectors.

2. The action recognition method according to claim 1, wherein, The method further includes: Obtain the first object spatiotemporal information, first structured attribute, and corresponding first confidence level for each frame in the first video frame sequence; Obtain the second object spatiotemporal information, second structured attribute, and corresponding second confidence level for each frame in the second video frame sequence. The first object's spatiotemporal information includes a first shooting time and a detection bounding box of the first object; the second object's spatiotemporal information includes a second shooting time and a detection bounding box of the second object.

3. The action recognition method according to claim 1, wherein, The construction of the preset feature library includes: Process the pre-acquired motion video frames to obtain the second feature vector; Obtain annotation information for the action video frame, wherein the annotation information is used to identify the interactive action category of the object in the action video frame; The second feature vector is matched with the interaction action category to construct the preset feature library.

4. An action recognition device, comprising: The building module is used to build a pre-defined feature library; The acquisition module is used to acquire the video data to be identified; The selection module is used to obtain a first video frame sequence containing a first object and a second video frame sequence containing a second object from the video data to be identified; determine one of the first video frame sequence and the second video frame sequence as an object frame sequence; for each video frame in the object frame sequence, find a video frame in the other video frame sequence that contains the object involved in the object frame sequence as a neighboring target frame, and form a spatiotemporal scene group of the first object and the second object in that frame with each video frame in the object frame sequence and the corresponding neighboring target frame; The determination module is used to determine the spatial action relationship between the first object and the second object based on each spatiotemporal scene group and the positional relationship between the detection bounding boxes of the first object and the second object. The aggregation module is used to aggregate the spatial action relationships of each video frame into a first feature vector. The matching module is used to match the first feature vector of the spatial action relationship of each video frame in the video data to be identified with the second feature vector in the preset feature library to obtain the interactive action category in the video data to be identified. The preset feature library includes multiple second feature vectors and interactive action categories corresponding to the second feature vectors.

5. An electronic device, comprising: Memory, used to store programs; A processor is configured to run the program stored in the memory to perform the action recognition method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon that can be executed by a processor, wherein, When the program is executed by the processor, it implements the action recognition method as described in any one of claims 1 to 3.

Images