Human motion recognition method, device, medium and electronic equipment based on video

Abstract

The present application relates to the field of computer vision, and in particular discloses a method, device, medium and electronic device for human action recognition based on video. The human action recognition method based on video of the present application includes: obtaining multiple frames of multiple human images in a video, wherein each frame of the multiple human images includes multiple human instances; generating a detection human body boundary box of the human body instance in each frame of the multiple human images; determining the image scene of each frame of the multiple human images, wherein one image scene corresponds to one or more types of actions; based on the detection human body boundary box and image scene of the multiple frames of the multiple human images, identifying the actions of the human body instances in the multiple frames of the multiple human images, and being able to post-process the action category and probability of the human body instance through the surrounding environment features in the video frames of the human body instances in the video clips, so that the accuracy of the action category of the human body instance finally predicted is higher.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular discloses a method, device, medium and electronic device for video-based human motion recognition. Background Art

[0002] In many scenarios, it is necessary to identify human movements in videos to determine the types of human movements. For example, in a monitoring system, in order to perceive human intentions, it is necessary to identify human movements in videos obtained by the monitoring system. In a human-computer interaction system, in order to understand human behavior, it is necessary to identify human movements in videos obtained by the human-computer interaction system.

[0003] In order to realize the recognition of human movements in videos, the existing scheme is to extract pictures of certain frames from the video to be processed, and then use a convolutional neural network to extract features from the pictures. Next, the human body movement type and the confidence of each movement type are determined based on the feature vector of the human body area extracted from the picture, and then the movement type with the largest confidence (and the confidence is greater than a preset threshold) is determined as the human body movement type in the video.

[0004] The existing solution directly extracts the features of some areas in the image to perform action recognition. However, directly and simply extracting the features of some areas of the image may not be able to well reflect the action characteristics of the human body in the video frame scene, resulting in poor action recognition effect and low accuracy. Summary of the invention

[0005] The embodiment of the present application provides a video-based human action recognition method, which can post-process the action category and probability of the human body instance through the surrounding environment features in the video frame of the human body instance in the video clip, so that the final predicted action category of the human body instance has a higher accuracy.

[0006] In the first aspect, the embodiment of the present application discloses a method for human action recognition based on video, the method comprising: obtaining multiple frames of multiple human images in a video, wherein each frame of the multiple human images includes multiple human instances; generating a detection human bounding box of the human instance in each frame of the multiple human images; determining the image scene of each frame of the multiple human images, wherein one image scene corresponds to one or more types of actions; based on the detection human bounding box and image scene of the multiple frames of the multiple human images, identifying the action of the human instance in the multiple frames of the multiple human images. It can be understood that the video containing multiple frames of multiple human images is a video in a crowded scene, and the multiple frames of multiple human images can be all images obtained by frame processing of the video. The initial human bounding box of the above human instance can be detected by a pre-trained human detection model or a conventional human detection model, wherein the pre-trained human detection model can be obtained by training a variety of training data sets in crowded scenes, and supports human instance detection in crowded scenes. In addition, in the process of identifying the action of the human instance in the multiple frames of multiple human images, several consecutive frames (such as 40 frames) of multiple human images can be processed at a time, that is, a video clip in the above video can be processed at a time. Moreover, an image scene (i.e., the scene category hereinafter) corresponds to one or more types of actions, that is, the scene category hereinafter is coupled with the action category, and when a human body instance is in a specific scene, it is usually in a corresponding specific action, that is, the image scene in which the human body instance is located can reflect the action of the human body instance. For example, when a human body instance is in the image scene "stairs", the action of the human body instance is usually "going up and down stairs". In this way, when the action of the human body instance in the detected human body boundary box of multiple frames of multi-human images is recognized, the accuracy of the action category of the human body instance finally recognized can be improved by combining the image scene in which the human body instance is located, that is, combining the relationship between the human body instance and the scene in which it is located.

[0007] It should be noted that in the embodiments of the present application, for the convenience of description, different names are used in different locations to describe the images involved in human motion recognition, such as "video frame" and "multiple human images", but their essence remains unchanged.

[0008] In a possible implementation of the first aspect, based on the detection human body bounding box and image scene of multiple frames of multiple human body images, the action of the human body instance in the multiple frames of multiple human body images is identified, including: based on the detection human body bounding box of the multiple frames of multiple human body images, the action of the human body instance in the multiple frames of multiple human body images is identified, and the initial probability of each predefined action in the multiple predefined actions corresponding to the human body instance is obtained; based on the image scene of the multiple frames of multiple human body images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected to obtain the corrected probability of each predefined action; the predefined action with the highest probability in each frame of the multiple human body images is used as the action recognized by the human body instance. It can be understood that the initial probability of each predefined action in the multiple predefined actions corresponding to the above human body instance can be determined by a neural network model for human action recognition (i.e., the action detection model hereinafter), and the multiple predefined actions are multiple action classifications provided by the action detection model. The action detection model can be a conventional or pre-trained network, and the embodiment of the present application does not specifically limit this. Specifically, the action detection model can determine the action category and initial probability distribution of the human body instance obtained by action recognition of the human body instance. Among them, the above correction of the initial probability distribution of each action category of the human body instance is specifically to adjust the weight of each probability, and normalize all the adjusted weights to obtain the corrected action with the highest probability of the human body instance. Therefore, based on the image scene of multiple frames and multiple human images, the probability of the action category of the human body instance predicted by the action detection model is re-weighted (i.e., post-processing below), so as to correct the initial probability distribution of the action category of the human body instance detected by the action detection model, so as to make the accuracy of the action category of the human body instance finally detected higher.

[0009] In a possible implementation of the first aspect, based on the image scene of multiple frames of multiple human images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected to obtain the corrected probability of each predefined action, including: based on the overlap between adjacent detected human body bounding boxes of multiple frames of multiple human images, and the image scene of multiple frames of multiple human images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected to obtain the corrected probability of each predefined action. It can be understood that the overlap between adjacent detected human body bounding boxes of each frame of multiple human images can reflect the overlap of adjacent human body instances in each frame of multiple human images. The overlap of adjacent human body instances can reflect the actions of these human body instances. For example, when the overlap of adjacent human body instances is high, the actions of these human bodies are usually multi-person actions, such as running together. On the contrary, when the overlap of adjacent human body instances is low, the actions of these human bodies are usually single-person actions, such as single-person running. In this way, based on the degree of overlap between adjacent detected human body boundary boxes in multi-frame multi-human images, the probability of the action category of the human body instance predicted by the action detection model is re-weighted, so that the initial probability of the action category of the human body instance detected by the action detection model is corrected, thereby making the accuracy of the action category of the human body instance finally detected higher.

[0010] In a possible implementation of the first aspect, the degree of overlap between adjacent detected human body bounding boxes is characterized by an intersection over union (IoU) ratio between adjacent detected human body bounding boxes. It can be understood that the higher the intersection over union (IoU) ratio between adjacent detected human body bounding boxes, the higher the degree of overlap between adjacent detected human body bounding boxes, and vice versa.

[0011] In a possible implementation of the first aspect, based on the degree of overlap between adjacent human body detection boundary boxes of multiple frames of multi-human images, the initial probability of each predefined action in multiple predefined actions corresponding to the human body instance is corrected, including: when the degree of overlap between adjacent human body detection boundary boxes of multiple frames of multi-human images is greater than a predetermined threshold, the probability of predefined actions related to multiple people in multiple predefined actions corresponding to the human body instance is increased; when the degree of overlap between adjacent human body detection boundary boxes of multiple frames of multi-human images is less than or equal to the predetermined threshold, the probability of predefined actions related to a single person in multiple predefined actions corresponding to the human body instance is increased. The value of the above-mentioned predetermined threshold can be determined according to actual needs, such as 5, and the embodiment of the present application does not make specific restrictions on this. It can be understood that when the degree of overlap between adjacent human body detection boundary boxes in multiple frames of multi-human images is greater than the predetermined threshold, it means that these human body instances are more likely to produce multi-person actions, so the probability of multi-person related actions of these human body instances can be increased, such as increasing the probability of the action (i.e., action category) "running together", and the probability of the action category related to a single person in these human body instances will be correspondingly reduced, such as reducing the probability of the action "single person running". Similarly, when the degree of overlap between adjacent detected human body boundary boxes in multiple frames of multi-human body images is less than or equal to a predetermined threshold (such as 4), it indicates that these human body instances are more likely to produce single-person actions. Therefore, the probability of single-person related actions of these human body instances can be increased, such as increasing the probability of the action "single-person running", and correspondingly reducing the probability of multi-person related action categories in these human body instances, such as reducing the probability of the action "multi-person running".

[0012] In a possible implementation of the first aspect, based on the image scene of multiple frames of multiple human images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected, including: based on the image scene of multiple frames of multiple human images, for the human body instances in the area where the image scene is located in the multiple frames of multiple human images, the probability of the human body instances corresponding to the predefined multiple categories of actions and the predefined actions corresponding to the determined image scene is increased. It can be understood that when the human body instances are in the area where the determined image scene is located in the multiple frames of multiple human images, it means that the possibility of these human body instances generating the actions corresponding to the image scene is relatively large, so the probability of the action categories corresponding to these human body instances and the determined image scene can be increased. For example, when the human body instances are in the area where the image background "stairs" is located, the probability of the action "going up and down the stairs" of these human body instances may be increased, and the probability of other actions of these human body instances (such as the action "playing football") may be reduced accordingly.

[0013] In a possible implementation of the first aspect, generating a detected human body bounding box of a human body instance in each frame of a multi-human body image includes: generating an initial human body bounding box of a human body instance in each frame of a multi-human body image; correcting the initial human body bounding box of the multi-human body image based on the position difference data between the multi-human body image and the previous frame of the multi-human body image to obtain a detected human body bounding box, wherein the position difference data includes the position difference between the human body instance in the multi-human body image in the previous frame and the multi-human body image; and recognizing the posture of the human body instance in the multi-human body image based on the detected human body bounding box of the multi-human body image. It can be understood that the position difference data between two adjacent images can be an optical flow between the two images. For example, the position difference data between the multi-human body image and the previous frame of the multi-human body image is the optical flow between the two images, so that the position difference data includes the position difference between the human body instance in the multi-human body image in the previous frame and the multi-human body image, that is, reflecting the running information of the human body instance from the previous frame and the multi-human body image in the time domain, and further reflecting the position change of the human body bounding box of the human body instance from the previous frame to the multi-human body image. Obviously, since the present solution uses the position difference data (i.e., optical flow) between adjacent multi-body images to correct the human body detection boundary frame of each human body embodiment, and the position difference data reflects the information in the time domain, it is achieved that the combination of the time domain information can avoid the situation of missing frames or dropped frames in the process of human body boundary frame detection for multi-frame multi-body images, thereby achieving the smoothing of the human body boundary frame of the human body instance in the multi-frame multi-body image (i.e., optical flow smoothing processing hereinafter). Furthermore, the human body boundary frame of the human body instance in the multi-frame multi-body image is corrected by the position difference data between adjacent multi-body images, that is, the human body boundary frame of the human body instance in the multi-frame multi-body image is optically smoothed, which improves the accuracy of the human body boundary frame detected by the human body instance in the video of the multi-frame multi-body image, thereby facilitating the improvement of the accuracy of subsequent human action recognition based on the human body boundary frame.

[0014] In addition, in some embodiments, the initial human body bounding box of the human body instance in the multi-human body image can also be corrected by using the position difference data between the next frame of the multi-human body image and the multi-human body image, which is not limited in the embodiments of the present application.

[0015] In a possible implementation manner of the first aspect, based on position difference data between the multiple human body images and a previous frame image of the multiple human body images, correcting the initial human body bounding box of the multiple human body images to obtain a detected human body bounding box includes: correcting the i-th initial human body bounding box in the t-th frame image of the multiple frames of multiple human body images to obtain the i-th detected human body bounding box by using the following formula:

[0016]

[0017] in, represents the coordinate set of the i-th initial human bounding box in the t-1-th frame image in multiple frames of human images, F t-1→t Represents the position difference data between the t-1th frame image and the tth frame image, Indicates that based on F t-1→t The coordinate set of the i-th predicted human bounding box in the t-th frame image is obtained. represents the coordinate set of the i-th initial human body bounding box in the t-th frame image, represents the coordinate set of the i-th detected human body bounding box in the t-th frame image, and α is and The synthetic weight coefficients, i and t are both positive integers.

[0018] It can be understood that a set of coordinate points of a bounding box of a human body instance is used to represent the human body bounding box. Specifically, the i-th initial human body bounding box in the t-1-th frame image is used as the benchmark, according to F t-1→t The reflected human instance in the t-1th frame image moves from the t-1th frame image to the tth frame image, that is, the position change of the i-th initial human bounding box in the t-1th frame image from the t-1th frame image to the tth frame image, and the i-th predicted human bounding box of the same human instance in the t-1th frame image is obtained. Then, by synthesizing the coordinate point set of the i-th predicted human bounding box in the t-1th frame image (i.e. ) and the coordinate point set of the i-th initial human body bounding box in the t-th frame image (i.e. ), that is, weighted averaging is performed on the corresponding coordinates of the two coordinate point sets, so that the i-th initial human body bounding box in the t-th frame image can be corrected to obtain a more accurate i-th detected human body bounding box.

[0019] In a possible implementation of the first aspect, the initial human body bounding box of the human body instance in each frame of the multi-human body image is generated by a first neural network model, and the model network of the first neural network model includes at least one of the following: ResNet152 network, ResNeXt101 network, SeNet154 network. It can be understood that when the backbone network of the first neural network model includes the above-mentioned multiple networks, the first neural network model has all the functions of these networks. Specifically, the first neural network model can be the human body detection model mentioned below.

[0020] In the second aspect, an embodiment of the present application discloses a video-based human action recognition device, which includes: an acquisition module, used to acquire multiple frames of multi-human images in a video, wherein each frame of the multi-human image includes multiple human instances; a first generation module, used to generate a detection human body boundary box of the human body instance in each frame of the multi-human image; a determination module, used to determine the image scene of each frame of the multi-human image, one image scene corresponds to one or more types of actions; and a recognition module, used to recognize the actions of the human body instances in the multiple frames of the multi-human images based on the detection human body boundary box and the image scene of the multiple frames of the multi-human images.

[0021] In a third aspect, an embodiment of the present application discloses a readable medium having instructions stored thereon, which, when executed on an electronic device, causes the electronic device to execute the video-based human motion recognition method of the first aspect described above.

[0022] In a fourth aspect, an embodiment of the present application discloses an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing the video-based human motion recognition method of the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 According to some embodiments of the present application, a schematic diagram of semantic segmentation results is shown;

[0024] Figure 2 According to some embodiments of the present application, a schematic diagram of optical flow between two consecutive frames is shown;

[0025] Figure 3 According to some embodiments of the present application, a human action recognition scene diagram is shown;

[0026] Figure 4 According to some embodiments of the present application, a schematic diagram of a composition architecture of an action recognition scenario is shown;

[0027] Figure 5 According to some embodiments of the present application, a flow chart of an action recognition method is shown;

[0028] Figure 6 According to some embodiments of the present application, a schematic diagram of a video clip sequence is shown;

[0029] Figure 7 According to some embodiments of the present application, a schematic diagram of a bounding box output by a human detection model is shown;

[0030] Figure 8 According to some embodiments of the present application, a schematic diagram of coordinate points of a human body bounding box corresponding to a human body instance is shown;

[0031] Fig. 9 According to some embodiments of the present application, a slowfast network composition block diagram is shown;

[0032] Fig.10 According to some embodiments of the present application, a schematic diagram of action recognition based on an AIA model is shown;

[0033] Fig.11 According to some embodiments of the present application, a schematic diagram of the structure of a multi-interaction relationship modeling fusion network is shown;

[0034] Fig.12a According to some embodiments of the present application, a visualization diagram of a staircase scene output by a semantic segmentation model is shown;

[0035] Figure 12b According to some embodiments of the present application, a visualization schematic diagram of a channel scene output by a semantic segmentation model is shown;

[0036] Fig.13 According to some embodiments of the present application, a schematic diagram of a coupling relationship established between a scene category and an action category is shown;

[0037] Fig.14 According to some embodiments of the present application, a schematic diagram of categories and probabilities that may be output by a human body detection model when detecting a human body instance is shown;

[0038] Fig.15 According to some embodiments of the present application, a block diagram of a motion recognition device is shown;

[0039] Fig.16 According to some embodiments of the present application, a block diagram of an electronic device is shown;

[0040] Fig.17 According to some embodiments of the present application, a block diagram of a system on chip (SoC) is shown. Specific embodiments

[0041] The illustrative embodiments of the present application include, but are not limited to, a video-based human motion recognition method and its apparatus, medium, and electronic device.

[0042] The video-based human motion recognition method in this application is suitable for various video processing scenarios, especially video processing scenarios of crowded scenes. For example, this scenario is applied to robot vision processing, video surveillance and human motion recognition, animal motion recognition, etc., and is specifically used for pedestrian motion recognition on the road, dangerous motion recognition in special scenes, etc.

[0043] The video-based human action recognition method disclosed in the present application can first extract the behavior representation information of the human instance in the video frame through the action detection model, then extract the position of the human instance in the video frame in combination with the human detection model, and perform action recognition on the human instance to obtain the probability distribution of the action category of the human instance. Then, the action category and probability of the human instance are post-processed in combination with the surrounding environment features of the human instance in the video clip, that is, the action category and probability of the human instance predicted by the action detection model are re-weighted. Among them, the surrounding environment features include the scene category in the video frame, the intersection and union ratio of the bounding box of the human instance in the video frame, etc. It can be understood that the action category of the human body is predicted in combination with the interactive relationship between the human body instance and the surrounding environment, and then the weight of the probability distribution of the action category is redistributed, so that the accuracy of the final predicted action category of the human body instance is higher.

[0044] To facilitate understanding of the technical solutions provided in the embodiments of the present application, the key terms used in the embodiments of the present application are explained here:

[0045] Semantic segmentation (SS): Semantic segmentation is an image processing method in computer vision. It can recognize images at the pixel level, make dense predictions for each pixel, and mark the object category to which each pixel in the image belongs. It can classify images into several categories, such as scenes, objects (including human bodies, animals, food, daily necessities, vehicles, etc.). It can be understood that if an image describes a scene of pedestrians going up and down the stairs, then semantic segmentation technology can classify the pixels representing people in the image into one category, and the pixels representing the background of the stairs into another category. For example, Figure 1 As shown, Figure 1 (a) is an image of pedestrians going up and down stairs. Figure 1 (b) is an image of the staircase scene extracted by semantic segmentation. As can be seen from the figure, the scene information extracted by semantic segmentation can reflect the surrounding environment information where the pedestrian is located.

[0046] Bounding box: A target detection box used to mark the position of a target object. The "target detection box" is also called a "bounding box". The target detection box can be a rectangular box, a circular box, or other shapes or an envelope that surrounds the target object. If the target detection box is a rectangular box, such as a long rectangular box or a square box, the position information of the target detection box includes the coordinates of the lower left corner point and the upper right corner point of the target detection box. If the target detection box is a circular box, the position information of the target detection box includes the coordinates of the center of the circle and the radius length of the target detection box. In the embodiments of the present application, the target detection box is a rectangular box as an example for explanation. In particular, the target object takes a person as an example. For the convenience of description, the "human body instance" and "human body" in this article both refer to a person, and the "human body bounding box" or "human body bounding box" in this article both refer to the position of the human body in the image.

[0047] Optical flow: It is used to define the movement of objects in an image sequence (such as images in a video stream), represented by a two-dimensional vector (or two-dimensional image), and is specifically used to represent the amount of movement of pixels representing the same object (object) in one frame of a video image to the next frame. In other words, the optical flow is the position difference of the target point when the time interval between video frames is very small, that is, between two consecutive frames of the video, that is, the change in the coordinates of the target point. In this way, the corresponding relationship between the previous frame and the current frame is found by using the change of pixels in the image sequence in the time domain and the correlation between adjacent frames, and the motion information of objects (such as the human body) between adjacent frames can be calculated. For example, Figure 2 The figure shows the optical flow between two consecutive frames (i.e., the t-1th frame and the tth frame) in a video stream. Figure 2 (a) is the image of the t-1th frame, Figure 2 (b) is the image of the tth frame, Figure 2 (c) is the optical flow from the t-1th frame to the tth frame. It can be seen from the figure that the optical flow is the displacement change in the time domain from the t-1th frame to the tth frame. In the embodiment of the present application, the optical flow is used to reflect the motion information of the human body between two consecutive frames in the video to be processed, that is, the position difference (i.e., displacement) of the human body. The optical flow can be expressed by the formula:

[0048] F t-1→t =I t (x+Δx,y+Δy)-I t-1 (x,y)

[0049] Among them, F t-1→t represents the optical flow from frame t-1 to frame t, (x, y) represents the non-boundary coordinate value of the image pixel, I t represents the t-th frame image, I t-1 represents the t-1th frame image. Δx, Δy represent the position difference between the t-1th frame and the tth frame.

[0050] In order to facilitate the description of the technical solution of the present application, the following takes human action recognition under video surveillance as an example.

[0051] like Figure 3 As shown, according to an embodiment of the present application, human motion recognition in a subway staircase scene under video surveillance is shown. Figure 3 As shown, the scene includes three cameras 301 and a video processing device 300, but it can be understood that the action recognition scene applicable to the technical solution of the present application can include any number of cameras, not limited to three.

[0052] The camera 301 is used to collect or shoot a video stream containing a human body, and the video processing device 300 is used to recognize human actions in the acquired video stream containing a human body, such as going up and down stairs, falling, fighting, stealing, etc.

[0053] In some embodiments, the camera 301 directly transmits the acquired video stream to the video processing device 300. In other embodiments, the camera 301 may also transmit the acquired video stream to a designated storage device for storage, and then the video processing device 300 obtains the video stream from the storage device, which is not limited in the embodiments of the present application.

[0054] In some embodiments of the present application, the video processing device 300 is a computer device with certain image or video processing capabilities, such as a personal computer (PC), a laptop or a server, etc. The server may be an independent physical server, or a server cluster consisting of multiple physical servers, or a server that provides basic cloud computing services such as cloud database, cloud storage and CDN. The scale of the server may be planned according to the number of video streams that need to be processed, and the embodiments of the present application do not limit this.

[0055] As an example, in a scenario of performing motion recognition on a human body in video surveillance, the camera 301 may be a surveillance camera installed in a place such as a road or a subway entrance, and is used to capture a video stream of pedestrians in these places. In practical applications, the above-mentioned motion recognition scenario may include a large number of cameras 301, for example, cameras 301 installed at various locations in a subway station.

[0056] In some embodiments, the camera 301 and the video processing device 300 are two independent devices, such as the camera is a surveillance camera and the video processing device 300 is a server that obtains and processes video streams from the surveillance camera. In other embodiments, the camera 301 and the video processing device 300 are different components in the same device, such as different components in the same mobile phone.

[0057] In addition, the camera 301 and the video processing device 300 can be connected to each other through one or more networks. The network can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, other possible networks, which are not limited in the embodiments of the present application.

[0058] In some embodiments of the present application, the above-mentioned video-based human motion recognition method can be executed through a component architecture of motion recognition, for example, the component architecture runs in the above-mentioned video processing device 300.

[0059] like Figure 4 As shown, it is a schematic diagram of a component architecture of an action recognition scenario provided by the present application, wherein the component architecture 40 includes a video clip sequence 41, a human detection branch 42, an action detection model 43, a semantic segmentation model 44, a fusion branch 45 and a video clip sequence action recognition result 46.

[0060] The following describes in detail the various parts of the structure 40 of the action recognition scene.

[0061] Human body detection branch 42: used to detect human bodies in continuous video frames of the input video clip sequence 41. The human body detection branch 42 includes a human body detection model for generating a bounding box of a human body, and an optical flow smoothing branch for correcting the bounding box generated by the human body detection model. Specifically, the human body detection model first classifies the objects in the video frame, obtains the area in the image where the human body instance is located, and then determines the position of the area where the human body instance is located in the video frame, and marks the bounding box of the human body instance at the corresponding position. The optical flow smoothing branch is used to detect multiple frames of images of human body bounding boxes according to the human body detection model, and optimizes these bounding boxes using the optical flow between the multiple frames, that is, adjusts the positions of these bounding boxes, that is, adjusts the positions of the detected human body instances in the multiple frames.

[0062] In some other embodiments, the above-mentioned human detection branch is not only used to detect the bounding box of the area where the human body is located in the image, but also to detect objects related to the human body in the image, such as objects that can reflect the human body's actions, such as the object "football" in contact with the human body's feet can reflect the human body's action as "kicking football".

[0063] Action detection model 43: used to integrate the interaction between the target person's own motion in the video clip and other people, objects and long-term memory information, and generate the action classification and probability distribution of the human instance in the video clip. The embodiment of the present application mainly takes the video test to be detected in a crowded scene as an example, and it is necessary to consider the person's own motion in the video clip and the interaction with other people and long-term memory information. Among them, the action detection model 43 includes a video clip feature extraction branch 431 and a human action detection branch 432.

[0064] Video clip feature extraction branch 431: A three-dimensional convolutional neural network for extracting representation information of video clips. Specifically, the SlowFast 8×8ResNet-101 (SlowFast8x8-R101) network is used as the specific structure of the three-dimensional convolutional neural network. The network is first pre-trained on the Kinetics-700 dataset, and the structure of the three-dimensional convolutional network is initialized using the pre-trained weights. The network uses two channels, fast and slow, for convolution operations. The slow channel is used to extract spatial features from multiple frames of images, and the fast channel is used to extract motion features in the temporal sequence of multiple frames of images. Then, different channels are used to process data of different dimensions of multiple frames of images to obtain recognition information, and then the behavior information of the video clip sequence 41 can be predicted based on the recognition information.

[0065] Human action detection branch 432: includes a human representation extraction module with a memory pool, a multi-interaction relationship modeling fusion network, three fully connected layers and a sigmoid regression layer. Specifically, the human representation extraction module uses RoIAlign to linearly interpolate each bounding box area on the multi-frame image behavior representation and obtains the regional representation of the person through pooling. At the same time, the human representation extraction module obtains the memory representation through the memory pool, and the multi-interaction relationship modeling fusion network models and fuses the regional representation and memory representation of the human body to obtain a robust behavior representation, and obtains the prediction probability of each category through the fully connected layer and the sigmoid regression layer.

[0066] Semantic segmentation model 44: used to output scene graphs and corresponding scene features. Specifically, firstly, the image depth features are extracted through a convolutional neural network, and the extracted image features are divided into two parts: object features and scene features. Then, image segmentation processing is performed on the object features and scene features respectively, and then the two segmentation processed features are fused (superimposed) to obtain the final full-image scene analysis result. After obtaining the full-image scene analysis result, in the embodiment of the present application, since we only need the scene graph and scene features, the pixels and features belonging to the object may not be output, and the scene graph and the corresponding scene features can be output.

[0067] Fusion branch 45: Establish a coupling relationship between multiple action categories and multiple scenes, which is used to increase the probability of the action category (output by the action detection model 43) coupled with the scene output by the semantic segmentation model 44. For example, the semantic segmentation model 44 outputs a stair scene, and the action detection model 43 outputs the action of going up and down stairs, standing action, bending action, etc. and the corresponding probability distribution. Among them, the action of going up and down stairs establishes a stair scene coupling relationship, then the probability of the action of going up and down stairs is increased, and the probability of other actions is normalized and the probability decreases. Finally, the action of the human instance with the highest probability is output.

[0068] Figure 5According to some embodiments of the present application, a flow chart of a method for human action recognition based on video is shown. Figures 1 to 5 , the human action recognition method based on video provided by this application is introduced in detail. Figure 5 As shown, the method specifically includes:

[0069] Step 502: the video processing device 300 obtains multiple frames of multi-human images in a video, wherein each frame of the multi-human image includes multiple human instances.

[0070] In some embodiments of the present application, the video to be processed in the above step 502 can be a surveillance video taken by the camera 301, and the video to be processed can also be a video uploaded by the user to a video platform or a short video social software. The video to be processed can also be a video in a test set, that is, the action recognition method of the present application is used to estimate the posture of the human body in the video in the test set. The following mainly takes the human body instance action recognition in the video of the crowded scene as an example to illustrate the video-based human action recognition method of the present application. For example, the video to be processed is generated after preprocessing Figure 4 The video clip sequence 41 is shown in FIG.

[0071] In other embodiments of the present application, step 502 may first obtain the video to be processed, and then pre-process the video to be processed in real time, such as performing a frame division operation to obtain a video clip sequence 41. For example, a video frame division software or a video capture software may be used to perform a frame division operation on the video to be processed, specifically, the video capture function of the Opencv library in the software may be used to perform a frame division operation on the video to be processed.

[0072] For example, if the length of the video to be processed is 15 minutes, and the frame division operation is performed according to the preset frame rate, assuming that the frame rate is 60fps, then the number of images obtained after the frame division and parsing of the video to be processed should be 60*15*60=54,000. It can be understood that 36,000 images are obtained after the 10-minute video is divided into frames. Among them, "*" means multiplication.

[0073] In the embodiment of the present application, the video to be processed is preprocessed, and the video segment sequence 41 is obtained after frame processing. Since the video to be processed is usually long or the video to be processed is a real-time video stream, it is impossible to obtain the motion features of the entire video to be processed, such as Figure 6As shown, the video to be processed after frame division can be divided into a video segment sequence according to a preset number of m video frames in time sequence. Specifically, the video segment sequence 41 can be obtained in the following manner: the video to be processed is pre-processed, and after frame division, the video frames of the video to be processed are divided according to a preset number of m video frames to obtain video segments containing m video frames, and the divided video segments are formed into a video frame sequence according to the time sequence of the video frames. Among them, the middle frame of the video segment is extracted as the key frame.

[0074] In the embodiment of the present application, taking the video to be processed of the staircase scene as an example, Figure 6 As shown, the video is evenly divided into n video segments, and the video segments are combined into a sequence of video segments to be processed {P1, P2, ...P t ,…P n}, where P t is the t-th video segment, and the t-th video segment is a video segment adjacent to the t+1-th video segment and located before the t+1-th video segment.

[0075] In specific implementation, the preset number can be based on how many frames in the training sample set used to train the action recognition model are used to add action annotations as a reference. For example, if the training sample set adds action annotations to all individuals every 20 frames, the preset number is 40 frames. The preset number can also be dynamically determined according to the length of the video to be processed. When the length of the video to be processed is long, the value of the preset number can be larger, and when the length of the video to be processed is short, the value of the preset number can be smaller. For example, the preset number is positively correlated with the length of the video to be processed.

[0076] It can be understood that the video clip sequence 41 is used to be input into the human detection branch 42 for human detection, the semantic segmentation model 44 for scene extraction, and the video clip feature extraction branch 431 for representation information extraction.

[0077] Step 504: The human body detection model generates an initial human body bounding box for each human body instance in each frame of the multi-human body image.

[0078] The human detection model can achieve three tasks. The first is the classification task, which is to traverse the continuous input video frames and classify the human and non-human parts of the video frames. The second is the regression task, which is to determine the position of the human body area in the video frame and mark the target detection bounding box at the corresponding position. The third is the coordinate point positioning task, which is to determine the position of the coordinate point of the human body's bounding box, where the bounding box can be a rectangular box surrounding the area where the human body is located. For example, for Figure 7 In the video frame shown, the human body detection model can generate a human body bounding box 1 for human body instance 1, and a human body bounding box 2 for human body instance 2. It can be understood that the above three tasks complement each other. For example, determining the position of the human body's bounding box coordinates is used to determine the position of the human body in the video frame.

[0079] Specifically, the human body detection model generates an initial human body bounding box of a human body instance in a video frame (i.e., multiple human body images in each frame), wherein a human body bounding box can be represented by a coordinate point set composed of coordinate points of its multiple vertices, for example, by the coordinates of the lower left corner and the upper right corner of the human body's bounding box. It can be understood that in the embodiment of the present application, a human body instance detected by the human body detection model and the human body bounding box determined in the area where the human body is located is the coordinate point set for determining the bounding box of the human body. That is to say, step 504 specifically detects and obtains the coordinate point set of the bounding box of each human body in the video frame in the video clip sequence.

[0080] In an embodiment of the present application, the process of human body detection by a human body detection model on multiple human body video frames in an input video clip sequence 41 includes: obtaining the position of each human body in the video frame, and predicting a bounding box of the human body for the area where each human body is located. Specifically, when performing human body detection on a video frame, the video frame can be input into a pre-trained human body detection model for human body detection, and the area of ​​the human body in the video frame is obtained. The human body detection model predicts a bounding box of the human body for the area where each human body is located. Among them, the human body detection model can be a pre-trained deep learning convolutional neural network. The human body detection model is trained using a large-scale training sample set composed of images with human body instance annotations. During model training, it is necessary to enable the model to accurately predict the area where the human body is located in the video frame, so that the predicted bounding box of the human body output by the human body detection model is more accurate.

[0081] In an embodiment of the present application, taking the video to be processed as a video of a crowded scene as an example, for each video to be processed, a video segment sequence 41 after frame division, the video frames are sequentially input into a trained human body detection model to obtain an initial human body boundary box output by the human body detection model.

[0082] Step 506: The optical flow smoothing branch 43 corrects the initial human body bounding box of the multi-human body image based on the position difference data between the multi-human body image and the previous frame image of the multi-human body image to obtain a detected human body bounding box, wherein the position difference data includes the position difference of the human body instance in the multi-human body image in the previous frame image and the multi-human body image.

[0083] In an embodiment of the present application, optical flow smoothing is performed on the bounding boxes of all human bodies in a multi-body video frame. Wherein, the optical flow smoothing is to correct the initial human body bounding box of the multi-body video frame to obtain a detected human body bounding box based on the position difference (i.e., displacement) data between the multi-body video frame and the previous video frame of the multi-body video frame, wherein the position difference data includes the position difference between the human body instance in the multi-body video frame in the previous video frame and the multi-body video frame. It can be understood that by predicting the bounding box of the human body in the current frame through the position difference in the time domain from the previous frame to the current frame, the situation of missing frames and dropped frames can be effectively avoided.

[0084] Specifically, for the same human body, the detection human body bounding box of the area where the human body is located is obtained based on the initial human body bounding box of the current frame output by the human body detection model and the initial human body bounding box output by the previous frame human body detection model combined with the optical flow from the previous frame to the current frame. Optical flow smoothing, that is, the motion information of the human body in the multi-frame video frame based on the time domain, corrects the detected initial bounding box (i.e., smoothing), avoids the phenomenon of missing frames when processing multi-frame video frames, and improves the accuracy of the final detection human body bounding box, that is, improves the accuracy of human action recognition.

[0085] It can be understood that in the embodiment of the present application, in the above step 504, the bounding box of the human body detected by the human body detection model in the current frame is subjected to optical flow smoothing, specifically, the coordinate points in the coordinate point set of the bounding box of these human bodies are smoothed to obtain the bounding box detection coordinate point set of each human body, that is, the detected human body bounding box of each human body in the current frame is obtained. Among them, in the embodiment of the present application, the detected human body bounding box of a human body is the bounding box of the human body optimized by optical flow smoothing, specifically, the coordinate points in the detection coordinate point set of the bounding box of a human body are the coordinate points optimized by optical flow smoothing. Among them, the detection coordinate point set of the detected human body bounding box of a human body obtained by optical flow smoothing is used for subsequent input into the action recognition model 43 for processing.

[0086] Specifically, the following formula is used as an example to take the optical flow smoothing processing of the bounding box of the i-th human body in the t-th frame video frame in the video clip (denoted as human body bounding box I1) to illustrate that in an embodiment of the present application, optical flow smoothing is performed on the bounding box of the human body detected by the human body detection model. For example, the t-th frame video frame is the multi-human video frame in the above step 506.

[0087] Figure 8 t is a schematic diagram of the coordinate points of the human body bounding box I1 corresponding to the human body instance I1 in the t-th frame of the multi-human video frame. Specifically, as shown in the figure, the coordinate set detected by the human body bounding box I1 is The upper right corner point w(x1, y1) of the bounding box of the human body and the lower left corner point z(x2, y2) of the bounding box of the human body are included. It can be understood that by obtaining the upper right corner point w(x1, y1) and the lower left corner point z(x2, y2) of the bounding box of the human body I1, the area of ​​the human instance I1 in the multi-human video frame can be known.

[0088] In a possible implementation, the embodiment of the present application calculates the coordinate set of the human body boundary box I1 after optical flow smoothing by the following formula (1):

[0089]

[0090] In the formula, The t in it represents the t-th video frame, and i represents the bounding box of the i-th person in the t-th video frame.

[0091] α represents the weight coefficient, which is an empirical parameter and is taken as 0.5 in most cases.

[0092] It represents the coordinate set of the human body bounding box I1 in the t-th frame of the video frame predicted according to the optical flow from the t-1th frame to the tth frame.

[0093] Represents the coordinate set of the initial human bounding box I1 in the t-th frame of the video output by the human detection model.

[0094] It can be understood that the above formula (1) can realize the coordinate set of the human body bounding box I1 of the t-th frame video frame predicted by the optical flow from the t-1th frame to the tth frame: And the initial human bounding box I1 coordinate set of the tth frame output by the human detection model Determine the coordinate set of the human body bounding box I1 detected in the t-th frame video frame

[0095] In addition, some embodiments of the present application can implement the above-mentioned "according to the optical flow from the t-1th frame to the tth frame, the coordinate set of the human body bounding box I1 of the tth frame video frame is predicted" by formula (2). ”:

[0096]

[0097] In the formula, F k-1→k : The optical flow of the joint coordinates of the human body bounding box I1 from frame t-1 to frame t.

[0098] Represents the coordinate set of the initial human bounding box I1 of the t-1th frame output by the human detection model.

[0099] Step 508: The video segment feature extraction branch 431 obtains the behavior representation information of the video segment.

[0100] In an embodiment of the present application, the action detection model 43 includes a video clip feature extraction branch 431 (i.e., a three-dimensional convolutional neural network) and a human action detection branch 432. Specifically, the action detection model can be an asynchronous interaction aggregation (AIA) model, wherein the backbone network of the AIA model 43 is a pre-trained three-dimensional convolutional neural network (3D convolutional neural network, 3D CNN). The backbone network can be an inflated convolutional network (Inflated 3D Conventional network, I3D) or a SlowFast8×8ResNet-101 (SlowFast8x8-R101) network. In an embodiment of the present application, the backbone network of the AIA model uses a SlowFast8x8-R101 network pre-trained on the Kinetics-700 dataset. The SlowFast8x8-R101 network is a network for video recognition based on a combination of fast and slow paths. The network uses fast and slow channels to capture several frames and then immediately performs a 3D convolution operation. It can be understood that the slow channel (i.e., high-resolution CNN) is used to analyze the spatial features in the video clip, while the fast channel (i.e., low-resolution CNN) is used to analyze the motion features in the video. The behavior actions in the video are classified by combining the features extracted from the two channels.

[0101] Specifically, the video to be processed is preprocessed, and a video clip sequence 41 is obtained frame by frame. The video clip sequence 41 is input into the SlowFast8x8-R101 network in sequence, and the video frames corresponding to the video clips are respectively input into the modules corresponding to the slow channel and the fast channel. Then, different channels are used to process the data of different dimensions in the video frames to obtain recognition information, and then the behavior representation information of the video clip sequence 41 can be predicted based on the recognition information. At the same time, the human body detection branch 42 generates a human body bounding box, and the human body action detection branch 432 obtains the behavior representation information in the human body bounding box. Among them, the behavior representation information contains the comprehensive information of human body action and timing in the video frame, specifically refers to the changes in human body action over time.

[0102] Specifically, Fig. 9 A schematic diagram of the fast and slow channel connection method based on the SlowFast8x8-R101 network is provided, such as Fig. 9 and Figure 6As shown, for example, the video to be processed is divided according to the preset number m of video frames, and the preset number m is 40. Then, a video segment in the video segment sequence includes 40 video frames. Taking a video segment Pt as an example, 8 frames and 32 frames of the video segment Pt are sampled and input into the slow channel and the fast channel respectively, and then different channels are used to process data of different dimensions in the video frames to obtain recognition information to predict the behavior representation information of the video segment sequence.

[0103] Step 510: According to the behavior representation information of the video clip generated by the video clip feature extraction branch 431 and the bounding box of the human instance generated by the human detection branch 42, the human behavior representation is extracted, and the action category and probability distribution of the human instance are predicted through a multi-interaction modeling fusion network and three connection layers and a sigmoid regression layer.

[0104] In an embodiment of the present application, the AIA model 43 includes a video clip feature extraction branch 431 (i.e., a three-dimensional convolutional neural network) and a human motion detection branch 432. Specifically, the human detection branch 42 generates a bounding box of the human body region of the video frame and the video clip feature extraction branch 431 extracts the behavior representation information of the video clip as the input of the human motion detection branch 432. After training, the human motion detection branch 432 uses the bounding box of the human body instance to obtain the regional representation and memory representation of the person, and further performs multi-classification judgment on this representation. Among them, the human motion detection branch 432 includes a human representation extraction module 4321 with a memory pool, a multi-interaction relationship modeling fusion network 4322, three fully connected layers and a sigmoid regression layer 4323.

[0105] Fig.10 A schematic diagram of action recognition based on the AIA model is provided. Fig.10 Describe the method of AIA model to predict human action categories.

[0106] In an embodiment of the present application, the human motion detection branch 432 includes a human representation extraction module 4321 with a memory pool, a multi-interaction relationship modeling fusion network, three fully connected layers and a sigmoid regression layer 4323. The representation extraction module 4321 uses RoIAlign to linearly interpolate each bounding box area on the video representation and obtains the regional representation of the human body through pooling. The representation extraction module 4321 obtains the memory representation through the memory pool, and the multi-interaction relationship modeling fusion network models and fuses the regional representation of people and objects and the memory representation to obtain a robust behavior representation, and obtains the prediction probability of each category through the fully connected layer and the sigmoid regression layer.

[0107] In the embodiment of the present application, the video clip is input into the video clip feature extraction branch 431 to obtain the representation of the video clip and the bounding box of the human body region generated by the human body detection branch 42. The representation extraction module 4321 is used to further pool the human body region representation on the behavior representation of the video clip using RoIAlign according to the human body bounding box. The video clips generating the human region representation are sequentially written into the memory pool, such as Fig.10 As shown, the sequence of video clips for obtaining the regional representation of a person is {P'1, P'2, ...P' t ,…P' n}.

[0108] For example, for the video segment P t To perform action recognition, 2L+1 video clips are read from the memory pool, where L is an integer greater than or equal to 1. Specifically, the 2L+1 video clips read include P' t The L adjacent video clips are spliced ​​into video clip P t The memory representation sequence {P' t-L ,…P' t ,…P' t+L}, the regional representation of the person and the memory representation of the person are input into the multi-interaction relationship modeling fusion network. Each module in the multi-interaction relationship modeling fusion network is used to model different interaction relationships, and a more robust video segment P' is obtained. t behavioral representation.

[0109] Specifically, Fig.11 A schematic diagram of the structure of the fusion network 4322 for modeling multiple interactive relationships, such as Fig.11 The multi-interaction relationship modeling fusion network 4322 shown includes: two human-to-human interaction modeling modules for receiving human regional representations, and two human-to-human memory modeling interaction modules for receiving human regional representations and memory representations respectively, wherein: the first human-to-human interaction modeling module, the first human-to-human memory modeling interaction module, the second human-to-human interaction modeling module, and the second human-to-human memory modeling interaction module are sequentially connected and transmit the sequentially enhanced human regional representations, and each interaction modeling module models an interaction relationship in human-to-human interaction and human-to-memory interaction, and transmits it to the next module after being fused with the human regional representation. The human regional representation outputted in the end is a comprehensive fusion of human-to-human interaction and human-to-memory interaction relationships, which is the robust behavior representation outputted in the end. Among them, human-to-human interaction refers to the video clip P t The interaction between different actors in the video clip P. t Video clip P t The memory representation sequence {P' t -L, ...P' t ,…P' t+L}interactions between actors in .

[0110] Specifically, the modeling refers to

[0111]

[0112] Among them: Q, K are two representations of the input, W Q , W K1 , W K2 , W0 is the weight of the fully connected layer, d is KW K1 , dimension. Depending on the input representation K, the module processes different interaction relationships. For example, the value of K includes the regional representation of a person and the memory representation. The corresponding modeling modules process the person-to-person interaction and the memory interaction in turn and output the corresponding representations that integrate the interaction information of this type; when the four modules are connected in series, the output of the previous modeling module is further used as the input of the next Q, and finally a variety of different interaction relationships are integrated.

[0113] In the embodiments of the present application, Fig.10 As shown, three fully connected layers and one sigmoid regression layer 4323 obtain the predicted probability of each category. Among them, the three fully connected layers include two hidden layers and one output layer. The sigmoid regression layer includes a sigmoid function and a cross entropy loss function. The output vector of the output layer can obtain the predicted probability of each category after passing through the sigmoid layer. The cross entropy loss function is used to train the entire network. The training here means: taking the samples in the sample set and the corresponding object bounding boxes and the regional representations of people in the adjacent video clips in the memory pool set in the representation extraction module 4321 as the input of the behavior detection network, using the cross entropy loss function, combined with the back propagation BP algorithm to adjust the network parameters, and updating the regional representations of people in the video clip to the memory pool.

[0114] In the embodiment of the present application, the action detection model 43 includes a video clip feature extraction branch 431 (i.e., a three-dimensional convolutional neural network) and a human action detection branch 432. The action detection model 43 is a deep learning convolutional neural network trained using a large-scale training sample set with human action annotations.

[0115] Step 512: The semantic segmentation model 44 determines the image scene of each frame of the multi-human image.

[0116] In an embodiment of the present application, the semantic segmentation model 44 extracts scenes from video clips, wherein the semantic segmentation model 44 is a pre-trained convolutional neural network. Specifically, the deep features of the video frame are first extracted through the convolutional neural network, and the extracted image features are divided into two parts: object features and scene features. Then, image segmentation processing is performed on the object features and scene features respectively, and then the two segmented features are fused (superimposed) to obtain the final full-image scene analysis result. After obtaining the full-image scene analysis result, the scene graph and the corresponding scene features can be output. Fig.12a This is a visualization diagram of the staircase scene output by the semantic segmentation model 44. Fig.12a The left picture shows the staircase scene after removing the human body instance in the video frame. Fig.12a The right picture is a stair position pixel map of the stair scene after removing the human body instance in the video frame, and the stair position pixel map represents the stair position information of the stair scene. Figure 12b A visualization diagram of the channel scene output by the semantic segmentation model 44. Figure 12b The left picture shows the channel scene diagram after removing the human body instance in the video frame. Figure 12b The right picture is a pixel map of the stair position of the channel scene after removing the human body instance in the video frame. The pixel map of the stair position represents the stair position information of the channel scene.

[0117] In an embodiment of the present application, the semantic segmentation model 44 is a deep convolutional neural network trained by a sample set calibrated with semantic region labels. For example, the semantic segmentation model is pre-trained using the ADE20K dataset. ADE20K is an image segmentation dataset for complex scenes, including 20,000 images as a training set, 2,000 images as a validation set, and 3,000 images as a test set.

[0118] In the embodiment of the present application, the common backbone deep convolutional neural network for semantic segmentation, such as AlexNet, VGG-16, GoogleNet, ResNet and PSPNet, etc., is used as an example to illustrate the scene of the semantic segmentation model extracting the video frame. Specifically, PSPNet will be used to implement the semantic segmentation model 44 of the convolutional neural network. Its input is the video frame of the video clip sequence 41, and the output result is the scene category of the video frame. In order to realize this process, firstly, PsNet is used to process the input video frame to generate a feature map, and then the generated feature map is subjected to a pyramid pooling operation, so that feature maps of different resolutions can be obtained, and then, a convolution operation is performed for each pooled feature map, and the results are stacked up in combination with the up-sampled feature map to form the final feature map, and finally, the score map of the object or scene category is obtained by convolution processing, and the Softmax activation function is used in the category output mapping to generate a probability distribution (the sum of the scores is 1). It can be understood that, taking the scene category score map as an example, it refers to the probability distribution map corresponding to the scene category that the video frame may output. For example, the probability of a passage scene is 0.5, that of a staircase scene is 0.4, and that of a subway entrance scene is 0.1. Then, according to the semantic fusion method, the semantic label with the highest probability is taken for each pixel. Here, these probabilities are referred to as the semantic confidence of the associated semantic category label. It can be understood that the semantic segmentation model 44 outputs the scene category with the highest confidence and the object category with the highest confidence. Among them, the scene category is used as the input of the fusion branch 45.

[0119] Step 514: The fusion branch 45 identifies the actions of the human instances in the multiple frames of multi-human images based on the detected human body bounding boxes and image scenes of the multiple frames of multi-human images.

[0120] In an embodiment of the present application, action recognition of human instances is performed on a video clip sequence 41. First, a human detection branch 42 performs human detection on the content of a video frame (i.e., multiple frames of multiple human images) to generate a bounding box of the human body. The action detection model 43 predicts the action category and probability of the human body instance in combination with the bounding box of the human body. Then, the predicted action category and probability of the human body instance are post-processed in combination with the environmental features of the human body instance in the video clip, that is, the action category and probability of the human body instance predicted by the action detection model 43 are re-weighted to increase the weight of the action category related to the environmental features. Among them, the environmental features include scene categories in the video frame (staircase scenes, restaurant scenes, etc.), interactive relationships between people (i.e., fighting, running together, gathering, etc.), interactive relationships between people and objects (people pushing carts, holding cups to drink water, etc.), etc.

[0121] Specifically, when two human instances are in action, fighting, or gathering together, the intersection over union (IoU) of the bounding boxes of the two human instances in the video frame will be high. The action category and probability distribution output by the action detection model 43 are further re-weighted according to the IoU. For example, when performing action recognition on a video clip sequence, the action category and probability distribution of the human instance predicted by the action detection model 43 are: the probability of running together is 0.6, the probability of standing is 0.2, and the probability of fighting is 0.2. Among them, the action categories of running together and fighting require the interaction of two or more people, and the fusion branch 45 further determines whether there is a human instance in the bounding box with a high degree of overlap with the bounding box where the human instance is located. If not, the fusion branch 45 re-weights the action category and probability of the human instance, that is, the probability of running together is reduced by 0.5, and the probability of fighting is reduced by 0.5, then the probability distribution of the action category of the human instance becomes: the probability of running together is 0.1, the probability of standing is 0.2, and the probability of fighting is -0.3; then the softmax operation is performed on the probabilities of all action categories of the weighted human instance to ensure that the overall probability sum is still 1. The action category and probability distribution of the reallocated weights are respectively the probability of running together is 0.36, the probability of standing is 0.40, and the probability of fighting is 0.24. Specifically, the fusion branch 45 performs post-processing on the predicted action categories and probabilities of the human body instances in combination with the scenes of the human body instances in the video clips, that is, the action categories and probabilities of the human body instances predicted by the action detection model 43 are re-weighted, and the weights of the action categories related to the environmental features are increased. When a staircase scene appears, the action of the person on the stairs is strongly correlated with the staircase scene. Therefore, the fusion branch 45 establishes a coupling relationship for all action categories and scenes that may be output by the action detection model 43. One image scene corresponds to one or more types of actions, or multiple image scenes correspond to one or more types of actions.

[0122] For example, Fig.13 A schematic diagram of the coupling relationship between a scene category and an action category is shown, such as Fig.13As shown, the action categories that may be output by the action classification in the action detection model 43 include going up and down stairs, running, backpack stealing, fighting, etc., and the scene categories that may be output by the semantic segmentation model 44 include intersections, restaurants, stairs, waiting rooms, etc. Among them, the arrows indicate the coupling relationship between the action categories and the scene categories, and the action category going up and down stairs establishes a coupling relationship with the stair scene and the channel scene. The action category running establishes a coupling relationship with the intersection scene and the waiting room scene, the backpack stealing establishes a coupling relationship with the restaurant scene, the waiting room scene, and the bus scene, the action category fighting establishes a coupling relationship with the restaurant scene, the bus scene, and the channel scene, the action category standing establishes a coupling relationship with the stair scene and the channel scene, the action category falling establishes a coupling relationship with the stair scene, the intersection scene, and the channel scene, and so on. The coupling relationship here is used to weight the action category and probability of the human body instance within the bounding box of the video clip output by the action detection model 43, and to increase the probability of the action category coupled with the scene output by the semantic segmentation model 44. It is understandable that this is just an example, and the action category output by the action detection model 43 and the scene category output by the semantic segmentation model 44 are not limited to this, and the coupling relationship between the action category and the scene category is not limited to this.

[0123] Specifically, take the action recognition of the video of the stair scene as an example. For example, when the scene output by the semantic segmentation model 44 is a stair scene, the action category and probability distribution of the human body instance output by the action detection model 43 are respectively the probability of going up and down the stairs 0.4, the probability of stealing from a backpack 0.4, and the probability of fighting 0.2. It can be understood that the action category going up and down the stairs is coupled with the stair scene, so the action category and probability are re-weighted. For example, the probability of going up and down the stairs of the human body instance is increased by 0.5, and the probability distribution of the action category of the human body instance becomes: the probability of going up and down the stairs 0.9, the probability of stealing from a backpack 0.4, and the probability of fighting 0.2; then the probability distribution of all actions is subjected to a softmax operation to ensure that the overall probability sum is still 1. The action category and probability distribution of the re-weighted action category are respectively the probability of going up and down the stairs 0.46, the probability of stealing from a backpack 0.3, and the probability of fighting 0.24, and the action of going up and down the stairs with the highest probability is taken as the final output action.

[0124] In the action recognition solution of the present application, a human body detection model is used to detect the human body in the video frames of the video clip sequence 41 to generate a human body boundary box. Specifically, the function and training process of the model are as follows:

[0125] 1) Training purpose of human body detection model

[0126] The embodiment of the present application uses a human body detection model to conduct a test experiment on a video to be detected in a crowded scene. The performance of the human body detection model is tested by two indicators: Average precision (AP) and MMR. Among them, AP reflects the accuracy and recall rate of the detection result, and the value of MMR is between 0.01-100. MMR refers to the logarithmic average loss rate of false positives per frame image (False Positive Per Image, FPPI). MMR is very sensitive to false positives (FP), especially FP with high confidence will seriously damage the MMR ratio. In the embodiment of the present application, the larger the AP and the smaller the MMR, the better the performance of the human body detection model.

[0127] 2) Selection of model framework for human body detection model

[0128] In the prior art, for human instance detection in a video to be detected in a crowded scene, the detection framework in the human detection model can be a one-stage detection framework, such as an SSD framework, a YOLO framework, etc., or a two-stage framework, wherein the one-stage detection framework detects human body recognition and human body positioning together, can run in real time, has a fast recognition speed, and has a lower accuracy than the two-stage framework. The two-stage detection framework divides human body recognition and human body positioning into two steps, which are completed separately, and has a high recognition accuracy.

[0129] Therefore, in an embodiment of the present application, for human instance detection in a video to be detected in a crowded scene, a two-stage framework is adopted, such as a cascade-region convolutional neural network architecture (cascade RCNN), a faster-region convolutional neural network architecture (faster RCNN), etc. Among them, the backbone network of the detection framework can be a ResNet152 network, a ResNeXt101 network, or a SeNet154 network.

[0130] The following table evaluates the frame selection results in the human detection model:

[0131] Table 1 gives the experimental results of human detection models using different backbone networks and detection frameworks and combining Feature-Pyramid Networks (FPN) for human detection on the HIE validation set. Among them, any human detection model composed in the table is trained for the HIE dataset using the training sample set. The "+" in the table indicates the components (or components) of the human detection model. For example, ResNet152+Faster RCNN+FPN means that the detection model consists of the backbone network ResNet50, the framework Faster RCNN and FPN. It should be noted that the baseline model is a human detection model with the Faster RCNN framework of ResNet50. The other detection models in the table search for hyperparameters on the baseline model and then apply it to a larger backbone or framework.

[0132] As can be seen from Table 1, the performance of the detection model composed of the backbone network ResNet152 or ResNeXt101 and the framework Cascade RCNN and FPN is significantly improved compared with the baseline model. For example, the detection model composed of the backbone network ResNet152 and the framework CascadeRCNN and FPN is 9.38AP higher and 11.46MMR lower than the baseline model. The detection model composed of the backbone network ResNeXt101 and the framework Cascade RCNN and FPN is 9.38AP higher and 11.46MMR lower than the baseline model. First, the performance of the model increased by 5.64AP and 5.84MMR lower by adding FPN. Secondly, in the selection of the model backbone network: ResNet152 has a significant improvement in model performance compared to ResNet50 or SENet154. ResNeXt101 has a significant improvement in performance compared to SENet154. Thirdly, in the selection of the model framework network, Cascade RCNN has a 1.79AP improvement and a 2.33MMR reduction compared to FasterRCNN.

[0133]

[0134] 2) Selection of training sample dataset for human detection model

[0135] In an embodiment of the present application, for human body detection in a video to be detected in a crowded scene, a human body instance in an image is detected by a pre-trained human body detection model. The human body training sample set used to train the human body detection model includes a human body bounding box marked with a human body instance, wherein each human body instance is annotated with a full body bounding box of the human body, and the human body training sample set contains many images of crowded scenes. The data in the human body training sample set is randomly scattered, and the sample set is divided into a training set, a verification set, and a test set according to a certain ratio. Specifically, based on the human body detection model, the human body detection model is trained using the video of the human body instance in the training set and the data of the human body instance bounding box marked, and multiple human body detection intermediate models are obtained; each human body detection intermediate model is verified using the video verification set, and the human body detection intermediate model with the best verification result is extracted; the human body detection intermediate model is tested using the video test set (to prevent overfitting), and when the test result matches the verification result (the difference between the test result and the verification result is less than a preset threshold), the human body detection intermediate model is output as the human body detection model.

[0136] The selection and verification results of the training sample set used to train the human detection model are evaluated in Table 2 below:

[0137] Table 2 shows the performance results of human detection on the HIE validation set by training human detection model 41 with different training sample sets, where human detection model 41 is the baseline model, which is a human detection model of the FasterRCNN framework with ResNet50. The "+" in the table indicates the dataset included in the training sample set. For example, HIE+CityPerson indicates that the training sample set used to train the baseline model includes the HIE dataset and the CityPerson dataset. allextra data represents a combined dataset of the COCO dataset, the CityPerson dataset, the CrowndHuman dataset, and the self-collected dataset.

[0138] As can be seen from Table 2, the training sample set based on the HIE dataset: adding the CrowndHuman dataset, the COCO dataset, and the CityPerson dataset improved the detection performance by 16.85AP and reduced 15.38MMR. Adding the CrowndHuman dataset and the self-collected dataset improved the detection performance by 19.39AP and reduced 18.43MMR, and adding the all extra data dataset improved the detection performance by 19.68AP and reduced 18.84MMR. It can be understood that the human body detection performance of the baseline model trained by the training sample set composed of the HIE dataset and the all extra data dataset is the best, but compared with the human body detection performance of the baseline model trained by the training sample set composed of the HIE dataset, the CrowndHuman dataset, and the self-collected dataset, the human body detection performance is improved by 0.33AP and reduced by 0.41MMR. The performance has not been significantly improved, but the training time is three times higher due to the merging of the COCO dataset. It can be understood that the training sample set used to train the human body detection model in the embodiment of the present application includes the HIE dataset, the CrowndHuman dataset, and the self-collected dataset, weighing the detection performance and training time.

[0139]

[0140] 3) Details of the training of the human body detection model

[0141] The detection method of the prior art is to predict a single candidate box for a single human instance. Since the human instances in the detection scene overlap with each other, during the model detection process, a part of the overlapping multiple human instances is likely to be eliminated by the set non-maximum suppression (NMS), and finally only the bounding boxes of the human body corresponding to the remaining human instances are obtained, wherein NMS is used to filter out the human bounding boxes with a high overlap rate (Intersection over Union, IOU) in the detection results of the human detection model 41. ) The present application applies a method to predict human instances in crowded scenes, called "Crowd-Det". The key idea of ​​Crowd-Det is to use the bounding box of the human body generated by the model as a candidate box, wherein each candidate box generated by the model does not identify a single instance, but predicts a candidate box for the highly overlapping human instances as a whole, and then post-processes the multiple human instances contained in a candidate box, and finally obtains a single human instance and a bounding box for a single human body.

[0142] Specifically, the Crowd-Det algorithm includes at least one of the following implementations. The following takes a candidate box including two or more human instances and a candidate box including person instance A or person instance B as an example for explanation:

[0143] a. EMD loss, which is used to determine which of the two bounding boxes of human instances A and B corresponds to respectively. Specifically, for example, the candidate boxes are generated by the convolutional neural network to generate bounding boxes of two human instances, namely bounding box 1 and bounding box 2. Assume that bounding box 1 is the bounding box of human instance A and bounding box 2 is the bounding box of human instance B, and calculate the EMD loss. Assume that bounding box 2 is the bounding box of human instance A and bounding box 1 is the bounding box of human instance B, and calculate the EMD loss. The bounding boxes corresponding to human instance A and human instance B with the smallest EMD loss values ​​are used as output results.

[0144] b. Optimization module, used for secondary prediction of bounding boxes. Specifically, in the embodiment of the present application, the bounding box of human instance A and the bounding box of human instance B obtained by the Crowd-Det algorithm through the EMD loss mechanism are input into the optimization module together with the prediction box generated by the model. The optimization module generates two bounding boxes of human bodies, and uses EMD loss to determine the bounding boxes of the human bodies corresponding to human instance A and human instance B respectively.

[0145] c. Set NMS is used to insert an additional judgment to check whether the two boxes are from the same candidate box before the bounding box of one person suppresses the bounding box of another person in the NMS mechanism each time. If so, the suppression is skipped. It can be understood that the above-mentioned suppression of the bounding box of one person by the bounding box of another person in the NMS mechanism means that: in the bounding boxes of the two overlapping (such as high overlap) human bodies respectively corresponding to the bounding boxes of the front human body suppresses the bounding box of the rear occluded human body, resulting in the elimination of the bounding box of the rear occluded human body. For example, in order to prevent the bounding box of the human body predicted by the overlapping person instance A or person instance B from being eliminated by NMS, the original NMS is simply improved to obtain the above-mentioned "Set NMS" mechanism, that is, each time when the bounding box of one person overlaps with the bounding box of another person, it is judged whether the bounding boxes of the two human bodies are from the same candidate box; if so, the NMS suppression is skipped.

[0146]

[0147] Table 3 shows the human detection model as the baseline model. The baseline model is trained on the training sample set and the human detection test verification results are performed on the HIE validation set. Among them, the baseline model is a human detection model with the Faster RCNN framework of ResNet50, and the training sample set includes the combined dataset of the HIE dataset, CityPerson dataset, COCO dataset, CrowndHuman dataset, and self-collected datasets.

[0148] As shown in Table 3, adding crowd-Det's emd loss processing to the baseline model improves the detection performance by 0.37AP and reduces 1.97MMR. Adding crowd-Det's emd loss processing and refine module processing to the baseline model improves the detection performance by 0.6AP and reduces 4.32MMR. Adding emd loss processing, refine module processing and crowd-Det's set NMS processing to the baseline model improves the detection performance by 0.69AP and reduces 5.54MMR. It can be understood that according to the results in the table, the Crowd-Det algorithm does improve the performance of crowded detection.

[0149] 4) KD regularization of human body detection model

[0150] In an embodiment of the present application, during the process of human body detection for each frame of an image, the output layer of the human body detection model outputs a candidate box corresponding to the human body in the image, as well as possible categories of the human body in the candidate box and the probabilities corresponding to the categories. The human body detection model selects the bounding box with the highest category probability of the human body. In order to reduce the overfitting of the human body by the human body detection model during the detection process, knowledge distillation (KD) regularization is additionally added during the training process of the human body detection model 41. Specifically, in the model training of the human body detection model 41, in addition to the conventional cross entropy (the first term in the formula below) as the loss function, KD regularization provides an additional regularization formula (the second term in the formula below). For example, during training, if Fig.14 As shown, for the human instance in the figure, the output layer of the human detection model 41 may output categories and probability distributions of categories with a probability of 0.5 for humans, 0.4 for trees, and 0.1 for chairs. Normal training only uses the cross entropy of the probability distribution and a one-hot label as the loss function. KD regularization also provides a virtual teacher model's probability distribution and the KL divergence of the model's probability distribution as an additional loss function. KD regularization can prevent the model from over-relying on pre-labeled labels when outputting categories and probability distributions in the result output layer, making the categories and probability distributions output by the human detection model 41 more accurate.

[0151] The mathematical formula for KD regularization provided in this application is as follows:

[0152] L reg =(1-α)H(q,p)+αD KL (p d τ ,p τ )

[0153] Among them, H is the cross entropy, p is the probability of the category output by the detection model, q is the ground truth of the category, and D KL is KL divergence, p τ is the probability distribution of the output category of the detection model after softening with temperature t, p d τ It is a manually designed probability distribution, and its distribution formula is as follows:

[0154]

[0155] Finally, based on the above analysis, two human detection models are trained using the HIE dataset, crowdhuman dataset and self-collected dataset, denoted as human detection model 1 and human detection model 2. Among them, the detection performance verified on the HIE validation set: the backbone network of human detection model 1 is ResNet152, the framework selects cascade RCNN combined with FPN, and the bounding box of the human body generated in the output layer of the model is processed by crowd-Det and KD regularization, and the detection performance is 83.21AP. The backbone network of human detection model 2 is ResNeXt101, the framework selects cascade RCNN combined with FPN, and the bounding box of the human body generated in the output layer of the model is processed by crowd-Det and KD regularization, and the detection performance is 83.78AP. The detection model after the fusion of human detection model 1 and human detection model 2 with a weight of 1:1 is used as the human detection model of the embodiment of the present application.

[0156] In the action recognition solution of the present application, the action detection model 43 is used to perform action recognition on the human body in the video clip to generate the action category of the human body. Specifically, taking the action detection model 43 as an AIA model as an example, the function and training process of the model are as follows:

[0157] 1) Selection of action training sample dataset for human action recognition model

[0158] In an embodiment of the present application, the training sample set used to train the AIA model includes an AVA dataset and a HIE dataset.

[0159] The AVA dataset contains 430 15-minute annotated Youtube videos. The 15-minute videos are uniformly divided into 300 non-overlapping 3-second segments. The sampling follows the time order of the action sequence, and the bounding box and action of the characters in the middle frame of each 3-second video segment are manually annotated. Specifically, the AVA dataset has 80 atomic visual actions (i.e., 80 action categories). The annotation of each action contains both descriptive text and the bounding box in the video frame. These actions can be divided into three groups: posture / movement actions, character interactions, and person-to-person interactions. For example, walking, kicking (something), shaking hands, running, etc., all actions have spatiotemporal positioning, resulting in 57.6k video clips, 96k annotated human actions, and 232k action labels.

[0160] The HIE training set contains 33,559 human action instances from 14 action categories. Action annotations are added for all individuals every 20 frames, and we call the frames containing action annotations keyframes. However, the distribution of all action categories in the HIE dataset is unbalanced. Classes such as running alone, running together, sitting and talking, fighting, and falling have less than 400 human instances each, while 11.7k actions are labeled as walking alone. Moreover, in the HIE dataset, for a video frame, the area where all human instances are located accounts for less than 2% of the total area of ​​the video frame on average. Therefore, performing spatiotemporal action localization on this dataset is an extremely challenging task. To alleviate the problem of insufficient data, we added some additional data, which contains 3k human instances from 2k video frames and adds 2.9k action annotations to the human instances.

[0161] 2) Training details of the human body detection model

[0162] a) Dataset Splitting

[0163] We try to split the dataset into training and validation sets by video, similar to detection training. However, several action labels have few positive instances in the training and / or validation sets, which makes it difficult to evaluate the performance using this split strategy. Therefore, we perform a few rounds of initial experiments using the train-validation split above to verify the correctness of our implementation. Afterwards, we switch to training on the full training set without validation, and then test on the given official test set.

[0164] b) Training details

[0165] For each trimmed clip around a keyframe, we sample 32 frames and 8 frames for the slow and fast branches in the feature extractor in the clip, respectively. For spatial dimensions, during training, we resize the short side of the frame to a length randomly sampled from (320, 352, 384, 416, 448) and pad the long side to 2.5 times the short side. We apply 10% bounding box jitter and random horizontal flips to improve generalization. Before classification prediction, we use a dropout rate of 0.2 and adopt a binary cross-entropy loss with sigmoid activation logits as supervision during training. Training was performed for a total of 160,000 iterations using the SGD optimizer with BatchNorm statistics frozen. We train each model on 8 GPUs with 2 clips per GPU, which brings the total number of batches to 16.

[0166] c) Weighted Sampling

[0167] Due to the lack of training samples for some classes, we observe that the score distribution of rare classes is close to zero after training. To alleviate the imbalance problem between action classes, we perform weighted sampling of clips during training, assigning higher weights to clips containing rare actions (such as sitting, running, etc.) and lower weights to clips containing frequent actions (such as walking and standing). After applying the weighted sampling strategy, the ratio of the number of occurrences of the largest sampled action to the number of occurrences of the smallest sampled action is reduced from about 120 to about 10 during training.

[0168] Fig.15 According to some embodiments of the present application, a structural block diagram of a human action recognition device based on video is shown. Fig.15 As shown, the human action recognition device 1500 based on video includes:

[0169] An acquisition module (1502) is used to acquire multiple frames of multiple human body images in a video, wherein each frame of the multiple human body image includes multiple human body instances;

[0170] A first generating module (1504) is used to generate a detected human body bounding box of a human body instance in each frame of the multi-human body image;

[0171] A determination module (1506) is used to determine the image scene of each frame of the multi-human image, where one image scene corresponds to one or more types of actions;

[0172] The recognition module (1508) is used to recognize the actions of human instances in the multiple frames of multi-human images based on the detected human body bounding boxes and image scenes of the multiple frames of multi-human images.

[0173] Understandably, Fig.15The video-based human action recognition device 1500 shown corresponds to the video-based human action recognition method provided in the present application. The technical details in the above specific description of the video-based human action recognition method provided in the present application are still applicable to the present invention. Fig.15 The video-based human motion recognition device 1500 shown is described in detail above and will not be repeated here.

[0174] Fig.16 FIG. 1 is a block diagram of an electronic device 1600 according to an embodiment of the present application. The electronic device may be the video processing device 300 described above. Fig.16 An example electronic device 1600 is schematically illustrated in accordance with various embodiments. In one embodiment, the electronic device 1600 may include one or more processors 1604, a system control logic 1608 coupled to at least one of the processors 1604, a system memory 1612 coupled to the system control logic 1608, a non-volatile memory (NVM) 1616 coupled to the system control logic 1608, and a network interface 1620 coupled to the system control logic 1608.

[0175] In some embodiments, the processor 1604 may include one or more single-core or multi-core processors. In some embodiments, the processor 1604 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where the electronic device 1600 uses an eNB (Evolved Node B) or RAN (Radio Access Network) controller, the processor 1604 may be configured to execute various embodiments, such as Figure 3 One or more of the multiple embodiments shown.

[0176] In some embodiments, system control logic 1608 may include any suitable interface controller to provide any suitable interface to at least one of processors 1604 and / or any suitable device or component in communication with system control logic 1608 .

[0177] In some embodiments, the system control logic 1608 may include one or more memory controllers to provide an interface to the system memory 1612. The system memory 1612 may be used to load and store data and / or instructions. In some embodiments, the memory 1612 of the system 1600 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0178] NVM / memory 1616 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 1616 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

[0179] NVM / storage 1616 may include a portion of storage resources on a device on which electronic device 1600 is installed, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM / storage 1616 may be accessed over a network via network interface 1620.

[0180] In particular, system memory 1612 and NVM / storage 1616 may include, respectively, a temporary copy and a permanent copy of instructions 1624. Instructions 1624 may include instructions that, when executed by at least one of processors 1604, cause electronic device 1600 to implement the following: Figure 5 In some embodiments, instructions 1624, hardware, firmware, and / or software components thereof may additionally / alternatively be placed in system control logic 1608, network interface 1620, and / or processor 1604.

[0181] The network interface 1620 may include a transceiver for providing a radio interface for the electronic device 1600, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 1620 may be integrated with other components of the electronic device 1600. For example, the network interface 1620 may be integrated with at least one of the processor 1604, the system memory 1612, the NVM / storage 1616, and a firmware device (not shown) having instructions. When at least one of the processors 1604 executes the instructions, the electronic device 1600 implements the following. Figure 5 The method shown.

[0182] The network interface 1620 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1620 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.

[0183] In one embodiment, at least one of the processors 1604 may be packaged together with logic for one or more controllers of the system control logic 1608 to form a system in package (SiP). In one embodiment, at least one of the processors 1604 may be integrated on the same die with logic for one or more controllers of the system control logic 1608 to form a system on a chip (SoC).

[0184] The electronic device 1600 may further include an input / output (I / O) device 1632. The I / O device 1632 may include a user interface to enable a user to interact with the electronic device 1600; the design of the peripheral component interface enables the peripheral components to also interact with the electronic device 1600. In some embodiments, the electronic device 1600 further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device 1600.

[0185] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0186] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0187] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 1620 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0188] According to an embodiment of the present application, Fig.17 A block diagram of a SoC (System on Chip) 1700 is shown. Fig.17 In the FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.17In the embodiment, SoC 1700 includes: an interconnect unit 1750, which is coupled to an application processor 1710; a system agent unit 1770; a bus controller unit 1780; an integrated memory controller unit 1740; a group or one or more coprocessors 1720, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random access memory (SRAM) unit 1730; and a direct memory access (DMA) unit 1760. In one embodiment, the coprocessor 1720 includes a special-purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor, etc.

[0189] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0190] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0191] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0192] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0193] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0194] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0195] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. Although the present application has been illustrated and described with reference to certain preferred embodiments of the present application, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present application.

Claims

1. A human action recognition method based on video, characterized in that: include: Acquire multiple frames of multiple human body images in a video, wherein each frame of the multiple human body images includes multiple human body instances; Generate a detected human body bounding box of a human body instance in each frame of the multiple human body images; Determine an image scene of the multiple human body images in each frame, where one image scene corresponds to one or more types of actions; Based on the detected human body bounding boxes and image scenes of the multiple frames of multi-human images, identifying actions of human body instances in the multiple frames of multi-human images; The detecting human body bounding boxes and image scenes based on the multiple frames of multiple human body images, and identifying the actions of human body instances in the multiple frames of multiple human body images, comprises: Based on the detected human body bounding boxes of the multiple frames of multi-human images, identifying the actions of human body instances in the multiple frames of multi-human images, and obtaining an initial probability that the human body instances correspond to each of the multiple predefined actions; Based on the image scene of the multiple frames of multiple human body images, correcting the initial probability of each action in the predefined multiple types of actions corresponding to the human body instance to obtain a corrected probability of each predefined action; Taking the predefined action with the highest probability in the multiple human body images in each frame as the action recognized by the human body instance; The method of correcting the initial probability of each action in the predefined multiple types of actions corresponding to the human body instance based on the image scene of the multiple frames of multiple human body images to obtain the corrected probability of each predefined action includes: Based on the degree of overlap between adjacent detected human body boundary boxes of the multiple frames of multi-human body images and the image scenes of the multiple frames of multi-human body images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected to obtain the corrected probability of each predefined action.

2. The method according to claim 1, characterized in that The degree of overlap between adjacent detected human bounding boxes is characterized by the intersection-over-union ratio between adjacent detected human bounding boxes.

3. The method according to claim 1, characterized in that The correcting the initial probability of each predefined action in the plurality of predefined actions corresponding to the human body instance based on the overlap degree between the adjacent detected human body boundary boxes of the plurality of frames of multi-human body images comprises: When the overlap between adjacent detected human body boundary boxes of the multiple frames of multi-human body images is greater than a predetermined threshold, increasing the probability of predefined actions related to multiple people among the multiple predefined actions corresponding to the human body instances; When the overlap degree between adjacent detected human body boundary boxes of the multiple frames of multi-human body images is less than or equal to the predetermined threshold, the probability of a predefined action related to a single person among the multiple predefined actions corresponding to the human body instances is increased.

4. The method according to any one of claims 1 to 3, characterized in that The method of correcting the initial probability of each action in the predefined multiple types of actions corresponding to the human body instance based on the image scene of the multiple frames of multiple human body images includes: Based on the image scenes of the multi-frame multi-human images, for human body instances in the multi-frame multi-human images that are located in the area where the image scenes are located, the probability that the human body instances correspond to predefined multiple types of actions and the predefined actions corresponding to the determined image scenes is increased.

5. The method according to claim 1, characterized in that The step of generating a detected human body bounding box of a human body instance in each frame of the multiple human body images comprises: Generate an initial human body bounding box of a human body instance in each frame of the multiple human body images; Based on position difference data between the multiple human body images and a previous frame image of the multiple human body images, correcting an initial human body bounding box of the multiple human body images to obtain a detected human body bounding box, wherein the position difference data includes a position difference between a human body instance in the multiple human body images in the previous frame image and the multiple human body images; The postures of human instances in the multiple human images are recognized based on the detected human body bounding boxes of the multiple human images.

6. The method according to claim 5, characterized in that The method of correcting the initial human body bounding box of the multiple human body images to obtain the detected human body bounding box based on the position difference data between the multiple human body images and the previous frame image of the multiple human body images comprises: The i-th initial human body bounding box in the t-th frame image of the multi-frame multi-human body images is corrected by the following formula to obtain the i-th detected human body bounding box: in, represents the coordinate set of the i-th initial human body bounding box in the t-1-th frame image in the multi-frame multi-human body image, F t-1→t represents the position difference data between the t-1th frame image and the tth frame image, Indicates that based on F t-1→t The obtained coordinate set of the i-th predicted human body bounding box in the t-th frame image, represents the coordinate set of the i-th initial human body bounding box in the t-th frame image, represents the coordinate set of the i-th detected human body bounding box in the t-th frame image, and α is and The synthetic weight coefficients, i and t are both positive integers.

7. The method according to claim 1, characterized in that The initial human body bounding boxes of the human body instances in the multiple human body images in each frame are generated by a first neural network model, and the model network of the first neural network model includes at least one of the following: ResNet152 network, ResNeXt101 network, and SeNet154 network.

8. A human action recognition device based on video, characterized in that: include: An acquisition module, used for acquiring multiple frames of multiple human body images in a video, wherein each frame of the multiple human body images includes multiple human body instances; A first generating module is used to generate a detected human body bounding box of a human body instance in each frame of the multiple human body images; A determination module, used to determine an image scene of the multiple human body images in each frame, where one image scene corresponds to one or more types of actions; A recognition module, configured to recognize actions of human instances in the multiple frames of multi-human images based on detected human bounding boxes and image scenes of the multiple frames of multi-human images; The identification module comprises: Based on the detected human body bounding boxes of the multiple frames of multi-human images, identifying the actions of human body instances in the multiple frames of multi-human images, and obtaining an initial probability that the human body instances correspond to each of the multiple predefined actions; Based on the image scene of the multiple frames of multiple human body images, correcting the initial probability of each action in the predefined multiple types of actions corresponding to the human body instance to obtain a corrected probability of each predefined action; Taking the predefined action with the highest probability in the multiple human body images in each frame as the action recognized by the human body instance; The method of correcting the initial probability of each action in the predefined multiple types of actions corresponding to the human body instance based on the image scene of the multiple frames of multiple human body images to obtain the corrected probability of each predefined action includes: Based on the degree of overlap between adjacent detected human body boundary boxes of the multiple frames of multi-human body images and the image scenes of the multiple frames of multi-human body images, the initial probability of each action in the predefined multiple categories of actions corresponding to the human body instance is corrected to obtain the corrected probability of each predefined action.

9. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the human motion recognition method based on video according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the video-based human action recognition method described in any one of claims 1 to 7.

Images