Pedestrian action continuous detection and recognition method and device, storage medium and equipment

Abstract

The application discloses a kind of pedestrian action continuous detection identification method, device, storage medium and equipment, belong to action detection identification field.It includes: the video to be detected is divided into multiple video segments and selects key frame;Frame image of video segment is input into pedestrian detection model, and detection frame is obtained on each frame image;The detection frame of the latter frame image is associated with the detection frame of the former frame image, and the detection frame of the two frame images on association is set with the same number;All frame images of video segment and its detection frame are input into action recognition model, and the action category of each detection frame of key frame is obtained;The action category of the detection frame of non-key frame is set as the action category of the detection frame of key frame with the same number thereof.The application realizes the frame-by-frame continuous detection identification of human action in video, solves the problem that only relying on key frame detection brings deviation to overall detection evaluation of video, and improves the visual experience of video action detection identification presentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of action detection and recognition, and particularly to a pedestrian action continuous detection and recognition method, device, storage medium and equipment. BACKGROUND

[0002] With the development of artificial intelligence technology, human behavior recognition has been preliminarily applied in various industries. Real-time detection and early warning of dangerous behaviors such as robbery and threat in banks, ATMs and other places, analysis of personnel flow in scenic spots, auxiliary nursing in the medical and health field, and detection of abnormal behaviors such as falling of the elderly and children in parks and community leisure and entertainment places have put forward strong demands for human action detection and recognition.

[0003] The existing pedestrian action detection and recognition method usually only detects and recognizes the key frames in the video clips, and the detection of the entire video completely depends on the detection effect of the key frames. If only relying on key frame detection, it is easy to bring deviation to the evaluation of the overall video detection effect. In addition, only the key frames are detected, and the detection box and the recognition result are presented in an intermittent manner, which is poor in visual experience. SUMMARY

[0004] To solve the defects of the prior art, the present application provides a pedestrian action continuous detection and recognition method, device, storage medium and equipment, which realizes frame-by-frame continuous detection and recognition of human actions in a video.

[0005] The technical solutions of the present application are as follows:

[0006] In a first aspect, the present application provides a pedestrian action continuous detection and recognition method, which comprises:

[0007] segmenting a to-be-detected video into a plurality of video clips, each video clip comprising a plurality of frames of images;

[0008] selecting a frame of image as a key frame in each video clip, and the remaining frames of images as non-key frames;

[0009] for each video clip, inputting all frames of images of the video clip into a pedestrian detection model to obtain a certain number of detection boxes on each frame of image of the video clip;

[0010] for all frames of images of each video clip, sequentially associating the detection boxes of a subsequent frame of image with the detection boxes of a previous frame of image in order, and setting the number of the detection boxes of the subsequent frame of image on the association as the number of the detection boxes of the previous frame of image associated therewith;

[0011] for each video clip, inputting all frames of images and their detection boxes of the video clip into an action recognition model to obtain an action category of each detection box of the key frame;

[0012] For each video segment, the action category of the non-key frame detection box is set to the action category of the key frame detection box with the same number.

[0013] Further, the action recognition model comprises a slow channel and a fast channel which are executed in parallel, and the slow channel and the fast channel are both convolutional neural networks, and the number of channels of the convolutional neural network of the fast channel is less than that of the slow channel.

[0014] For each video segment, all frame images and their detection boxes of the video segment are input into an action recognition model to obtain the action category of each detection box of the key frame, comprising:

[0015] The video segment is sampled at different frame sampling rates to obtain a first frame sequence containing fewer frame images and a second frame sequence containing more frame images;

[0016] The first frame sequence and the second frame sequence are input into the slow channel and the fast channel respectively to extract features and obtain a first feature map matrix and a second feature map matrix respectively;

[0017] The first feature map matrix and the second feature map matrix are respectively subjected to a time sequence pooling operation, and the features of the region of interest are extracted based on the detection box of the key frame on the obtained two time sequence pooling results, and a spatial pooling operation is respectively performed to obtain the features of the slow channel and the features of the fast channel, wherein the features of the slow channel represent the static information of the video segment, and the features of the fast channel represent the dynamic information of the video segment.

[0018] The features of the slow channel and the features of the fast channel are fused, and the fusion result is sequentially subjected to a full connection operation and a softmax operation to obtain the probability of each action category.

[0019] The action category corresponding to the maximum probability is taken as the action category of the detection box of the key frame.

[0020] Further, the action recognition model further comprises a lateral connection from the fast channel to the slow channel, and the lateral connection sends the data of the fast channel into the slow channel.

[0021] Further, for each video segment, all frame images of the video segment are input into a pedestrian detection model to obtain a certain number of detection boxes on each frame image of the video segment, comprising:

[0022] input all frame images of the video clip into a YOLOX detection model, obtain a plurality of candidate detection boxes on each frame image of the video clip and a confidence score of the candidate detection boxes identified as a person;

[0023] perform a non-maximum suppression operation on the candidate detection boxes according to a set NMS threshold value;

[0024] filter out the candidate detection boxes with a confidence score lower than a set confidence threshold value from the result of the non-maximum suppression operation, to obtain the certain number of detection boxes and the confidence scores thereof.

[0025] Further, for all frame images of each video clip, the detection boxes of a subsequent frame image are sequentially and orderly data-associated with the detection boxes of a previous frame image, and the number of the detection boxes of the subsequent frame image associated with the detection boxes of the previous frame image is set as the number of the detection boxes of the previous frame image, comprising:

[0026] performing Kalman filter processing on the detection boxes of the previous frame image to obtain a predicted box corresponding to the subsequent frame image;

[0027] comparing the confidence score of the detection boxes of the subsequent frame image with a set high-score threshold value and a set low-score threshold value, regarding the detection boxes of the subsequent frame image with a confidence score greater than the high-score threshold value as high-score detection boxes, and regarding the detection boxes of the subsequent frame image with a confidence score greater than the low-score threshold value and less than or equal to the high-score threshold value as low-score detection boxes;

[0028] performing a first matching between the predicted box and the high-score detection boxes to obtain a first-type matching pair composed of the predicted box and the high-score detection boxes that pass the first matching, and high-score detection boxes and predicted boxes that do not pass the first matching;

[0029] for each first-type matching pair, associating the high-score detection box in the first-type matching pair with the detection box of the previous frame image corresponding to the predicted box in the first-type matching pair, and setting the number of the high-score detection box associated with the detection box of the previous frame image as the number of the detection box of the previous frame image;

[0030] comparing the confidence score of the high-score detection box that does not pass the first matching with a set screening threshold value, setting a new number for the high-score detection box that does not pass the first matching and has a confidence score greater than the screening threshold value, and discarding the high-score detection box that does not pass the first matching and has a confidence score not greater than the screening threshold value;

[0031] performing a second matching between the first-matching-failed prediction box and the low-score detection box to obtain a second-matching-passed second-type matching pair composed of the prediction box and the low-score detection box, a second-matching-failed low-score detection box and a second-matching-failed prediction box;

[0032] for each second-type matching pair, associating the low-score detection box in the second-type matching pair with a detection box in a previous frame image corresponding to the prediction box in the second-type matching pair, and setting a number of the associated low-score detection box as a number of the detection box in the previous frame image corresponding to the prediction box in the second-type matching pair;

[0033] discarding the second-matching-failed low-score detection box, retaining a detection box in a previous frame image corresponding to the second-matching-failed prediction box, and performing data association with a detection box in a subsequent frame image, and discarding the detection box in the subsequent frame image when the data association is unsuccessful.

[0034] Further, in each video segment, an intermediate frame image is selected as a key frame, and the rest of the frame images are non-key frame images.

[0035] Further, the method further comprises:

[0036] displaying and presenting all detection boxes in the key frame and their action categories, and detection boxes in non-key frame images corresponding to the detection boxes in the key frame and their action categories, and displaying and presenting detection boxes in non-key frame images not corresponding to the detection boxes in the key frame.

[0037] In a second aspect, the present application provides a pedestrian action continuous detection and recognition device, which comprises:

[0038] a video segmentation module, configured to segment a to-be-detected video into a plurality of video segments, each video segment comprising a plurality of frame images;

[0039] a key frame selection module, configured to select a frame image as a key frame in each video segment, and select the rest of the frame images as non-key frame images;

[0040] a pedestrian detection module, configured to, for each video segment, input all frame images of the video segment into a pedestrian detection model to obtain a certain number of detection boxes on each frame image of the video segment;

[0041] an ID matching module, configured to, for all frame images of each video segment, sequentially perform data association between a detection box in a subsequent frame image and a detection box in a previous frame image in sequence, and set a number of the detection box in the subsequent frame image on the association as a number of the detection box in the previous frame image associated therewith;

[0042] The action recognition module inputs all frame images and their detection boxes of each video segment into an action recognition model to obtain an action category of each detection box of the key frame.

[0043] The result sharing module sets the action category of the detection box of the non-key frame to the action category of the detection box of the key frame with the same number.

[0044] Further, the action recognition model comprises a slow channel and a fast channel which are executed in parallel, and the slow channel and the fast channel are both convolutional neural networks, and the number of channels of the convolutional neural network of the fast channel is less than that of the slow channel.

[0045] The action recognition module comprises:

[0046] The sampling unit samples the video segment at different frame sampling rates to obtain a first frame sequence containing fewer frame images and a second frame sequence containing more frame images.

[0047] The feature map matrix extraction unit inputs the first frame sequence and the second frame sequence into the slow channel and the fast channel respectively to extract features to obtain a first feature map matrix and a second feature map matrix.

[0048] The feature calculation unit performs a time sequence pooling operation on the first feature map matrix and the second feature map matrix respectively, extracts features of a region of interest based on the detection box of the key frame on the obtained two time sequence pooling results, and performs a spatial pooling operation respectively to obtain a feature of the slow channel and a feature of the fast channel, wherein the feature of the slow channel represents static information of the video segment, and the feature of the fast channel represents dynamic information of the video segment.

[0049] The probability calculation unit fuses the feature of the slow channel and the feature of the fast channel, and sequentially performs a full connection operation and a softmax operation on the fusion result to obtain a probability of each action category.

[0050] The category determination unit sets the action category corresponding to the maximum probability as the action category of the detection box of the key frame.

[0051] Further, the action recognition model further comprises a lateral connection from the fast channel to the slow channel, and the lateral connection sends data of the fast channel into the slow channel.

[0052] Further, the pedestrian detection module comprises:

[0053] a candidate bounding box acquisition unit, configured to input all frame images of the video clip into a YOLOX detection model to obtain a plurality of candidate bounding boxes and confidence scores of the candidate bounding boxes identified as persons on each frame image of the video clip;

[0054] an NMS unit, configured to perform a non-maximum suppression operation on the candidate bounding boxes according to a set NMS threshold value;

[0055] a filtering unit, configured to filter out, from a result of the non-maximum suppression operation, candidate bounding boxes with confidence scores lower than a set confidence threshold value to obtain the certain number of bounding boxes and the confidence scores thereof.

[0056] Further, the ID matching module comprises:

[0057] a prediction unit, configured to perform Kalman filter processing on the bounding boxes of the previous frame image to obtain predicted bounding boxes corresponding to the next frame image;

[0058] a classification unit, configured to compare the confidence scores of the bounding boxes of the next frame image with a set high-score threshold value and a set low-score threshold value, to regard the bounding boxes of the next frame image with confidence scores greater than the high-score threshold value as high-score bounding boxes, and to regard the bounding boxes of the next frame image with confidence scores greater than the low-score threshold value and less than or equal to the high-score threshold value as low-score bounding boxes;

[0059] a first matching unit, configured to perform a first matching on the predicted bounding boxes and the high-score bounding boxes to obtain first-type matching pairs composed of the predicted bounding boxes and the high-score bounding boxes that pass the first matching, high-score bounding boxes that fail the first matching, and predicted bounding boxes that fail the first matching;

[0060] a first association unit, configured to, for each first-type matching pair, associate the high-score bounding box in the first-type matching pair with the bounding box of the previous frame image corresponding to the predicted bounding box in the first-type matching pair, and set a number of the associated high-score bounding box to a number of the bounding box of the previous frame image;

[0061] a first post-processing unit, configured to compare the confidence scores of the high-score bounding boxes that fail the first matching with a set screening threshold value, to set a new number to the high-score bounding boxes that fail the first matching and have confidence scores greater than the screening threshold value, and to discard the high-score bounding boxes that fail the first matching and have confidence scores not greater than the screening threshold value;

[0062] a second matching unit, configured to perform a second matching between the prediction boxes that fail the first matching and the low-scored detection boxes to obtain second-class matching pairs composed of the prediction boxes that pass the second matching and the low-scored detection boxes, low-scored detection boxes that fail the second matching, and prediction boxes that fail the second matching;

[0063] a second association unit, configured to, for each second-class matching pair, associate the low-scored detection box in the second-class matching pair with a detection box of a previous frame image corresponding to the prediction box in the second-class matching pair, and set a number of the associated low-scored detection box as a number of the detection box of the previous frame image corresponding to the prediction box in the second-class matching pair;

[0064] a second post-processing unit, configured to discard the low-scored detection boxes that fail the second matching, retain the detection boxes of the previous frame image corresponding to the prediction boxes that fail the second matching, and perform data association with detection boxes of subsequent frame images, and discard the detection boxes of the previous frame image corresponding to the prediction boxes that fail the second matching when the association with the detection boxes of the subsequent frame images fails.

[0065] Further, in each video segment, a middle frame image is selected as a key frame, and the remaining frame images are selected as non-key frame images.

[0066] Further, the device further comprises:

[0067] a presentation module, configured to display and present all the detection boxes of the key frame and the action categories thereof, and the detection boxes of the non-key frame images with the same numbers as the detection boxes of the key frame and the action categories thereof, and display and present the detection boxes of the non-key frame images with different numbers from the detection boxes of the key frame.

[0068] In a third aspect, the present application provides a computer readable storage medium for continuous detection and recognition of pedestrian actions, comprising a memory for storing processor executable instructions, the instructions being executed by the processor to implement the steps of the method for continuous detection and recognition of pedestrian actions according to the first aspect.

[0069] In a fourth aspect, the present application provides a device for continuous detection and recognition of pedestrian actions, comprising at least one processor and a memory for storing computer executable instructions, the processor executing the instructions to implement the steps of the method for continuous detection and recognition of pedestrian actions according to the first aspect.

[0070] The present application has the following beneficial effects:

[0071] The application realizes sharing of key frame recognition results for non-key frames through ID matching of detection boxes, thereby realizing frame-by-frame continuous detection and recognition of human body actions in a video, solving the problem of deviation caused by relying on key frame detection for overall video detection and evaluation, and realizing better and more stable detection and recognition effects, better assisting in completing functions such as behavior analysis and abnormal detection in application scenarios; and the application solves the problem of discontinuity of pedestrian action detection and recognition, and detection boxes and recognition results of each frame of image are presented in a continuous manner, thereby improving the visual experience of presentation. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 A flowchart of the pedestrian action continuous detection and recognition method of the application;

[0073] Figure 2 A schematic diagram of an example of the action recognition model of the application;

[0074] Figure 3 A process diagram for data association between a detection box of a t-th frame of image and a detection box of a t-1-th frame of image;

[0075] Figure 4 A schematic diagram of the pedestrian action continuous detection and recognition device of the application. DETAILED DESCRIPTION

[0076] In order to make the technical problems, technical solutions and advantages of the application clearer, the technical solutions of the application will be clearly and completely described below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the application.

[0077] Embodiment 1:

[0078] The embodiment of the application provides a pedestrian action continuous detection and recognition method, as shown in the figure, the method comprises the following steps: Figure 1

[0079] S100: dividing a video to be detected into a plurality of video segments, each video segment comprising a plurality of frames of image.

[0080] ​In this step, for a given piece of video V containing a specified action to be detected, it is first divided into frames and segmented into multiple video clips, each of which includes multiple frames of images. The segmentation can be performed according to a certain length s, for example, when s is 16, the segmented video clip includes 16 frames of images. In order to avoid information loss, the adjacent two clips can also be overlapped by several frames, such as 5 frames.

[0081] S200: Select a frame of image as a key frame in each video clip, and the remaining frames of images as non-key frames.

[0082] For example, for each video clip, the middle frame of image is selected as the key frame (key_frame), and the remaining frames of images are selected as the non-key frames (norm_frame).

[0083] S300: For each video clip, input all frames of images of the video clip into the pedestrian detection model to obtain a certain number of detection boxes on each frame of image of the video clip.

[0084] The pedestrian detection model is used to detect a certain number of detection boxes representing people on each frame of image, and the present application does not limit the specific implementation manner of the pedestrian detection model.

[0085] For example, each frame of image obtains a matrix with a dimension of Nx5 through the pedestrian detection model, where N is the number of detection boxes, and the five dimensions correspond to the upper left corner coordinates (x1, y1), the lower right corner coordinates (x2, y2) of the detection box and the confidence score score of the detection box identified as a person, respectively.

[0086] S400: For all frames of images of each video clip, the detection boxes of the next frame of image are sequentially data-associated with the detection boxes of the previous frame of image in order, and the number of the detection boxes of the next frame of image on the association is set to the number of the detection boxes of the previous frame of image associated therewith.

[0087] In this step, the detection boxes of the previous and next frames of image are data-associated, and the IDs of the detection boxes of the previous and next frames of image on the association are matched.

[0088] Specifically, assuming that the detection box set of the first frame image of the video clip is bboxes_1, a number (ID) is assigned to each detection box in bboxes_1, and then the detection boxes in the detection box set bboxes_2 of the second frame image are data-associated with the detection boxes in bboxes_1. The detection boxes in bboxes_2 that are associated are recorded as bbox_j, and the detection boxes in bboxes_1 that are associated with bbox_j are recorded as bbox_i. The ID value of bbox_i is assigned to bbox_j. For the detection boxes in bboxes_2 that are not associated, other processing is performed. In this way, data association of all frame images of the entire video clip is completed.

[0089] S500: For each video clip, all frame images and their detection boxes of the video clip are input into the action recognition model to obtain the action category of each detection box of the key frame.

[0090] The action recognition model is used to determine the action category pred of the detection box of the key frame according to the information of all frame images of the video clip. The specific implementation of the action recognition model is not limited in the present application.

[0091] S600: For each video clip, the action category of the detection box of the non-key frame is set to the action category of the detection box of the key frame with the same number.

[0092] This step is used to share the key frame recognition result based on the ID matching result. For the key frame of the video clip, the detection box in the key frame and its action category are obtained through the action recognition model. For each non-key frame in the video clip, the detection box with the same ID as the key frame detection box is found, and the action category of the detection box with the same ID in the key frame is shared with the corresponding detection box of the non-key frame. For the detection box of the non-key frame that does not match the ID of the key frame, no processing is performed.

[0093] The prior art can only detect and recognize the key frame in the video clip, and the recognition result of the key frame represents the entire video clip. The detection of the entire video to be detected is evaluated by the recognition result of the key frame of all video clips. This method completely depends on the detection effect of the key frame for the detection of the entire video, which is easy to cause deviation in the evaluation of the overall detection effect of the video. In addition, only the key frame is detected, and the detection box and the recognition result are presented in an intermittent manner, which is poor in visual experience.

[0094] The application associates the detection boxes of the two frames of images, assigns the same number to the detection boxes on the association, after obtaining the detection and recognition result of the key frame (each detection box of the key frame and the action category thereof), assigns the action category of the detection box of the key frame to the detection box of the non-key frame with the same number, and realizes the action detection and recognition of each frame of image.

[0095] Moreover, the application can also output and display the detection boxes, for the key frame, display and present all the detection boxes of the key frame and the action category thereof, for the non-key frame, display and present the detection box of the non-key frame with the same number as the detection box of the key frame and the action category thereof, and only display and present the detection box of the non-key frame with a different number from the detection box of the key frame.

[0096] The application realizes the sharing of the key frame recognition result to the non-key frame through the ID matching of the detection box, thereby realizing the frame-by-frame continuous detection and recognition of the human body action in the video, solving the problem of the deviation of the overall detection and evaluation of the video caused by the key frame detection only, realizing better and more stable detection and recognition effect, and better assisting the behavior analysis and abnormal detection in the application scene; and the application solves the problem of discontinuous detection and recognition of the pedestrian action, and the detection box and the recognition result of each frame of image are presented in a continuous manner, thereby improving the visual experience.

[0097] The prior art pedestrian action detection and recognition method detects pedestrians through Faster RCNN, and in the scene with dense pedestrians and serious occlusion, there is a serious missed detection. In order to improve the detection effect of the pedestrian detection model in the scene with dense pedestrians and serious occlusion, the application adopts the following method for pedestrian detection:

[0098] S310: input all the frame images of the video segment into the YOLOX detection model, and obtain a plurality of candidate detection boxes and the confidence score of the candidate detection boxes recognized as a person on each frame image of the video segment.

[0099] The application uses the YOLOX detection model to realize pedestrian detection, in order to improve the pedestrian detection effect of the model in the complex crowded scene, the YOLOX detection model is retrained with pedestrian data. In the detection process, the video segment is input into the YOLOX detection model to complete the detection of pedestrians in all frame images, and the detection box of the pedestrian and the confidence score score recognized as a person are output.

[0100] S320: perform a non-maximum suppression operation on the candidate detection boxes according to the set NMS threshold.

[0101] Non-maximum suppression (NMS) is a post-processing method applied in object detection, which can remove redundant detection boxes.

[0102] S330: filtering out the candidate detection boxes with confidence scores lower than the set confidence threshold from the result of the non-maximum suppression operation, to obtain a certain number of detection boxes and their confidence scores.

[0103] This step filters out the candidate detection boxes with confidence scores lower than the threshold through the confidence threshold, and finally outputs the detection boxes remaining in the image and their confidence scores score.

[0104] As an improvement of an embodiment of the present application, as shown in the figure, the aforementioned S400 comprises: Figure 3

[0105] S410: processing the detection boxes of the previous frame image through the Kalman filter to obtain the predicted boxes corresponding to the next frame image.

[0106] The Kalman filter is an algorithm that uses the state equation of a linear system to optimally estimate the state of the system through system input and output observation data. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process.

[0107] The present application predicts human motion based on the Kalman filter, and predicts the position of the detection box of the next frame (t time) based on the position of the detection box of the previous frame (t-1 time).

[0108] The specific expression is as follows:

[0109] x t =F t x t-1 +B t u t +ω t

[0110] Where x t and x t-1 are state vectors at t time and t-1 time respectively (i.e. detection boxes), which can be represented by (x, y, a, h, vx, vy, va, vh), (x, y) is the center point coordinate of the detection box, h is the height of the detection box, a is the aspect ratio of the detection box, vx, vy, va, vh are the change rates of x, y, a, h;

[0111] u t is the motion prediction value;

[0112] F t is the state transition equation of the Kalman filter, which converts the state at t-1 time to the state at t time;

[0113] B t is the control input matrix, which converts the motion measurement value u​t the action mapping to the state vector;

[0114] ω t a predicted Gaussian noise with mean 0 and covariance matrix Q t .

[0115] Exemplarily, bboxes_t represents the detection boxes of the t-th frame image, and bboxes_t* represents the predicted boxes of the t-th frame image based on the detection boxes bboxes_t-1 of the (t-1)-th frame image via the Kalman filter.

[0116] S420: comparing the confidence scores of the detection boxes of the subsequent frame image with a set high-score threshold and a set low-score threshold, taking the detection boxes of the subsequent frame image with the confidence scores greater than the high-score threshold as high-score detection boxes, and taking the detection boxes of the subsequent frame image with the confidence scores greater than the low-score threshold and less than or equal to the high-score threshold as low-score detection boxes.

[0117] Exemplarily, the detection boxes bboxes_t of the t-th frame image are classified based on a high-score threshold τ high and a low-score threshold τ low The detection boxes bboxes_t with the confidence scores greater than τ high are classified as high-score detection boxes (denoted as high-score bboxes), and the detection boxes bboxes_t with the confidence scores lower than τ high but higher than τ low are classified as low-score detection boxes (denoted as low-score bboxes).

[0118] S430: performing a first matching between the predicted boxes and the high-score detection boxes, to obtain a first type of matching pair composed of the predicted boxes and the high-score detection boxes that pass the first matching, and the high-score detection boxes and the predicted boxes that fail the first matching.

[0119] This step is used for performing the first matching. Exemplarily, the predicted boxes bboxes_t* of the t-th frame image are matched with the high-score detection boxes in the detection boxes bboxes_t of the t-th frame image, to finally obtain the matched first type of matching pair, the high-score detection boxes that fail the first matching, and the predicted boxes that fail the first matching.

[0120] During the matching, the IOU (IOU represents the ratio of the intersection of the predicted boxes and the high-score detection boxes to their union) of the predicted boxes and the high-score detection boxes can be calculated, and the matching can be completed via the Hungarian algorithm.

[0121] S440: For each first-class matching pair, associate the high-resolution detection box in the first-class matching pair with the detection box of the previous frame image corresponding to the prediction box in the first-class matching pair, and set the number of the associated high-resolution detection box to the number of the detection box of the previous frame image associated with it.

[0122] This step is used to perform data association. When performing data association, the final association is between the detection box bboxes_t-1 of the (t-1)th frame image and the detection box bboxes_t of the tth frame image. The determination of whether to create an association depends on the prediction box bboxes_t* of the tth frame image and the detection box bboxes_t of the tth frame image.

[0123] S450: Compare the confidence scores of high-scoring detection boxes that failed the first match with the set filtering threshold. Assign new numbers to high-scoring detection boxes that failed the first match and whose confidence scores are greater than the filtering threshold. Discard high-scoring detection boxes that failed the first match and whose confidence scores are not greater than the filtering threshold.

[0124] S460: Perform a second match between the predicted bounding box that failed the first match and the low-scoring detection box to obtain a second type of matching pair consisting of the predicted bounding box and the low-scoring detection box that passed the second match, as well as the low-scoring detection box and the predicted bounding box that failed the second match.

[0125] S470: For each second-class matching pair, associate the low-scoring detection box in the second-class matching pair with the detection box of the previous frame image corresponding to the predicted box in the second-class matching pair, and set the number of the associated low-scoring detection box to the number of the detection box of the previous frame image associated with it.

[0126] S480: Discard the low-scoring detection boxes that fail the second match, retain the detection boxes of the previous frame corresponding to the predicted boxes that fail the second match, and associate them with the detection boxes of subsequent frames. Discard the detection boxes when they fail to associate with the detection boxes of several subsequent frames (e.g., 30 frames).

[0127] As another improvement to the embodiments of the present invention, such as Figure 2 As shown, the action recognition model of the present invention includes a slow channel and a fast channel that are executed in parallel. Both the slow channel and the fast channel are convolutional neural networks, and the number of channels in the convolutional neural network of the fast channel is less than the number of channels in the convolutional neural network of the slow channel.

[0128] The inventors have found that a series of frame images in a video scene usually contains two different parts: a static part that changes little or slowly, and a dynamic part that is changing. For example, a video of a plane taking off contains a relatively static airport and a dynamic plane moving quickly in the static airport scene. For another example, in daily life, when two people meet, a handshake usually happens quickly while other parts of the scene are relatively static.

[0129] According to this finding, the present application designs an action recognition model to include a slow lane and a fast lane that are executed in parallel. The slow lane is a slow high-resolution convolutional neural network with fewer input frame sequences and more channel numbers, which is used to analyze the spatial static content in the video. The fast lane is a fast low-resolution convolutional neural network with more input frame sequences and fewer channel numbers, which is used to analyze the temporal dynamic content in the video. The fast lane uses fewer channel numbers (i.e., uses fewer filter numbers) to keep the network lightweight, and its ability to represent static spatial semantics is weaker.

[0130] This is similar to the principle of primate retinal ganglion cells, in which about 80% of the cells (P-cells) operate at low frequency and can recognize static details, and about 20% of the cells (M-cells) operate at high frequency and are responsible for responding to rapid changes.

[0131] Based on this action recognition model, the aforementioned S500 includes:

[0132] S510: Sampling the video segment at different frame sampling rates to obtain a first frame sequence containing fewer frame images and a second frame sequence containing more frame images.

[0133] For example, the frame sampling rate is set to 2 and 1, i.e., sampling a 16-frame video segment every two frames or every frame to obtain an 8-frame first frame sequence and a 16-frame second frame sequence.

[0134] S520: Inputting the first frame sequence and the second frame sequence into the slow lane and the fast lane respectively to extract features and obtain a first feature map matrix and a second feature map matrix respectively.

[0135] For example, inputting the 8-frame first frame sequence into the slow lane to extract a feature map for each frame image representing static information of the video segment, and the feature maps of all images of the first frame sequence form the first feature map matrix.

[0136] At the same time, inputting the 16-frame second frame sequence into the fast lane to extract a feature map for each frame image representing dynamic information of the video segment, and the feature maps of all images of the second frame sequence form the second feature map matrix.

[0137] S530: respectively performing a time sequence pooling operation on the first feature map matrix and the second feature map matrix, extracting features of a region of interest (ROI) based on a detection frame of the key frame on the obtained two time sequence pooling results, respectively performing a spatial pooling operation, obtaining features of a slow channel and features of a fast channel, the features of the slow channel representing static information of the video segment, and the features of the fast channel representing dynamic information of the video segment.

[0138] Pooling is a process of compressing one or more matrices created by a preceding convolution layer into a smaller matrix. In deep learning, pooling generally refers to spatial pooling. Application of pooling on a time sequence is referred to as time sequence pooling.

[0139] Taking a video segment of 16 frames of video segments as an example, after each frame of image passes through a convolutional neural network, 16 frames of feature maps are obtained. Since action category recognition is usually based on a video level rather than a frame level, a time sequence pooling method is needed to convert the frame features into video level features.

[0140] After time sequence pooling, ROI Align operation is performed on the two time sequence pooling results, respectively, to complete region feature aggregation, and then spatial pooling is performed to obtain features of a fast channel and a slow channel.

[0141] S540: fusing the features of the slow channel and the features of the fast channel, sequentially performing a full connection operation and a softmax operation on the fusion result, and obtaining probabilities of each action category of each detection frame of the key frame.

[0142] During fusion, the fusion is performed through a concat operation in the channel dimension, and then a full connection layer is used to obtain a feature of num_classes dimension, num_classes is the number of action categories, and then a softmax activation is performed to obtain probabilities of being recognized as each action category.

[0143] S550: taking an action category corresponding to a maximum probability as an action category of the detection frame of the key frame.

[0144] In this step, probabilities lower than a score threshold can be filtered out through the set score threshold, and then a maximum value is selected from the remaining probabilities, and an action category corresponding to the maximum value is the action category (pred) of the detection frame of the key frame.

[0145] The action recognition model further includes a lateral connection from the fast channel to the slow channel, and the lateral connection sends data of the fast channel into the slow channel.

[0146] Because the information of the fast lane and the slow lane is fused, one path needs to know the representation learned by the other path, and the data of the fast lane is sent into the slow lane through a lateral connection. Exemplarily, the connection mode of the lateral connection can be implemented by using a 3D convolution with a convolution kernel of 5x1. 2

[0147] Embodiment 2

[0148] The embodiment of the present application provides a pedestrian action continuous detection and recognition device, as shown in the figure, the device comprises: Figure 4

[0149] a video segmentation module 1, used for segmenting a to-be-detected video into a plurality of video clips, each video clip comprising a plurality of frame images.

[0150] a key frame selection module 2, used for selecting a frame image as a key frame in each video clip, and the remaining frame images as non-key frames.

[0151] For example, the middle frame image can be selected as the key frame in each video clip, and the remaining frame images are non-key frames.

[0152] a pedestrian detection module 3, which inputs all frame images of each video clip into a pedestrian detection model to obtain a certain number of detection boxes on each frame image of the video clip.

[0153] an ID matching module 4, which sequentially associates the detection boxes of a later frame image with the detection boxes of a previous frame image according to the order, and sets the number of the detection boxes of the later frame image on the association as the number of the detection boxes of the previous frame image associated therewith.

[0154] an action recognition module 5, which inputs all frame images and their detection boxes of each video clip into an action recognition model to obtain the action category of each detection box of the key frame.

[0155] a result sharing module 6, which sets the action category of the detection boxes of the non-key frame as the action category of the detection boxes of the key frame with the same number.

[0156] The device of the present application can further comprise:

[0157] a presentation module, used for displaying and presenting all detection boxes of the key frame and their action categories, and the detection boxes of the non-key frame with the same number as the detection boxes of the key frame and their action categories, and displaying and presenting the detection boxes of the non-key frame with a number different from that of the detection boxes of the key frame.

[0158] ​​The application realizes sharing of key frame recognition results for non-key frames through ID matching of detection boxes, thereby realizing frame-by-frame continuous detection and recognition of human body actions in a video, solving the problem of deviation caused by relying on key frame detection for overall video detection and evaluation, and achieving better and more stable detection and recognition effects, and better assisting in completing functions such as behavior analysis and abnormal detection in application scenarios; and the application solves the problem of discontinuity of pedestrian action detection and recognition, and detection boxes and recognition results of each frame of image are presented in a continuous manner, thereby improving the visual experience.

[0159] The action recognition model of the application comprises a slow lane and a fast lane which are executed in parallel, the slow lane and the fast lane are both convolutional neural networks, and the number of channels of the convolutional neural network of the fast lane is less than that of the convolutional neural network of the slow lane. Figure 2 The action recognition model further comprises a lateral connection from the fast lane to the slow lane, and the lateral connection sends data of the fast lane into the slow lane.

[0160] Based on the above-mentioned action recognition model, the action recognition module comprises:

[0161] The sampling unit is configured to sample the video segment according to different frame sampling rates to obtain a first frame sequence containing fewer frames of images and a second frame sequence containing more frames of images.

[0162] The feature map matrix extraction unit is configured to input the first frame sequence and the second frame sequence into the slow lane and the fast lane respectively to extract features, and obtain a first feature map matrix and a second feature map matrix respectively.

[0163] The feature calculation unit is configured to perform a time sequence pooling operation on the first feature map matrix and the second feature map matrix respectively, extract features of a region of interest based on the detection box of the key frame on the obtained two time sequence pooling results respectively, and perform a spatial pooling operation respectively to obtain features of the slow lane and features of the fast lane, wherein the features of the slow lane represent static information of the video segment, and the features of the fast lane represent dynamic information of the video segment.

[0164] The probability calculation unit is configured to fuse the features of the slow lane and the features of the fast lane, and sequentially perform a full connection operation and a softmax operation on the fusion result to obtain probabilities of each action category.

[0165] The category determination unit is configured to take an action category corresponding to a maximum value of the probabilities as an action category of the detection box of the key frame.

[0166] As an improvement of an embodiment of the application, the pedestrian detection module comprises:

[0167] a candidate detection frame obtaining unit, configured to input all frame images of the video segment into a YOLOX detection model, and obtain a plurality of candidate detection frames and confidence scores of the candidate detection frames on each frame image of the video segment.

[0168] an NMS unit, configured to perform a non-maximum suppression operation on the candidate detection frames according to a set NMS threshold.

[0169] a filtering unit, configured to filter out the candidate detection frames with confidence scores lower than a set confidence threshold from a result of the non-maximum suppression operation, to obtain the certain number of detection frames and the confidence scores.

[0170] As another improvement of the embodiment of the present application, the ID matching module comprises:

[0171] a prediction unit, configured to perform Kalman filter processing on the detection frames of the previous frame image, to obtain a prediction frame corresponding to the next frame image.

[0172] a classification unit, configured to compare the confidence scores of the detection frames of the next frame image with a set high-score threshold and a set low-score threshold, to take the detection frames of the next frame image with confidence scores greater than the high-score threshold as high-score detection frames, and take the detection frames of the next frame image with confidence scores greater than the low-score threshold and less than or equal to the high-score threshold as low-score detection frames.

[0173] a first matching unit, configured to perform a first matching on the prediction frame and the high-score detection frame, to obtain a first type of matching pair composed of the prediction frame and the high-score detection frame that pass the first matching, and the high-score detection frame and the prediction frame that fail the first matching.

[0174] a first association unit, configured to, for each first type of matching pair, associate the high-score detection frame in the first type of matching pair with the detection frame of the previous frame image corresponding to the prediction frame in the first type of matching pair, and set a number of the associated high-score detection frame as a number of the detection frame of the previous frame image.

[0175] a first post-processing unit, configured to compare the confidence scores of the high-score detection frames that fail the first matching with a set screening threshold, to set a new number for the high-score detection frame that fails the first matching and has a confidence score greater than the screening threshold, and discard the high-score detection frame that fails the first matching and has a confidence score not greater than the screening threshold.

[0176] The second matching unit is configured to perform second matching between the prediction frame that fails the first matching and the low-score bounding box to obtain a second type of matching pair composed of the prediction frame that fails the second matching and the low-score bounding box, a low-score bounding box that fails the second matching, and a prediction frame that fails the second matching.

[0177] The second association unit is configured to, for each second type of matching pair, associate the low-score bounding box in the second type of matching pair with the bounding box of the previous frame image corresponding to the prediction frame in the second type of matching pair, and set the number of the associated low-score bounding box as the number of the bounding box of the previous frame image corresponding to the prediction frame in the second type of matching pair.

[0178] The second post-processing unit is configured to discard the low-score bounding box that fails the second matching, retain the bounding box of the previous frame image corresponding to the prediction frame that fails the second matching, and perform data association with the bounding box of a subsequent frame image, and discard the bounding box of the subsequent frame image when the association is unsuccessful.

[0179] The device provided in the embodiment of the present application has the same implementation principle and technical effects as the method embodiment 1 described above. For brevity, the part of the device embodiment not mentioned in the description can be referred to the corresponding content in the method embodiment 1 described above. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device and unit described above can be referred to the corresponding process in the method embodiment 1 described above, which will not be described herein again.

[0180] Embodiment 3:

[0181] The method described in the embodiment 1 described above can be realized by a computer program to implement business logic and recorded on a storage medium, the storage medium can be read and executed by a computer to realize the effects of the scheme described in the embodiment 1. Therefore, the present application further provides a computer readable storage medium for continuous detection and recognition of pedestrian actions, including a memory for storing processor executable instructions, the instructions are executed by the processor to realize the steps of the pedestrian action continuous detection and recognition method including the embodiment 1.

[0182] The present application realizes the sharing of key frame recognition results for non-key frames through the ID matching of the bounding box, thereby realizing the frame-by-frame continuous detection and recognition of human actions in the video, solving the problem of deviation caused by relying only on key frame detection for overall detection and evaluation of the video, and realizing better and more stable detection and recognition effects, and better assisting the completion of behavior analysis and abnormal detection functions in the application scene; and the present application solves the problem of discontinuous detection and recognition of pedestrian actions, and the detection bounding box and the recognition result of each frame image are presented in a continuous manner, thereby improving the visual experience of presentation.

[0183] The storage medium can include a physical device for storing information, usually after digitizing the information and then storing it with media using electricity, magnetism, or optics. The storage medium can include devices that store information using electrical energy, such as various types of memory, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, U disks; devices that store information using optical methods, such as CDs or DVDs. Of course, there are other ways of reading the storage medium, such as quantum memory, graphene memory, and so on.

[0184] The storage medium described above can also include other embodiments according to the description of method embodiment 1. The implementation principle and technical effects of the present embodiment are the same as those of the foregoing method embodiment 1. For details, please refer to the description of the related method embodiment 1, which will not be repeated here.

[0185] Embodiment 4:

[0186] The present application also provides a device for continuous detection and recognition of pedestrian actions. The device can be a single computer or can include an actual operating device that uses one or more methods or one or more embodiment devices described in the specification. The device for continuous detection and recognition of pedestrian actions can include at least one processor and a memory storing computer executable instructions, and the processor executes the instructions to implement the steps of the pedestrian action continuous detection and recognition method described in any one or more embodiments 1.

[0187] The present application realizes the sharing of key frame recognition results for non-key frames by matching the ID of the detection frame, thereby realizing the frame-by-frame continuous detection and recognition of human actions in the video, solving the problem of deviation caused by relying only on key frame detection for overall video detection and evaluation, and achieving better and more stable detection and recognition effects, better assisting the completion of behavior analysis and abnormal detection functions in application scenarios; and the present application solves the problem of discontinuous pedestrian action detection and recognition, and the detection frame and recognition result of each image are presented in a continuous manner, improving the visual experience of presentation.

[0188] The device described above can also include other embodiments according to the description of method embodiment 1. The implementation principle and technical effects of the present embodiment are the same as those of the foregoing method embodiment 1. For details, please refer to the description of the related method embodiment 1, which will not be repeated here.

[0189] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the present application, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features therein, within the technical scope disclosed by the present application, and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for continuous detection and recognition of pedestrian actions, characterized in that, The method includes: The video to be detected is divided into multiple video segments, and each video segment includes multiple frames of images; In each video segment, select one frame as the keyframe and the remaining frames as non-keyframes. For each video segment, all frame images of the video segment are input into the pedestrian detection model, and a certain number of detection boxes are obtained on each frame image of the video segment. For all frames of each video segment, the detection boxes of the next frame are sequentially associated with the detection boxes of the previous frame, and the number of the associated detection box of the next frame is set to the number of the associated detection box of the previous frame. For each video segment, all frame images of the video segment and their detection boxes are input into the action recognition model to obtain the action category of each detection box of the key frame; For each video segment, the action category of the detection box of the non-keyframe is set to the action category of the detection box of the keyframe with the same number. The action recognition model includes a slow channel and a fast channel that are executed in parallel. Both the slow channel and the fast channel are convolutional neural networks, and the number of channels in the convolutional neural network of the fast channel is less than the number of channels in the convolutional neural network of the slow channel. For each video segment, all frame images and their detection boxes of the video segment are input into the action recognition model to obtain the action category of each detection box of the keyframe, including: The video segment is sampled according to different frame sampling rates to obtain a first frame sequence containing fewer frames and a second frame sequence containing more frames. The first frame sequence and the second frame sequence are respectively input into the slow channel and the fast channel to extract features, and the first feature map matrix and the second feature map matrix are obtained respectively. Temporal pooling operations are performed on the first feature map matrix and the second feature map matrix respectively. Based on the two temporal pooling results, features of the region of interest are extracted from the detection boxes of the keyframes respectively, and spatial pooling operations are performed respectively to obtain the features of the slow channel and the features of the fast channel. The features of the slow channel represent the static information of the video segment, and the features of the fast channel represent the dynamic information of the video segment. The features of the slow channel and the features of the fast channel are fused together, and the fusion result is subjected to a fully connected operation and a softmax operation in sequence to obtain the probability of each action category. The action category corresponding to the highest probability is used as the action category of the detection box in the keyframe.

2. The pedestrian action continuous detection and recognition method according to claim 1, characterized in that, The action recognition model also includes a lateral connection from the fast channel to the slow channel, wherein the lateral connection sends data from the fast channel to the slow channel.

3. The pedestrian action continuous detection and recognition method according to claim 1, characterized in that, For each video segment, all frame images of the video segment are input into the pedestrian detection model, and a certain number of detection boxes are obtained on each frame image of the video segment, including: Input all frame images of the video segment into the YOLOX detection model to obtain several candidate detection boxes and a confidence score of the candidate detection boxes identifying people on each frame image of the video segment; Non-maximum suppression is performed on the candidate detection boxes according to the set NMS threshold; Candidate detection boxes with confidence scores lower than a set confidence threshold are filtered out from the results of the non-maximum suppression operation to obtain the specified number of detection boxes and their confidence scores.

4. The pedestrian action continuous detection and recognition method according to claim 3, characterized in that, The step of sequentially associating the detection bounding boxes of the next frame with the detection bounding boxes of the previous frame for all frames of each video segment, and setting the number of the associated detection bounding box of the next frame to the number of the associated detection bounding box of the previous frame, includes: The detection bounding box of the previous frame image is processed by a Kalman filter to obtain the prediction bounding box corresponding to the next frame image. The confidence score of the detection box in the next frame image is compared with the set high score threshold and low score threshold. The detection boxes in the next frame image with a confidence score greater than the high score threshold are designated as high score detection boxes, and the detection boxes in the next frame image with a confidence score greater than the low score threshold and less than or equal to the high score threshold are designated as low score detection boxes. The predicted bounding box is matched with the high-scoring detection box for the first time to obtain a first-class matching pair consisting of the predicted bounding box and the high-scoring detection box that pass the first match, and a high-scoring detection box and a predicted bounding box that fail the first match. For each first-class matching pair, the high-resolution detection box in the first-class matching pair is associated with the detection box of the previous frame image corresponding to the prediction box in the first-class matching pair, and the number of the associated high-resolution detection box is set to the number of the detection box of the previous frame image associated with it. The confidence scores of high-scoring detection boxes that failed the first match are compared with the set filtering threshold. High-scoring detection boxes that failed the first match with a confidence score greater than the filtering threshold are assigned a new number, and high-scoring detection boxes that failed the first match with a confidence score not greater than the filtering threshold are discarded. The predicted boxes that failed the first match are matched with the low-scoring detection boxes a second time to obtain a second type of matching pair consisting of the predicted boxes and the low-scoring detection boxes that succeed in the second match, as well as the low-scoring detection boxes and the predicted boxes that failed the second match. For each second-class matching pair, associate the low-scoring detection box in the second-class matching pair with the detection box of the previous frame image corresponding to the predicted box in the second-class matching pair, and set the number of the associated low-scoring detection box to the number of the detection box of the previous frame image associated with it. Discard the low-scoring detection boxes that fail the second match, retain the detection boxes of the previous frame corresponding to the predicted boxes that fail the second match, and associate them with the detection boxes of subsequent frames. Discard the boxes if the association with the detection boxes of several subsequent frames fails.

5. The method for continuous detection and recognition of pedestrian actions according to claim 1, characterized in that, In each video segment, select the middle frame as the keyframe and the remaining frames as non-keyframes.

6. The method for continuous detection and recognition of pedestrian actions according to any one of claims 1-5, characterized in that, The method further includes: Display all detection boxes and their action categories of the keyframe, as well as the detection boxes and their action categories of non-keyframes with the same detection box number as the keyframe. Display the detection boxes of non-keyframes with different detection box numbers than the keyframe.

7. A pedestrian motion continuous detection and recognition device, characterized in that, The device includes: The video segmentation module is used to segment the video to be detected into multiple video segments, each of which includes multiple frames of images; The keyframe selection module is used to select one frame as a keyframe in each video segment, and the remaining frames as non-keyframes. The pedestrian detection module inputs all frame images of the video segment into the pedestrian detection model for each video segment, and obtains a certain number of detection boxes on each frame image of the video segment. The ID matching module sequentially associates the detection boxes of the next frame with the detection boxes of the previous frame for all frames of each video segment, and sets the number of the associated detection box of the next frame to the number of the associated detection box of the previous frame. The action recognition module inputs all frame images and their detection boxes of each video segment into the action recognition model to obtain the action category of each detection box of the key frame. The results sharing module sets the action category of the detection boxes of non-keyframes to the action category of the detection boxes of keyframes with the same number for each video segment. The action recognition model includes a slow channel and a fast channel that are executed in parallel. Both the slow channel and the fast channel are convolutional neural networks, and the number of channels in the convolutional neural network of the fast channel is less than the number of channels in the convolutional neural network of the slow channel. The action recognition module includes: The sampling unit is used to sample the video segment according to different frame sampling rates to obtain a first frame sequence containing fewer frame images and a second frame sequence containing more frame images. The feature map matrix extraction unit is used to input the first frame sequence and the second frame sequence into the slow channel and the fast channel respectively to extract features, and obtain the first feature map matrix and the second feature map matrix respectively. The feature calculation unit is used to perform temporal pooling operations on the first feature map matrix and the second feature map matrix respectively, extract the features of the region of interest based on the detection box of the key frame on the two temporal pooling results respectively, and perform spatial pooling operations respectively to obtain the features of the slow channel and the features of the fast channel. The features of the slow channel represent the static information of the video segment, and the features of the fast channel represent the dynamic information of the video segment. The probability calculation unit is used to fuse the features of the slow channel and the features of the fast channel, and perform a fully connected operation and a softmax operation on the fusion result in sequence to obtain the probability of each action category. The category determination unit is used to use the action category corresponding to the maximum probability as the action category of the detection box of the keyframe.

8. A computer-readable storage medium for continuous detection and recognition of pedestrian actions, characterized in that, It includes a memory for storing processor-executable instructions, which, when executed by the processor, implement the steps of the pedestrian motion continuous detection and recognition method according to any one of claims 1-6.

9. A device for continuous detection and recognition of pedestrian movements, characterized in that, It includes at least one processor and a memory storing computer-executable instructions, wherein the processor executes the instructions to implement the steps of the pedestrian action continuous detection and recognition method according to any one of claims 1-6.

Images