Behavior recognition method, behavior recognition device, computer equipment and storage medium
By using behavior recognition methods on mobile devices, the combination of pedestrian detection model and behavior recognition model is used to trace back the video frame image for behavior judgment, which solves the problem of high requirements for device resources by behavior recognition algorithms and realizes the mobile device Continuous behavior recognition.
Patent Information
- Application Number
- CN202210302176.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-25
AI Technical Summary
Current behavior recognition algorithms have high requirements for the computing resources and storage resources of the device, which makes it difficult for a long time to run on mobile devices.
Propose a behavior recognition method, obtain the original video data to be detected, input it into the pre-trained pedestrian detection model to determine the target pedestrian, perform target tracking and key point information, traverse the video frame image to determine whether the preset conditions are met, and before backtracking n Frame video frames and images to judge behaviors to reduce the load pressure on the mobile terminal.
It effectively reduces the load pressure on mobile devices during behavior recognition operation, and realizes continuous behavior recognition on mobile devices.
Smart Images

Figure CN114708613B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a behavior recognition method, a behavior recognition device, a computer device and a storage medium. Background Art
[0002] With the development of artificial intelligence technology, deep learning behavior recognition algorithms are often used in today's society to detect whether the target object has performed certain actions, and then give prompts or warnings, such as detecting pedestrians falling or fighting. However, most of the current behavior recognition algorithms have high requirements for the computing and storage resources of the device, and it is often difficult to run them continuously for a long time on mobile devices. Summary of the invention
[0003] The main purpose of the embodiments of the present disclosure is to propose a behavior recognition method, a behavior recognition device, a computer device and a storage medium, which can reduce the load pressure on the mobile terminal when the behavior recognition is running.
[0004] To achieve the above objective, a first aspect of an embodiment of the present disclosure provides a behavior recognition method, the method comprising:
[0005] Acquire original video data to be detected, and input the original video data into a pre-trained pedestrian detection model, so as to determine a target pedestrian from a video frame image of the original video data through the pedestrian detection model;
[0006] Tracking the target pedestrian to obtain a target video frame image and target trajectory information corresponding to the target pedestrian;
[0007] Determining key point information for the target pedestrian from the target video frame image according to the target trajectory information;
[0008] Traversing each video frame image of the original video data, and determining whether key point information for the target pedestrian in the currently traversed video frame image meets a preset behavior recognition condition;
[0009] When the key point information satisfies the preset behavior recognition condition, obtaining the first n video frame images of the currently traversed video frame image, where n is an integer greater than 1;
[0010] The first n video frame images of the currently traversed video frame images are input into a pre-trained behavior recognition model for behavior judgment, so as to obtain a behavior recognition result for the target pedestrian through the behavior recognition model.
[0011] In some embodiments, the target pedestrian is tracked to obtain a target video frame image and target trajectory information corresponding to the target pedestrian, including:
[0012] Inputting each video frame image of the original video data into a pre-trained appearance vector prediction network model, so as to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model;
[0013] Acquire motion information of the target pedestrian in each of the video frame images;
[0014] According to the appearance information and the motion information, a target video frame image and target trajectory information corresponding to the target pedestrian are obtained.
[0015] In some embodiments, obtaining a target video frame image and target trajectory information corresponding to the target pedestrian according to the appearance information and the motion information includes:
[0016] Traverse each of the video frame images, and perform the following processing on the currently traversed video frame image:
[0017] Acquire first state information of the target pedestrian in the currently traversed video frame image, wherein the first state information includes first motion information and first appearance information;
[0018] Predicting the appearance information of the target pedestrian in the next video frame image according to the first appearance information to obtain predicted appearance information;
[0019] Acquire second state information of the target pedestrian in the next video frame image, wherein the second state information includes second motion information and second appearance information;
[0020] Obtaining a first distance according to the first motion information and the second motion information;
[0021] Obtaining a second distance according to the predicted appearance information and the second appearance information;
[0022] Performing weighted calculation on the first distance and the second distance to obtain a cost matrix;
[0023] The target video frame image and target trajectory information corresponding to the target pedestrian are obtained according to the cost matrix.
[0024] In some embodiments, the appearance vector prediction network model includes a convolution module, a pooling module, a residual module and a fully connected module, and the inputting of each video frame image of the original video data into the pre-trained appearance vector prediction network model to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model includes:
[0025] Preprocessing each of the video frame images according to the target pedestrian to obtain a preprocessed image;
[0026] Performing first feature processing on the preprocessed image through the convolution module to obtain a first feature image;
[0027] Performing second feature processing on the first feature image by the pooling module to obtain a second feature image;
[0028] Performing third feature processing on the second feature image by the residual module to obtain a third feature image;
[0029] The third feature image is subjected to fourth feature processing by the fully connected module to obtain appearance information of the target pedestrian in each video frame image.
[0030] In some embodiments, the residual module includes a first residual unit and a second residual unit, the first residual unit does not include a residual connection, the second residual unit includes the residual connection, and the performing a third feature processing on the second feature image through the residual module to obtain a third feature image includes:
[0031] Performing fifth feature processing on the second feature image by the first residual unit to obtain a fifth feature image;
[0032] The sixth feature processing is performed on the fifth feature image by the second residual unit to obtain a third feature image.
[0033] In some embodiments, determining key point information for the target pedestrian from the target video frame image according to the target trajectory information includes:
[0034] Determining a candidate range corresponding to the target pedestrian in the target video frame image according to the target trajectory information;
[0035] Expanding the candidate range to obtain a target range;
[0036] The target range is input into a preset key point algorithm model to perform key point calculation to determine key point information for the target pedestrian.
[0037] In some embodiments, inputting the target range into a preset key point algorithm model to perform key point calculation to determine key point information for the target pedestrian includes:
[0038] Input the target range into a preset key point algorithm model to perform key point calculation to obtain a plurality of heat map data, wherein each heat map data uniquely corresponds to a key point of the target pedestrian in the target video frame image;
[0039] Position mapping is performed on the multiple heat map data to determine key point information for the target pedestrian.
[0040] A second aspect of the present disclosure provides a behavior recognition device, the device comprising:
[0041] A pedestrian detection module, used for acquiring raw video data to be detected, and inputting the raw video data into a pre-trained pedestrian detection model, so as to determine a target pedestrian from a video frame image of the raw video data through the pedestrian detection model;
[0042] A target tracking module is used to track the target pedestrian and obtain a target video frame image and target trajectory information corresponding to the target pedestrian;
[0043] A key point determination module, used to determine key point information for the target pedestrian from the target video frame image according to the target trajectory information;
[0044] A condition judgment module is used to traverse each video frame image of the original video data to determine whether the key point information of the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition;
[0045] A video frame image acquisition module, used for acquiring n video frame images before the currently traversed video frame image when the key point information meets the preset behavior recognition condition, where n is an integer greater than 1;
[0046] The behavior judgment module is used to input the first n video frame images of the currently traversed video frame image into a pre-trained behavior recognition model for behavior judgment, so as to obtain a behavior recognition result for the target pedestrian through the behavior recognition model.
[0047] The third aspect of the embodiments of the present disclosure proposes a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the behavior recognition method as described in any one of the embodiments of the first aspect of the present application.
[0048] The fourth aspect of the embodiments of the present disclosure proposes a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the behavior recognition method as described in any one of the embodiments of the first aspect of the present application.
[0049] The behavior recognition method, behavior recognition device, computer equipment and storage medium proposed in the embodiments of the present disclosure obtain the original video data to be detected and input the original video data into the pre-trained pedestrian detection model, so as to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model. The target pedestrian is tracked to obtain the target video frame image and target trajectory information corresponding to the target pedestrian, and the key point information for the target pedestrian is determined from the target video frame image according to the target trajectory information. Each video frame image of the original video data is traversed, and the behavior state of the target pedestrian is judged by determining whether the key point information for the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition. When the key point information meets the preset behavior recognition condition, the first n video frame images of the currently traversed video frame image are obtained, where n is an integer greater than 1. The first n video frame images of the currently traversed video frame image are input into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model. The disclosed embodiment makes a judgment by backtracking the behaviors of the first n video frames of the target video frame image, thereby reducing the load pressure on the mobile terminal when the behavior recognition is running. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a flow chart of a behavior recognition method provided by an embodiment of the present disclosure;
[0051] Figure 2 yes Figure 1 Flow chart of step S120 in FIG.
[0052] Figure 3 yes Figure 2 Flow chart of step S230 in FIG.
[0053] Figure 4 yes Figure 2 Flow chart of step S210 in FIG.
[0054] Figure 5 yes Figure 4 Flowchart of step S440 in FIG.
[0055] Figure 6 is a module schematic diagram of an appearance vector prediction network model provided by an embodiment of the present disclosure;
[0056] Figure 7 yes Figure 6 A module schematic diagram of the first residual module 630 provided in FIG.
[0057] Figure 8 yes Figure 1 Flow chart of step S130 in FIG.
[0058] Fig. 9 yes Figure 8 Flowchart of step S830 in FIG.
[0059] Fig.10 is a module structure block diagram of a behavior recognition device provided by an embodiment of the present disclosure;
[0060] Fig.11 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present disclosure. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0064] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.
[0065] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0066] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0067] First, some nouns involved in this application are analyzed:
[0068] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0069] Kalman filter: It is a motion model used to predict the trajectory of a target and use tracking results with higher confidence to correct the prediction results.
[0070] Hungarian algorithm: It is an algorithm for finding the maximum matching of a bipartite graph. In the multi-target tracking problem, it is an algorithm for finding the optimal matching solution for several targets in the previous and next frames.
[0071] Sort algorithm: It is an online tracking method that uses a method based on Kalman filter and Hungarian algorithm for tracking. That is, it introduces a linear velocity model and Kalman filter to predict the position. Position prediction is performed first and then matching. In the absence of a suitable matching detection frame, the result of the motion model can be used to predict the position of the object. Afterwards, in the data association stage, the Sort algorithm uses the Hungarian algorithm for frame-by-frame association.
[0072] DeepSort multi-target tracking algorithm (DeepSort algorithm for short) introduces an appearance feature extraction network based on the Sort algorithm for target tracking. The network is trained offline on the pedestrian re-identification dataset. During the target tracking process, the appearance features of the target are extracted for nearest neighbor matching, which can effectively improve the target tracking effect under occlusion.
[0073] Mahalanobis Distance: It is a commonly used distance indicator in metric learning. Like Euclidean distance, Manhattan distance, Hamming distance, etc., it is used as a similarity indicator for evaluating data. It can deal with the problem of non-independent and identically distributed dimensions in high-dimensional linearly distributed data.
[0074] Cosine distance: also known as cosine similarity, uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals.
[0075] Heatmap: It is a visualization method that is widely used in various big data analysis scenarios due to its rich color changes and vivid and full information expression.
[0076] With the development of artificial intelligence technology, deep learning behavior recognition algorithms are often used in today's society to detect whether the target object has performed certain actions, and then give prompts or warnings, such as detecting pedestrians falling or fighting. However, most of the current behavior recognition algorithms have high requirements for the computing and storage resources of the device, and it is often difficult to run them continuously for a long time on mobile devices.
[0077] Based on this, the embodiment of the present disclosure proposes a behavior recognition method, a behavior recognition device, a computer device and a storage medium, which obtains the original video data to be detected and inputs the original video data into a pre-trained pedestrian detection model, so as to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model. The target pedestrian is tracked to obtain the target video frame image and target trajectory information corresponding to the target pedestrian, and the key point information for the target pedestrian is determined from the target video frame image according to the target trajectory information. Each video frame image of the original video data is traversed, and the behavior state of the target pedestrian is judged by determining whether the key point information for the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition. When the key point information meets the preset behavior recognition condition, the first n video frame images of the currently traversed video frame image are obtained, where n is an integer greater than 1. The first n video frame images of the currently traversed video frame image are input into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model. The disclosed embodiment makes a judgment by backtracking the behaviors of the first n video frames of the target video frame image, thereby reducing the load pressure on the mobile terminal when the behavior recognition is running.
[0078] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0079] The behavior recognition method provided in the embodiments of the present disclosure relates to the field of artificial intelligence. The behavior recognition method provided in the embodiments of the present disclosure can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of a training method for implementing a model, etc., but is not limited to the above forms.
[0080] The disclosed embodiments can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0081] Please refer to Figure 1 According to the behavior recognition method of the first aspect of the embodiment of the present disclosure, it includes but is not limited to steps S110 to S160.
[0082] Step S110, obtaining original video data to be detected, and inputting the original video data into a pre-trained pedestrian detection model, so as to determine a target pedestrian from a video frame image of the original video data through the pedestrian detection model;
[0083] Step S120, tracking the target pedestrian to obtain a target video frame image and target trajectory information corresponding to the target pedestrian;
[0084] Step S130, determining key point information for the target pedestrian from the target video frame image according to the target trajectory information;
[0085] Step S140, traversing each video frame image of the original video data, and determining whether the key point information of the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition;
[0086] Step S150, when the key point information meets the preset behavior recognition condition, obtaining the previous n video frame images of the currently traversed video frame image, where n is an integer greater than 1;
[0087] Step S160, inputting the first n video frame images of the currently traversed video frame image into a pre-trained behavior recognition model for behavior judgment, so as to obtain a behavior recognition result for the target pedestrian through the behavior recognition model.
[0088] In step S110 of some embodiments, the original video data to be detected is obtained, wherein the original video data may be specially recorded video data, or video data stored from an existing video surveillance device. After the original video data is obtained, the video frame image corresponding to each frame in the original video data is extracted. The original video data is input into a pre-trained pedestrian detection model to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model. Specifically, after the original video data is input into the pre-trained pedestrian detection model, the output detection result includes one or more pedestrian frames in each video frame image, that is, multiple target pedestrians in the video frame image are determined based on the pedestrian frames, thereby achieving target tracking of each target pedestrian.
[0089] It should be noted that the pre-trained pedestrian detection model can use the pedestrian detection algorithm to perform data training on the pedestrian training set. Among them, the pedestrian detection algorithm can use the Yolox-nano network, which is a lightweight visual structure network. Yolox is an Anchor-free (no anchor node) that can avoid the non-optimal selection of the preset Anchor (also known as Anchor box, which is a group of rectangular boxes clustered on the training set using methods such as k-means before training, representing the length and width scale of the main distribution of the target in the data set) on the new data set.
[0090] It should be noted that in order to improve the robustness of the pre-trained pedestrian detection model, the multi-selected pedestrian training sets include the COCO dataset (Person class, i.e. the COCO dataset containing people), the PennFuDan pedestrian detection dataset, the Stanford-40 behavior recognition dataset, and the multi-target tracking-MOT16 dataset, so as to perform network training based on a variety of datasets and achieve efficient pedestrian recognition.
[0091] In step S120 of some embodiments, in order to accurately distinguish and identify the behavioral states of different target pedestrians in a multi-person scenario, the target pedestrian is tracked to obtain the target video frame image and target trajectory information corresponding to the target pedestrian. The present application can achieve target tracking of target pedestrians in different scenarios.
[0092] The disclosed embodiment proposes a behavior recognition method, a behavior recognition device, a computer device and a storage medium, which obtains the original video data to be detected and inputs the original video data into a pre-trained pedestrian detection model, so as to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model. The target pedestrian is tracked to obtain the target video frame image and target trajectory information corresponding to the target pedestrian, and the key point information for the target pedestrian is determined from the target video frame image according to the target trajectory information. Each video frame image of the original video data is traversed, and the behavior state of the target pedestrian is judged by determining whether the key point information for the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition. When the key point information meets the preset behavior recognition condition, the first n video frame images of the currently traversed video frame image are obtained, where n is an integer greater than 1. The first n video frame images of the currently traversed video frame image are input into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model. The disclosed embodiment can reduce the load pressure on the mobile terminal when the behavior recognition is running by tracing back the behaviors of the first n frames of the target video frame image to make a judgment.
[0093] Please refer to Figure 2 In some embodiments, step S120 specifically includes but is not limited to steps S210 to S230.
[0094] Step S210, inputting each video frame image of the original video data into a pre-trained appearance vector prediction network model, so as to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model;
[0095] Step S220, obtaining motion information of the target pedestrian in each video frame image;
[0096] Step S230, obtaining a target video frame image and target trajectory information corresponding to the target pedestrian according to the appearance information and the motion information.
[0097] In step S210 of some embodiments, in order to accurately distinguish and identify the behavior states of different target pedestrians in a multi-person scenario, each video frame image of the original video data is input into a pre-trained appearance vector prediction network model, so as to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model. Specifically, the target tracking method for the target pedestrian can use the target tracking model of DeepSort to achieve the tracking of the target pedestrian, and the DeepSort algorithm combines the motion information and appearance information of the target pedestrian, and then each video frame image of the original video data is input into the pre-trained appearance vector prediction network model to obtain the appearance information of the target pedestrian in each video frame image.
[0098] In step S220 of some embodiments, the motion information of the target pedestrian in each video frame image is obtained. Specifically, the motion information of the target pedestrian includes the motion state of the target pedestrian and the pedestrian frame corresponding to the target pedestrian obtained in step S110. The motion information of the target pedestrian is obtained by: Indicates, where (u, v) represents the coordinate position of the center of the pedestrian frame corresponding to the target pedestrian, γ represents the aspect ratio of the pedestrian frame corresponding to the target pedestrian, and h represents the height and width of the pedestrian frame corresponding to the target pedestrian. Indicates the component running speed of the center of the pedestrian box corresponding to the target pedestrian, Indicates the running speed of the height-to-width ratio of the pedestrian box corresponding to the target pedestrian, Indicates the height, width and running speed of the pedestrian box corresponding to the target pedestrian.
[0099] In step S230 of some embodiments, a target video frame image and target trajectory information corresponding to the target pedestrian are obtained based on the appearance information and the motion information, and then the running trajectory state of the target pedestrian is determined based on the target video frame image and the target trajectory information. The disclosed embodiment can reduce the load pressure on the mobile terminal when the behavior recognition is running by tracing back the previous n frames of the target video frame image.
[0100] Please refer to Figure 3 In some embodiments, step S230 specifically includes but is not limited to steps S310 to S380.
[0101] Step S310, traverse each video frame image, and perform the following processing on the currently traversed video frame image:
[0102] Step S320, obtaining first state information of the target pedestrian in the currently traversed video frame image, wherein the first state information includes first motion information and first appearance information;
[0103] Step S330, predicting state information of the target pedestrian in the next video frame image according to the first appearance information to obtain predicted appearance information;
[0104] Step S340, obtaining second state information of the target pedestrian in the next video frame image, wherein the second state information includes second motion information and second appearance information;
[0105] Step S350, obtaining a first distance according to the first motion information and the second motion information;
[0106] Step S360, obtaining a second distance according to the predicted appearance information and the second appearance information;
[0107] Step S370, performing weighted calculation on the first distance and the second distance to obtain a cost matrix;
[0108] Step S380, obtaining the target video frame image and target trajectory information corresponding to the target pedestrian according to the cost matrix.
[0109] In some embodiments, in steps S310 to S340, in order to accurately locate the target video frame image in the original video data, each video frame image in the original video data is traversed, and the following processing is performed on the currently traversed video frame image, namely, obtaining the first state information of the target pedestrian in the currently traversed video frame image, wherein the first state information includes first motion information and first appearance information, and the representation method of the first motion information is the same as the representation method of the above-mentioned motion information, and the representation method of the first appearance information is the same as the representation method of the above-mentioned appearance information, which will not be repeated here. The appearance information of the target pedestrian in the next frame of the video frame image is predicted based on the first appearance information to obtain the predicted appearance information. Specifically, the Kalman filtering method can be used to predict the state information of the currently traversed video frame image in the next frame, wherein the appearance information can be predicted by a CNN network. In a specific embodiment, each pedestrian frame calculates a 128-dimensional vector as the appearance vector of the pedestrian frame, that is, the corresponding predicted appearance information is predicted. In order to track the running trajectory of the target pedestrian, the second state information of the target pedestrian in the next frame of video frame image is obtained, wherein the second state information includes second motion information and second appearance information, and the representation method of the second motion information is the same as the representation method of the above motion information, and the representation method of the second appearance information is the same as the representation method of the above appearance information, which will not be repeated here. The disclosed embodiment locates the target video frame image with key point information in the original video data, and then judges by tracing back the behavior of the previous n frames of the target video frame image, thereby reducing the load pressure on the mobile terminal during the uninterrupted operation of behavior recognition.
[0110] In steps S340 to S380 of some embodiments, after associating and obtaining the second state information of the pedestrian frame corresponding to the target pedestrian in the next frame of video frame image, the first distance is obtained according to the first motion information and the second motion information, and the second distance is obtained according to the predicted appearance information and the second appearance information. Specifically, the first distance d1 between the first motion information and the second motion information is calculated according to the measurement method of Mahalanobis distance, and the second distance d2 between the predicted appearance information and the second appearance information is calculated according to the measurement method of cosine distance. The first distance d1 and the second distance d2 are weightedly calculated as shown in formula (1) to obtain a cost matrix as a comprehensive matching degree. In order to obtain the running trajectory of the target pedestrian, the Hungarian algorithm is used to match the cost matrix, the corresponding pedestrian frame in the next frame of video frame image is associated with the running trajectory of the target pedestrian, and the state parameters of the Kalman filter in the predicted trajectory are updated, and finally the target video frame image and target trajectory information corresponding to the target pedestrian are obtained. The disclosed embodiment locates the target video frame image with key information in the original video data, and then makes a judgment by tracing back the behavior of the first n frames of the target video frame image, thereby reducing the load pressure on the mobile terminal during the uninterrupted operation of behavior recognition.
[0111] It should be noted that the specific calculation method of the first distance and the second distance in the present invention is not limited to the measurement method of the Mahalanobis distance and the measurement method of the cosine distance, and can be selected according to the actual needs of those skilled in the art, which will not be repeated here.
[0112] Please refer to Figure 4 In some embodiments, the appearance vector prediction network model includes a convolution module, a pooling module, a residual module and a fully connected module, and step S210 specifically includes but is not limited to steps S410 to S450.
[0113] Step S410, preprocessing each video frame image according to the target pedestrian to obtain a preprocessed image;
[0114] Step S420, performing first feature processing on the preprocessed image through a convolution module to obtain a first feature image;
[0115] Step S430, performing second feature processing on the first feature image through a pooling module to obtain a second feature image;
[0116] Step S440, performing third feature processing on the second feature image through a residual module to obtain a third feature image;
[0117] Step S450: Perform fourth feature processing on the third feature image through a fully connected module to obtain appearance information of the target pedestrian in each video frame image.
[0118] In steps S410 to S450 of some embodiments, in order to improve the convergence speed of the appearance prediction network model and improve the efficiency of model training, the appearance vector prediction network model includes a convolution module, a pooling module, a residual module and a fully connected module. Specifically, after the target pedestrian is determined from the video frame image of the original video data by the pedestrian detection model, the pedestrian frame about the target pedestrian in each video frame image is obtained, and the image area corresponding to the pedestrian frame is preprocessed. For example, the image area corresponding to the pedestrian frame is adjusted to an image with a data format of 3*256*128, and a preprocessed image is obtained. The preprocessed image is subjected to the first feature processing by the convolution module to obtain a first feature image with a feature map size of 16*256*128 (16 represents the number of convolution kernels in the convolution layer, 256 represents the height of the feature map, and 128 represents the width of the feature map). In order to simplify the complexity of the network model, the first feature image is processed with the second feature through the pooling module, that is, the size of the first feature image is reduced by using the maximum pooling (MaxPool) to obtain a second feature image of 16*128*64. After that, the second feature image is processed with the third feature through the residual module to obtain a third feature image of 128*1*1. Finally, the third feature image is processed with the fourth feature through the fully connected module to obtain the appearance information of the target pedestrian in each video frame image, that is, the output is a 128-dimensional appearance vector.
[0119] Please refer to Figure 5 In some embodiments, the residual module includes multiple residual units, wherein each residual unit includes a first residual unit and a second residual unit, the first residual unit does not include a residual connection, and the second residual unit includes a residual connection, and step S440 specifically includes but is not limited to steps S510 to S520.
[0120] Step S510, performing fifth feature processing on the second feature image through the first residual unit to obtain a fifth feature image;
[0121] Step S520: Perform sixth feature processing on the fifth feature image through the second residual unit to obtain a third feature image.
[0122] In some embodiments of the present invention, in step S510 to step S520, the residual module includes a plurality of residual units, wherein each residual unit includes a first residual unit and a second residual unit, and the first residual unit does not include a residual connection. Specifically, the first residual unit performs the fifth feature processing on the second feature image, that is, the second feature image is first subjected to the first convolution subunit, and the first convolution subunit is used to adjust the number of convolution channels of the second feature image. In order to reduce the number of parameters of the network model and improve the calculation speed of the network model, the second feature image after the first convolution subunit is subjected to the first depth convolution subunit, and the first depth convolution subunit is used to perform feature calculation on the second feature image and reduce the image size of the second feature image. The second feature image after the first depth convolution subunit is subjected to the second convolution subunit to obtain the fifth feature image, and the second convolution subunit is used to fuse the information between the convolution channels. Then, the sixth feature processing is performed on the fifth feature image through the second residual unit, that is, the third convolution subunit is performed on the fifth feature image, and then the second depth convolution subunit is performed on the fifth feature image after the third convolution subunit. Finally, the fourth convolution subunit is performed on the fifth feature image after the second depth convolution subunit to obtain the third feature image.
[0123] In a specific embodiment, please refer to Figure 6 and Figure 7, the appearance vector prediction network model includes a convolution module 610, a pooling module 620, a first residual module 630, a second residual module 640, a third residual module 650, and a fully connected module 660. Obtain a preprocessed image, perform a first feature processing on the preprocessed image through the convolution module 610 to obtain a first feature image, perform a second feature processing on the first feature image through the pooling module 620 to obtain a second feature image, and perform a third feature processing on the second feature image through the residual module to obtain a third feature image, specifically including: performing a third feature processing on the second feature image through the first residual module 630 to obtain a first residual feature image, performing a seventh feature processing on the first residual feature image through the second residual module 640 to obtain a second residual feature image, and performing an eighth feature processing on the second residual feature image through the third residual module 650 to obtain a third feature image. Afterwards, perform a fourth feature processing on the third feature image through the fully connected module 660 to obtain the appearance information of the target pedestrian in each video frame image. Among them, the first residual module 630 includes a first residual unit 631 and a second residual unit 632. In the first residual unit 631, a 1*1 first convolution subunit 6311 is first performed on the second feature image, and then a first depth convolution subunit 6312 with a step size of 2 is performed on the second feature image. Finally, a 1*1 second convolution subunit 6313 is performed on the second feature image to obtain a fifth feature image. In the second residual unit 632, a 1*1 third convolution subunit 6321 is first performed on the fifth feature image, and then a second depth convolution subunit 6322 with a step size of 1 is performed on the fifth feature image. Finally, a 1*1 fourth convolution subunit 6323 is performed on the fifth feature image to obtain a sixth feature image. It should be noted that the image processing process of the second residual module 640 and the third residual module 650 is the same as that of the first residual module 630, which will not be repeated here.
[0124] Please refer to Figure 8 In some embodiments, step S130 specifically includes but is not limited to steps S810 to S830.
[0125] Step S810, determining a candidate range corresponding to the target pedestrian in the target video frame image according to the target trajectory information;
[0126] Step S820, expanding the candidate range to obtain a target range;
[0127] Step S830: input the target range into a preset key point algorithm model to perform key point calculation to determine key point information for the target pedestrian.
[0128] In step S810 of some embodiments, after the target pedestrian is determined from the video frame images of the original video data by the pedestrian detection model, a pedestrian frame about the target pedestrian in each video frame image is obtained, and after the target pedestrian is tracked, a target video frame image and target trajectory information corresponding to the target pedestrian are obtained, and the pedestrian frame in the target video frame image is used to frame the target pedestrian being tracked, and the candidate range corresponding to the target pedestrian is determined based on the pedestrian frame. The disclosed embodiment determines the candidate range corresponding to the target pedestrian, obtains the key point information of the target pedestrian, and then makes a judgment by backtracking the previous n frames of the target video frame image with key point information, thereby reducing the load pressure on the mobile terminal during the operation of behavior recognition.
[0129] In some embodiments, in steps S820 to S830, in order to more accurately calculate the key point information of the target pedestrian in the target video frame image, the candidate range in the pedestrian frame is expanded. For example, taking the candidate range as the original pedestrian frame as an example, the original pedestrian frame is expanded to 1.1 times the range to obtain the target range corresponding to the target pedestrian. The target range is input into the preset key point algorithm model for key point calculation to determine the key point information for the target pedestrian. Among them, the preset key point algorithm model can be constructed using a full convolutional network.
[0130] Please refer to Fig. 9 In some embodiments, step S830 specifically includes but is not limited to step S910 and step S920.
[0131] Step S910, inputting the target range into a preset key point algorithm model to perform key point calculation, and obtaining a plurality of heat map data, wherein each heat map data uniquely corresponds to a key point of the target pedestrian in the target video frame image;
[0132] Step S920: Position mapping is performed on the plurality of heat map data to determine key point information for the target pedestrian.
[0133] In step S910 and step S920 of some embodiments, the target range is input into a preset key point algorithm model for key point calculation, and multiple heat map data corresponding to the pedestrian frame in the target video frame image are obtained, wherein each heat map data uniquely corresponds to the key point of the target pedestrian in the target video frame image. For example, 17 heat map data corresponding to the pedestrian frame in the target video frame image are obtained according to the preset key point algorithm model, and each heat map data uniquely corresponds to the key point of the target pedestrian in the target video frame image. The position of the maximum value in each heat map data is mapped back to the target video frame image to obtain the key point coordinates corresponding to the target pedestrian in the target video frame image. Therefore, the preset key point algorithm model is used to calculate the key points. When this method is applied to a mobile device, high-speed computing and analysis can be achieved. The disclosed embodiment can reduce the load pressure on the mobile terminal during the operation of behavior recognition by locating the target video frame image with key information in the original video data.
[0134] In step S140 of some embodiments, it is determined whether the key point information for the target pedestrian in the currently traversed video frame image satisfies the preset behavior recognition conditions. Specifically, the relationship between the coordinates of the obtained key point coordinates is judged. For example, the preset behavior recognition condition is the falling state, and its corresponding judgment condition is that the vector formed between key points such as the head and buttocks satisfies the set inclination angle. Then, when the key point information of the target pedestrian satisfies the behavior recognition condition of the falling state, the spontaneous cause of the target pedestrian falling is judged. Among them, the preset behavior recognition conditions also include a shaking state. It should be noted that the preset behavior recognition conditions can be set according to actual behavior needs, which will not be repeated here.
[0135] In steps S150 to S160 of some embodiments, in order to more completely and accurately identify the behavior state of the target pedestrian, by backtracking the behavior of the first n frames of the target video frame image, that is, using the timing information of the video to make a judgment, when the behavior recognition condition is met, the first n frames of the video frame image of the currently traversed video frame image are obtained, where n is an integer greater than 1, and the first n frames of the video frame image of the currently traversed video frame image are input into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model. Specifically, when it is determined that the target pedestrian is in a state of falling down, but it is impossible to determine whether the reason for the falling down is spontaneous lying down to rest or falling down due to an accident, the behavior recognition result of the target pedestrian is determined by obtaining the first n frames of the video frame image of the currently traversed video frame image for judgment. For example, in fall detection, the position of the target pedestrian is first determined in the video frame image, and the key point information of the target pedestrian is obtained by target tracking and key point detection of the target pedestrian. When the angles between the key points of the head, hips, and ankles of the target pedestrian in the currently traversed video frame image are greater than the preset threshold, the target pedestrian is considered to be in a fallen state, and the first 10 video frame images of the currently traversed video frame image are taken and sent to the behavior recognition model for judgment, and then it is judged whether the falling state of the target pedestrian is a spontaneous fall or an accidental fall, and then it is decided whether to give an alarm prompt. When the falling state of the target pedestrian is a spontaneous fall, no alarm prompt is given; and when the falling state of the target pedestrian is an accidental fall, an alarm prompt is given in time. The disclosed embodiment can reduce the load pressure on the mobile terminal when the behavior recognition is running by tracing back the behavior of the first n frames of the target video frame image.
[0136] The behavior recognition method, behavior recognition device, computer equipment and storage medium proposed in the embodiment of the present disclosure obtain the original video data to be detected, and input the original video data into the pre-trained pedestrian detection model, so as to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model; track the target pedestrian to obtain the target video frame image and target trajectory information corresponding to the target pedestrian; determine the key point information for the target pedestrian from the target video frame image according to the target trajectory information; determine whether the key point information for the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition to judge the behavior state of the target pedestrian; when the key point information meets the preset behavior recognition condition, obtain the first n frames of the currently traversed video frame image, where n is an integer greater than 1; input the first n frames of the currently traversed video frame image into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model. The embodiment of the present disclosure can reduce the load pressure on the mobile terminal when the behavior recognition is running by tracing back the first n frames of the target video frame image for behavior judgment.
[0137] The present disclosure also provides a behavior recognition device, please refer to Fig.10 , the above-mentioned behavior recognition method can be implemented. The device includes: a pedestrian detection module 1010, a target tracking module 1020, a key point determination module 1030, a condition judgment module 1040, a video frame image acquisition module 1050 and a behavior judgment module 1060. Among them, the pedestrian detection module 1010 is used to obtain the original video data to be detected, and input the original video data into the pre-trained pedestrian detection model, so as to determine the target pedestrian from the video frame image of the original video data through the pedestrian detection model; the target tracking module 1020 is used to track the target pedestrian and obtain the target video frame image and target trajectory information corresponding to the target pedestrian; the key point determination module 1030 is used to determine the key point information for the target pedestrian from the target video frame image according to the target trajectory information; the condition judgment module 1040 is used to traverse each video frame image of the original video data, and determine whether the key point information for the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition; the video frame image acquisition module 1050 is used to obtain the first n frames of the currently traversed video frame image when the key point information meets the preset behavior recognition condition, where n is an integer greater than 1; the behavior judgment module 1060 is used to input the first n frames of the currently traversed video frame image into the pre-trained behavior recognition model for behavior judgment, so as to obtain the behavior recognition result for the target pedestrian through the behavior recognition model.
[0138] The behavior recognition device of the embodiment of the present disclosure is used to execute the behavior recognition method in the above embodiment. Its specific processing process is the same as the behavior recognition method in the above embodiment, and will not be repeated here.
[0139] The embodiments of the present disclosure also provide a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute any method as described in any one of the embodiments of the first aspect of the present application.
[0140] Combine the following Fig.11 The hardware structure of the computer device is described in detail. The computer device includes: a processor 1110 , a memory 1120 , an input / output interface 1130 , a communication interface 1140 and a bus 1150 .
[0141] The processor 1110 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;
[0142] The memory 1120 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1120 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1120, and the processor 1110 calls and executes the behavior recognition method of the embodiments of this disclosure;
[0143] Input / output interface 1130, used to implement information input and output;
[0144] The communication interface 1140 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0145] bus 1150 , which transmits information between the various components of the device (e.g., processor 1110 , memory 1120 , input / output interface 1130 , and communication interface 1140 );
[0146] The processor 1110 , the memory 1120 , the input / output interface 1130 , and the communication interface 1140 are connected to each other in communication within the device via the bus 1150 .
[0147] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the behavior recognition method of the embodiment of the present disclosure.
[0148] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0149] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.
[0150] It can be understood by those skilled in the art that Figures 1 to 10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.
[0151] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0152] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0154] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0155] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0159] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the scope of the rights of the present disclosure is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the rights of the present disclosure.
Claims
1. A behavior recognition method, characterized in that: The method comprises: Acquire original video data to be detected, and input the original video data into a pre-trained pedestrian detection model, so as to determine a target pedestrian from a video frame image of the original video data through the pedestrian detection model; Track the target pedestrian to obtain a target video frame image and target trajectory information corresponding to the target pedestrian; the tracking of the target pedestrian to obtain a target video frame image and target trajectory information corresponding to the target pedestrian includes: inputting each video frame image of the original video data into a pre-trained appearance vector prediction network model to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model; obtaining the motion information of the target pedestrian in each video frame image; obtaining the target video frame image and target trajectory information corresponding to the target pedestrian based on the appearance information and the motion information; wherein, obtaining the target video frame image and target trajectory information corresponding to the target pedestrian based on the appearance information and the motion information includes: traversing each video frame image, The following processing is performed in the currently traversed video frame image: obtaining the first state information of the target pedestrian in the currently traversed video frame image, wherein the first state information includes first motion information and first appearance information; predicting the appearance information of the target pedestrian in the next frame of video frame image according to the first appearance information to obtain predicted appearance information; obtaining the second state information of the target pedestrian in the next frame of video frame image, wherein the second state information includes second motion information and second appearance information; obtaining a first distance according to the first motion information and the second motion information; obtaining a second distance according to the predicted appearance information and the second appearance information; performing weighted calculation on the first distance and the second distance to obtain a cost matrix; obtaining a target video frame image and target trajectory information corresponding to the target pedestrian according to the cost matrix; Determining key point information for the target pedestrian from the target video frame image according to the target trajectory information; Traversing each video frame image of the original video data, and determining whether the key point information for the target pedestrian in the currently traversed video frame image meets a preset behavior recognition condition; When the key point information satisfies the preset behavior recognition condition, obtaining the first n video frame images of the currently traversed video frame image, where n is an integer greater than 1; The first n video frame images of the currently traversed video frame images are input into a pre-trained behavior recognition model for behavior judgment, so as to obtain a behavior recognition result for the target pedestrian through the behavior recognition model.
2. The behavior recognition method according to claim 1, characterized in that: The appearance vector prediction network model includes a convolution module, a pooling module, a residual module and a fully connected module. The inputting each video frame image of the original video data into the pre-trained appearance vector prediction network model to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model includes: Preprocessing each of the video frame images according to the target pedestrian to obtain a preprocessed image; Performing first feature processing on the preprocessed image through the convolution module to obtain a first feature image; Performing second feature processing on the first feature image by the pooling module to obtain a second feature image; Performing third feature processing on the second feature image by the residual module to obtain a third feature image; The third feature image is subjected to fourth feature processing by the fully connected module to obtain appearance information of the target pedestrian in each video frame image.
3. The behavior recognition method according to claim 2, characterized in that: The residual module includes a first residual unit and a second residual unit, the first residual unit does not include a residual connection, the second residual unit includes the residual connection, and the third feature image is obtained by performing a third feature processing on the second feature image through the residual module, including: Performing fifth feature processing on the second feature image by the first residual unit to obtain a fifth feature image; The sixth feature processing is performed on the fifth feature image by the second residual unit to obtain a third feature image.
4. The behavior recognition method according to any one of claims 1 to 3, characterized in that: The determining key point information for the target pedestrian from the target video frame image according to the target trajectory information includes: Determining a candidate range corresponding to the target pedestrian in the target video frame image according to the target trajectory information; Expanding the candidate range to obtain a target range; The target range is input into a preset key point algorithm model to perform key point calculation to determine key point information for the target pedestrian.
5. The behavior recognition method according to claim 4, characterized in that: The step of inputting the target range into a preset key point algorithm model to perform key point calculation to determine key point information for the target pedestrian includes: Input the target range into a preset key point algorithm model to perform key point calculation to obtain a plurality of heat map data, wherein each heat map data uniquely corresponds to a key point of the target pedestrian in the target video frame image; Position mapping is performed on the multiple heat map data to determine key point information for the target pedestrian.
6. A behavior recognition device, characterized in that: The device comprises: A pedestrian detection module, used for acquiring raw video data to be detected, and inputting the raw video data into a pre-trained pedestrian detection model, so as to determine a target pedestrian from a video frame image of the raw video data through the pedestrian detection model; A target tracking module is used to track the target pedestrian and obtain a target video frame image and target trajectory information corresponding to the target pedestrian; the target tracking of the target pedestrian and obtaining a target video frame image and target trajectory information corresponding to the target pedestrian includes: inputting each video frame image of the original video data into a pre-trained appearance vector prediction network model to obtain the appearance information of the target pedestrian in each video frame image through the appearance vector prediction network model; obtaining the motion information of the target pedestrian in each video frame image; obtaining the target video frame image and target trajectory information corresponding to the target pedestrian based on the appearance information and the motion information; wherein, obtaining the target video frame image and target trajectory information corresponding to the target pedestrian based on the appearance information and the motion information includes: traversing each video frame image, performing the following processing on the currently traversed video frame image: obtaining the first state information of the target pedestrian in the currently traversed video frame image, wherein the first state information includes first motion information and first appearance information; predicting the appearance information of the target pedestrian in the next frame of video frame image according to the first appearance information to obtain predicted appearance information; obtaining the second state information of the target pedestrian in the next frame of video frame image, wherein the second state information includes second motion information and second appearance information; obtaining a first distance according to the first motion information and the second motion information; obtaining a second distance according to the predicted appearance information and the second appearance information; performing weighted calculation on the first distance and the second distance to obtain a cost matrix; obtaining the target video frame image and target trajectory information corresponding to the target pedestrian according to the cost matrix; A key point determination module, used to determine key point information for the target pedestrian from the target video frame image according to the target trajectory information; A condition judgment module, used to traverse each video frame image of the original video data, and determine whether the key point information of the target pedestrian in the currently traversed video frame image meets the preset behavior recognition condition; A video frame image acquisition module, used for acquiring n video frame images before the currently traversed video frame image when the key point information meets the preset behavior recognition condition, where n is an integer greater than 1; The behavior judgment module is used to input the first n video frame images of the currently traversed video frame image into a pre-trained behavior recognition model for behavior judgment, so as to obtain a behavior recognition result for the target pedestrian through the behavior recognition model.
7. Computer device, characterized in that The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute: The behavior recognition method according to any one of claims 1 to 5.
8. Storage medium, the storage medium being a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is configured to perform: The behavior recognition method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Skeleton-based behavior detection method, terminal device, and computer storage medium
CN109508688A
Target monitoring method and device, computer equipment and storage medium
CN110222577A
Darkflow-deepsort-based multi-target tracking detection method, device, and storage medium
WO2021017291A1
Behavior detection method, apparatus, and computer-readable storage medium
WO2021208735A1
