Behavior recognition method, apparatus, device, storage medium, and product

Abstract

The application discloses a behavior recognition method and device, equipment, storage medium and product, relates to the technical field of behavior recognition, and the behavior recognition method is applied to the edge end, and includes dividing a captured video segment into multiple video units; feature extraction is performed on the video unit, and a to-be-reasoned video unit is determined according to a feature extraction result; a sampling frame rate is determined according to the to-be-reasoned video unit, frame sampling is performed on the to-be-reasoned video unit according to the sampling frame rate, and a sampling result is sent to the cloud end for behavior recognition. Since the application divides the collected video segment into multiple video units, determines the to-be-reasoned video unit based on the video unit, performs frame sampling from the to-be-reasoned video unit, and sends the sampling result to the cloud end for behavior recognition, compared with the existing mode of sending all the collected videos to the cloud end for behavior recognition, the above mode of the application can reduce the analysis delay and reduce data transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior recognition, in particular to a behavior recognition method, device, equipment, storage medium and product. BACKGROUND

[0002] Video analysis applications based on human activity recognition (HAR) are of great significance in many fields, such as robots, human-computer interaction, security monitoring, and child education. For example, in security monitoring, the response time can be greatly shortened and public safety can be improved by accurately identifying abnormal behavior. With the rapid development of deep learning technology, these applications are increasingly deployed to the edge of the network (such as smart cameras) to reduce analysis latency, reduce bandwidth consumption, and improve privacy and security. Existing video analysis systems can analyze video frames directly on edge devices without transmitting them to cloud servers for processing. However, video analysis systems usually rely on computationally intensive deep learning models for inference, and the limited computing resources on edge devices cannot support real-time model inference. If video frames are uploaded to the cloud for analysis, the cost of data transmission will increase, and the bandwidth and latency will also increase. Therefore, how to improve the efficiency of behavior recognition has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the present application is to provide a behavior recognition method, device, equipment, storage medium and product, which aims to solve the technical problem of low efficiency of existing behavior recognition.

[0004] To achieve the above-mentioned purpose, the present application provides a behavior recognition method, which is applied to an edge, and the behavior recognition method comprises the following steps:

[0005] dividing a captured video segment into a plurality of video units;

[0006] performing feature extraction on the video units, and determining a video unit to be inferred according to the feature extraction result;

[0007] determining a sampling frame rate according to the video unit to be inferred, sampling frames of the video unit to be inferred according to the sampling frame rate, and sending the sampling result to the cloud for behavior recognition.

[0008] Optionally, the step of performing feature extraction on the video units and determining a video unit to be inferred according to the feature extraction result comprises:

[0009] performing feature extraction on the video units to obtain a feature extraction result;

[0010] determining the feature difference between the current video unit and the previous feature unit according to the feature extraction result;

[0011] determine a sampling frame rate according to the to-be-reasoned video unit, and sample frames of the to-be-reasoned video unit according to the sampling frame rate.

[0012] Optionally, the step of determining a sampling frame rate according to the to-be-reasoned video unit and sampling frames of the to-be-reasoned video unit according to the sampling frame rate comprises:

[0013] determining a rhythm category corresponding to the to-be-reasoned video unit according to the rhythm category information;

[0014] determining a sampling frame rate of the to-be-reasoned video unit according to the rhythm category;

[0015] sampling frames of the to-be-reasoned video unit according to the sampling frame rate.

[0016] Optionally, before the step of determining a sampling frame rate according to the to-be-reasoned video unit and sampling frames of the to-be-reasoned video unit according to the sampling frame rate, the method further comprises:

[0017] sampling sample data according to different frame sampling rates to obtain sample data sampling results;

[0018] determining a behavior recognition accuracy corresponding to the sample data sampling results;

[0019] determining a best frame rate corresponding to each behavior category according to the behavior recognition accuracy;

[0020] performing clustering based on the best frame rate to obtain clustering results;

[0021] determining rhythm category information according to the clustering results.

[0022] The application divides a captured video segment into multiple video units, extracts features of the video units, determines a to-be-reasoned video unit according to the feature extraction results, determines a sampling frame rate according to the to-be-reasoned video unit, samples frames of the to-be-reasoned video unit according to the sampling frame rate, and sends the sampling results to the cloud for behavior recognition. Since the application divides the collected video segment into multiple video units, determines a to-be-reasoned video unit based on the video units, samples frames from the to-be-reasoned video unit, and sends the sampling results to the cloud for behavior recognition, the above-mentioned method of the application can reduce analysis delay and reduce data transmission compared with the existing method of sending all collected videos to the cloud for behavior recognition.

[0023] In addition, to achieve the above-mentioned purpose, the application provides a behavior recognition method, which is applied to the cloud, and comprises the following steps:

[0024] receiving sampling results sent by the edge end;

[0025] adjust the sampling result into a target feature map through an input adjustment module;

[0026] infer a behavior category of the target feature map based on a time delay adjustment module to obtain a behavior recognition result, and send the behavior recognition result to the edge end.

[0027] The application receives a sampling result sent by an edge end, adjusts the sampling result into a target feature map through an input adjustment module, and infers a behavior category of the target feature map based on a time delay adjustment module to obtain a behavior recognition result, and sends the behavior recognition result to the edge end. Compared with the existing behavior analysis method in the edge end, the above method can avoid the limitation of the computing resources of the edge end and improve the behavior analysis efficiency.

[0028] In addition, to achieve the above-mentioned purpose, the application further provides a behavior recognition device applied to an edge end, which comprises:

[0029] a division module configured to divide a captured video segment into a plurality of video units;

[0030] a feature extraction module configured to perform feature extraction on the video units and determine a video unit to be inferred according to the feature extraction result;

[0031] a sampling module configured to determine a sampling frame rate according to the video unit to be inferred, perform frame sampling on the video unit to be inferred according to the sampling frame rate, and send a sampling result to a cloud end for behavior recognition.

[0032] In addition, to achieve the above-mentioned purpose, the application further provides a behavior recognition device applied to a cloud end, which comprises:

[0033] a receiving module configured to receive a sampling result sent by an edge end;

[0034] an input adjustment module configured to adjust the sampling result into a target feature map through the input adjustment module;

[0035] a time delay adjustment module configured to infer a behavior category of the target feature map based on the time delay adjustment module to obtain a behavior recognition result, and send the behavior recognition result to the edge end.

[0036] In addition, to achieve the above-mentioned purpose, the application further provides a behavior recognition device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned behavior recognition method.

[0037] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the behavior recognition method.

[0038] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the behavior recognition method. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and serve to explain the principles of the application, together with the description.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0041] Figure 1 The flowchart provided for the behavior recognition method embodiment one of the present application;

[0042] Figure 2 The edge structure schematic diagram provided for the behavior recognition method embodiment one of the present application;

[0043] Figure 3 The flowchart provided for the behavior recognition method embodiment two of the present application;

[0044] Figure 4 The structure schematic diagram provided for the behavior recognition method embodiment two of the present application;

[0045] Figure 5 The elastic model adaptation structure schematic diagram provided for the behavior recognition method embodiment two of the present application;

[0046] Figure 6 The module structure schematic diagram of the behavior recognition device embodiment one of the present application;

[0047] Figure 7 The module structure schematic diagram of the behavior recognition device embodiment two of the present application;

[0048] Figure 8 The device structure schematic diagram of the hardware running environment involved in the behavior recognition method in the embodiments of the present application.

[0049] The object implementation, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely exemplary of the application and do not limit the application.

[0051] In order to better understand the technical scheme of the application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0052] The edge end divides the captured video clip into a plurality of video units; feature extraction is performed on the video units, and a to-be-reasoned video unit is determined according to the feature extraction result; a sampling frame rate is determined according to the to-be-reasoned video unit, frame sampling is performed on the to-be-reasoned video unit according to the sampling frame rate, and the sampling result is sent to the cloud end for behavior recognition. Since the application divides the collected video clip into a plurality of video units, and then determines a to-be-reasoned video unit based on the video units, and performs frame sampling on the to-be-reasoned video unit, and sends the sampling result to the cloud end for behavior recognition, compared with the existing method of sending all the collected video to the cloud end for behavior recognition, the above-mentioned method of the application can reduce the analysis delay and reduce the data transmission.

[0053] The cloud end receives the sampling result sent by the edge end; the sampling result is adjusted to a target feature map through an input adjustment module; the behavior category of the target feature map is inferred based on a time delay adjustment module, and a behavior recognition result is obtained, and the behavior recognition result is sent to the edge end. Compared with the existing behavior analysis method in the edge end, the above-mentioned method of the application can avoid the limitation of the computing resources of the edge end and improve the behavior analysis efficiency.

[0054] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a behavior recognition device, etc. capable of realizing the above functions. The following will take the behavior recognition device as an example to describe the embodiment and the following embodiments.

[0055] Based on this, the embodiment of the application provides a behavior recognition method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the behavior recognition method of the application is shown in the figure.

[0056] In the embodiment, the behavior recognition method includes steps S10-S30:

[0057] Step S10, divide the captured video clip into a plurality of video units;

[0058] It should be noted that the video segment can be a segment of video collected for behavior recognition. It can be a fixed number of video frames, such as 15 frames. Dividing the captured video segment into a plurality of video units can be to recognize and segment the video segment to obtain a plurality of video units, a video unit being composed of a series of image blocks belonging to the same target. For the convenience of description, the nth video unit of a certain target is denoted as u n .

[0059] Step S20, feature extraction is performed on the video unit, and a video unit to be inferred is determined according to the feature extraction result;

[0060] It should be noted that for each video unit u n , the embodiment constructs a state vector containing three elements <flag, thresh, class>. Among them, flag indicates whether inference is needed, thresh is a threshold for evaluating semantic consistency, which can be a pre-defined value, and class specifies the behavior category of the video unit. u n The state vector of u n is determined by the feature map difference between u n-1 . The feature extraction on the video unit can be to extract features f n for the video unit u n using the X3D model. The determination of the video unit to be inferred according to the feature extraction result can be to determine the feature difference between the current video unit and the previous video unit according to the feature extraction result. If the feature difference is less than thresh, it is determined whether the behavior category of the previous video unit is known. If it is known, it is determined that the behavior category of the current video unit is consistent with the behavior category of the previous video unit. If it is not known or the current video unit is the first video unit, it is determined that the current video unit is a video unit that needs to be recognized, i.e., a video unit to be inferred. If the feature difference is greater than or equal to thresh, it indicates that there may be a behavior transition, and inference is needed immediately.

[0061] Step S30, determining a sampling frame rate according to the video unit to be inferred, frame sampling is performed on the video unit to be inferred according to the sampling frame rate, and the sampling result is sent to the cloud for behavior recognition.

[0062] It should be noted that, in order to avoid too many video frames being transmitted to the cloud, in the embodiment, the to-be-inferred video unit can be frame-sampled according to a sampling frame rate, and the sampled video frames are sent to the cloud. The sampling frame rate determined according to the to-be-inferred video unit can be determined according to a first video unit in the to-be-inferred video unit, and the same sampling frame rate is used for subsequent video units. The sampling frame rate determined according to the first video unit in the to-be-inferred video unit can be that the accuracy of the behavior analysis result obtained under different sampling frame rates is calculated, and the sampling frame rate with the highest accuracy is taken as the sampling frame rate of the to-be-inferred video unit. Then, the to-be-inferred video unit is frame-sampled according to the sampling frame rate, and the sampling result is sent to the cloud for behavior recognition.

[0063] Further, in view of the finiteness of edge computing and network resources, it is not realistic to transmit each frame to the cloud for inference. Therefore, the step S30 can include:

[0064] determining a rhythm category corresponding to the to-be-inferred video unit according to the rhythm category information;

[0065] determining a sampling frame rate of the to-be-inferred video unit based on the rhythm category;

[0066] frame-sampling the to-be-inferred video unit according to the sampling frame rate.

[0067] It should be noted that the rhythm category information can include the optimal sampling frame rate corresponding to each behavior category. The determination of the rhythm category corresponding to the to-be-inferred video unit according to the rhythm category information can be the determination of the behavior category of the first video unit in the to-be-inferred video unit, and the determination of the rhythm category in the rhythm category information corresponding to the behavior category. The determination of the sampling frame rate of the to-be-inferred video unit based on the rhythm category can take the optimal sampling frame rate corresponding to the rhythm category as the sampling frame rate of the to-be-inferred video unit.

[0068] Further, in order to determine the optimal sampling frame rate corresponding to each behavior category, before the step S30, the method can include:

[0069] sampling the sample data according to different frame sampling rates to obtain sample data sampling results;

[0070] determining a behavior recognition accuracy corresponding to the sample data sampling results;

[0071] determining the optimal frame rate corresponding to each behavior category according to the behavior recognition accuracy;

[0072] clustering based on the optimal frame rate to obtain a clustering result;

[0073] Determine rhythm category information according to the clustering result.

[0074] It should be noted that the sample data can include sample video frame data of different behavior categories. Sampling the sample data according to different frame sampling rates can be sampling according to different frame sampling rates for each behavior category to obtain sample data sampling results of different behavior categories and different sampling frame rates. Then, the sampling frame rate corresponding to each behavior category with the highest behavior recognition accuracy, that is, the optimal frame rate, is determined. Clustering is performed based on the optimal frame rate to obtain a clustering result. According to the clustering result, the rhythm category information can be determined according to different behavior categories in the same cluster and the corresponding sampling frame rate. The sampling frame rate under the same cluster can be the average of the optimal frame rates of each behavior category under the cluster.

[0075] Reference can be made to Figure 2 , Figure 2 An edge structure schematic diagram provided by the behavior recognition method embodiment one of the present application; after capturing a video segment (i.e., a fixed number of frames, such as 15 frames), the target detection and tracking module (i.e., target detection and target tracking in Figure 2 is responsible for identifying and segmenting frames, dividing them into multiple video units. A video unit is composed of a series of image blocks belonging to the same target. For ease of description, the nth video unit of a certain specific target in this embodiment is denoted as u n . Subsequently, the sequence boundary locator is used to evaluate u n to determine whether to send it to the cloud for inference, while the behavior rhythm sampler selects some frames from it for transmission. Specifically, for each video unit u n , a state vector containing three elements <flag, thresh, class> is constructed. Among them, flag indicates whether u n needs to be inferred, thresh is used as a threshold for evaluating semantic consistency, and class specifies the behavior category of the video unit. The state vector of u n is determined by the feature map difference between u n and u n-1 . The key idea here is that the difference between the feature maps of video units reflects semantic consistency, that is, a difference less than thresh indicates that the two video units belong to the same behavior. To ensure the optimal inference frequency, thresh is dynamically adjusted. Once u n is selected for inference, the behavior rhythm sampler determines the appropriate frame sampling rate according to the estimated behavior rhythm category. Then, the video unit is encoded and uploaded to the cloud.

[0076] In practice, actions in a video stream typically last for a period of time. For example, bending over might last a few seconds, while walking might last a few minutes or even longer. Performing multiple inferences about the same action is a waste of computational resources. Therefore, identifying the frame sequence corresponding to the same action is crucial; this embodiment refers to it as a Single-Activity Sequence (SAS). In this embodiment, an efficient rule-based algorithm 1 is proposed to identify SAS by analyzing the semantic differences between consecutive frames. Specifically, Algorithm 1 is as follows:

[0077] Algorithm 1: Calculate the state S for the video unit n

[0078] Input: u n The nth video unit, S n-1 Video unit u n-1 The state vector, f n-1 :u n-1 Feature map, count: number of blocks to be inferred, S n [flag]: State vector S n The flag item (the other tags are similar).

[0079] Output: S n :u n state vector

[0080] f n ←X3D(u n )

[0081]

[0082] if diff n-1 [thresh]then

[0083] S n [flag]←S n-1 [flag]

[0084] Update S according to the threshold update formula. n [thresh]

[0085] S n [class] = S n-1 [class]

[0086] if S n [flag] == to be deduced then

[0087] count←count-1

[0088] if count≤0then​

[0089] Infer Cached Video Unit and Update S n [class]

[0090] S n [flag]←No Need to Infer

[0091] Reset count

[0092] end

[0093] end

[0094] else

[0095] if S n-1 [flag] == To Infer then

[0096] Infer Cached Video Unit

[0097] end S n [flag]←To Infer

[0098] Reset S n [thresh]

[0099] S n [class]←Unknown (To Infer)

[0100] end

[0101] Specifically, given a video stream, it is first divided into multiple video segments, each of which consists of N (e.g., 15) consecutive frames. Subsequently, target detection and tracking are performed on these video segments to generate multiple video units. Each video unit consists of image patches belonging to the same person. Thus, the number of image patches in a video unit can be less than N. Specifically, the bounding box of target detection is enlarged by 50% in each direction so as to include interaction with the surrounding environment (e.g., a ball) when generating image patches. For a video unit u n of a certain target, a state vector S n = <flag, thresh, class> is defined, where flag e {0: To Infer, 1: No Need to Infer} represents the inference state, thresh e (0, 1) is used to evaluate the semantic consistency between u n and u n-1 , and class represents the behavior category of u n . For convenience of discussion, flag in S n-1 is denoted as S n-1 [flag]. Other elements use similar notations.

[0102] The key to deciding whether the current video unit needs inference is semantic consistency. In this embodiment, a behavior classification model (i.e., X3D) is used to extract feature maps as semantic information. The feature difference is used to evaluate semantic consistency. Video units with small semantic difference are considered to belong to the same behavior, because the semantic difference of video units from different behaviors is usually large.

[0103] Specifically, first, the X3D model is used to extract feature maps for video unit u n . n Then, the feature difference, denoted as diff ∈ (0, 1), is calculated, i.e., f n and f n-1 . If diff is less than thresh, it indicates that u n and u n-1 belong to the same behavior. In this case, u n-1 is assigned the same class or sent to the cloud for inference, depending on whether the class of u n is known. On the other hand, diff greater than thresh indicates that there may be a behavior transition, marking the end of the current SAS at u n-1 , requiring immediate inference. Note that the X3D model on the edge device is only used to evaluate the semantic consistency between video units, while the video units are streamed to the cloud server for inference.

[0104] However, using a fixed thresh to evaluate semantic consistency is problematic. A higher thresh may fail to detect transitions between similar behaviors (e.g., walking and running), while a lower thresh may incorrectly split a sequence of the same behavior into multiple SAS. To address this issue, this embodiment dynamically adjusts thresh based on the state vector. The initial value of thresh can be set to 0.4, and once a behavior transition occurs, it will be reset to the initial value; otherwise, thresh decays as follows:

[0105] S n [thresh] = S n-1 [thresh] × 2 -σ

[0106] where σ = diff n / S n-1 [thresh] represents the likelihood of a behavior transition. A higher σ indicates a greater probability of behavior change, so the threshold needs to decay faster. In addition, to avoid caching too many video units for inference, once a sufficient number of video units are cached, immediate inference is triggered.

[0107] Once a video unit is decided to be sent to the cloud for inference, a behavior rhythm sampler will be used to select a minimum but representative subset of frames for transmission. The behavior rhythm sampler estimates the rhythm of the behavior and determines the optimal frame rate for transmission. It includes an offline phase and an online phase. In the offline phase, the optimal frame rate for each behavior class is pre-analyzed and classified into different rhythm categories. In the online phase, the most appropriate frame rate is selected according to the estimated rhythm category.

[0108] Specifically, in the offline phase, first calculate the average accuracy AC of behavior k at a series of frame rates k , that is, AC k = <a1, a2, …, a n >, where a i represents the accuracy when the frame rate is f i . In order to determine the optimal frame rate of the behavior, calculate the change of accuracy with respect to frame rate, the formula is as follows:

[0109]

[0110] As the frame rate increases, R i will generally decrease. Therefore, the first frame rate with R i below the preset threshold can be selected as the optimal frame rate of the behavior. After determining the optimal frame rate of each behavior, use clustering algorithm (such as K-Means) to group them into k rhythm categories. In this embodiment, k = 4 can be set. This helps to preliminarily classify the rhythm categories on the edge device.

[0111] Online phase: the first video unit in SAS is used to determine the rhythm category and the optimal frame rate (i.e. determine the behavior category of the first video unit in SAS, query the rhythm category information according to the determined behavior category, and determine the corresponding optimal frame rate). Since a single video unit provides sufficient information for preliminary classification of the rhythm, subsequent video units in the same SAS will be sampled at the same frame rate.

[0112] The embodiment is applied to the edge end, including dividing a captured video segment into a plurality of video units; performing feature extraction on the video units, and determining a to-be-inferred video unit according to a feature extraction result; determining a sampling frame rate according to the to-be-inferred video unit, sampling frames of the to-be-inferred video unit at the sampling frame rate, and sending a sampling result to the cloud for behavior recognition. Since the embodiment divides the captured video segment into a plurality of video units, and then determines the to-be-inferred video unit based on the video units, and samples frames from the to-be-inferred video unit, and sends the sampling result to the cloud for behavior recognition, compared with the existing mode of sending all captured videos to the cloud for behavior recognition, the above mode of the embodiment can reduce analysis delay and reduce data transmission.

[0113] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned embodiment one can refer to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , Figure 3 The flowchart provided by the second embodiment of the behavior recognition method of the present application is shown in FIG. 6.

[0114] In this embodiment, the behavior recognition method is applied to the cloud, and the behavior recognition method comprises the following steps:

[0115] Step S40: receiving the sampling result sent by the edge side;

[0116] Step S50: adjusting the sampling result to a target feature map through an input adjustment module;

[0117] It should be noted that the above-mentioned input adjustment module can refer to Figure 4 , Figure 4 The structural diagram provided by the second embodiment of the behavior recognition method of the present application is shown in FIG. 7. In this embodiment, after the cloud receives the video unit (i.e. the sampling result), it will decide whether to cache it for subsequent processing or immediately perform reasoning according to the flag. However, the frame sampling process leads to inconsistent frame intervals and quantities, which has an adverse effect on the accuracy of the model. In order to solve this problem, the elastic model adaptation module of this embodiment introduces a super model with adaptive different inputs. This super model is divided into two stages: the input adjustment stage and the latency adjustment stage. The goal of the input adjustment stage is to convert input frames with different intervals and quantities into a unified feature map. Subsequently, the latency adjustment stage is responsible for predicting the final behavior class while complying with the latency requirement. Once the behavior class is obtained, the reasoning result will be sent back to the edge device. Please refer to Figure 5 , Figure 5 The elastic model adaptation structure diagram provided by the second embodiment of the behavior recognition method of the present application is shown in FIG. 8. The overview of elastic model adaptation is shown in Figure 5 It can be divided into two key stages: the input adjustment stage, i.e. the input adjustment module (Input Adjustment Phase), and the latency adjustment stage, i.e. the latency adjustment module (Latency Adjustment Phase). The input adjustment stage aims to adapt to different frame intervals and quantities, while the goal of the latency adjustment stage is to adjust the model to continuously meet the latency target.

[0118] The input adjustment stage aims to convert input data with different frame intervals and quantities into a unified feature map. It is composed of a series of neural network modules, each of which has an input and produces an output. The checkpoints in the input adjustment stage are denoted as <cp1, cp2, …, cp n>, and define the path R connecting two checkpoints cp i and cp j as R i,j . Each path consists of multiple rhythm modules, each of which is responsible for processing frames sampled at a specific interval. Frames sampled at a specific interval work best when processed by the rhythm module designed for that specific rhythm, and accuracy decreases when processed by other rhythm modules. During each inference, only one rhythm module in each path is selected for execution.

[0119] To accommodate different frame numbers, the model skips checkpoints as necessary to ensure uniform dimensions of the final feature map. Each rhythm module is designed to reduce the same time dimension in the feature map, denoted as Δt. Inputs containing more frames will pass through more rhythm modules, while inputs containing fewer frames will follow a shorter path. This approach ensures flexibility in adapting to different frame numbers while generating uniform feature maps.

[0120] During training, a reference feature map is first generated for each rhythm module within the path. This is achieved by training a baseline model using frames sampled at intervals matching the specific rhythm. Subsequently, each rhythm module is trained using a feature-based knowledge distillation method. For example, rhythm module r i,j,v is trained using reference feature maps f i,v and f j,v , which are obtained from the baseline model matching rhythm v. The loss function during training is:

[0121]

[0122] During inference, based on the initial time dimension T input and the target time dimension T mid , the number of checkpoints J that need to be skipped is calculated, i.e., J = n - (T mid - T input ) / Δt, and the probability of each checkpoint being skipped is During this process, each path selects a rhythm module matching the current frame rate for execution to ensure optimal accuracy.

[0123] Step S60: Based on the time delay adjustment module, the behavior class of the target feature map is inferred to obtain a behavior recognition result, and the behavior recognition result is sent to the edge end.

[0124] It should be noted that in the time delay adjustment stage, the model infers the behavior class based on the uniform feature map. Similar to the input adjustment stage, the time delay adjustment stage also contains multiple checkpoints, and between each two checkpoints is a neural network module. The checkpoints in the time delay adjustment stage are denoted asn path between checkpoints cp i and cp j is defined as R i,j Each path contains multiple depth modules. Each depth module is configured with different number of neural network layers, with deeper modules providing higher accuracy but also larger inference latency.

[0125] To meet the requirement of real-time video analysis, the complexity of the model needs to be adjusted according to the latency target. In this embodiment, the latency target of a specific inference task is affected by two factors: (i) the remaining latency budget L remain : as more video units are cached for inference, the remaining latency budget decreases in order to maintain the same end-to-end latency (i.e. the time difference from the occurrence of the behavior to the generation of the recognition result); (ii) the likelihood of new behavior occurrence σ: the greater the likelihood of new behavior occurrence in the newly arrived video units, the more the current inference latency needs to be reduced to always ensure low latency. Based on these factors, the latency target L target of the current inference task is defined as:

[0126] L target = L remain - β · σ

[0127] where L target = L remain - β · σ, and β is a weight coefficient. A higher σ value indicates a more significant semantic change, meaning a higher probability of behavior transition.

[0128] During the training process, these modules are also trained through distillation learning similar to the input adjustment phase. After training is completed, the latency of each depth module is calculated, denoted as T = {t1, …, t k}. Note that the latency of a certain depth module is consistent across different paths. In addition, the average latency of all different depth modules is calculated as

[0129] During the inference process, the number of checkpoints that need to be skipped is first calculated, i.e. the average latency of different depth modules, where L all is the latency when no checkpoint is skipped. Since skipping checkpoints can affect accuracy, a portion of the remaining available latency is randomly selected to use deeper depth blocks for inference.

[0130] The embodiment receives a sampling result sent by an edge end; adjusts the sampling result into a target feature map through an input adjustment module; infers a behavior category of the target feature map based on a time delay adjustment module to obtain a behavior recognition result, and sends the behavior recognition result to the edge end. Compared with the existing behavior analysis method in the edge end, the above method can avoid limited computing resources in the edge end and improve the behavior analysis efficiency.

[0131] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the behavior recognition method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0132] The present application also provides a behavior recognition device, please refer to Figure 6 , Figure 6 The module structure diagram of the behavior recognition device embodiment one of the present application is shown in the figure; the behavior recognition device includes an application in an edge end, and the behavior recognition device includes:

[0133] The division module 10 is used for dividing a captured video segment into a plurality of video units;

[0134] The feature extraction module 20 is used for performing feature extraction on the video unit, and determining a to-be-reasoned video unit according to the feature extraction result;

[0135] The sampling module 30 is used for determining a sampling frame rate according to the to-be-reasoned video unit, performing frame sampling on the to-be-reasoned video unit according to the sampling frame rate, and sending a sampling result to a cloud end for behavior recognition.

[0136] The embodiment is applied to an edge end, and includes dividing a captured video segment into a plurality of video units; performing feature extraction on the video unit, and determining a to-be-reasoned video unit according to the feature extraction result; determining a sampling frame rate according to the to-be-reasoned video unit, performing frame sampling on the to-be-reasoned video unit according to the sampling frame rate, and sending a sampling result to a cloud end for behavior recognition. Since the embodiment divides the collected video segment into a plurality of video units, determines a to-be-reasoned video unit based on the video unit, performs frame sampling from the to-be-reasoned video unit, and sends a sampling result to a cloud end for behavior recognition, compared with the existing method of sending all the collected videos to a cloud end for behavior recognition, the above method can reduce the analysis time delay and reduce data transmission.

[0137] The present application also provides a behavior recognition device, please refer to Figure 7 , Figure 7 The module structure diagram of the behavior recognition device embodiment two of the present application is shown in the figure; the behavior recognition device is applied to a cloud end, and the behavior recognition device includes:

[0138] The receiving module 40 is configured to receive the sampling result sent by the edge side.

[0139] The input adjusting module 50 is configured to adjust the sampling result to a target feature map by the input adjusting module.

[0140] The time delay adjusting module 60 is configured to infer a behavior category of the target feature map based on the time delay adjusting module to obtain a behavior recognition result, and send the behavior recognition result to the edge side.

[0141] The embodiment receives the sampling result sent by the edge side, adjusts the sampling result to a target feature map by the input adjusting module, and infers a behavior category of the target feature map based on the time delay adjusting module to obtain a behavior recognition result, and sends the behavior recognition result to the edge side. Compared with the existing behavior analysis method at the edge side, the above method can avoid the limited computing resources at the edge side and improve the behavior analysis efficiency.

[0142] The behavior recognition device provided by the application adopts the behavior recognition method in the above embodiment, and can solve the technical problem of low behavior recognition efficiency. Compared with the prior art, the behavior recognition device provided by the application has the same beneficial effects as the behavior recognition method provided by the above embodiment, and other technical features in the behavior recognition device are the same as the features disclosed in the above embodiment, which will not be repeated here.

[0143] The application provides a behavior recognition device, which comprises at least one processor and a memory connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the behavior recognition method in the above embodiment.

[0144] Reference will be made to the following description Figure 8 which shows a structural schematic diagram of a behavior recognition device suitable for implementing the embodiments of the application. The behavior recognition device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 8 The behavior recognition device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the application.

[0145] As shown in Figure 8 The behavior recognition device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the behavior recognition device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the behavior recognition device to communicate wirelessly or wired with other devices to exchange data. Although the behavior recognition device with various systems is shown in the figure, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0146] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0147] The behavior recognition device provided by the present disclosure adopts the behavior recognition method in the above embodiments, and can solve the technical problem of low behavior recognition efficiency in the prior art. Compared with the prior art, the behavior recognition device provided by the present disclosure has the same beneficial effects as the behavior recognition method provided by the above embodiments, and other technical features in the behavior recognition device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0148] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0149] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes and modifications that can be made to the application in light of the teachings described herein are contemplated in the broad scope of the application. Accordingly, the scope of the application should be determined not with reference to the above description but with reference to the claims appended hereto.

[0150] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the behavior recognition method in the above-described embodiments.

[0151] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0152] The above computer readable storage medium can be contained in the behavior recognition device; or can exist separately and not be assembled into the behavior recognition device.

[0153] The above computer readable storage medium carries one or more programs, which, when executed by the behavior recognition device, cause the behavior recognition device to perform the above behavior recognition method.

[0154] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0155] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0156] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0157] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above-mentioned behavior recognition method, and can solve the technical problem of low behavior recognition efficiency. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the behavior recognition method provided by the above-mentioned embodiments, which will not be described here.

[0158] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the behavior recognition method as described above.

[0159] The computer program product provided by the application can solve the technical problem of low behavior recognition efficiency. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the behavior recognition method provided by the above-mentioned embodiments, and are not described here.

[0160] The above-mentioned embodiments are only part of the embodiments of the application, and do not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the content of the specification and drawings of the application within the technical concept of the application is included in the patent protection scope of the application.

Claims

1. A behavior recognition method, characterized by, Applied to the edge end, the behavior recognition method comprises the following steps: Divide the captured video clip into multiple video units; Feature extraction is performed on the video unit, and the video unit to be inferred is determined according to the feature extraction result; Determine the sampling frame rate according to the video unit to be inferred, sample the video unit to be inferred according to the sampling frame rate, and send the sampling result to the cloud for behavior recognition; Before the step of determining the sampling frame rate according to the video unit to be inferred and sampling the video unit to be inferred according to the sampling frame rate, it further comprises: Sample the sample data at different frame sampling rates to obtain sample data sampling results; Determine the behavior recognition accuracy corresponding to the sample data sampling results; Determine the best frame rate corresponding to each behavior category according to the behavior recognition accuracy; Based on the best frame rate, clustering is performed to obtain a clustering result; Determine the rhythm category information according to the clustering result; The step of determining the sampling frame rate according to the video unit to be inferred and sampling the video unit to be inferred according to the sampling frame rate comprises: Determine the rhythm category corresponding to the video unit to be inferred according to the rhythm category information; Determine the sampling frame rate of the video unit to be inferred based on the rhythm category; Sample the video unit to be inferred according to the sampling frame rate.

2. The behavior recognition method of claim 1, wherein, The step of performing feature extraction on the video unit and determining the video unit to be inferred according to the feature extraction result comprises: Feature extraction is performed on the video unit to obtain a feature extraction result; Determine the feature difference between the current video unit and the previous feature unit according to the feature extraction result; Determine the video unit to be inferred based on the feature difference and a preset determination threshold.

3. A behavior recognition method characterized by, Applied to the cloud, the cloud cooperates with the edge end of any one of claims 1-2 to realize behavior recognition, and the behavior recognition method comprises the following steps: Receive the sampling result sent by the edge end; Adjust the sampling result to a target feature map through an input adjustment module; Infer the behavior category of the target feature map based on a time delay adjustment module to obtain a behavior recognition result, and send the behavior recognition result to the edge end.

4. The behavior recognition apparatus according to claim 1, wherein Applied to the edge end, the behavior recognition device comprises: A division module for dividing the captured video clip into multiple video units; A feature extraction module for performing feature extraction on the video unit and determining the video unit to be inferred according to the feature extraction result; A sampling module for determining the sampling frame rate according to the video unit to be inferred, sampling the video unit to be inferred according to the sampling frame rate, and sending the sampling result to the cloud for behavior recognition; Before the step of determining the sampling frame rate according to the video unit to be inferred and sampling the video unit to be inferred according to the sampling frame rate, it further comprises: Sample the sample data at different frame sampling rates to obtain sample data sampling results; Determine the behavior recognition accuracy corresponding to the sample data sampling results; Determine the best frame rate corresponding to each behavior category according to the behavior recognition accuracy; Based on the best frame rate, clustering is performed to obtain a clustering result; Determine rhythm category information according to the clustering result; The determining the sampling frame rate according to the to-be-reasoned video unit, and the frame sampling of the to-be-reasoned video unit according to the sampling frame rate, includes: Determine the rhythm category corresponding to the to-be-reasoned video unit according to the rhythm category information; Determine the sampling frame rate of the to-be-reasoned video unit based on the rhythm category; Frame sample the to-be-reasoned video unit according to the sampling frame rate.

5. The behavior recognition apparatus according to claim 1, wherein Applied to the cloud, the cloud cooperates with the edge of claim 4 to realize behavior recognition, and the behavior recognition device comprises: A receiving module for receiving the sampling result sent by the edge; An input adjustment module for adjusting the sampling result to a target feature map through the input adjustment module; A time delay adjustment module for inferring the behavior category of the target feature map based on the time delay adjustment module to obtain a behavior recognition result, and sending the behavior recognition result to the edge.

6. A behavior recognition device, characterized by, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the behavior recognition method according to any one of claims 1 to 2 and / or 3.

7. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the behavior recognition method according to any one of claims 1 to 2 and / or 3.

8. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the steps of the behavior recognition method according to any one of claims 1 to 2 and / or 3.

Images