Behavior recognition method and device, equipment, storage medium and product

Abstract

The invention discloses a behavior recognition method and device, equipment, a storage medium and a product, and relates to the technical field of behavior recognition, and the behavior recognition method is applied to an edge end, and comprises the steps: dividing a captured video clip into a plurality of video units; performing feature extraction on the video unit, and determining a to-be-reasoned video unit according to a feature extraction result; and determining a sampling frame rate according to the to-be-reasoned video unit, performing frame sampling on the to-be-reasoned video unit according to the sampling frame rate, and sending a sampling result to a cloud for behavior recognition. According to the invention, the collected video clip is divided into a plurality of video units, the to-be-reasoned video unit is determined based on the video units, the frame sampling is carried out from the to-be-reasoned video unit, and the sampling result is sent to the cloud for behavior recognition, so that the behavior recognition efficiency is improved compared with the existing mode that all collected videos are sent to the cloud for behavior recognition. According to the method, the analysis time delay can be reduced, and data transmission is reduced.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition technology, and in particular to behavior recognition methods, devices, equipment, storage media and products. Background Art

[0002] Video analysis applications based on human activity recognition (HAR) are of great significance in many fields, such as robotics, human-computer interaction, security monitoring, and children's education. For example, in security monitoring, accurate identification of abnormal behavior can greatly shorten response time and improve public safety. With the rapid development of deep learning technology, these applications are increasingly deployed to the edge of the network (such as smart cameras) to reduce analysis latency, reduce bandwidth consumption, and improve privacy and security. Existing video analysis systems can analyze video frames directly on edge devices without transmitting them to cloud servers for processing. However, video analysis systems usually rely on computationally intensive deep learning models for reasoning, and the limited computing resources on edge devices cannot support real-time model reasoning. If video frames are uploaded to the cloud for analysis, the cost of data transmission will increase, and bandwidth and latency will increase. Therefore, how to improve the efficiency of behavior recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of this application is to provide a behavior recognition method, device, equipment, storage medium and product, aiming to solve the technical problem of low efficiency of existing behavior recognition.

[0004] To achieve the above objectives, the present application proposes a behavior recognition method, which is applied to an edge terminal and includes the following steps:

[0005] Dividing the captured video clip into a plurality of video units;

[0006] Extracting features from the video unit, and determining the video unit to be inferred according to the feature extraction result;

[0007] A sampling frame rate is determined according to the video unit to be inferred, frame sampling is performed on the video unit to be inferred according to the sampling frame rate, and the sampling result is sent to the cloud for behavior recognition.

[0008] Optionally, the step of extracting features from the video unit and determining the video unit to be inferred according to the feature extraction result includes:

[0009] Performing feature extraction on the video unit to obtain a feature extraction result;

[0010] Determining a feature difference between a current video unit and a previous feature unit according to the feature extraction result;

[0011] The video unit to be inferred is determined based on the feature difference and a preset decision threshold.

[0012] Optionally, the step of determining a sampling frame rate according to the video unit to be inferred, and performing frame sampling on the video unit to be inferred according to the sampling frame rate includes:

[0013] Determine the rhythm category corresponding to the video unit to be inferred according to the rhythm category information;

[0014] Determining a sampling frame rate of the video unit to be inferred based on the rhythm category;

[0015] Frame sampling is performed on the video unit to be inferred according to the sampling frame rate.

[0016] Optionally, before the step of determining a sampling frame rate according to the video unit to be inferred and performing frame sampling on the video unit to be inferred according to the sampling frame rate, the step further includes:

[0017] The sample data is sampled according to different frame sampling rates to obtain the sample data sampling results;

[0018] Determine the behavior recognition accuracy corresponding to the sample data sampling result;

[0019] Determining the optimal frame rate corresponding to each behavior category according to the behavior recognition accuracy;

[0020] Performing clustering based on the optimal frame rate to obtain a clustering result;

[0021] The rhythm category information is determined according to the clustering result.

[0022] The present application divides the captured video clips into multiple video units; performs feature extraction on the video units, and determines the video units to be inferred based on the feature extraction results; determines the sampling frame rate based on the video units to be inferred, performs frame sampling on the video units to be inferred according to the sampling frame rate, and sends the sampling results to the cloud for behavior recognition. Since the present application divides the collected video clips into multiple video units, determines the video units to be inferred based on the video units, performs frame sampling from the video units to be inferred, and sends the sampling results to the cloud for behavior recognition, compared to the existing method of sending all collected videos to the cloud for behavior recognition, the above method of the present application can reduce analysis delays and data transmission.

[0023] In addition, to achieve the above purpose, the present application proposes a behavior recognition method, which is applied in the cloud and includes the following steps:

[0024] Receive the sampling results sent by the edge end;

[0025] The sampling result is adjusted into a target feature map by an input adjustment module;

[0026] Based on the delay adjustment module, the behavior category of the target feature graph is inferred to obtain a behavior recognition result, and the behavior recognition result is sent to the edge end.

[0027] The present application receives the sampling results sent by the edge end; adjusts the sampling results to a target feature graph through an input adjustment module; infers the behavior category of the target feature graph based on the delay adjustment module, obtains the behavior recognition result, and sends the behavior recognition result to the edge end. Compared with the existing method of performing behavior analysis at the edge end, the above method of the present application can avoid the limited computing resources of the edge end and improve the efficiency of behavior analysis.

[0028] In addition, to achieve the above purpose, the present application also proposes a behavior recognition device, which is applied to the edge end, and the behavior recognition device includes:

[0029] A division module, for dividing the captured video clip into a plurality of video units;

[0030] A feature extraction module, used to extract features from the video unit and determine the video unit to be inferred according to the feature extraction result;

[0031] The sampling module is used to determine the sampling frame rate according to the video unit to be inferred, perform frame sampling on the video unit to be inferred according to the sampling frame rate, and send the sampling result to the cloud for behavior recognition.

[0032] In addition, to achieve the above purpose, the present application also proposes a behavior recognition device, which is applied to the cloud, and the behavior recognition device includes:

[0033] A receiving module, used to receive the sampling results sent by the edge end;

[0034] An input adjustment module, used for adjusting the sampling result into a target feature map through the input adjustment module;

[0035] The delay adjustment module is used to infer the behavior category of the target feature graph based on the delay adjustment module, obtain the behavior recognition result, and send the behavior recognition result to the edge end.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a behavior recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior recognition method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior recognition method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the behavior recognition method described above are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of the first embodiment of the behavior recognition method of the present application is provided;

[0042] Figure 2 A schematic diagram of the edge structure provided for the first embodiment of the behavior recognition method of the present application;

[0043] Figure 3 A flowchart of the second embodiment of the behavior recognition method of the present application is provided;

[0044] Figure 4 A structural diagram provided for the second embodiment of the behavior recognition method of this application;

[0045] Figure 5 A schematic diagram of the elastic model adaptation structure provided in Example 2 of the behavior recognition method of this application;

[0046] Figure 6 This is a schematic diagram of the module structure of the first embodiment of the behavior recognition device of the present application;

[0047] Figure 7 This is a schematic diagram of the module structure of the second embodiment of the behavior recognition device of the present application;

[0048] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the behavior recognition method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The edge of the present application divides the captured video clips into multiple video units; performs feature extraction on the video units, and determines the video units to be inferred based on the feature extraction results; determines the sampling frame rate based on the video units to be inferred, performs frame sampling on the video units to be inferred according to the sampling frame rate, and sends the sampling results to the cloud for behavior recognition. Since the present application divides the collected video clips into multiple video units, determines the video units to be inferred based on the video units, performs frame sampling from the video units to be inferred, and sends the sampling results to the cloud for behavior recognition, compared to the existing method of sending all collected videos to the cloud for behavior recognition, the above method of the present application can reduce analysis delays and data transmission.

[0053] The cloud of the present application receives the sampling results sent by the edge; adjusts the sampling results to a target feature map through an input adjustment module; infers the behavior category of the target feature map based on the delay adjustment module, obtains the behavior recognition result, and sends the behavior recognition result to the edge. Compared with the existing method of performing behavior analysis at the edge, the above method of the present application can avoid the limited computing resources of the edge and improve the efficiency of behavior analysis.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a behavior recognition device, etc. The following takes the behavior recognition device as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the present application embodiment provides a behavior recognition method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the behavior recognition method of the present application.

[0056] In this embodiment, the behavior recognition method includes steps S10 to S30:

[0057] Step S10, dividing the captured video clip into a plurality of video units;

[0058] It should be noted that the video clip may be a captured video that needs to be used for behavior recognition. It may be a fixed number of video frames, such as 15 frames. Dividing the captured video clip into multiple video units may be identifying and segmenting the video clip to obtain multiple video units. A video unit consists of a series of image blocks belonging to the same target. For ease of explanation, the nth video unit of a specific target is recorded as u n .

[0059] Step S20, extracting features from the video unit, and determining the video unit to be inferred according to the feature extraction result;

[0060] It should be noted that for each video unit u n , this example will construct a system with three elements<flag,thresh,class> The state vector of the video unit. Among them, flag indicates whether reasoning is needed, thresh is the threshold for evaluating semantic consistency, which can be a predefined value, and class specifies the behavior category of the video unit. n The state vector is given by u n with u n-1 The feature extraction of the video unit may be performed by using an X3D model for the video unit u n Extract feature f n . Determining the video unit to be inferred based on the feature extraction result may be determining the feature difference between the current video unit and the previous video unit based on the feature extraction result. If the feature difference is less than thresh, it is determined whether the behavior category of the previous video unit is known. If known, it is determined that the behavior category of the current video unit is consistent with the behavior category of the previous video unit. If it is unknown or the current video unit is the first video unit, it is determined that the current video unit is a video unit that requires behavior recognition, that is, the video unit to be inferred. If the feature difference is greater than or equal to thresh, it indicates that there may be a behavior conversion and inference needs to be performed immediately.

[0061] Step S30, determining a sampling frame rate according to the video unit to be inferred, performing frame sampling on the video unit to be inferred according to the sampling frame rate, and sending the sampling result to the cloud for behavior recognition.

[0062] It should be noted that in order to avoid transmitting too many video frames to the cloud, in this embodiment, the video unit to be inferred can be frame sampled according to the sampling frame rate, and the sampled video frames can be sent to the cloud. The sampling frame rate determined according to the video unit to be inferred can be determined according to the first video unit in the video unit to be inferred, and the subsequent video units all use the same sampling frame rate. The sampling frame rate determined according to the first video unit in the video unit to be inferred can be calculated at different sampling frame rates. The accuracy of the behavior analysis results is obtained, and the sampling frame rate with the highest accuracy is used as the sampling frame rate of the video unit to be inferred. Then, the video unit to be inferred is frame sampled according to the sampling frame rate, and the sampling results are sent to the cloud for behavior recognition.

[0063] Furthermore, given the limited computing and network resources at the edge, it is unrealistic to transmit every frame to the cloud for inference. Therefore, step S30 may include:

[0064] Determine the rhythm category corresponding to the video unit to be inferred according to the rhythm category information;

[0065] Determining a sampling frame rate of the video unit to be inferred based on the rhythm category;

[0066] Frame sampling is performed on the video unit to be inferred according to the sampling frame rate.

[0067] It should be noted that the rhythm category information may include the best sampling frame rate corresponding to each behavior category. Determining the rhythm category corresponding to the video unit to be inferred based on the rhythm category information may be to determine the behavior category of the first video unit in the video unit to be inferred, and determine the rhythm category in the rhythm category information to which the behavior category belongs. Determining the sampling frame rate of the video unit to be inferred based on the rhythm category may use the best sampling frame rate corresponding to the rhythm category as the sampling frame rate of the video unit to be inferred.

[0068] Furthermore, in order to determine the optimal sampling frame rate corresponding to each behavior category, before step S30, the following steps may be included:

[0069] The sample data is sampled according to different frame sampling rates to obtain the sample data sampling results;

[0070] Determine the behavior recognition accuracy corresponding to the sample data sampling result;

[0071] Determining the optimal frame rate corresponding to each behavior category according to the behavior recognition accuracy;

[0072] Performing clustering based on the optimal frame rate to obtain a clustering result;

[0073] The rhythm category information is determined according to the clustering result.

[0074] It should be noted that the sample data may include sample video frame data of different behavior categories. The sampling of sample data according to different frame sampling rates may be for each behavior category, sampling is performed according to different frame sampling rates to obtain sample data sampling results of different behavior categories and different sampling frame rates. Then, the sampling frame rate with the highest behavior recognition accuracy corresponding to each behavior category is determined, that is, the optimal frame rate. Clustering is performed based on the optimal frame rate to obtain a clustering result. Determining the rhythm category information according to the clustering result may be determining different behavior categories and the corresponding sampling frame rates in the same cluster according to the clustering result, and the sampling frame rate under the same cluster may be the average value of the optimal frame rates of each behavior category under the cluster.

[0075] Please refer to Figure 2 , Figure 2 Schematic diagram of the edge structure provided in Example 1 of the behavior recognition method of the present application; in this embodiment, after capturing a video clip (i.e., a fixed number of frames, such as 15 frames), the target detection and tracking module (i.e. Figure 2 The object detection and object tracking in the video processing unit is responsible for identifying and segmenting the frame into multiple video units. A video unit consists of a series of image blocks belonging to the same object. For ease of explanation, the nth video unit of a specific object in this embodiment is denoted as u n . The sequence boundary locator is then used to evaluate u n , to decide whether to send it to the cloud for inference, and the behavior rhythm sampler selects some frames for transmission. Specifically, for each video unit u n , will construct a list containing three elements<flag,thresh,class> The state vector of . Among them, flag indicates u n Whether reasoning is needed, thresh is the threshold for evaluating semantic consistency, and class specifies the behavior category of the video unit. n The state vector is given by u n with u n-1 The key idea here is that the difference in feature maps between video units reflects semantic consistency, that is, a difference less than thresh indicates that the two video units belong to the same behavior. To ensure the optimal inference frequency, thresh is dynamically adjusted. Once u n After being selected for inference, the behavior rhythm sampler determines the appropriate frame sampling rate based on the estimated behavior rhythm category. Subsequently, the video unit is encoded and uploaded to the cloud.

[0076] In a specific implementation, the behavior in the video stream usually lasts for a period of time. For example, a behavior such as bending over may last for a few seconds, while walking may last for several minutes or even longer. It is a waste of computing resources to perform multiple inferences on the same behavior. Therefore, it is crucial to identify the frame sequence corresponding to the same behavior, which is referred to as a single-activity sequence (SAS) in this embodiment. In this embodiment, an efficient rule-based algorithm 1 is proposed to identify SAS by analyzing the semantic differences between consecutive frames. The specific algorithm 1 is:

[0077] Algorithm 1: Calculate the state S for the video unit n

[0078] Input: u n : The nth video unit, S n-1 : Video unit u n-1 The state vector, f n-1 :u n-1 feature map, count: number of blocks to be inferred, S n [flag]: state vector S n Flag item (other tags are similar)

[0079] Output: S n :u n The state vector

[0080] f n ←X3D(u n )

[0081]

[0082] if diff n-1 [thresh]then

[0083] S n [flag]←S n-1 [flag]

[0084] Update S according to the threshold update formula n [thresh]

[0085] S n [class] = S n-1 [class]

[0086] if S n [flag] == to be inferred then

[0087] count←count-1

[0088] if count≤0then​

[0089] Infer the cached video unit and update S n [class]

[0090] S n [flag]←No reasoning required

[0091] Reset count

[0092] end

[0093] end

[0094] else

[0095] if S n-1 [flag] == to be inferred then

[0096] Inference cached video units

[0097] end S n [flag]←To be inferred

[0098] Reset S n [thresh]

[0099] S n [class]←Unknown (to be inferred)

[0100] end

[0101] Specifically, given a video stream, it is first divided into multiple video clips, each of which consists of N (e.g., 15) consecutive frames. Subsequently, target detection and tracking are performed on these video clips to generate multiple video units. Each video unit consists of image blocks belonging to the same person. Therefore, the number of image blocks in a video unit can be less than N. Specifically, the bounding box of the target detection is enlarged by 50% in each direction so that the interaction with the surrounding environment (e.g., a ball) is included when generating image blocks. For a video unit u of a certain target n , define a state vector (statevector) as S n =<flag,thresh,class> , where flag∈{0: to be inferred, 1: no inference required} represents the inference state, and thresh∈(0,1) is used to evaluate u n and u n-1 The semantic consistency between them, class represents u n For the convenience of discussion, S n-1 The flag in is denoted as S n-1 [flag]. Similar notations are used for other elements.

[0102] The key to determining whether the current video unit needs to be inferred is semantic consistency. In this embodiment, a behavior classification model (i.e., X3D) is used to extract feature maps as semantic information. Feature differences are used to evaluate semantic consistency. Video units with smaller semantic differences are considered to belong to the same behavior, because the semantic differences of video units from different behaviors are usually larger.

[0103] Specifically, first use the X3D model for the video unit u n Extract feature f n Then, the feature difference is calculated, denoted as diff∈(0,1), that is, f n and f n-1 If diff is less than thresh, this indicates that u n and u n-1 belongs to the same behavior. In this case, according to u n-1 If the category of is known, n Assign the same category or send it to the cloud for inference. On the other hand, diff greater than thresh indicates that there may be a behavior transition, indicating that the current SAS is in u n-1 Note that the X3D model on the edge device is only used to evaluate the semantic consistency between video units, while the video units are streamed to the cloud server for inference.

[0104] However, using a fixed thresh to evaluate semantic consistency is problematic. A higher thresh may fail to detect transitions between similar behaviors (e.g., walking and running), while a lower thresh may incorrectly segment a sequence of the same behavior into multiple SASs. To address this issue, the present embodiment dynamically adjusts thresh based on the state vector. The initial value of thresh can be set to 0.4, and it will be reset to the initial value once a behavior transition occurs; otherwise, thresh decays as follows:

[0105] S n [thresh]=S n-1 [thresh]×2 -σ

[0106] Where σ = diff n / S n-1 [thresh] represents the probability of behavior transition. A higher σ indicates a greater probability of behavior change, so the threshold needs to be decayed faster. In addition, to avoid caching too many video units for inference, immediate inference is triggered once a sufficient number of video units are cached.

[0107] Once a video unit is decided to be sent to the cloud for inference, the behavior rhythm sampler is used to select a minimal but representative subset of frames for transmission. The behavior rhythm sampler estimates the rhythm of the behavior and determines the optimal frame sampling rate for transmission. It consists of an offline stage and an online stage. In the offline stage, the best frame rate for each behavior category is pre-analyzed and classified into different rhythm categories. In the online stage, the most appropriate frame rate is selected based on the estimated rhythm category.

[0108] Specifically, in the offline stage, we first calculate the average accuracy AC of behavior k at a series of frame rates k , that is, AC k =<α1,α2,…,α n >, where α i Indicates the frame rate is f i To determine the optimal frame rate for a behavior, calculate the change in accuracy relative to the frame rate using the following formula:

[0109]

[0110] As the frame rate increases, R i Usually it will decrease. Therefore, R i The first frame rate that is lower than the preset threshold is taken as the best frame rate for the behavior. After determining the best frame rate for each behavior, a clustering algorithm (such as K-Means) is used to group them into k rhythm categories. In this embodiment, k=4 can be set. This helps to perform preliminary classification of rhythm categories on the edge device.

[0111] Online stage: The first video unit in the SAS is used to determine the rhythm category and the optimal frame rate (i.e., determine the behavior category of the first video unit in the SAS, query the rhythm category information according to the determined behavior category, and determine the corresponding optimal frame rate). Since a single video unit provides sufficient information for the preliminary classification of the rhythm, subsequent video units in the same SAS will be sampled at the same frame rate.

[0112] This embodiment is applied to the edge, including dividing the captured video clips into multiple video units; extracting features from the video units, and determining the video units to be inferred based on the feature extraction results; determining the sampling frame rate based on the video units to be inferred, sampling frames from the video units to be inferred according to the sampling frame rate, and sending the sampling results to the cloud for behavior recognition. Since this embodiment divides the collected video clips into multiple video units, determines the video units to be inferred based on the video units, performs frame sampling from the video units to be inferred, and sends the sampling results to the cloud for behavior recognition, compared with the existing method of sending all collected videos to the cloud for behavior recognition, the above method of this embodiment can reduce analysis delay and data transmission.

[0113] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 A flow chart of the second embodiment of the behavior recognition method of the present application.

[0114] In this embodiment, the behavior recognition method is applied to the cloud, and the behavior recognition method includes the following steps:

[0115] Step S40: receiving the sampling result sent by the edge end;

[0116] Step S50: adjusting the sampling result into a target feature map through an input adjustment module;

[0117] It should be noted that, please refer to Figure 4 , Figure 4 A structural diagram provided for Example 2 of the behavior recognition method of the present application; in this embodiment, after receiving the video unit (i.e., the sampling result), the cloud will decide whether to cache it for subsequent processing or perform inference immediately based on the flag. However, the frame sampling process leads to inconsistent frame intervals and numbers, which has an adverse effect on the accuracy of the model. In order to solve this problem, the elastic model adaptation module of this embodiment introduces a super model that can adapt to different inputs. This super model is divided into two stages: an input adjustment stage and a delay adjustment stage. The goal of the input adjustment stage is to convert input frames with different intervals and numbers into a unified feature map. Subsequently, the delay adjustment stage is responsible for predicting the final behavior category while complying with the delay requirements. Once the behavior category is obtained, the inference result will be sent back to the edge device. Please refer to Figure 5 , Figure 5 The schematic diagram of the elastic model adaptation structure provided in the second embodiment of the behavior recognition method of this application; the overview of the elastic model adaptation is as follows Figure 5 It can be divided into two key phases: the input adjustment phase, i.e., the input adjustment module (Input Adjustment Phase) and the delay adjustment phase, i.e., the delay adjustment module (LatencyAdjustment Phase). The input adjustment phase aims to adapt to different frame intervals and numbers, while the goal of the delay adjustment phase is to adjust the model to continuously meet the delay target.

[0118] The input adjustment phase aims to convert input data with different frame intervals and numbers into a unified feature map. It consists of a series of neural network modules, each of which has an input and produces an output. The checkpoints in the input adjustment phase are represented as <cp1,cp2,…,cp n>, and define the connection between the two checkpoints cp i and cp j The path is R i,j Each path consists of multiple rhythm modules, each of which is responsible for processing frames sampled at a specific interval. Frames sampled at a specific interval are best processed by a rhythm module designed for that specific rhythm, while the accuracy is reduced when processed by other rhythm modules. During each inference process, only one rhythm module in each path is selected for execution.

[0119] To adapt to different numbers of frames, the model skips checkpoints when necessary to ensure that the dimensions of the final feature map are uniform. Each rhythm module is designed to reduce the same time dimension in the feature map, denoted as Δt. Inputs with more frames will pass through more rhythm modules, while inputs with fewer frames will follow shorter paths. This approach ensures flexible adaptation to different numbers of frames while generating uniform feature maps.

[0120] During training, a reference feature map is first generated for each rhythm module within a pathway. This is achieved by training a baseline model using frames sampled at intervals matching a specific rhythm. Subsequently, each rhythm module is trained using a feature-based knowledge distillation approach. For example, a rhythm module r i,j,v Using the reference feature map f i,v and f j,v For training, these feature maps are obtained from the baseline model matched with the rhythm v. The loss function during training is:

[0121]

[0122] During the reasoning process, according to the initial time dimension T input and the target time dimension T mid , calculate the number of checkpoints J that need to be skipped, that is: J = n-(T mid -T input ) / Δt, the probability of each checkpoint being skipped is In this process, each path selects a rhythm module that matches the current frame rate to execute to ensure the best accuracy.

[0123] Step S60: inferring the behavior category of the target feature graph based on the delay adjustment module, obtaining a behavior recognition result, and sending the behavior recognition result to the edge end.

[0124] It should be noted that in the delay adjustment stage, the model infers the behavior category based on the unified feature map. Similar to the input adjustment stage, the delay adjustment stage also contains multiple checkpoints, and there is a neural network module between every two checkpoints. The checkpoints in the delay adjustment stage are represented as <cp1′,cp2′,…,cpn ′>, checkpoint cp i ′ and cp j The path between ′ is defined as R i,j Each path contains multiple deep modules. Each deep module is configured with a different number of neural network layers. Modules with deeper layers provide higher accuracy, but also produce greater inference latency.

[0125] In order to meet the requirements of real-time video analysis, the complexity of the model needs to be adjusted according to the latency target. In this embodiment, the latency target of a specific reasoning task is affected by two factors: (i) the remaining latency budget L due to the video unit cache remain : As more video units are cached for inference, in order to maintain the same end-to-end latency (i.e., the time difference from the occurrence of an action to the generation of a recognition result), the remaining latency budget will decrease; (ii) The probability of a new action occurring σ: The greater the probability of a new action occurring in a newly arrived video unit, the more it is necessary to reduce the current inference latency to ensure low latency at all times. Based on these factors, the latency target L for the current inference task is target Defined as:

[0126] L target =L remain -β·σ

[0127] Among them, L target =L remain -β·σ, β is the weight coefficient. A higher σ value indicates a more significant semantic change, which means a higher probability of behavior conversion.

[0128] During the training process, these modules are also trained through distillation learning similar to the input adjustment stage. After training, the delay of each deep module is calculated and recorded as T = {t1,…,t k}. Note that the latency of a certain depth module is consistent across all paths. In addition, the average latency of all modules with different depths is calculated as

[0129] During the inference process, we first calculate the number of checkpoints that need to be skipped, i.e. The average delay of modules with different depths, where L all is the latency when no checkpoints are skipped. Since skipping checkpoints may affect accuracy, the remaining available latency is A portion of the paths are randomly selected to use deeper depth blocks for inference.

[0130] This embodiment receives the sampling results sent by the edge end; adjusts the sampling results to a target feature graph through an input adjustment module; infers the behavior category of the target feature graph based on the delay adjustment module, obtains a behavior recognition result, and sends the behavior recognition result to the edge end. Compared with the existing method of performing behavior analysis at the edge end, the above method of this embodiment can avoid the limited computing resources of the edge end and improve the efficiency of behavior analysis.

[0131] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the behavior identification method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0132] This application also provides a behavior recognition device, please refer to Figure 6 , Figure 6 Schematic diagram of the module structure of the first embodiment of the behavior recognition device of the present application; the behavior recognition device includes an edge end, and the behavior recognition device includes:

[0133] A division module 10, for dividing the captured video clip into a plurality of video units;

[0134] A feature extraction module 20 is used to extract features from the video unit and determine the video unit to be inferred according to the feature extraction result;

[0135] The sampling module 30 is used to determine the sampling frame rate according to the video unit to be inferred, perform frame sampling on the video unit to be inferred according to the sampling frame rate, and send the sampling result to the cloud for behavior recognition.

[0136] This embodiment is applied to the edge, including dividing the captured video clips into multiple video units; extracting features from the video units, and determining the video units to be inferred based on the feature extraction results; determining the sampling frame rate based on the video units to be inferred, sampling frames from the video units to be inferred according to the sampling frame rate, and sending the sampling results to the cloud for behavior recognition. Since this embodiment divides the collected video clips into multiple video units, determines the video units to be inferred based on the video units, performs frame sampling from the video units to be inferred, and sends the sampling results to the cloud for behavior recognition, compared with the existing method of sending all collected videos to the cloud for behavior recognition, the above method of this embodiment can reduce analysis delay and data transmission.

[0137] This application also provides a behavior recognition device, please refer to Figure 7 , Figure 7 This is a schematic diagram of the module structure of the second embodiment of the behavior recognition device of the present application; the behavior recognition device is applied to the cloud, and the behavior recognition device includes:

[0138] The receiving module 40 is used to receive the sampling result sent by the edge end;

[0139] An input adjustment module 50, configured to adjust the sampling result into a target feature map through an input adjustment module;

[0140] The delay adjustment module 60 is used to infer the behavior category of the target feature graph based on the delay adjustment module, obtain the behavior recognition result, and send the behavior recognition result to the edge end.

[0141] This embodiment receives the sampling results sent by the edge end; adjusts the sampling results to a target feature graph through an input adjustment module; infers the behavior category of the target feature graph based on the delay adjustment module, obtains a behavior recognition result, and sends the behavior recognition result to the edge end. Compared with the existing method of performing behavior analysis at the edge end, the above method of this embodiment can avoid the limited computing resources of the edge end and improve the efficiency of behavior analysis.

[0142] The behavior recognition device provided by the present application adopts the behavior recognition method in the above embodiment, which can solve the technical problem of low efficiency of existing behavior recognition. Compared with the prior art, the beneficial effects of the behavior recognition device provided by the present application are the same as the beneficial effects of the behavior recognition method provided by the above embodiment, and other technical features in the behavior recognition device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0143] The present application provides a behavior recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the behavior recognition method in the above-mentioned embodiment one.

[0144] Reference below Figure 8 , which shows a schematic diagram of the structure of a behavior recognition device suitable for implementing the embodiment of the present application. The behavior recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The behavior recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0145] like Figure 8 As shown, the behavior recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the behavior recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the behavior recognition device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a behavior recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or provided alternatively.

[0146] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0147] The behavior recognition device provided by this application adopts the behavior recognition method in the above embodiment, which can solve the technical problem of low efficiency of existing behavior recognition. Compared with the prior art, the beneficial effects of the behavior recognition device provided by this application are the same as the beneficial effects of the behavior recognition method provided by the above embodiment, and the other technical features in the behavior recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0150] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the behavior recognition method in the above-mentioned embodiment.

[0151] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0152] The computer-readable storage medium may be included in the behavior recognition device; or may exist independently without being assembled into the behavior recognition device.

[0153] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the behavior recognition device, the behavior recognition device executes the behavior recognition method.

[0154] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0157] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned behavior recognition method, and can solve the technical problem of low efficiency of existing behavior recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the behavior recognition method provided in the above-mentioned embodiment, and will not be repeated here.

[0158] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned behavior recognition method when executed by a processor.

[0159] The computer program product provided by this application can solve the technical problem of low efficiency of existing behavior recognition. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the behavior recognition method provided by the above embodiment, which will not be repeated here.

[0160] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: Applied to the edge, the behavior recognition method includes the following steps: Dividing the captured video clip into a plurality of video units; Extracting features from the video unit, and determining the video unit to be inferred according to the feature extraction result; A sampling frame rate is determined according to the video unit to be inferred, frame sampling is performed on the video unit to be inferred according to the sampling frame rate, and the sampling result is sent to the cloud for behavior recognition.

2. The behavior recognition method according to claim 1, characterized in that: The step of extracting features from the video unit and determining the video unit to be inferred according to the feature extraction result includes: Performing feature extraction on the video unit to obtain a feature extraction result; Determining a feature difference between a current video unit and a previous feature unit according to the feature extraction result; The video unit to be inferred is determined based on the feature difference and a preset decision threshold.

3. The behavior recognition method according to any one of claims 1 to 2, characterized in that: The step of determining a sampling frame rate according to the video unit to be inferred, and performing frame sampling on the video unit to be inferred according to the sampling frame rate comprises: Determine the rhythm category corresponding to the video unit to be inferred according to the rhythm category information; Determining a sampling frame rate of the video unit to be inferred based on the rhythm category; Frame sampling is performed on the video unit to be inferred according to the sampling frame rate.

4. The behavior recognition method according to claim 3, characterized in that: Before the step of determining a sampling frame rate according to the video unit to be inferred and performing frame sampling on the video unit to be inferred according to the sampling frame rate, the step further includes: The sample data is sampled according to different frame sampling rates to obtain the sample data sampling results; Determine the behavior recognition accuracy corresponding to the sample data sampling result; Determining the optimal frame rate corresponding to each behavior category according to the behavior recognition accuracy; Performing clustering based on the optimal frame rate to obtain a clustering result; The rhythm category information is determined according to the clustering result.

5. A behavior recognition method, characterized in that: Applied to the cloud, the behavior recognition method includes the following steps: Receive the sampling results sent by the edge end; The sampling result is adjusted into a target feature map by an input adjustment module; Based on the delay adjustment module, the behavior category of the target feature graph is inferred to obtain a behavior recognition result, and the behavior recognition result is sent to the edge end.

6. A behavior recognition device, characterized in that: Applied to the edge, the behavior recognition device includes: A division module, for dividing the captured video clip into a plurality of video units; A feature extraction module, used to extract features from the video unit and determine the video unit to be inferred according to the feature extraction result; The sampling module is used to determine the sampling frame rate according to the video unit to be inferred, perform frame sampling on the video unit to be inferred according to the sampling frame rate, and send the sampling result to the cloud for behavior recognition.

7. A behavior recognition device, characterized in that: Applied in the cloud, the behavior recognition device includes: A receiving module, used to receive the sampling results sent by the edge end; An input adjustment module, used for adjusting the sampling result into a target feature map through the input adjustment module; The delay adjustment module is used to infer the behavior category of the target feature graph based on the delay adjustment module, obtain the behavior recognition result, and send the behavior recognition result to the edge end.

8. A behavior recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior recognition method according to any one of claims 1 to 4 and / or 5.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior recognition method according to any one of claims 1 to 4 and / or 5 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the behavior recognition method according to any one of claims 1 to 4 and / or 5 are implemented.

Images