Target behavior recognition method and electronic device

Abstract

The application discloses a target behavior recognition method and an electronic device, relates to the technical field of target behavior recognition, and comprises the following steps: inputting a first standard frame into a multi-branch detection model, determining whether a first detection target exists in the first standard frame; in response to the existence, determining and cropping a first detection region in the first standard frame to obtain a second standard frame, inputting the second standard frame into the multi-branch detection model, determining whether a second detection target corresponding to a second detection branch exists, and sequentially performing the determination until the step-by-step detection based on a preset detection branch structure is completed; determining a frame-level behavior recognition result of the first detection target according to a root branch detection result; and determining a comprehensive behavior recognition result of the first detection target based on a plurality of frame-level behavior recognition results. The application can detect video frames through the multi-branch detection model, dynamically control the detection process according to the detection result, improve the recognition accuracy while reducing the calculation overhead, and solve the problems of low recognition accuracy and efficiency of related technologies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target behavior recognition, and particularly relates to a target behavior recognition method and an electronic device. BACKGROUND

[0002] With the wide application of video monitoring and intelligent recognition technology, the demand for automatic detection and recognition of personnel behavior is increasing. The existing target behavior recognition method is generally based on a deep learning model, which detects and classifies the whole video frame to identify whether a specific behavior target exists. However, although the target behavior can be recognized, there are still many defects in the lightness and recognition accuracy.

[0003] On the one hand, the existing target behavior recognition method generally performs global detection on each image frame, extracts features and performs recognition calculation on the whole frame content, resulting in large calculation resource consumption, which is not conducive to deployment on real-time monitoring systems or embedded terminal devices. On the other hand, the detection structure of the existing method is a fixed process, which cannot dynamically adjust or terminate the detection process according to the detection result, resulting in the need to perform complete detection in the case of target missing, causing redundant calculation and efficiency reduction, and even affecting the recognition stability due to feature redundancy or error accumulation. SUMMARY

[0004] The present application provides a target behavior recognition method capable of detecting video frames through a multi-branch detection model and dynamically controlling the detection process according to the detection result, which reduces the calculation overhead while improving the recognition accuracy, thereby facilitating the lightness deployment and real-time application of the system.

[0005] The present application provides a target behavior recognition method, comprising:

[0006] obtaining video stream data and extracting video frames, and pre-processing the video frames to obtain first standard frames;

[0007] inputting the first standard frames into a pre-trained multi-branch detection model to determine whether a first detection target corresponding to a first detection branch in the multi-branch detection model exists in the first standard frames;

[0008] in response to the existence of the first detection target, determining and cropping a first detection region in the first standard frames, generating a second standard frame according to the first detection region and inputting the second standard frame into the multi-branch detection model to determine whether a second detection target corresponding to a second detection branch in the multi-branch detection model exists in the second standard frame;

[0009] In response to the existence of the second detection target, a third standard frame is generated and input into the multi-branch detection model, and the step is sequentially executed until the step-by-step detection based on the preset detection branch structure is completed.

[0010] According to one or more branch detection results output by the multi-branch detection model, a frame-level behavior recognition result of the first detection target in the first standard frame is determined.

[0011] Based on the multiple frame-level behavior recognition results of the first detection target in the continuous video frames, a comprehensive behavior recognition result of the first detection target is determined.

[0012] The application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any one of the target behavior recognition methods.

[0013] According to the application, since the multi-branch detection model is used for step-by-step detection of video frames, a dynamic behavior recognition process based on a detection target hierarchical structure is realized. Specifically, after the first detection target is detected, the next level detection branch is triggered, avoiding redundant calculation of complete detection in the case of target loss, and significantly reducing the resource consumption of the recognition process. At the same time, the step-by-step detection realized by the branch structure can focus on more fine detection targets at each level, improving the accuracy of local feature recognition and the precision of behavior recognition. In addition, by performing temporal fusion based on the frame-level recognition results on continuous video frames, the continuity and robustness of behavior recognition can be enhanced, so as to balance the light weight and recognition accuracy. Therefore, the problems of large calculation resource consumption, fixed detection process, difficult lightweight deployment, and low recognition accuracy and efficiency of the prior art by using traditional models for recognition can be solved. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 A target behavior recognition method application environment schematic diagram provided for the embodiments of the application;

[0016] Figure 2 A target behavior recognition method schematic diagram provided for the embodiments of the application;

[0017] Figure 3 A multi-branch detection model architecture schematic diagram provided for the embodiments of the application;

[0018] Figure 4An input frame feature vector generation schematic diagram provided by an embodiment of the present application;

[0019] Figure 5 A personnel abnormal behavior recognition flowchart provided by an embodiment of the present application;

[0020] Figure 6 An electronic device schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0022] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0024] The target behavior recognition method provided by the present application can be applied to the application environment as shown in Figure 1 The image acquisition terminal 101 communicates with the operation server 102 through a network or a data transmission cable. The image acquisition terminal 101 is used to acquire video frame data in a monitoring scene, and sends the acquired video data to the operation server 102 for analysis and processing. The operation server 102 is used to receive video frames from the image acquisition terminal 101, and detects the input video frames based on a pre-deployed multi-branch detection model. The image acquisition terminal 101 can be, but is not limited to, a fixed or mobile video monitoring camera, a smart camera, an image acquisition device carried by a drone, a mobile terminal or other devices with image acquisition function. The operation server 102 can be an independent server device, a cloud computing platform, an edge computing node, or an embedded intelligent processing unit, etc. with operation and model inference capability.

[0025] In an embodiment of the present application, the multi-branch detection model in the operation server 102 includes a plurality of detection branches, each detection branch corresponding to a different detection depth or feature extraction level. After input of a video frame, the model sequentially calls each branch for detection in a predetermined order, and when any branch detects that the target behavior does not exist in the current video frame, the detection process of the subsequent branch is terminated, thereby dynamically controlling the overall detection process and avoiding invalid inference calculation; the obtained multi-branch detection results are fused to obtain the final target behavior recognition result. Through the above structure, the target behavior recognition method of the present application can significantly reduce the detection calculation amount while ensuring the recognition accuracy, realize dynamic adjustment and lightweight deployment of the detection process, and is suitable for various application scenarios such as real-time video monitoring, intelligent security, behavior analysis, etc.

[0026] In one embodiment, as shown in FIG. 1, Figure 2 The present application provides a target behavior recognition method, applied to an operation server 102, comprising:

[0027] Step 201, acquiring video stream data and extracting video frames, pre-processing the video frames to obtain first standard frames;

[0028] Step 202, inputting the first standard frame into the pre-trained multi-branch detection model to determine whether a first detection target corresponding to a first detection branch in the multi-branch detection model exists in the first standard frame;

[0029] Step 203, in response to the existence of the first detection target, determining and cropping a first detection region in the first standard frame, generating a second standard frame according to the first detection region and inputting it into the multi-branch detection model to determine whether a second detection target corresponding to a second detection branch in the multi-branch detection model exists in the second standard frame;

[0030] Step 204, in response to the existence of the second detection target, generating a third standard frame and inputting it into the multi-branch detection model, and sequentially executing until the step-by-step detection based on the preset detection branch structure is completed;

[0031] Step 205, determining the frame-level behavior recognition result of the first detection target in the first standard frame according to one or more branch detection results output by the multi-branch detection model;

[0032] Step 206, determining the comprehensive behavior recognition result of the first detection target based on a plurality of frame-level behavior recognition results of the first detection target in consecutive video frames.

[0033] The target behavior recognition method provided in the embodiment realizes a dynamic behavior recognition process based on a detection target hierarchical structure through the step-by-step detection of video frames by the multi-branch detection model. The method triggers the next level of detection branch after detecting the first detection target, thereby avoiding redundant calculation of performing complete detection in the case of target loss and significantly reducing the resource consumption of the recognition process. Meanwhile, the step-by-step detection realized by the branch structure can focus on more fine detection targets at each level, thereby improving the accuracy of local feature recognition and the precision of behavior recognition. In addition, the time sequence fusion based on the frame-level recognition result on the continuous video frames can enhance the continuity and robustness of behavior recognition, thereby balancing the light weight and recognition accuracy.

[0034] In one embodiment, the video stream data is acquired and video frames are extracted, and the video frames are preprocessed to obtain first standard frames, including:

[0035] Frame extraction is performed on the video stream data based on a preset frame extraction time interval to obtain a plurality of original video frames;

[0036] The size of the plurality of original video frames is scaled based on the receiving size of the multi-branch detection model to obtain a plurality of corrected video frames, wherein the scaling of the size is preferably realized by bilinear interpolation;

[0037] The format of the plurality of corrected video frames is converted based on the receiving format of the multi-branch detection model to obtain a plurality of candidate video frames, wherein the conversion of the format includes but is not limited to color space conversion, resolution unification, and channel format unification;

[0038] The first standard frame is selected from the plurality of candidate video frames to input the multi-branch detection model.

[0039] Specifically, in the embodiment, the frame extraction, size scaling, and format conversion steps are set in the input stage, which can optimize the preprocessing process of video data while ensuring the consistency of model input. The frame extraction can reduce input redundancy in the time dimension and reduce repeated calculation of continuous frames. The size scaling makes the input frame match the receiving size of the detection model, thereby avoiding feature extraction deviation caused by size mismatch. The format conversion ensures that the data channel and color space are consistent with the model, thereby improving the stability and generalization performance of the model input. In summary, the front-end data processing burden can be reduced to provide standardized input for subsequent detection and improve the recognition efficiency and stability.

[0040] In one embodiment, the first standard frame is selected from the plurality of candidate video frames, including:

[0041] A candidate frame sequence is constructed based on a preset number of continuous candidate video frames;

[0042] determine frame variation degrees of the plurality of candidate video frames according to feature difference degrees of adjacent video frames in the candidate frame sequence;

[0043] based on the frame variation degrees, determine whether there exists a target frame segment composed of a plurality of candidate video frames that are continuous and have frame variation degrees greater than a preset frame variation degree threshold value;

[0044] in response to the existence, select a video frame located at a middle position of the target frame segment as the first standard frame according to a time position;

[0045] in response to the non-existence, select a video frame at a middle position of the candidate frame sequence as the first standard frame based on a time sequence.

[0046] Specifically, in the embodiment, the scheme of selecting the first standard frame based on the frame variation degrees can adaptively select a key frame containing significant change information in the video stream, thereby avoiding the target missing problem caused by random or fixed interval frame extraction; meanwhile, by calculating the feature difference degrees between adjacent frames and screening out a frame segment with significant changes, and then selecting a middle frame thereof as an input frame, the selected frame can be ensured to have representativeness and detection significance, thereby improving the responsiveness and accuracy of behavior detection, and especially suitable for behavior period detection, which can enhance the spatiotemporal sensitivity of the detection frame while maintaining the calculation efficiency.

[0047] In one specific embodiment, frame variation degrees of the plurality of candidate video frames are determined according to feature difference degrees of adjacent video frames in the candidate frame sequence, and are represented as:

[0048]

[0049] wherein D i represents the frame variation degree of the i-th candidate video frame relative to the i-1-th candidate video frame, F i (x, y) represents a pixel gray value of the i-th candidate video frame, F i-1 (x, y) represents a pixel gray value of the i-th candidate video frame, and H represents a frame size height of the candidate video frame, and W represents a frame size width of the candidate video frame.

[0050] based on the frame variation degrees, determine whether there exists a target frame segment composed of a plurality of candidate video frames that are continuous and have frame variation degrees greater than a preset frame variation degree threshold value, including: setting 1.2 times of an average frame variation degree as the frame variation degree threshold value based on historical statistical samples, denoted as T D , if there exist k consecutive candidate video frames that satisfy D i > T D , the k consecutive candidate video frames are defined as the target frame segment, and k is preferably 3-5;

[0051] select a video frame located at a middle position of the target frame segment as the first standard frame according to a time position, represented as:​

[0052] ;

[0053] wherein, F mid denotes the middle frame of the selected target frame segment, i start denotes the serial number of the first candidate video frame in the target frame segment, i end denotes the serial number of the last candidate video frame in the target frame segment, is a floor function.

[0054] In an embodiment, before the first standard frame is input into the pre-trained multi-branch detection model, further comprising:

[0055] parsing the received target behavior recognition task to determine a detection target, the detection target at least including: a first detection target and a plurality of branch detection targets hierarchically associated with the first detection target;

[0056] determining a plurality of detection branches of the multi-branch detection model according to the first detection target and the plurality of branch detection targets, and generating a detection branch structure;

[0057] setting a model interface to receive an input frame and a current detection target identifier conforming to a preset format and size;

[0058] setting a backbone extraction path to perform feature extraction on the received input frame to obtain an input frame feature vector;

[0059] setting a plurality of branch detection paths corresponding to the detection targets to extract a branch feature vector corresponding to the current detection target based on the input frame feature vector and the current detection target identifier, and compare the branch feature vector with a detection target feature vector corresponding to the detection target to generate a branch detection result;

[0060] setting a model exit to output the branch detection result;

[0061] constructing the multi-branch detection model based on the model interface, the backbone extraction path, the plurality of branch detection paths, and the model exit.

[0062] Specifically, in the embodiment, the multi-branch detection model is dynamically constructed according to the identification task in the input stage, so that the detection branch structure can be adaptively generated according to the hierarchical relationship of the detection target, and the associated detection of multi-level targets is supported. Specifically, in the embodiment, the multi-branch detection model constructed by the cooperative configuration of the model interface, the main extraction path and the plurality of branch detection paths can flexibly cope with different detection tasks, rather than relying on a fixed structure. In addition, the setting of the model outlet ensures the independent output and unified management of each branch detection result. Therefore, the configurability and expansibility of the model are improved, and the rapid adaptation and deployment for different task scenarios are realized on the premise of ensuring the detection accuracy.

[0063] In one embodiment, as shown in Figure 3 The multi-branch detection model provided by the embodiment includes a model interface, a main extraction path, a plurality of branch detection paths, and a model outlet, wherein,

[0064] The model interface is used to receive an input frame and a current detection target identifier corresponding to the input frame, and the current detection target identifier is used to indicate the detection type or the detection task category of the current input frame. The model interface transmits the detection target identifier to the main extraction path and the branch selection module at the same time as receiving the input frame, so as to determine the subsequent detection branch path;

[0065] The main extraction path is used to extract features of the input frame to generate a main feature representation, i.e., an input frame feature vector, and after output, the input frame feature vector is input to the corresponding branch detection path according to the branch selection result determined by the detection target identifier;

[0066] The plurality of branch detection paths includes but is not limited to a first branch detection path, a second branch detection path, and a third branch detection path. Different branch detection paths perform feature refinement and result prediction for different detection tasks or target categories, for example, can be used for human body detection, posture detection and object recognition respectively. Each branch detection path outputs a corresponding detection result, including a first branch detection result, a second branch detection result and a third branch detection result.

[0067] The model outlet is used to receive and output each branch detection result for subsequent fusion of the branch detection result.

[0068] Through the above structural design, the multi-branch detection model of the embodiment can dynamically select the branch detection path according to different detection target identifiers, realize task-adaptive detection processing, improve the accuracy and real-time performance of detection, and improve the reusability of the model, reduce the model size, and reduce the calculation and deployment overhead.

[0069] As Figure 4As shown, in one embodiment, feature extraction is performed on the received input frame to obtain an input frame feature vector, including:

[0070] An initial frame feature is generated according to the input frame, and the initial frame feature is copied to obtain at least a first frame feature stream and a second frame feature stream;

[0071] A three-dimensional convolution, normalization and activation operation are sequentially performed on the first frame feature stream to extract deep semantic features to obtain a first sub-frame feature;

[0072] A shallow three-dimensional convolution processing is performed on the second frame feature stream to extract low-level spatial structure features to obtain a second sub-frame feature;

[0073] The first sub-frame feature is divided into a first grand-frame feature and a second grand-frame feature in the channel dimension;

[0074] A multi-level three-dimensional convolution processing and channel compression are performed on the first grand-frame feature to extract high-order semantic features to obtain a first great-grand-frame feature;

[0075] The channel dimension of the second sub-frame feature, the second grand-frame feature and the first great-grand-frame feature is adjusted, spliced and fused to obtain a fused feature and generate an input frame feature vector.

[0076] Specifically, in the embodiment, through multi-stream and multi-level extraction and fusion of input frame features, spatial and semantic information of video frames can be fully captured. Specifically, three-dimensional convolution combined with deep and shallow feature stream extraction mechanism makes deep stream pay attention to semantic features and shallow stream pay attention to spatial structure features. Further, through multi-level convolution and channel compression, high-order semantic features are extracted and fused with low-level features, which can effectively retain spatio-temporal structure and semantic context information. Such multi-level fusion feature extraction method improves feature expression capability without increasing computational complexity, provides high-resolution and high-semantic-density input feature vectors for subsequent branch detection, and thus improves overall recognition performance.

[0077] In one specific embodiment, the process of feature extraction on the received input frame to generate an input frame feature vector includes the following steps:

[0078] An initial frame feature is generated according to the input frame, which can be extracted by a three-dimensional convolution layer, wherein the convolution kernel size is preferably 3x3x3 and the step is 1, to simultaneously capture local feature information of the input frame in the spatial and temporal dimensions;

[0079] The initial frame feature is copied to obtain at least a first frame feature stream and a second frame feature stream.

[0080] The three-dimensional convolution, batch normalization and nonlinear activation (such as ReLU) operations are sequentially performed on the first frame feature stream to extract deep semantic features, and a first sub-frame feature is obtained, which can enhance the time-dependent information of the input frame and extract high-level semantic representation;

[0081] The shallow three-dimensional convolution processing is performed on the second frame feature stream, and the convolution kernel size is preferably 1x3x3, which is used to extract the low-level spatial structure features of the input frame to obtain a second sub-frame feature, wherein the shallow convolution can retain local texture, edge and spatial layout information, and provide a basis for subsequent multi-scale feature fusion;

[0082] The first sub-frame feature is divided into a first grandson frame feature and a second grandson frame feature in the channel dimension, and the division ratio can be 1:1 or self-adaptively adjusted according to the network structure;

[0083] The multi-level three-dimensional convolution processing is performed on the first grandson frame feature, including a plurality of continuous 3x3x3 convolution layers, to extract high-order semantic features layer by layer, and the channel compression is realized through 1x1x1 convolution to obtain a first great-grandson frame feature, which can improve the abstract level of feature expression while maintaining the compactness of the feature space;

[0084] The channel dimension of the second sub-frame feature, the second grandson frame feature and the first great-grandson frame feature is aligned, and the feature splicing fusion (such as concat operation) is used to combine them to obtain a fusion feature, which can be mapped through a fully connected layer to generate an input frame feature vector as the input of the subsequent multi-branch detection model.

[0085] Through the above design, the embodiment establishes an effective fusion relationship between deep semantic features and shallow spatial features, which can enhance the expression ability of time sequence semantics while ensuring the fidelity of local spatial details, thereby improving the overall feature representation quality of the input frame and providing more accurate and stable feature support for the subsequent detection task.

[0086] In one embodiment, a branch feature vector is extracted based on the input frame feature vector, and the branch feature vector is compared with the corresponding detection target feature vector to generate a branch detection result, including:

[0087] In response to the current branch detection path receiving the input frame feature vector, the pre-stored current detection target feature vector corresponding to the current branch detection path is obtained;

[0088] Based on the current detection target feature vector and the input frame feature vector, the response degree of a plurality of channels in the input frame feature vector to the current detection target is determined, and a target correlation weight coefficient is generated;

[0089] According to the target correlation weight coefficient, a plurality of channels in the input frame feature vector are weighted to generate a branch feature vector;

[0090] A similarity calculation is performed on the branch feature vector and the current detection target feature vector to generate a current branch feature similarity;

[0091] In response to the current branch feature similarity being greater than or equal to a preset branch feature similarity threshold, it is determined that the current detection target exists in the input frame;

[0092] In response to the current branch feature similarity being less than the preset branch feature similarity threshold, it is determined that the current detection target does not exist in the input frame;

[0093] A current branch detection confidence is generated based on the current branch feature similarity through a confidence mapping function, and a current branch detection result is generated and output in combination with the current determination result.

[0094] Specifically, in the embodiment, the target correlation weight and the similarity calculation mechanism are introduced in the branch detection path to realize accurate detection based on target feature adaptive response. Specifically, the channels in the input frame feature that respond strongly to the target are determined using the pre-stored target feature vector, and a branch feature vector is generated by weighting, thereby strengthening the model's attention to the target area. Further, through the dual calculation mechanism of similarity and confidence, the reliability of the detection result can be quantified at the feature matching level, which helps to improve the determination accuracy and interpretability of the detection result, thereby significantly reducing the false detection rate of the model and enhancing the model's discrimination ability in multi-target detection scenarios.

[0095] In one specific embodiment, in response to the current branch detection path receiving an input frame feature vector, the system obtains a current detection target feature vector corresponding to the branch detection path from a feature storage module, including:

[0096] The input frame feature vector is denoted as F = [f1, f2,..., fn], where n represents the number of channels of the input frame feature vector, and the value range is 64-1024, f n i represents the input frame feature component of the i-th channel; i

[0097] The current detection target feature vector is denoted as T = [t1, t2,..., tn], and ti represents the current detection target feature component of the i-th channel, which can be obtained in advance by prior sample feature extraction, template averaging or feature clustering method and stored in the model database;

[0098] Based on the current detection target feature vector and the input frame feature vector, the response degree of a plurality of channels in the input frame feature vector to the current detection target is determined, and the correlation response value of the feature component of each channel is calculated, denoted as:​

[0099] ;

[0100] wherein, r i represents the correlation response value between the i-th channel in the input frame feature vector and the corresponding i-th channel in the current detection target feature vector, used to measure the matching degree of the channel feature to the current detection target, the larger the value, the more the channel feature can reflect the characteristic attribute of the current detection target; ε is a stable constant to prevent the denominator from being zero, the value range is 10-8 to 10-6; ||·|| represents L2 norm operation;

[0101] The correlation response values of all channels are normalized to obtain the target correlation weight coefficient, denoted as: ; wherein, w i represents the target correlation weight of the i-th channel in the input frame feature vector and the i-th channel in the current detection target feature vector;

[0102] Based on the target correlation weight coefficient, the channel dimension of the input frame feature vector is weighted to obtain the branch feature vector, denoted as F*=[w1f1,w2f2,...,w n f n ];

[0103] The similarity between the branch feature vector and the current detection target feature vector is calculated to generate the current branch feature similarity, denoted as: ; wherein, S represents the current branch feature similarity;

[0104] Based on the current branch feature similarity, the current branch detection confidence is generated through the confidence mapping function, denoted as:

[0105] ;

[0106] wherein, C represents the current branch detection confidence obtained by mapping, α represents the confidence curve slope adjustment coefficient, usually taking the value of 5 to 15, used to control the confidence rising speed, β is the offset term, the value range is 0.3 to 0.6, used to balance the response distribution of positive and negative samples;

[0107] Combined with the current determination result, the current branch detection result is generated and output, wherein the current branch detection result includes but is not limited to: current detection target class label, current determination result (existence or nonexistence), current branch feature similarity, current branch detection confidence, detection frame or spatial position coordinate information.

[0108] In one specific embodiment, after constructing the multi-branch detection model, the model is further trained, specifically including:

[0109] Set training parameters, including batch size, learning rate, training rounds, and optimizer type, etc.

[0110] Extract original image samples containing multiple detection targets from the training data set, the detection targets at least including a first detection target corresponding to a first detection branch in the multi-branch detection model, and a second detection target corresponding to a second detection branch in the multi-branch detection model;

[0111] Preprocess the extracted original image samples to ensure consistency with the input requirements of the multi-branch detection model. Specifically, perform noise removal, brightness normalization, and geometric correction operations on the original image; and scale the image size to the receiving size of the multi-branch detection model, preferably the receiving size is 640x640, keep the aspect ratio of the image unchanged during the scaling process, scale the longest side to 640 pixels, and fill the shorter side in gray scale to obtain a standard training frame that meets the input format;

[0112] After completing the image preprocessing, perform feature enhancement operations on the standard training frame. Specifically, use Mosaic splicing enhancement and MixUp fusion enhancement strategies to combine the training samples in space and mix the features to improve the generalization ability of the model under different scene, scale and occlusion conditions;

[0113] The enhanced training sample is input into the multi-branch detection model, which includes: a model interface for receiving input frames and current detection target identifiers that meet the preset format and size; a backbone extraction path for extracting features from the input frame to obtain an input frame feature vector; a plurality of branch detection paths corresponding to the first detection target and a plurality of branch detection targets hierarchically associated with it; and a model outlet for outputting each branch detection result;

[0114] In the forward propagation stage, the backbone extraction path extracts multiple layers of semantic features from the input frame, and the obtained input frame feature vector is transmitted to each branch detection path respectively;

[0115] Each branch detection path extracts a branch feature vector and generates a corresponding branch detection result according to the features of the corresponding detection target, including: a determination result (yes / no) of whether the current detection target exists, and a branch detection confidence of the current detection target;

[0116] In the back propagation stage, the model calculates the loss value and updates the model parameters according to the difference between the predicted branch detection result and the true annotation information in the training sample. Specifically, the total loss function of the multi-branch detection model includes the weighted combination of multiple sub-loss terms, mainly including: feature similarity loss, confidence loss, and branch consistency loss;

[0117] The total loss function of the multi-branch detection model can be expressed as:

[0118] L total =τL sim +φL conf +μL beh ;

[0119] Among them, L total L represents the total loss of the multi-branch detection model. sim L represents the feature similarity loss. conf L represents the confidence loss. beh The branch consistency loss is represented by τ, φ, and μ, which represent the feature similarity loss weight, confidence loss weight, and branch consistency loss weight, respectively, and are used to balance the optimization rate of each loss term. Through the above joint optimization design, the model can simultaneously take into account feature discriminability, confidence reliability, and branch detection consistency, and achieve higher accuracy and stronger robustness in hierarchical multi-target detection.

[0120] Specifically, the feature similarity loss is used to constrain the similarity measure between the input frame feature vector and the detected target feature vector, enabling the model to learn the discriminative power of different targets in the feature space. Preferably, the feature similarity loss can be a weighted function with a logarithmic adjustment term to enhance the training effect on difficult samples, expressed as:

[0121] ;

[0122] Where k1 represents the feature similarity loss balance coefficient, θ1 represents the adjustment power used to control the gradient scaling strength, and S i The similarity of branch features is calculated for the i-th branch detection path, and N represents the total number of branch detection paths. Through the above design, the discriminative power and similarity measurement accuracy between different detection targets can be improved while maintaining the feature aggregation degree.

[0123] Specifically, the confidence loss is used to constrain the consistency between the branch detection confidence generated by the similarity through the confidence mapping function and the actual existence label, so that the model can accurately reflect the probability of target existence in the output layer, thereby improving the stability and calibration accuracy of the detection confidence; preferably, the confidence loss can be expressed as a weighted binary cross-entropy function with a logarithmic term, as follows:

[0124] ;

[0125] Where k2 represents the confidence loss balance coefficient, used to control the weights of positive and negative samples; y i P represents the actual existence label of the target corresponding to the detection path of the i-th branch (1 for existence, 0 for non-existence); irepresents the branch detection confidence output by the i-th branch detection path; through the above loss design, effective calibration of confidence prediction can be realized, so that the model has higher target existence judgment accuracy in complex scenes.

[0126] Specifically, the branch consistency loss is used to constrain the consistency of the detection results of the multi-branch detection path under the same input frame, so as to ensure that the logical correspondence between the main detection target and the hierarchical associated detection target remains stable; preferably, the branch consistency loss can adopt the form of a square constraint function based on branch response difference, which is expressed as:

[0127] ;

[0128] wherein k3 represents a consistency constraint coefficient; R i , R j respectively represent the response vectors output by the i-th and j-th branch detection paths; N represents the total number of branch detection paths; through the above design, the detection conflicts between different detection branches can be effectively reduced, the overall coordination and stability of the hierarchical detection results can be maintained, and thus the overall robustness of the multi-branch detection model can be improved.

[0129] In one embodiment, in response to the existence of the first detection target, a first detection region is determined and cropped in the first standard frame, and a second standard frame is generated according to the first detection region and input to the multi-branch detection model, including:

[0130] based on the first branch detection result output by the first detection branch in the multi-branch detection model, determining the detection frame position of the first detection target, and extracting the target center coordinates and target boundary range of the first detection target in the first standard frame;

[0131] based on the target center coordinates, expanding the target boundary in a circumferential direction according to a preset first expansion ratio, and cropping in the first standard frame according to the expanded target boundary to obtain a first detection region;

[0132] performing geometric transformation and image correction on the first detection region according to the input size and format requirements of the multi-branch detection model to obtain a second standard frame;

[0133] inputting the second standard frame into the multi-branch detection model to detect whether a second detection target exists in the second standard frame in the second detection branch.

[0134] Specifically, in the embodiment, by means of the way of cropping and generating a new standard frame based on the bounding box position and the expansion scale, gradual refinement detection from a global frame to a local frame can be realized. Specifically, after the first detection target is identified, the region where the first detection target is located is expanded and corrected in space, so that the target subject can be ensured to be kept complete, and the interference of redundant background on detection can be inhibited. The second standard frame generated subsequently meets the model input requirements in size and format, ensuring the continuity and accuracy of the model inference process, so that the regional focusing capability of multi-level detection and the behavior fine-grained recognition effect can be improved while maintaining the continuity of detection.

[0135] In one embodiment, according to one or more branch detection results output by the multi-branch detection model, a frame-level behavior recognition result of the first detection target in the first standard frame is determined, including:

[0136] The multi-branch detection model outputs a plurality of branch detection results, and the branch detection result at least includes the existence of the first detection target;

[0137] In response to the branch detection result further at least including the existence of a second detection target, the first detection target is taken as a detection subject, and the remaining detection targets existing are taken as detection appendages. For example, when the detection result includes a “person target” and an “object target”, the “person target” is taken as the detection subject, and the “object target” is taken as the detection appendage.

[0138] According to the spatio-temporal correlation of the detection subject and one or more detection appendages, an associated weight coefficient of the one or more detection appendages relative to the detection subject is determined.

[0139] Based on the associated weight coefficient, joint inference is performed on the plurality of branch detection results to determine a subject frame behavior category of the detection subject in the first standard frame.

[0140] According to the branch detection confidence distribution corresponding to the plurality of branch detection results, a frame-level behavior confidence of the subject frame behavior category corresponding to the detection subject is determined.

[0141] According to the subject frame behavior category and the frame-level behavior confidence, a frame-level behavior recognition result is output.

[0142] Specifically, in the embodiment, by introducing spatio-temporal correlation analysis of the detection subject and the detection appendage at the frame level, the recognition and inference of interactive behaviors among multiple targets can be realized. Specifically, by calculating the associated weight of the detection appendage relative to the detection subject and performing joint inference, the behavior relationship (such as interaction, contact or confrontation) among multiple targets in a single frame can be captured. In addition, combined with the distribution information of the branch detection confidence, the frame-level confidence estimation of the behavior category can be dynamically corrected, so as to improve the discrimination and robustness of the frame-level behavior recognition, break through the limitation of single-target independent recognition, and realize accurate behavior semantic modeling in a multi-target scene.

[0143] In one specific embodiment, the correlation weight coefficient of the one or more detection appendages relative to the detection subject is determined according to the spatio-temporal correlation between the detection subject and the one or more detection appendages, denoted as:

[0144] ;

[0145] wherein w ij represents the correlation weight coefficient between the detection subject i and the jth detection appendage, reflecting the comprehensive correlation degree of the two in the spatial and temporal dimensions, p i , p j respectively represent the spatial position vectors of the detection subject and the detection appendage in the current frame, which can be the center point coordinates or key point position vectors of the corresponding target detection frame in the first labeled frame, with the unit being pixels or normalized coordinates; t i , t j respectively represent the time index values of the detection subject and the jth detection appendage, which can be frame numbers or time stamps, with the unit being frames or seconds; ||p i -p j || represents the Euclidean spatial distance between the detection subject and the jth detection appendage, used to characterize the spatial proximity; ||t i -t j || represents the interval of the subject and the jth detection appendage in the time dimension, used to characterize the time synchronization of the action occurrence; is a spatial scale parameter, controlling the sensitivity of the weight decay to the spatial distance, which can be in the range of 5-50 pixels; is a time scale parameter, controlling the sensitivity of the weight decay to the time interval, which can be in the range of 1-10 frames; Z is a normalization constant, used to ensure that the sum of all correlation weights is 1.

[0146] In one specific embodiment, based on the correlation weight coefficient, the multiple branch detection results are jointly inferred to determine the subject behavior class of the detection subject in the first standard frame, specifically including the following steps:

[0147] According to the branch detection result of each detection appendage, the corresponding class label and detection confidence value of each appendage are obtained, wherein the class label is used to represent the type information of the detection appendage (such as personnel, object, vehicle, etc.), and the branch detection confidence is used to represent the credibility of the detection result;

[0148] The branch detection confidence of each detection appendage is weighted according to the corresponding correlation weight coefficient, so that the appendages with higher spatio-temporal correlation with the detection subject have greater influence in joint inference;

[0149] In the weight fusion process, according to the category attributes of different detection appendages, corresponding category response factors are respectively set to reflect the influence degree of different appendage types on the judgment of the behavior of the subject, for example, for the appendages representing dangerous objects, higher category response factors can be assigned to enhance their contribution in behavior reasoning; for the appendages of ordinary static objects, lower category response factors can be assigned to weaken their influence on the final result.

[0150] The weighted confidence values of all appendages are summarized and normalized to obtain comprehensive response values for each candidate behavior category, which are used to reflect the overall matching degree of the detected subject under different candidate behavior categories.

[0151] According to the size of the comprehensive response value of each candidate behavior category, the behavior category with the maximum comprehensive response value is selected as the frame-level behavior category of the detected subject in the current standard frame.

[0152] Through the above process, the embodiment realizes the dynamic judgment of the frame-level behavior on the basis of comprehensively considering the spatio-temporal correlation between the detected appendages and the detected subject, the confidence distribution of the detected appendages and the category attributes. This method not only can accurately reflect the action characteristics of the detected subject itself, but also can combine the semantic information of the surrounding targets for joint reasoning, thereby significantly improving the accuracy and stability of frame-level behavior recognition in complex interactive scenes.

[0153] In one specific embodiment, the frame-level behavior confidence of the subject behavior category corresponding to the detected subject is determined according to the branch detection confidence distribution corresponding to the plurality of branch detection results, comprising:

[0154] Obtain each branch detection confidence value related to the detected subject in the plurality of branch detection results, and construct a branch detection confidence distribution to reflect the detection stability of the model on different detection targets and different branches;

[0155] For each branch detection result, combine its branch detection confidence with the spatio-temporal correlation weight coefficient to obtain a weighted branch detection confidence, so that the detection result with higher correlation with the detected subject in space and time has higher weight in frame-level confidence calculation;

[0156] Statistical aggregation is performed on all weighted branch detection confidences, which can be calculated by weighted average, maximum likelihood estimation or confidence interval fusion, to calculate the overall confidence index of the detected subject in the current standard frame, wherein the weighted average method can reflect the global confidence trend, and the maximum likelihood method can strengthen the confidence information of the dominant branch.

[0157] Further, in order to ensure the numerical stability and comparability of the confidence, the overall confidence index is normalized, which can be realized by linear transformation, Sigmoid function mapping or Softmax normalization, etc.

[0158] The calculated frame-level behavior confidence is associated with the determined frame-level behavior class to form a final frame-level behavior recognition result of the detection subject in the current standard frame.

[0159] Through the above process, the embodiment realizes adaptive calculation of frame-level confidence based on the fusion of confidence distribution information of multiple branch detection results and in combination with the spatio-temporal correlation weight, which can effectively suppress the influence of single branch detection error on the overall judgment and improve the reliability and robustness of the frame-level behavior recognition result.

[0160] In one embodiment, based on the multiple frame-level behavior recognition results of the first detection target in the continuous video frames, a comprehensive behavior recognition result of the first detection target is determined, including:

[0161] Obtaining multiple frame-level behavior recognition results of the first detection target in the continuous video frames, and parsing to obtain corresponding multiple subject frame behavior classes and frame-level behavior confidences;

[0162] Based on the time sequence, the multiple frame-level behavior recognition results are sorted to obtain a frame behavior sequence;

[0163] According to the similarity of the subject frame behavior classes of adjacent frames in the frame behavior sequence, a behavior consistency score of the first detection target between the continuous video frames is determined;

[0164] According to the behavior consistency score and the frame-level behavior confidence, a time weighting coefficient of the multiple frame-level behavior recognition results is respectively determined;

[0165] Based on the time weighting coefficient, the multiple frame-level behavior recognition results are weighted and fused to determine a comprehensive behavior class and a corresponding comprehensive behavior confidence of the first detection target in the frame behavior sequence;

[0166] According to the comprehensive behavior class and the comprehensive behavior confidence, a comprehensive behavior recognition result is output.

[0167] Specifically, in the embodiment, the time sequence consistency optimization of comprehensive behavior recognition is realized by weighting and fusing the continuous frame behavior results at the time sequence level. Specifically, the behavior consistency score is calculated based on the inter-frame behavior category similarity, and the time weighting coefficient is generated combined with the frame-level confidence, so as to dynamically strengthen the influence of high-consistency and high-confidence frames in the fusion process, effectively suppress the interference of accidental frame false detection or low-confidence frames, and improve the time stability and comprehensive confidence of the overall recognition result; thus, a more continuous, accurate and robust target comprehensive behavior recognition result can be output, meeting the needs of real-time monitoring and behavior analysis scenes.

[0168] In one specific embodiment, based on the multiple frame-level behavior recognition results of the first detection target in the continuous video frames, the comprehensive behavior recognition result of the first detection target is determined, specifically including:

[0169] Obtain multiple subject frame behavior recognition results of the first detection target in the continuous video frames, and parse the corresponding multiple subject frame behavior categories and subject frame behavior confidences therefrom, wherein each subject frame behavior recognition result is used to represent the behavior state information of the first detection target in a single frame;

[0170] Sort the multiple subject frame behavior recognition results in the time sequence of the video frames to obtain a subject frame behavior sequence, which is used to describe the behavior change process of the first detection target in the time dimension;

[0171] According to the similarity between the subject frame behavior categories of adjacent frames in the subject frame behavior sequence, determine the subject behavior consistency score of the first detection target between the continuous video frames, wherein the subject behavior consistency score is used to reflect the motion continuity and behavior stability of the first detection target in the time dimension, specifically, when the subject frame behavior categories of adjacent frames are the same or change smoothly, the consistency score is higher; when there is obvious jump or sudden abnormality, the consistency score is lower; the calculation of the behavior consistency score can be realized based on category label matching, feature vector similarity or time sequence smoothing processing, etc.;

[0172] According to the aforementioned subject behavior consistency score and the subject frame behavior confidences corresponding to each frame, respectively determine the frame-level time weighting coefficients of the multiple subject frame behavior recognition results, specifically, in the weight calculation process, the frames with high confidence and high consistency with adjacent frames will obtain larger time weighting coefficients; the time weighting coefficients of the frames with low confidence or poor consistency will be correspondingly reduced, and this weighting strategy can adaptively highlight the continuous and reliable behavior information in the time dimension, and weaken the influence of noise frames or abnormal frames;

[0173] Based on the above frame-level time weighting coefficient, the multiple subject frame behavior recognition results are weighted and fused to determine a subject comprehensive behavior category of the first detection target in the subject frame behavior sequence and a corresponding subject comprehensive behavior confidence, wherein the subject comprehensive behavior category is used to represent the main behavior type of the first detection target in a time range, and the subject comprehensive behavior confidence is used to represent the overall credibility of the behavior judgment result.

[0174] According to the subject comprehensive behavior category and the subject comprehensive behavior confidence, the comprehensive behavior recognition result of the first detection target is output, which can be used as a judgment basis for subsequent event recognition, behavior prediction or anomaly detection.

[0175] Through the above process, the embodiment introduces behavior consistency modeling and time weighting fusion mechanism on the basis of fusing multiple subject frame behavior recognition results, realizes dynamic reasoning from frame-level behavior recognition result to time period-level comprehensive behavior recognition result, can effectively improve the time sequence consistency and overall robustness of the first detection target behavior recognition, significantly reduces the influence of single frame misjudgment on the final recognition result, and thus improves the accuracy and stability of comprehensive behavior recognition in complex scenes.

[0176] As shown in Figure 5 The embodiment of the application also provides a target behavior recognition process taking a person as a first detection target, an abnormal body posture as a second detection target, and an abnormal entity as a third detection target, comprising:

[0177] Receiving video stream data input, performing video preprocessing on the input video stream data to extract effective frames and perform basic image enhancement or noise reduction operation;

[0178] Firstly calling a multi-branch detection model, extracting features of the input frame via a backbone feature extraction path, and then selecting a first branch detection path to detect the preprocessed video frame to determine whether there is a person target in the video picture;

[0179] If the person target is detected, a first detection region is extracted, and the multi-branch detection model is called again, and after the features are extracted again via the backbone feature extraction path, a second branch detection path is selected according to the current detection task identifier to detect the person target in the first detection region to determine whether the person has an abnormal posture, such as falling down, holding or unnatural body posture, etc.

[0180] If the detection result indicates that there is an abnormal posture target, a second detection region is further extracted, and the multi-branch detection model is called for the third time. After the third extraction of features through the backbone feature extraction path, the third branch detection path is selected to identify the abnormal entity in the second detection region, so as to determine whether there is an abnormal entity target related to the abnormal posture in the video image, such as a dangerous object, an obstacle or other target object.

[0181] If the abnormal entity target is detected, according to the association relationship of the first detection result, the second detection result and the third detection result and the corresponding branch detection confidence, it is determined that the personnel abnormal behavior frame-level recognition result in the current video frame is an abnormal behavior, for example, the personnel holds dangerous goods in the current image;

[0182] If the personnel abnormal behavior frame-level recognition results of a plurality of continuous video frames are all abnormal behaviors, and the integrated abnormal behavior recognition result of the personnel determined after fusing the personnel abnormal behavior frame-level recognition results of the plurality of continuous video frames is also an abnormal behavior, it is determined that the personnel target has an abnormal behavior, for example, the personnel holds dangerous goods in a plurality of images;

[0183] If no abnormal posture target or abnormal entity target is detected, it is determined that no abnormal behavior occurs in the video frame, and the identification process ends.

[0184] Through the above process, the embodiment can progressively analyze different detection targets in the video stream based on the hierarchical calling mechanism of the multi-branch detection model, realize the step-by-step judgment from personnel detection to posture recognition and then to abnormal entity recognition, effectively improve the accuracy and robustness of abnormal behavior recognition, and reduce redundant calculation of irrelevant scenes, thereby improving the real-time response capability of the system.

[0185] In one embodiment, the method further comprises:

[0186] Based on the branch feature similarity and the branch detection confidence output by the multi-branch detection model for the same detection target within a preset time window, a template stability score and a target appearance drift risk score of the detection target are calculated;

[0187] When the template stability score is greater than a first preset threshold and the target appearance drift risk score is less than a second preset threshold, the detection target feature vector corresponding to the current detection target is updated online in an exponential moving average manner, so that the subsequent branch detection path can adapt to the slow change of the appearance of the target when performing feature comparison;

[0188] When the target appearance drift risk score is greater than the second preset threshold for consecutive multiple frames, the online updating is suspended, the historical steady-state detection target feature vector is rolled back, the calling of the deeper branch is terminated, and the frame-level behavior recognition result to be reviewed is output, so as to suppress the false detection caused by the feature template drift while ensuring the recognition accuracy.

[0189] In one embodiment, the method further comprises:

[0190] The input video frame is preprocessed at the edge side, and the shallow branch of the multi-branch detection model is called to obtain the first branch detection result and the branch detection confidence thereof;

[0191] According to the branch detection confidence, the branch feature similarity, and the current network bandwidth and time delay measure, a branch sinking strategy score is determined;

[0192] When the branch sinking strategy score is lower than a preset sinking threshold, the frame-level behavior recognition result is generated at the edge side based on the first branch detection result;

[0193] When the branch sinking strategy score is higher than the preset sinking threshold and the uploading condition is met, the corresponding video cropping region and / or intermediate feature are uploaded to the cloud end, the deep branch of the multi-branch detection model is called at the cloud end to generate the second branch detection result, and the second branch detection result is returned to the edge side, and the first branch detection result and the second branch detection result are fused at the edge side to obtain the final frame-level behavior recognition result;

[0194] The detection benefits in one branch sinking process and the network state are recorded, so as to adaptively adjust the branch sinking threshold in subsequent recognition tasks, and realize the dynamic cooperation and calculation overhead optimization of the multi-branch detection between the edge and the cloud.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0196] As shown in Figure 6 The embodiments of the present application also provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above target behavior recognition method embodiments.

[0197] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the claimed application. Accordingly, modifications and / or additions, other than those explicitly described herein, can be obvious to those skilled in the art in the light of this disclosure. The claimed application is intended to embrace all such modifications and / or additions.

[0198] The above provides a target behavior recognition method and an electronic device. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only applicable to help understand the method and the core idea of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A target behavior recognition method, characterized in that, include: Acquiring video stream data and extracting video frames, and preprocessing the video frames to obtain a first standard frame, includes: extracting frames from the video stream data based on a preset frame extraction time interval to obtain multiple original video frames; scaling the size of the multiple original video frames based on the receiving size of the multi-branch detection model to obtain multiple corrected video frames; converting the format of the multiple corrected video frames based on the receiving format of the multi-branch detection model to obtain multiple candidate video frames; and selecting the first standard frame from the multiple candidate video frames to input into the multi-branch detection model. The first standard frame is input into the pre-trained multi-branch detection model to determine whether there is a first detection target in the first standard frame that corresponds to the first detection branch in the multi-branch detection model. In response to the existence of the first detection target, a first detection region is determined and cropped from the first standard frame, and a second standard frame is generated based on the first detection region and input into the multi-branch detection model to determine whether there is a second detection target in the second standard frame that corresponds to the second detection branch in the multi-branch detection model; In response to the existence of the second detection target, a third standard frame is generated and input into the multi-branch detection model, and the process is repeated until the step-by-step detection based on the preset detection branch structure is completed. Based on one or more branch detection results output by the multi-branch detection model, determine the frame-level behavior recognition result of the first detection target in the first standard frame; Based on the frame-level behavior recognition results of the first detected target in consecutive video frames, a comprehensive behavior recognition result of the first detected target is determined. The selection of the first standard frame from the plurality of candidate video frames is performed in the following manner: A candidate frame sequence is constructed based on a preset number of consecutive candidate video frames; the frame variation degree of multiple candidate video frames is determined according to the feature difference degree of adjacent video frames in the candidate frame sequence; based on the frame variation degree, it is determined whether there exists a target frame segment composed of multiple consecutive candidate video frames whose frame variation degree is greater than a preset frame variation degree threshold; if it exists, a video frame located in the middle position of the target frame segment is selected as the first standard frame according to the time position; if it does not exist, a video frame located in the middle position of the candidate frame sequence is selected as the first standard frame according to the time order.

2. The target behavior recognition method according to claim 1, characterized in that, Before inputting the first standard frame into the pre-trained multi-branch detection model, the method further includes: The received target behavior recognition task is parsed to determine the detection target, which includes at least the first detection target and multiple branch detection targets that are hierarchically related to the first detection target. Based on the first detection target and the multiple branch detection targets, determine multiple detection branches of the multi-branch detection model and generate the detection branch structure; Configure the model interface to receive input frames and the current detection target identifier that conform to the preset format and size; Set the main extraction path to extract features from the received input frame and obtain the input frame feature vector; Multiple branch detection paths are set up, each corresponding to a detection target. Based on the input frame feature vector and the current detection target identifier, the branch feature vector corresponding to the current detection target is extracted, and the branch feature vector is compared with the corresponding detection target feature vector to generate a branch detection result. Set the model output to output the branch detection results; The multi-branch detection model is constructed based on the model interface, the main extraction path, the multiple branch detection paths, and the model exit.

3. The target behavior recognition method according to claim 2, characterized in that, The step of extracting features from the received input frame to obtain the input frame feature vector includes: An initial frame feature is generated based on the input frame, and the initial frame feature is copied to obtain at least a first frame feature stream and a second frame feature stream; Three-dimensional convolution, normalization, and activation operations are sequentially performed on the feature stream of the first frame to extract deep semantic features and obtain the features of the first subframe. Shallow 3D convolution processing is performed on the feature stream of the second frame to extract low-level spatial structure features to obtain the features of the second subframe; The first subframe features are divided into first grandchild frame features and second grandchild frame features in the channel dimension; Multi-level three-dimensional convolution processing and channel compression are performed on the features of the first grandchild frame to extract high-order semantic features and obtain the features of the first grandchild frame. Adjust the channel dimensions of the second subframe feature, the second grandchild frame feature, and the first great-grandchild feature, and then concatenate and fuse them to obtain the fused features and generate the input frame feature vector.

4. The target behavior recognition method according to claim 2, characterized in that, The step of extracting branch feature vectors based on the input frame feature vectors and comparing the branch feature vectors with the corresponding detection target feature vectors to generate branch detection results includes: In response to the current branch detection path receiving the input frame feature vector, the pre-stored current detection target feature vector corresponding to the current branch detection path is obtained; Based on the current target feature vector and the input frame feature vector, determine the response degree of multiple channels in the input frame feature vector to the current target and generate target relevance weight coefficients; Based on the target relevance weight coefficient, the multiple channels in the input frame feature vector are weighted to generate the branch feature vector; Perform similarity calculation on the branch feature vector and the current detection target feature vector to generate the current branch feature similarity; If the current branch feature similarity is greater than or equal to a preset branch feature similarity threshold, then it is determined that the current detection target exists in the input frame; If the similarity of the current branch feature is less than a preset branch feature similarity threshold, it is determined that the current detection target does not exist in the input frame. Based on the feature similarity of the current branch, the confidence of the current branch detection is generated through a confidence mapping function. Combined with the current judgment result, the current branch detection result is generated and output.

5. The target behavior recognition method according to claim 1, characterized in that, In response to the presence of the first detection target, the first detection region is determined and cropped from the first standard frame, and a second standard frame is generated based on the first detection region and input into the multi-branch detection model, including: Based on the first branch detection result output by the first detection branch in the multi-branch detection model, the detection box position of the first detection target is determined, and the target center coordinates and target boundary range of the first detection target in the first standard frame are extracted. Based on the target center coordinates, the target boundary is expanded circumferentially according to a preset first expansion ratio, and the expanded target boundary is cropped in the first standard frame to obtain the first detection area; Based on the input size and format requirements of the multi-branch detection model, geometric transformation and image correction are performed on the first detection region to obtain the second standard frame; The second standard frame is input into the multi-branch detection model to detect whether the second detection target exists in the second standard frame in the second detection branch.

6. The target behavior recognition method according to claim 1, characterized in that, The step of determining the frame-level behavior recognition result of the first detected target in the first standard frame based on one or more branch detection results output by the multi-branch detection model includes: Obtain multiple branch detection results output by the multi-branch detection model, wherein the branch detection results include at least the presence of the first detection target; If the branch detection result further includes at least the existence of the second detection target, then the first detection target is taken as the detection subject, and the remaining detection targets are taken as the detection appendages. Based on the spatiotemporal correlation between the detection subject and one or more detection attachments, determine the correlation weight coefficient of one or more detection attachments relative to the detection subject; Based on the association weight coefficient, joint inference is performed on multiple branch detection results to determine the subject frame behavior category of the detected subject in the first standard frame; Based on the branch detection confidence distribution corresponding to multiple branch detection results, determine the frame-level behavior confidence of the subject frame behavior category corresponding to the detection subject; Based on the main frame behavior category and the frame-level behavior confidence level, the frame-level behavior recognition result is output.

7. The target behavior recognition method according to claim 6, characterized in that, The step of determining the comprehensive behavior recognition result of the first detected target based on multiple frame-level behavior recognition results of the first detected target in consecutive video frames includes: Obtain multiple frame-level behavior recognition results of the first detection target in consecutive video frames, and parse them to obtain the corresponding multiple subject frame behavior categories and the frame-level behavior confidence scores; The frame-level behavior recognition results are sorted according to time order to obtain a frame behavior sequence. Based on the similarity of the main frame behavior categories of adjacent frames in the frame behavior sequence, the behavior consistency score of the first detection target among the consecutive video frames is determined. Based on the behavior consistency score and the frame-level behavior confidence, time weighting coefficients are determined for multiple frame-level behavior recognition results. Based on the time weighting coefficient, multiple frame-level behavior recognition results are weighted and fused to determine the comprehensive behavior category of the first detected target in the frame behavior sequence and the corresponding comprehensive behavior confidence. Based on the comprehensive behavior category and the comprehensive behavior confidence level, the comprehensive behavior recognition result is output.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the target behavior recognition method as described in any one of claims 1 to 7 when executing the computer program.

Images