A behavior detection method and device based on space-time context, equipment and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JINAN BOGUAN INTELLIGENT TECH CO LTD
- Filing Date
- 2022-08-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing behavior detection methods suffer from high sample dependence and computational complexity when dealing with the interaction between the target and its context, making it difficult to effectively distinguish complex behaviors.
By detecting the target position in a video sequence based on a preset frame interval, extracting absolute and relative position features, and performing attention operations, spatiotemporal behavioral features are generated for classification.
It reduces sample dependence and computational complexity, improves accuracy for complex behaviors, and reduces dependence on large amounts of data.
Smart Images
Figure CN115359570B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a behavior detection method, apparatus, device, and medium based on spatiotemporal context. Background Technology
[0002] The main task of behavior detection is to detect the location and actions of pedestrians in a video sequence. It is usually used in applications such as surveillance cameras and video review. By reporting abnormal behavior of pedestrians in video data in a timely manner, it can maintain public safety and protect people's property. It is currently a hot research topic.
[0003] Current behavior detection methods are divided into single-frame behavior detection and spatiotemporal behavior detection. Single-frame behavior detection involves inputting a single image frame and using the detection result of that frame as the target's behavior category. Spatiotemporal behavior detection, on the other hand, involves inputting an entire video sequence and fusing information from multiple frames of the target using techniques such as 3D convolution and target tracking. Compared to single-frame behavior detection, spatiotemporal behavior detection is more robust and accurate, and is currently the mainstream method. Typically, methods such as 3D convolution primarily enhance the analysis of features of individual targets. However, in practical applications, target behavior often interacts with the context. Without considering the states of other targets, some behaviors are difficult to define, such as fighting and exercising. Considering a single target makes behavior difficult to distinguish; only when the relationships between targets are taken into account can behavior be easily determined.
[0004] One existing technology proposes a context-based behavior recognition method. The main idea is to detect targets in intermediate frames using a detection model, extract spatiotemporal features of the target region from frames before and after using networks like SlowFast, perform attention operations on the target features, and finally classify the target. This approach only uses attention operations to assess the correlation between features, thus requiring a large amount of data to achieve good results.
[0005] Another approach is similar in concept to the existing technologies mentioned above, but differs in that it first performs a correlation analysis between targets as a first-order relationship during attention computation, and then performs a correlation analysis between relationships as a second-order relationship. This approach considers more complex relationships, is more dependent on the amount of data, and has higher computational complexity.
[0006] Therefore, reducing sample dependence and computational complexity are urgent problems to be solved in the process of behavior detection. Summary of the Invention
[0007] In view of this, the purpose of this invention is to provide a behavior detection method, apparatus, device, and medium based on spatiotemporal context, which can reduce sample dependence and computational complexity. The specific solution is as follows:
[0008] In a first aspect, this application discloses a behavior detection method based on spatiotemporal context, including:
[0009] The video sequence is detected based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames;
[0010] Feature extraction is performed on the video sequence to obtain a first feature map corresponding to the video sequence. The first feature map is then pooled based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, the action features of all targets are determined by the position information of the intermediate detection frames of the video sequence and the second feature map. The intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames.
[0011] Based on the location information of all the targets, determine the absolute location features of all the targets and the relative location features between every two targets;
[0012] Attention operations are performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets. The spatiotemporal behavior features are then classified to obtain the corresponding behavior detection results.
[0013] Optionally, the step of detecting the video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames includes:
[0014] The video sequence is detected based on a preset frame interval to obtain the center point coordinates of all target detection boxes in the video sequence corresponding to several detection frames, as well as the size information of the detection boxes; wherein, the size information is the width and height information of the detection boxes.
[0015] Optionally, the absolute positional features of all targets are determined based on their positional information, including:
[0016] The maximum, minimum, and average values of the change rate of each target between every two consecutive detection frames, as well as the output of the change rate after passing through the multilayer perceptron, are all concatenated to obtain the concatenated feature of the change rate of each target between every two consecutive detection frames. The change rate includes a first change rate, a second change rate, a third change rate, and a fourth change rate. The first and second change rates represent the change rates of the center point coordinates of the detection frame, while the third and fourth change rates represent the change rates of the width and height of the detection frame.
[0017] For each consecutive two detection frames, the concatenated features of the first change rate, the concatenated features of the second change rate, the concatenated features of the third change rate, and the concatenated features of the fourth change rate of each target are concatenated to obtain the absolute position features of all targets.
[0018] Optionally, before determining the absolute position features of all targets based on their position information, the method further includes:
[0019] Based on the center point coordinates of the detection box containing each target in each detection frame and the size information of the detection box, the rate of change of each target between every two consecutive detection frames is determined.
[0020] The maximum value, minimum value, and average value of the change rate of each target between each consecutive two detection frames are determined from the change rate of each target between each consecutive two detection frames. The change rate of each target is calculated based on the multilayer perceptron to obtain the output result of the change rate of each target between each consecutive two detection frames through the multilayer perceptron.
[0021] Optionally, determining the relative positional features between any two targets based on the positional information of all targets includes:
[0022] The maximum value, minimum value, average value, and output result of the relative change rate between each pair of targets between every two consecutive detection frames are all concatenated to obtain the concatenated feature of the relative change rate between each pair of targets between every two consecutive detection frames. The relative change rate includes a first relative change rate, a second relative change rate, a third relative change rate, and a fourth relative change rate. The first and second relative change rates are the relative change rates of the center point coordinates of the detection frame, and the third and fourth relative change rates are the relative change rates of the width and height of the detection frame.
[0023] The concatenated features of the first relative change rate, the second relative change rate, the third relative change rate, and the fourth relative change rate between each pair of targets between every two consecutive detection frames are concatenated to obtain the position features between each pair of targets between every two consecutive detection frames. The position features are then calculated based on the multilayer perceptron to obtain the relative position features between each pair of targets between every two consecutive detection frames.
[0024] Optionally, before determining the relative positional features between any two targets based on the positional information of all targets, the method further includes:
[0025] Based on the coordinates of the center point of the detection box containing each target in each detection frame and the size information of the detection box, the relative change rate between each pair of targets between each consecutive two detection frames is determined.
[0026] The maximum value, minimum value, and average value of the relative change rate are determined from the relative change rate between each pair of targets between each two consecutive detection frames. Based on the multilayer perceptron, the relative change rate between each pair of targets is calculated to obtain the output result of the multilayer perceptron of the relative change rate between each pair of targets between each two consecutive detection frames.
[0027] Optionally, the attention operation performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavioral features of all targets includes:
[0028] Based on a preset attention calculation formula, attention operations are performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavioral features of all targets. The preset attention calculation formula is as follows:
[0029] Q i ,K i V i =conv2D(cat(FC) act FC abs ));
[0030]
[0031]
[0032] Among them, FC act The action characteristics of the target are represented by FC. abs This represents the absolute positional characteristics of the target. The relative positional features between target i and target j are represented by d, where d represents the dimension, and FC is the most common feature. i This represents the spatiotemporal behavioral characteristics of the target.
[0033] Secondly, this application discloses a behavior detection device based on spatiotemporal context, comprising:
[0034] The location information determination module is used to detect the video sequence based on a preset frame interval to obtain the location information of all targets in the video sequence corresponding to several detection frames;
[0035] An action feature determination module is used to extract features from the video sequence to obtain a first feature map corresponding to the video sequence, and to pool the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, the action features of all targets are determined by the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames.
[0036] The location feature determination module is used to determine the absolute location features of all targets and the relative location features between any two targets based on the location information of all targets.
[0037] The behavior detection module is used to perform attention operations on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and to classify the spatiotemporal behavior features to obtain the corresponding behavior detection results.
[0038] Thirdly, this application discloses an electronic device, including:
[0039] Memory, used to store computer programs;
[0040] A processor is used to execute the computer program to implement the aforementioned behavior detection method based on spatiotemporal context.
[0041] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned behavior detection method based on spatiotemporal context.
[0042] As can be seen, this application proposes a behavior detection method based on spatiotemporal context, comprising: detecting a video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to a plurality of detection frames; extracting features from the video sequence to obtain a first feature map corresponding to the video sequence, and pooling the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence; then determining the action features of all targets through the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames; determining the absolute position features of all targets and the relative position features between every two targets based on the position information of all targets; performing attention operation on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and classifying the spatiotemporal behavior features to obtain the corresponding behavior detection results. Therefore, this application calculates the correlation between targets by using the positional change information of the targets, and generates the contextual behavioral features of the targets through an attention mechanism, that is, the spatiotemporal behavioral features of all targets. Furthermore, by classifying the spatiotemporal behavioral features, the corresponding behavior detection results are obtained. Compared with traditional methods that rely on a large amount of sample data for detection, this application does not require a large number of samples or need to consider more complex relationships, thus reducing sample dependence and computational complexity. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0044] Figure 1 This is a flowchart of a behavior detection method based on spatiotemporal context disclosed in this application;
[0045] Figure 2 A schematic diagram of a behavior detection architecture based on spatiotemporal context proposed in this application.
[0046] Figure 3 This is a flowchart of a specific behavior detection method based on spatiotemporal context disclosed in this application;
[0047] Figure 4 This is a schematic diagram of the structure of a location feature network disclosed in this application;
[0048] Figure 5This is a structural schematic diagram of an absolute position feature disclosed in this application;
[0049] Figure 6 This is a structural schematic diagram of a relative position feature disclosed in this application;
[0050] Figure 7 This is a schematic diagram of the behavior detection device based on spatiotemporal context disclosed in this application;
[0051] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] One existing technique uses attention operations to calculate the correlation between features, which requires a large amount of data to achieve good results. This invention directly models the distance and position information between targets, reducing the difficulty of model learning and decreasing sample dependence. Another existing technique is similar in concept to the first, but differs in that it first calculates the correlation between targets as a first-order relationship during attention operations, and then calculates the correlation between relationships as a second-order relationship. This approach considers more complex relationships, relies more on data, and has higher computational complexity.
[0054] To address this, this application proposes a behavior detection scheme based on spatiotemporal context, which can reduce sample dependency and computational complexity.
[0055] This application discloses a behavior detection method based on spatiotemporal context. See also Figure 1 As shown, the method includes:
[0056] Step S11: Detect the video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames.
[0057] In this embodiment, a video sequence X (with a total of T frames) is detected based on a preset frame interval t to obtain the center point coordinates (x, y) of all targets in the video sequence corresponding to several detection frames, as well as the size information of the detection frames. The several detections are specifically T / t times. The size information is the width and height information (w, h) of the detection frames. The shape of the detection frames is (T / t × n × 4), where n is the number of detected targets.
[0058] Step S12: Extract features from the video sequence to obtain a first feature map corresponding to the video sequence, and pool the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, determine the action features of all targets through the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames, that is, the T / (t×2)th detection frame.
[0059] In this embodiment, feature extraction is performed on the video sequence to obtain a first feature map Y, the shape of which is (T×h×w×C), where C is the number of channels. Pooling is then performed on Y based on the time dimension to obtain a second feature map Z, the shape of which is (h×w×C). It should be noted that feature extraction can specifically employ 3D convolution, I3D (Interactive Three Dimensions), or slowfast; pooling can employ average pooling or max pooling. In this embodiment, determining the action features of all targets using the position information of the intermediate detection frames in the video sequence and the second feature map specifically includes: extracting features for each target based on the second feature map and the position information of the intermediate detection frames to obtain a feature map P for all targets, the shape of which is (n×7×7×C). Then, pooling is performed on P to obtain the action features of all targets, the shape of which is (n×C). The intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames.
[0060] Step S13: Determine the absolute position features of all targets and the relative position features between every two targets based on the position information of all targets.
[0061] In this embodiment, the absolute position feature represents the target's movement information within the T-frame, while the relative position feature mainly reflects the relative changes in position and size of two different targets within the T-frame.
[0062] Step S14: Perform attention operations on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and classify the spatiotemporal behavior features to obtain the corresponding behavior detection results.
[0063] Specifically, attention operations are performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets based on a preset attention operation formula to obtain the spatiotemporal behavioral features of all targets. The preset attention operation formula is as follows:
[0064] Q i ,K i V i =conv2D(cat(FC) act FC abs ));
[0065]
[0066]
[0067] Among them, FC act The action characteristics of the target are represented by FC. abs This represents the absolute positional characteristics of the target. The relative positional features between target i and target j are represented by d, where d represents the dimension, and FC is the most common feature. i This represents the spatiotemporal behavioral characteristics of the target.
[0068] Figure 2 This diagram illustrates a spatiotemporal context-based behavior detection architecture proposed in this application, specifically including: a detection module, a feature extraction module, a ROI Align module, and a position feature module. The detection module detects the position information of all targets, the feature extraction module extracts feature maps corresponding to the video sequence, the ROI Align module determines the action features of the targets, and the position feature module determines the absolute position features of the targets and the relative position features between any two targets. Figure 2 It can be seen that when determining action features, the position information of intermediate detection frames is needed, and when performing attention operations, the position information of all detection frames is needed.
[0069] As can be seen, this application proposes a behavior detection method based on spatiotemporal context, comprising: detecting a video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to a plurality of detection frames; extracting features from the video sequence to obtain a first feature map corresponding to the video sequence, and pooling the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence; then determining the action features of all targets through the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames; determining the absolute position features of all targets and the relative position features between every two targets based on the position information of all targets; performing attention operation on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and classifying the spatiotemporal behavior features to obtain the corresponding behavior detection results. Therefore, this application calculates the correlation between targets by using the positional change information of the targets, and generates the contextual behavioral features of the targets through an attention mechanism, that is, the spatiotemporal behavioral features of all targets. Furthermore, by classifying the spatiotemporal behavioral features, the corresponding behavior detection results are obtained. Compared with traditional methods that rely on a large amount of sample data for detection, this application does not require a large number of samples or need to consider more complex relationships, thus reducing sample dependence and computational complexity.
[0070] This application discloses a specific behavior detection method based on spatiotemporal context. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. See also Figure 3 As shown, it specifically includes:
[0071] Step S21: Detect the video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames.
[0072] Step S22: Extract features from the video sequence to obtain a first feature map corresponding to the video sequence, and pool the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, determine the action features of all targets by using the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames.
[0073] Step S23: The maximum value, minimum value, average value, and output result of the multilayer perceptron of the change rate of each target between every two consecutive detection frames are all spliced together to obtain the spliced feature of the change rate of each target between every two consecutive detection frames.
[0074] In this embodiment, before feature stitching is performed on the maximum, minimum, and average values of the change rates of each target between every two consecutive detection frames, as well as the output results of the change rates after passing through the multilayer perceptron, it is first necessary to determine the change rates of each target between every two consecutive detection frames based on the center point coordinates of the detection box containing each target in each detection frame and the size information of the detection box. From the change rates of each target between every two consecutive detection frames, the maximum, minimum, and average values of the change rates of each target between every two consecutive detection frames are determined, and the change rates of each target are calculated based on the multilayer perceptron to obtain the output results of the change rates of each target between every two consecutive detection frames after passing through the multilayer perceptron.
[0075] Specifically, the first rate of change is The second rate of change is The third rate of change is The fourth rate of change is In this embodiment, the dx, dy, dw, and dh from the k-th and (k-1)-th detections are respectively input to... Figure 4 In the location feature network, the location feature network calculates the maximum value, minimum value, average value, and output result of the change rate of each target after passing through the multilayer perceptron. It should be noted that the change rate includes a first change rate, a second change rate, a third change rate, and a fourth change rate. The first and second change rates are the change rates of the center point coordinates of the detection box, and the third and fourth change rates are the change rates of the width and height of the detection box. Specifically, the first change rate is the change rate of the center point coordinate x of the detection box, the second change rate is the change rate of the center point coordinate y of the detection box, the third change rate is the change rate of the width w of the detection box, and the fourth change rate is the change rate of the height h of the detection box.
[0076] In this embodiment, the maximum value, minimum value, average value, and output result of the first change rate of each target between the k-th and (k-1)-th detection frames are spliced together to obtain the spliced features of the first change rate of each target between the k-th and (k-1)-th detection frames, which have a shape of (1×4). Similarly, the spliced features of the second, third, and fourth change rates are determined.
[0077] Step S24: For each consecutive two detection frames, the concatenated features of the first change rate, the concatenated features of the second change rate, the concatenated features of the third change rate, and the concatenated features of the fourth change rate of each target are concatenated to obtain the absolute position features of all targets.
[0078] In this embodiment, see Figure 5 As shown, the spliced features of the first change speed, the spliced features of the second change speed, the spliced features of the third change speed, and the spliced features of the fourth change speed are spliced together to obtain the absolute position features of the target. In this way, the k-th detection and the (k-1)-th detection frames are applied to all detection frames, and this embodiment can obtain the absolute position features of all targets.
[0079] Step S25: The maximum value, minimum value, average value of the relative change rate between each pair of targets between each two consecutive detection frames, as well as the output result of the relative change rate after passing through the multilayer perceptron, are all concatenated to obtain the concatenated feature of the relative change rate between each pair of targets between each two consecutive detection frames.
[0080] In this embodiment, before feature stitching is performed on the maximum value, minimum value, average value, and output result of the relative change rate between each pair of targets between each consecutive two detection frames, it is necessary to determine the relative change rate between each pair of targets between each consecutive two detection frames based on the center point coordinates of the detection boxes containing the two targets in each detection frame and the size information of the detection boxes. The maximum value, minimum value, and average value of the relative change rate are determined from the relative change rate between each pair of targets between each consecutive two detection frames, and the relative change rate between each pair of targets is calculated based on the multilayer perceptron to obtain the output result of the relative change rate between each pair of targets between each consecutive two detection frames.
[0081] Specifically, the first relative rate of change is The second equivalent rate of change is The third relative change rate is The fourth relative rate of change is In this embodiment, the dx values of the k-th and (k-1)-th detections are used. ij dy ij , Enter them separately Figure 4 In the location feature network, the location feature network calculates the maximum value, minimum value, and average value of the relative change rate of two targets, as well as the output result of the relative change rate after passing through the multilayer perceptron. It should be noted that the relative change rate includes a first relative change rate, a second relative change rate, a third relative change rate, and a fourth relative change rate; the first relative change rate and the second relative change rate are the relative change rates of the center point coordinates of the detection box, and the third relative change rate and the fourth relative change rate are the relative change rates of the width and height of the detection box; specifically, the first relative change rate is the relative change rate of the center point coordinate x of the detection box, the second relative change rate is the relative change rate of the center point coordinate y of the detection box, the third relative change rate is the relative change rate of the width w of the detection box, and the fourth relative change rate is the relative change rate of the height h of the detection box.
[0082] In this embodiment, the maximum value, minimum value, average value, and output result of the first relative change velocity of the two targets between the k-th and (k-1)-th detection frames are concatenated to obtain the concatenated feature of the first relative change velocity of the two targets between the k-th and (k-1)-th detection frames, which has a shape of (1×4). Similarly, the concatenated features of the second, third, and fourth relative change velocities are determined.
[0083] Step S26: Perform feature concatenation on the concatenated features of the first relative change rate, the second relative change rate, the third relative change rate, and the fourth relative change rate between each pair of targets between every two consecutive detection frames to obtain the position features between each pair of targets between every two consecutive detection frames. Calculate the position features based on the multilayer perceptron to obtain the relative position features between each pair of targets between every two consecutive detection frames.
[0084] In this embodiment, see Figure 6As shown, the stitched features of the first relative change velocity, the stitched features of the second relative change velocity, the stitched features of the third relative change velocity, and the stitched features of the fourth relative change velocity are stitched together to obtain the position features between the two targets. The position features are then calculated based on the multilayer perceptron to obtain the relative position features between the two targets. In this way, the k-th detection and the (k-1)-th detection frames are applied to all detection frames, and this embodiment can obtain the relative position features between every two targets.
[0085] Step S27: Perform attention operations on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and classify the spatiotemporal behavior features to obtain the corresponding behavior detection results.
[0086] For more detailed working processes of steps S21, S22, and S27, please refer to the aforementioned disclosed embodiments, which will not be elaborated here.
[0087] As can be seen, in calculating the absolute position features of the target and the relative position features of every two targets, this application calculates the correlation between targets based on the target's speed, detection box size, and changes in target position. It then generates contextual behavioral features of the targets through an attention mechanism, i.e., the spatiotemporal behavioral features of all targets. Furthermore, by classifying these spatiotemporal behavioral features, the corresponding behavior detection results are obtained. This application enables the model to utilize the dependencies between target contexts, improving the model's accuracy for complex actions. In addition, this application combines the absolute position information of the target with behavioral features to distinguish the features of the same target at different positions, replacing the positional encoding that needs to be learned in traditional attention networks. Compared to traditional methods that rely on large amounts of sample data for detection, this application does not require a large number of samples or consider more complex relationships, thus reducing sample dependence and computational complexity.
[0088] Accordingly, embodiments of this application also disclose a behavior detection device based on spatiotemporal context, see [link to relevant documentation]. Figure 7 As shown, the device includes:
[0089] The location information determination module 11 is used to detect the video sequence based on a preset frame interval to obtain the location information of all targets in the video sequence corresponding to several detection frames;
[0090] Action feature determination module 12 is used to extract features from the video sequence to obtain a first feature map corresponding to the video sequence, and to pool the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, the action features of all targets are determined by the position information of the intermediate detection frames of the video sequence and the second feature map. The intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames.
[0091] The position feature determination module 13 is used to determine the absolute position features of all targets and the relative position features between every two targets based on the position information of all targets;
[0092] The behavior detection module 14 is used to perform attention operations on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and to classify the spatiotemporal behavior features to obtain the corresponding behavior detection results.
[0093] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0094] As can be seen, this application proposes a behavior detection method based on spatiotemporal context, comprising: detecting a video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to a plurality of detection frames; extracting features from the video sequence to obtain a first feature map corresponding to the video sequence, and pooling the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence; then determining the action features of all targets through the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames; determining the absolute position features of all targets and the relative position features between every two targets based on the position information of all targets; performing attention operation on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and classifying the spatiotemporal behavior features to obtain the corresponding behavior detection results. Therefore, this application calculates the correlation between targets by using the positional change information of the targets, and generates the contextual behavioral features of the targets through an attention mechanism, that is, the spatiotemporal behavioral features of all targets. Furthermore, by classifying the spatiotemporal behavioral features, the corresponding behavior detection results are obtained. Compared with traditional methods that rely on a large amount of sample data for detection, this application does not require a large number of samples or need to consider more complex relationships, thus reducing sample dependence and computational complexity.
[0095] Furthermore, embodiments of this application also provide an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0096] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the spatiotemporal context-based behavior detection method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.
[0097] In this embodiment, the power supply 26 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0098] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon may include computer programs 221, and the storage method may be temporary storage or permanent storage. The computer programs 221 may include, in addition to computer programs capable of performing the spatiotemporal context-based behavior detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, computer programs capable of performing other specific tasks.
[0099] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned behavior detection method based on spatiotemporal context.
[0100] For the specific steps of this method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0101] The various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to in the method section.
[0102] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0103] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0104] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0105] The above provides a detailed description of the behavior detection method, apparatus, device, and storage medium based on spatiotemporal context provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A behavior detection method based on spatiotemporal context, characterized in that, include: The video sequence is detected based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames; Feature extraction is performed on the video sequence to obtain a first feature map corresponding to the video sequence. The first feature map is then pooled based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, the action features of all targets are determined by the position information of the intermediate detection frames of the video sequence and the second feature map. The intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames. Based on the location information of all the targets, determine the absolute location features of all the targets and the relative location features between every two targets; Attention operations are performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets. The spatiotemporal behavior features are then classified to obtain the corresponding behavior detection results. Wherein, the absolute position feature represents the target's movement information within T frames, and the relative position feature represents the relative changes in position and size of two different targets within T frames, where T is the total number of frames in the video sequence.
2. The behavior detection method based on spatiotemporal context according to claim 1, characterized in that, The method of detecting the video sequence based on a preset frame interval to obtain the position information of all targets in the video sequence corresponding to several detection frames includes: The video sequence is detected based on a preset frame interval to obtain the center point coordinates of all target detection boxes in the video sequence corresponding to several detection frames, as well as the size information of the detection boxes; wherein, the size information is the width and height information of the detection boxes.
3. The behavior detection method based on spatiotemporal context according to claim 2, characterized in that, Determining the absolute positional features of all targets based on their positional information includes: The maximum, minimum, and average values of the change rate of each target between every two consecutive detection frames, as well as the output of the change rate after passing through the multilayer perceptron, are all concatenated to obtain the concatenated feature of the change rate of each target between every two consecutive detection frames. The change rate includes a first change rate, a second change rate, a third change rate, and a fourth change rate. The first and second change rates represent the change rates of the center point coordinates of the detection frame, while the third and fourth change rates represent the change rates of the width and height of the detection frame. For each consecutive two detection frames, the concatenated features of the first change rate, the concatenated features of the second change rate, the concatenated features of the third change rate, and the concatenated features of the fourth change rate of each target are concatenated to obtain the absolute position features of all targets.
4. The behavior detection method based on spatiotemporal context according to claim 3, characterized in that, Before determining the absolute position features of all targets based on their position information, the method further includes: Based on the center point coordinates of the detection box containing each target in each detection frame and the size information of the detection box, the rate of change of each target between every two consecutive detection frames is determined. The maximum value, minimum value, and average value of the change rate of each target between each consecutive two detection frames are determined from the change rate of each target between each consecutive two detection frames. The change rate of each target is calculated based on the multilayer perceptron to obtain the output result of the change rate of each target between each consecutive two detection frames through the multilayer perceptron.
5. The behavior detection method based on spatiotemporal context according to claim 2, characterized in that, Determining the relative positional characteristics between any two targets based on the positional information of all targets includes: The maximum value, minimum value, average value, and output result of the relative change rate between each pair of targets between every two consecutive detection frames are all concatenated to obtain the concatenated feature of the relative change rate between each pair of targets between every two consecutive detection frames. The relative change rate includes a first relative change rate, a second relative change rate, a third relative change rate, and a fourth relative change rate. The first and second relative change rates are the relative change rates of the center point coordinates of the detection frame, and the third and fourth relative change rates are the relative change rates of the width and height of the detection frame. The concatenated features of the first relative change rate, the second relative change rate, the third relative change rate, and the fourth relative change rate between each pair of targets between every two consecutive detection frames are concatenated to obtain the position features between each pair of targets between every two consecutive detection frames. The position features are then calculated based on the multilayer perceptron to obtain the relative position features between each pair of targets between every two consecutive detection frames.
6. The behavior detection method based on spatiotemporal context according to claim 5, characterized in that, Before determining the relative positional features between any two targets based on the positional information of all the targets, the method further includes: Based on the coordinates of the center point of the detection box containing each target in each detection frame and the size information of the detection box, the relative change rate between each pair of targets between each consecutive two detection frames is determined. The maximum value, minimum value, and average value of the relative change rate are determined from the relative change rate between each pair of targets between each two consecutive detection frames. Based on the multilayer perceptron, the relative change rate between each pair of targets is calculated to obtain the output result of the multilayer perceptron of the relative change rate between each pair of targets between each two consecutive detection frames.
7. The behavior detection method based on spatiotemporal context according to any one of claims 1 to 6, characterized in that, The attention operation is performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavioral features of all targets, including: Based on a preset attention calculation formula, attention operations are performed on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavioral features of all targets. The preset attention calculation formula is as follows: ； ； ； in, This represents the action characteristics of the target. This represents the absolute positional characteristics of the target. Indicate target With the goal The relative positional characteristics between them Representing dimension, This represents the spatiotemporal behavioral characteristics of the target.
8. A behavior detection device based on spatiotemporal context, characterized in that, include: The location information determination module is used to detect the video sequence based on a preset frame interval to obtain the location information of all targets in the video sequence corresponding to several detection frames; An action feature determination module is used to extract features from the video sequence to obtain a first feature map corresponding to the video sequence, and to pool the first feature map based on the time dimension to obtain a second feature map corresponding to the video sequence. Then, the action features of all targets are determined by the position information of the intermediate detection frames of the video sequence and the second feature map; wherein, the intermediate detection frame is the detection frame located in the middle position among the plurality of detection frames. The location feature determination module is used to determine the absolute location features of all targets and the relative location features between any two targets based on the location information of all targets. The behavior detection module is used to perform attention operations on the action features of all targets, the absolute position features of all targets, and the relative position features between every two targets to obtain the spatiotemporal behavior features of all targets, and to classify the spatiotemporal behavior features to obtain the corresponding behavior detection results. Wherein, the absolute position feature represents the target's movement information within T frames, and the relative position feature represents the relative changes in position and size of two different targets within T frames, where T is the total number of frames in the video sequence.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the behavior detection method based on spatiotemporal context as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the behavior detection method based on spatiotemporal context as described in any one of claims 1 to 7.