Video behavior detection method and device, equipment and storage medium

Abstract

The application discloses a video behavior detection method and device, equipment and a storage medium. The application obtains video features of a current video segment, obtains character information in the current video segment based on the video features, and obtains multi-scale context information representing different background information in the current video segment. The application obtains a current bidirectional interaction relationship feature between the character and the multi-scale context information based on the character information and the multi-scale context information. The current bidirectional interaction relationship feature fully considers the interaction relationship between the character and the context information in the background in complex interaction, and thus can accurately predict the behavior category of the character in the current video segment based on the current bidirectional interaction relationship feature. The current bidirectional interaction relationship feature obtained in the application can perceive the context information associated with the character through the character information, and can perceive the character information associated with the context information through the context information, thereby providing more abundant data support for behavior detection and further improving the detection accuracy.

Description

Technical Field

[0001] This application relates to the field of video detection technology, and more specifically, to a video behavior detection method, apparatus, device, and storage medium. Background Technology

[0002] Video action detection requires locating each individual in a video and classifying their actions. Current video action detection methods typically extract temporal change information about individuals from images preceding and following keyframes in the video, and then predict the individual's behavior category based on this temporal change information.

[0003] However, for videos involving complex interactions, characters may interact intricately with other information in the background. These complex interactions can affect the detection of character behavior categories. Existing methods that rely solely on temporal changes in character behavior are ill-suited for detecting complex interactions. For example, in analyzing classroom or exam behavior, existing methods struggle to detect instances of students talking to classmates or passing notes during exams. Therefore, a new video behavior detection method is urgently needed to improve the accuracy of character behavior category detection in videos. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a video behavior detection method, apparatus, device, and storage medium to improve the accuracy of detecting human behavior categories in videos. The specific solution is as follows:

[0005] Firstly, a video behavior detection method is provided, including:

[0006] Obtain the video features of the current video segment;

[0007] Based on the video features, obtain the information of the people in the current video segment, as well as the multi-scale context information representing different background information in the current video segment;

[0008] Based on the character information and the multi-scale context information, obtain the current bidirectional interaction relationship features between the character and the multi-scale context information in the current video segment;

[0009] Based on the current two-way interaction relationship characteristics, predict the behavior category of the person in the current video segment.

[0010] Secondly, a video behavior detection device is provided, comprising:

[0011] The video feature extraction module is used to obtain the video features of the current video segment;

[0012] The person information acquisition module is used to acquire person information in the current video segment based on the video features;

[0013] The multi-scale context information extraction module, based on the video features, obtains multi-scale context information representing different background information in the current video segment;

[0014] The first two-way interaction relationship extraction module is used to obtain the current two-way interaction relationship features between the person and the multi-scale context information in the current video segment based on the person information and the multi-scale context information.

[0015] The behavior category prediction module is used to predict the behavior category of the person in the current video segment based on the current two-way interaction relationship features.

[0016] Thirdly, a video behavior detection device is provided, including: a memory and a processor;

[0017] The memory is used to store programs;

[0018] The processor is used to execute the program to implement the various steps of the video behavior detection method as described above.

[0019] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the video behavior detection method as described above.

[0020] By employing the above technical solution, this application obtains video features of the current video segment, and based on these video features, obtains information about the people in the current video segment, as well as multi-scale contextual information representing different background information in the current video segment, such as information about people, objects, and scenes at different scales in the background. Then, based on the obtained information about people and multi-scale contextual information, it obtains the current two-way interaction relationship features between people and multi-scale contextual information in the current video segment. These current two-way interaction relationship features fully consider the interaction relationship between people and contextual information in the background during complex interactions. Therefore, based on the current two-way interaction relationship features, it is possible to accurately predict the behavior category of people in the current video segment. In particular, for complex interaction scenarios, the solution of this application can improve the detection accuracy of the behavior category of people in the video.

[0021] Furthermore, this application obtains the current bidirectional interaction relationship features between the person and the multi-scale context information in the current video segment based on person information and multi-scale context information. These bidirectional interaction relationship features can perceive the context information associated with the person through the person information, and vice versa. This provides richer data support for determining the person's behavior category, thereby further improving the detection accuracy of the person's behavior category. For example, using the solution of this application, information about the person associated with an object can be perceived through that object; for instance, a basketball can be used to find the person who threw the basketball. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0023] Figure 1 This is a flowchart illustrating a video behavior detection method provided in an embodiment of this application.

[0024] Figure 2 A schematic diagram illustrating the extraction of position codes using a position coding network is provided.

[0025] Figure 3 This example illustrates a schematic diagram of a video behavior detection network structure.

[0026] Figure 4 This example illustrates padding an input feature sequence and adding mask values ​​to attention weights.

[0027] Figure 5 This example illustrates the correspondence between objects and video segments in the input feature sequence of a second bidirectional interaction relationship extraction module.

[0028] Figure 6 This is a schematic diagram of the structure of a video behavior detection device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of the video behavior detection device provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] This application provides a video behavior detection solution that can be applied to scenarios where the behavior categories of people in a video are detected, such as classroom scenes, examination room scenes, etc., to determine whether people have performed actions of a specified category, such as students talking to their classmates or passing notes during an exam.

[0032] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, or cloud platform.

[0033] Next, combined Figure 1 The video behavior detection method of this application may include the following steps:

[0034] Step S100: Obtain the video features of the current video segment.

[0035] Specifically, the current video segment can be a video clip extracted from the original video, and its duration can be adjusted. For example, it could be a 2-second video segment centered on a keyframe image. The current video segment is then uniformly sampled to obtain the image sequence input_frames.

[0036] Furthermore, a video feature extraction network (Backbone Network) can be used to process the image sequence to extract spatially and temporally aware video feature tensors X:

[0037] X∈R C×T×H×W =Backbone Network(Input_frames)

[0038] The dimensions of the video features are C×T×H×W, representing channel, time, height, and width, respectively.

[0039] Optionally, to reduce the size of video features and thus decrease the computational cost and parameter count of the network, this embodiment can reduce the number of channels in the video features and perform average pooling in the time dimension to generate simplified video features X. 1 ∈R C / 2×2×H×W Subsequently, in order to preserve information in the temporal dimension, the temporal dimension can be compressed into the channel dimension to generate the final video feature X. 2 ∈R C×H×W .

[0040] Step S110: Based on the video features, obtain the character information in the current video segment, as well as the multi-scale context information representing different background information in the current video segment.

[0041] The character information can include character features, which may include motion information and appearance information. Furthermore, considering that different characters may be located at different positions in the video, character position encoding can be added to the character information to represent the character's position within the video. In this step, the video features can be processed to obtain the character information for the current video segment.

[0042] Furthermore, considering that the current video segment also contains background information, the size of different background information may be different, and there may be an interactive relationship between the background information and the person, which can greatly help in judging the behavior category of the person. Therefore, in this step, multi-scale contextual information representing different background information in the current video segment can be extracted based on video features. This multi-scale contextual information contains background information of different scales, such as background objects, background people, background scenes, etc.

[0043] Considering that different background information may have different sizes, this step uses a multi-scale feature extraction method, which can extract context information at different scales and ensure the completeness of context information extraction.

[0044] Step S120: Based on the character information and the multi-scale context information, obtain the current bidirectional interaction relationship features between the character and the multi-scale context information in the current video segment.

[0045] Specifically, after obtaining the character information and multi-scale context information, we can model the two-way interaction relationship between different objects, that is, model the current two-way interaction relationship features between the character and the multi-scale context information in the current video segment, and provide sufficient supporting information for the judgment of the character's behavior category.

[0046] Step S130: Based on the current two-way interaction relationship features, predict the behavior category of the person in the current video segment.

[0047] Specifically, after obtaining the current two-way interaction relationship features in the aforementioned steps, the behavior category of the person in the current video segment can be predicted based on these features. Since the current two-way interaction relationship features provide sufficient supporting information for the judgment of the behavior category of the person, the behavior category of the person in the current video segment can be predicted more accurately. In particular, for videos containing complex interactive behaviors, the solution of this application can obtain the current two-way interaction relationship features between the person and multi-scale contextual information, thereby accurately predicting the behavior category of the person.

[0048] The video behavior detection method provided in this application obtains video features of the current video segment, and obtains character information and multi-scale context information representing different background information in the current video segment based on the video features. Examples include character information, object information, and scene information at different scales in the background. Then, based on the obtained character information and multi-scale context information, the method obtains the current two-way interaction relationship features between the character and the multi-scale context information in the current video segment. The current two-way interaction relationship features fully consider the interaction relationship between the character and the context information in the background in complex interactions. Therefore, the method can accurately predict the behavior category of the character in the current video segment based on the current two-way interaction relationship features. In particular, for complex interaction scenarios, the method of this application can improve the detection accuracy of character behavior categories in the video.

[0049] Furthermore, this application obtains the current bidirectional interaction relationship features between the person and the multi-scale context information in the current video segment based on person information and multi-scale context information. These bidirectional interaction relationship features can perceive the context information associated with the person through the person information, and vice versa. This provides richer data support for determining the person's behavior category, thereby further improving the detection accuracy of the person's behavior category. For example, using the solution of this application, information about the person associated with an object can be perceived through that object; for instance, a basketball can be used to find the person who threw the basketball.

[0050] In some embodiments of this application, an optional implementation of step S130 above, which predicts the behavior category of a person in the current video segment based on the current two-way interaction relationship features, is provided.

[0051] Considering that the temporal changes of a person can affect the detection of their behavior, this embodiment can simultaneously consider the current two-way interaction features and the two-way interaction features between the person and the multi-scale contextual information in the neighboring video segments before and / or after the current video segment in the original video when predicting the behavior category. This introduces the two-way interaction features of the context of the current video segment, thereby improving the accuracy of the person's behavior detection.

[0052] Specific methods may include:

[0053] Based on the current two-way interaction relationship features and the nearest neighbor two-way interaction relationship features, target two-way interaction relationship features between people and multi-scale contextual information in long-term video segments are obtained.

[0054] The nearest-neighbor two-way interaction relationship feature refers to the two-way interaction relationship feature between characters and multi-scale contextual information in the nearest video segments located before and / or after the current video segment in the original video. This application can pre-determine the nearest-neighbor two-way interaction relationship feature for the nearest video segments according to the aforementioned steps S100-S120, and store it in the database.

[0055] Based at least on the target two-way interaction relationship features, predict the behavior category of the person in the current video segment.

[0056] In this embodiment, the current two-way interaction relationship features of the current video segment and the neighboring two-way interaction relationship features of the neighboring video segments are considered simultaneously to obtain the target two-way interaction relationship features between the person and multi-scale contextual information in the long video segment. Based on this, the behavior category of the person in the current video segment can be predicted more accurately.

[0057] Taking the act of a waiter serving tea to a customer as an example, the waiter's current actions, the interaction between the waiter and the teacup and the customer, the interaction between the waiter and objects such as the teacup in nearby video segments before or after the current moment, and the waiter's movement towards the customer are all crucial for judging the waiter's behavior. Therefore, when predicting behavior categories based on the aforementioned two-way interaction features, we can further consider the interaction between the information of people in the current video segment and the information of people in nearby video segments to obtain the information of people in long-term video clips.

[0058] Based on this, the behavior category of the person in the current video segment can be predicted based on the characteristics of the two-way interaction relationship of the target and the information of the person in the long segment.

[0059] The solution provided in this embodiment takes into account the bidirectional interaction characteristics between people and multi-scale contextual information over a long period of time, as well as the information about people in the long period of time, which can more accurately predict the behavior category of people.

[0060] In some embodiments of this application, the process of obtaining the person information in the current video segment based on the video features in the aforementioned step S110 is described.

[0061] Person information can include person characteristics and person location code, with the person location code indicating the person's position information in the video frame.

[0062] The process of obtaining information about people in the current video segment can include the following steps:

[0063] S1. Generate a location code corresponding to the video features.

[0064] Specifically, in this embodiment, a positional coding network, PosNet, can be pre-constructed. For example, a lightweight positional coding network with a receptive field covering the input video frame size can be constructed by stacking multiple extremely narrow (e.g., 2 channels or other values) convolutional layers, thereby reducing the computational cost of the positional coding network. The convolutional layers in the positional coding network can be set to a small size, such as 3×3, to reduce the number of convolutional parameters.

[0065] It is understandable that the receptive field of a network (the range of input tensors (or images) that the last convolutional layer can perceive) increases with the network depth. The more convolutional layers in a series, i.e. the deeper the network, the larger the range that the last convolutional layer can perceive. Therefore, by controlling the number of convolutional layers in a positional coding network, adaptive positional coding at any scale can be obtained.

[0066] like Figure 2 As shown, the position encoding network contains multiple concatenated convolutional layers (Dilated Blocks). After the last convolutional layer, a Conv1×1 output convolution result is passed, which converts the position encoding extracted by the position encoding network into a fixed-channel position encoding.

[0067] In the process of generating the positional encoding corresponding to the video features, a tensor Input_pos∈R with the same scale as the video features (i.e., the same height H and width W as the video features) can first be randomly initialized. 1×1×H×W The data is then input into a location encoding network to generate location codes Pos corresponding to the video features. The processing steps are as follows:

[0068] Pos∈R 1×C×H×W =PosNet(Input_pos)

[0069] The code for each position in the output location code can represent the location information of the corresponding position in the video features.

[0070] S2. The human detection module detects the target bounding box of the person in the key frame image of the current video segment.

[0071] S3. Extract the character features corresponding to the target box position of the character from the video features.

[0072] Specifically, video features are the features corresponding to the current input video segment, containing all feature information from each video frame. Since the bounding box of the person on the keyframe image can be obtained in the previous step, the video features corresponding to the location of this bounding box can be extracted from the video features as the person's features.

[0073] S4. Extract the person's position code corresponding to the target box position of the person from the position code corresponding to the video feature, and combine the person's feature and the person's position code to form person information.

[0074] Specifically, the bounding box containing the person was obtained in the aforementioned steps. Therefore, the person's positional code pos_person, corresponding to the bounding box position, can be extracted from the global positional encoding corresponding to the video features. Finally, the extracted person features and the person's positional code constitute the person information.

[0075] Reference Figure 3 The example illustrates a schematic diagram of a video behavior detection network structure.

[0076] The input video segment is processed by the Backbone Network to obtain video features X. 2 .

[0077] At the same time, a human detection module can be used to extract the target box (Box) containing the person.

[0078] The video behavior detection network may also include an Interactive Relationship Extraction Module and a Neighbour Feature Bank and Interaction Module.

[0079] The interaction relationship extraction module may include: a location encoding network PosNet, a multi-scale context information extraction module Context Module, an interest pooling module ROI Align, and a first bidirectional high-order interaction relationship extraction module (which may also be referred to as the first bidirectional interaction relationship extraction module in this paper).

[0080] The nearest neighbor feature storage and interaction module may include: a first feature library, a second bidirectional interaction relationship extraction module, and a third bidirectional interaction relationship extraction module. The first, second, and third bidirectional interaction relationship extraction modules have the same structure.

[0081] Location coding network generation and video feature X 2 The corresponding position code Pos.

[0082] Interest pooling module in video feature X 2The human feature A is extracted from the video. Furthermore, based on the bounding box (Box) of the human detection module, the human position feature (pos_person) is extracted from the position code (pos) corresponding to the video features. The human information is composed of human feature A and the human position code (pos_person).

[0083] Specifically, the interest pooling module can extract the corresponding person position code pos_person based on the person's target bounding box (Box) using mean pooling, as follows:

[0084] Pos_person∈R C =MeanPooling(RoiAlign(Pos,Box))

[0085] The multi-scale context information extraction module can extract information based on video features X. 2 This allows us to obtain multi-scale contextual information representing different background information in the current video segment.

[0086] Specifically, video features X are processed through a multi-scale context information extraction module. 2 The multi-scale context features and their corresponding position codes pos are obtained by processing the context features and their corresponding position codes pos_context. The multi-scale context information is composed of the context features and their corresponding position codes pos_context.

[0087] The multi-scale context information extraction module can include several parallel adaptive pooling layers or convolutional layers of different sizes. By using these parallel adaptive pooling layers or convolutional layers of different sizes, context information at different scales can be extracted from video features, thus obtaining multi-scale context information. The specific processing procedure is as follows:

[0088] Context∈R C Pos_context∈R C =ContextModule(X 2 Pos)

[0089] Further optionally, considering that context features at different scales may have the same location and location encoding, in order to distinguish context features at different scales, this embodiment can add corresponding scale encoding to context features at different scales to indicate the scale to which the context feature belongs.

[0090] Specifically, the scale code corresponding to each context feature can be obtained, and multi-scale context information is composed of the context feature, the corresponding position code, and the scale code.

[0091] The first bidirectional interaction relationship extraction module can extract the current bidirectional interaction relationship features between the characters and multi-scale context information in the current video segment based on the character information obtained by the interest pooling module and the multi-scale context information obtained by the multi-scale context information extraction module. Specifically:

[0092] Character information and multi-scale contextual information can be combined into an input feature sequence I:

[0093]

[0094] Where N represents the number of people and M represents the number of multi-scale contextual features.

[0095] The input feature sequence is calculated through the first bidirectional interaction relationship extraction module. The attention weights of each feature in the input feature sequence are calculated, and the features in the input feature sequence are weighted and summed according to the attention weights to obtain the current bidirectional interaction relationship features between the object corresponding to each feature in the input feature sequence and the other objects corresponding to the other features.

[0096] The objects corresponding to each feature in the input feature sequence include people in the current video segment and multi-scale context information.

[0097] In this embodiment, the first bidirectional interaction relationship extraction module may include several stacked multi-head self-attention (MHSA) layers to model the bidirectional high-order interaction relationship between the person and the multi-scale contextual information in the background.

[0098] Input feature sequence Q is generated using 1×1 convolution. i ,K i V i Then, the attention weights between each feature in the input feature sequence and each feature in the input feature sequence are calculated. Current bidirectional interactive relationship characteristics H i via Attn i,j For V j The linear weighting is obtained. The calculation process is as follows:

[0099] Q i ,K i V i =Conv2d(I i )

[0100]

[0101]

[0102] Where C represents the number of channels of the input feature.

[0103] Furthermore, to enable the method of this application to perform parallel behavior detection on different videos, and considering that the number of characters in different videos may vary, the length of the feature sequence I input to the first bidirectional interaction relationship extraction module for each video in the same batch during parallel detection may be inconsistent. Therefore, this embodiment can expand the input feature sequence I to a uniform length using padding. However, the expanded portion has no contribution to the original portion of the input sequence; therefore, this embodiment can eliminate the interference of the expanded portion on the original portion of the input sequence by adding a mask. The specific implementation is as follows:

[0104] When combining character information and multi-scale context information into an input feature sequence, the character information and multi-scale context information can be concatenated and then padded to an input feature sequence of a set uniform length. In this input feature sequence I, each object is a tensor with a scale of 1xC; therefore, the padding elements can also be 1xC all-zero tensors.

[0105] Reference Figure 4 As shown, it illustrates the input feature sequences extracted from multiple (3) different videos. Figure 4 The uniform length of the input feature sequence is set to 5. Each line represents the input feature sequence extracted from a video. Figure 4 In the diagram, blank boxes represent the original features extracted from the video, and black boxes represent the padding element features.

[0106] As can be seen, the length of the input feature sequence extracted in the first row is exactly 5, so no padding is needed. The length of the input feature sequence extracted from the video corresponding to the second row is 3, so 2 padding elements are needed. The length of the input feature sequence extracted from the video corresponding to the third row is 2, so 3 padding elements are needed.

[0107] After calculating the attention weight between each feature in the input feature sequence and all other features in the input feature sequence, and before performing a weighted summation of the features in the input feature sequence according to the attention weights, the following processing steps can be added:

[0108] Add a corresponding mask value to each feature in the input feature sequence and the attention weight of each feature in the input feature sequence to obtain the masked attention weight.

[0109] In this context, the mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value -inf.

[0110] Reference Figure 4 Taking the second row of input feature sequences as an example:

[0111] The first, second, and third features are the original features, while the fourth and fifth features are padding element features. Taking the calculation of the attention weight between the first feature and the entire input feature sequence as an example, Attn11, Attn12, Attn13, Attn14, and Attn15 are calculated. To eliminate the influence of the fourth and fifth padding element features on the first feature, Attn14 and Attn15 should be set to infinitesimals, while the values ​​of Attn11, Attn12, and Attn13 remain unchanged. Therefore, the corresponding mask is [0 0 0 – inf – inf].

[0112] For each row of input feature sequence, the corresponding mask result is as follows: Figure 4 As shown, the mask result in the i-th row represents the attention weight between the i-th feature in the corresponding input feature sequence and each feature in the input feature sequence.

[0113] After adopting the solution provided in this embodiment, the formula for calculating attention weights in the above formula should be replaced with the following formula:

[0114]

[0115] Comparing the formula before and after the replacement, we can see that the formula after the replacement adds a mask term, which is used to remove the interference of the expanded part in the input sequence on the original part.

[0116] The second two-way interaction relationship extraction module can obtain target two-way interaction relationship features between people and multi-scale contextual information in long-term video segments based on the current two-way interaction relationship features extracted by the first two-way interaction relationship extraction module and the nearest neighbor two-way interaction relationship features stored in the first feature library. Specifically:

[0117] To support the modeling of interactions between characters and multi-scale contextual information in longer video clips, this application can establish a first feature library to store the bidirectional interaction features extracted from each video clip. For example... Figure 3 As shown, the current two-way interaction relationship features extracted from the current video segment can also be updated and stored in the first feature library.

[0118] Furthermore, the nearest neighbor bidirectional interaction relationship features can be extracted from the first feature library for video segments that are closest to the current video segment. These video segments that are closest to the current video segment can be several video segments preceding and / or following the current video segment.

[0119] Considering the different temporal positions of different video segments in the original video, we can further obtain the temporal position encoding of the current video segment in the original video, as well as the temporal position encoding of the nearest neighbor video segments in the original video. Then, we add the current two-way interaction relationship features and the temporal position encoding of the current video segment in the original video to obtain the current two-way interaction relationship information; similarly, we add the nearest neighbor two-way interaction relationship features and the temporal position encoding of the nearest neighbor video segments in the original video to obtain the nearest neighbor two-way interaction relationship information. We combine the current two-way interaction relationship information and the nearest neighbor two-way interaction relationship information into an input feature sequence, which serves as the input to the second two-way interaction relationship extraction module. The input feature sequence can be expressed as the following formula:

[0120]

[0121] Where H represents the current bidirectional interaction relationship feature extracted from the current video segment, w represents the position number of the current video segment in the original video, and tempoaral_pos w The current video segment is represented by the time position code, N represents the number of current two-way interaction relationship features extracted from the current video segment, F represents the neighboring two-way interaction relationship features extracted from the neighboring video segments. In this embodiment, w neighboring video segments are obtained forward and backward respectively with the current video segment as the center, so the number of neighboring video segments is 2w, and M represents the number of neighboring two-way interaction relationship features extracted from the neighboring video segments.

[0122] Reference Figure 5 This example illustrates the correspondence between each object in the input feature sequence L and the video segment.

[0123] The second bidirectional interaction relationship extraction module can calculate the attention weight of each feature in the input feature sequence and the attention weight of each feature in the input feature sequence, and perform weighted summation of each feature in the input feature sequence according to the attention weight to obtain the target bidirectional interaction relationship features between the object corresponding to each feature in the input feature sequence and the other objects corresponding to the other features.

[0124] The objects corresponding to each feature in the input feature sequence include people in a long video segment composed of the current video segment and neighboring video segments, as well as multi-scale contextual information.

[0125] In this embodiment, the second bidirectional interaction relationship extraction module may include several stacked multi-head self-attention (MHSA) layers to model the bidirectional high-order interaction relationship between people and multi-scale contextual information in the background in a long video clip.

[0126] The input feature sequence L can be generated by 1×1 convolution to produce Q. i ,K i V i Then, the attention weight Attn between each feature in the input feature sequence and each feature in the input feature sequence is calculated. i,j Target two-way interactive relationship feature O i via Attn i,j For V j The linear weighting is obtained. The calculation process is as follows:

[0127] Q i ,K i V i =conv2d(L)

[0128]

[0129]

[0130] Furthermore, to enable the method of this application to perform parallel behavior detection on different videos, the number of bidirectional interaction features in different videos may not be the same, resulting in inconsistent lengths of the feature sequence L input to the second bidirectional interaction extraction module. Therefore, this embodiment can expand the input feature sequence L to a uniform length using padding. However, the expanded portion has no contribution to the original part of the input sequence; therefore, this embodiment can remove the interference of the expanded portion on the original part of the input sequence by adding a mask. The specific implementation is as follows:

[0131] When combining the current two-way interaction relationship information with the nearest two-way interaction relationship information into an input feature sequence, the current two-way interaction relationship information and the nearest two-way interaction relationship information can be concatenated and expanded into an input feature sequence of a set uniform length through padding.

[0132] The padding method can be referenced. Figure 4 As shown.

[0133] After calculating the attention weight between each feature in the input feature sequence and all other features in the input feature sequence, and before performing a weighted summation of the features in the input feature sequence according to the attention weights, the following processing steps can be added:

[0134] Add a corresponding mask value to each feature in the input feature sequence and the attention weight of each feature in the input feature sequence to obtain the masked attention weight.

[0135] In this context, the mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value -inf.

[0136] The process of adding mask values ​​to the attention weights in this embodiment can be referred to in the previous introduction, as the principle is the same, and will not be repeated here.

[0137] After adopting the solution provided in this embodiment, the formula for calculating attention weights in the above formula should be replaced with the following formula:

[0138]

[0139] Comparing the formula before and after the replacement, we can see that the formula after the replacement adds a mask term, which is used to remove the interference of the expanded part in the input sequence on the original part.

[0140] The third two-way interaction relationship extraction module can obtain person information in long video segments based on the person information extracted by the interest pooling module in the current video segment and the person information in neighboring video segments stored in the second feature library. Specifically:

[0141] To support the modeling of interactive relationships between characters in longer video clips, this application can establish a second feature library to store character information extracted from each video clip (including character appearance information and character motion information). For example... Figure 3 As shown, the information about the people extracted from the current video segment can also be updated and stored in the second feature library.

[0142] Furthermore, the person information extracted from the video segments adjacent to the current video segment can be obtained from the second feature library. These adjacent video segments can be several video segments before and / or after the current video segment.

[0143] The information of people in the current video segment and the information of people in neighboring video segments are combined into an input feature sequence, which is then used as the input to the third bidirectional interaction relationship extraction module.

[0144] The third two-way interactive relationship extraction module calculates the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, and then weights and sums the features in the input feature sequence according to the attention weight to obtain the information of the people in the long video segment composed of the current video segment and the nearest video segments.

[0145] In this embodiment, the third bidirectional interaction relationship extraction module may include several stacked multi-head self-attention (MHSA) layers to model the bidirectional high-order interaction relationships between people in long video clips.

[0146] The specific calculation process of the third two-way interactive relationship extraction module can be referred to the calculation process of the first and second two-way interactive relationship extraction modules mentioned above. The principle is the same, so it will not be repeated here.

[0147] Furthermore, to enable the method of this application to perform parallel behavior detection on different videos, and considering that the number of characters in different videos may vary, resulting in inconsistent feature sequence lengths input to the third bidirectional interaction extraction module, this embodiment can expand the input feature sequence to a uniform length using padding. However, the expanded portion contributes ineffectively to the original part of the input sequence; therefore, this embodiment can eliminate the interference of the expanded portion on the original part by adding a mask. The specific implementation is as follows:

[0148] When combining the information of people in the current video segment and the information of people in the neighboring video segments into an input feature sequence, one can concatenate the information of people in the current video segment and the information of people in the neighboring video segments and expand it into an input feature sequence of a set uniform length through padding.

[0149] The padding method can be referenced. Figure 4 As shown.

[0150] After calculating the attention weight of each feature in the input feature sequence and before performing a weighted summation of the features in the input feature sequence according to the attention weights, the following processing steps can be added:

[0151] Add a corresponding mask value to each feature in the input feature sequence and the attention weight of each feature in the input feature sequence to obtain the masked attention weight.

[0152] In this context, the mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value -inf.

[0153] The process of adding mask values ​​to the attention weights in this embodiment can be referred to in the previous introduction, as the principle is the same, and will not be repeated here.

[0154] Combination Figure 3 As shown:

[0155] The second two-way interaction feature extraction module can obtain the target two-way interaction feature, and the third two-way interaction feature extraction module can obtain the information of people in the long video segment. Furthermore, based on the target two-way interaction feature and the information of people in the long video segment, the behavior category of people in the current video segment can be predicted, which can specifically include:

[0156] The characteristics of the two-way interaction between the target and the information of the people in the long video clip are fused to obtain the fused features.

[0157] Based on the fused features, the behavior category of the person in the current video segment is predicted.

[0158] Specifically, the fused features can be input into a fully connected classification layer to output behavior category labels.

[0159] In summary, the proposed solution establishes bidirectional interaction features between a person and other people, objects, and scenes in the background, capturing the multi-scale contextual information upon which complex behavior decisions depend, thus helping to more accurately predict the behavior category of a person. Furthermore, this application uses a multi-scale context extraction module to extract multi-scale contextual features representing information about people, objects, and scenes at different scales in the background. It does not rely on any object detector to extract object information from the background, exhibiting high flexibility and transferability, and can be easily transferred to different scenes.

[0160] Furthermore, in order to model the interaction information between a person and the context, as well as the person's own information (including appearance information and action information) in long video clips, this case proposes to use two parallel two-way interaction relationship extraction modules (i.e., the second and third two-way interaction relationship extraction modules) to model the interaction information between a person and the context and the person's own information, respectively.

[0161] In order to add positional encoding to each character or context in the first two-way interaction relationship extraction module, this application proposes a lightweight positional encoding network, which can be composed of multiple convolutional layers, to generate adaptive positional encoding at arbitrary scales, thus avoiding the performance degradation problem caused by interpolation in fixed-scale positional encoding.

[0162] The video behavior detection apparatus provided in the embodiments of this application is described below. The video behavior detection apparatus described below can be referred to in correspondence with the video behavior detection method described above.

[0163] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a video behavior detection device disclosed in an embodiment of this application.

[0164] like Figure 6 As shown, the device may include:

[0165] Video feature extraction module 11 is used to obtain video features of the current video segment;

[0166] Person information acquisition module 12 is used to acquire person information in the current video segment based on the video features;

[0167] The multi-scale context information extraction module 13, based on the video features, obtains multi-scale context information representing different background information in the current video segment;

[0168] The first two-way interaction relationship extraction module 14 is used to obtain the current two-way interaction relationship features between the person and the multi-scale context information in the current video segment based on the person information and the multi-scale context information.

[0169] The behavior category prediction module 15 is used to predict the behavior category of the person in the current video segment based on the current two-way interaction relationship features.

[0170] Optionally, the process by which the behavior category prediction module predicts the behavior category of a person in the current video segment based on the current two-way interaction relationship features may include:

[0171] Based on the current two-way interaction relationship features and the nearest two-way interaction relationship features, the target two-way interaction relationship features between people and multi-scale contextual information in long video segments are obtained; wherein, the nearest two-way interaction relationship features are the two-way interaction relationship features between people and multi-scale contextual information in the nearest video segments located before and / or after the current video segment in the original video.

[0172] Based at least on the target two-way interaction relationship features, predict the behavior category of the person in the current video segment.

[0173] Optionally, the behavior category prediction module may include: a second two-way interaction relationship extraction module and a prediction module, wherein:

[0174] The second two-way interaction relationship extraction module is used to obtain target two-way interaction relationship features between people and multi-scale contextual information in a long video segment based on the current two-way interaction relationship features and the nearest two-way interaction relationship features.

[0175] The prediction module is used to predict the behavior category of the person in the current video segment, based at least on the target two-way interaction relationship features.

[0176] Optionally, the apparatus of this application may further include:

[0177] The third two-way interaction relationship extraction module is used to obtain person information in a long video segment based on the person information in the current video segment and the person information in the neighboring video segments. Based on this, the behavior category prediction module, at least based on the target two-way interaction relationship features, predicts the behavior category of the person in the current video segment, which may include:

[0178] Based on the target two-way interaction relationship features and the character information in the long video segment, predict the behavior category of the character in the current video segment.

[0179] Optionally, the process by which the aforementioned person information acquisition module acquires person information in the current video segment based on the video features may include:

[0180] Generate a location code corresponding to the video features;

[0181] The human detection module detects the bounding boxes of people in keyframe images of the current video segment.

[0182] Extract the character features corresponding to the location of the target box where the character is located from the video features;

[0183] The person's location code corresponding to the target box position of the person is extracted from the location code corresponding to the video features, and the person's information is composed of the person's features and the person's location code.

[0184] The aforementioned character information acquisition module may include: a location encoding network, a human detection module, and an interest pooling module, wherein:

[0185] The location coding network is used to generate location codes corresponding to the video features;

[0186] The human detection module is used to detect the target bounding box of a person in the keyframe image of the current video segment;

[0187] The interest pooling module is used to extract the person features corresponding to the target box position of the person in the video features; and to extract the person position code corresponding to the target box position of the person in the position code corresponding to the video features, and the person information is composed of the person features and the person position code.

[0188] Optionally, the process by which the multi-scale context information extraction module obtains multi-scale context information representing different background information in the current video segment based on the video features may include:

[0189] The video features and their corresponding location codes are processed by the multi-scale context information extraction module to obtain the processed multi-scale context features and the location codes corresponding to each context feature. The multi-scale context information is composed of the context features and their corresponding location codes.

[0190] The multi-scale context information extraction module includes several parallel adaptive pooling layers or convolutional layers of different sizes.

[0191] Optionally, the process by which the multi-scale context information extraction module obtains multi-scale context information representing different background information in the current video segment based on the video features may further include:

[0192] Obtain the scale code corresponding to each of the context features, and the multi-scale context information is composed of the context features, the corresponding position codes, and the scale codes.

[0193] Optionally, the process by which the first two-way interaction relationship extraction module obtains the current two-way interaction relationship features between the person and the multi-scale context information in the current video segment based on the person information and the multi-scale context information may include:

[0194] The character information and the multi-scale context information are combined into an input feature sequence;

[0195] The first bidirectional interaction relationship extraction module calculates the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, and adds each feature in the input feature sequence according to the attention weight to obtain the current bidirectional interaction relationship feature between the object corresponding to each feature in the input feature sequence and the other objects corresponding to the other features.

[0196] The objects corresponding to each feature in the input feature sequence include people in the current video segment and multi-scale context information. The first bidirectional interaction relationship extraction module includes several stacked multi-layer multi-head self-attention layers.

[0197] Optionally, the process by which the first bidirectional interaction relationship extraction module combines the character information and the multi-scale context information into an input feature sequence may include:

[0198] The character information and the multi-scale context information are concatenated and padded to an input feature sequence of a set uniform length. Based on this, after calculating the attention weights of each feature in the input feature sequence and before weighted summation of the features according to the attention weights, the first bidirectional interaction relationship extraction module can also be used for:

[0199] Add a corresponding mask value to the attention weights of each feature in the input feature sequence and the other features in the input feature sequence to obtain the masked attention weights. The mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value.

[0200] Optionally, the process by which the second bidirectional interaction relationship extraction module obtains the target bidirectional interaction relationship features between a person and multi-scale contextual information in a long-term video segment based on the current bidirectional interaction relationship features and the nearest neighbor bidirectional interaction relationship features may include:

[0201] The current two-way interaction relationship feature and the temporal position encoding of the current video segment in the original video are added together to obtain the current two-way interaction relationship information; the nearest two-way interaction relationship feature and the temporal position encoding of the nearest video segment in the original video are added together to obtain the nearest two-way interaction relationship information.

[0202] The current two-way interaction relationship information and the nearest neighbor two-way interaction relationship information are combined into an input feature sequence;

[0203] The second bidirectional interaction relationship extraction module calculates the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, and adds each feature in the input feature sequence according to the attention weight to obtain the target bidirectional interaction relationship feature between the object corresponding to each feature in the input feature sequence and the other objects corresponding to the other features.

[0204] The objects corresponding to each feature in the input feature sequence include people in a long video segment composed of the current video segment and neighboring video segments, as well as multi-scale context information. The second bidirectional interaction relationship extraction module includes several stacked multi-layer multi-head self-attention layers.

[0205] Optionally, the process by which the second bidirectional interaction relationship extraction module combines the current bidirectional interaction relationship information with the nearest neighbor bidirectional interaction relationship information to form an input feature sequence may include:

[0206] The current bidirectional interaction relationship information is concatenated with the nearest neighbor bidirectional interaction relationship information, and then padded to an input feature sequence of a set uniform length. Based on this, after calculating the attention weights of each feature in the input feature sequence and before weighted summation of the features in the input feature sequence according to the attention weights, the second bidirectional interaction relationship extraction module can also be used for:

[0207] Add a corresponding mask value to the attention weights of each feature in the input feature sequence and the other features in the input feature sequence to obtain the masked attention weights. The mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value.

[0208] Optionally, the process by which the third bidirectional interaction relationship extraction module obtains person information in a long video segment based on the person information in the current video segment and the person information in the neighboring video segments may include:

[0209] The information of people in the current video segment and the information of people in the neighboring video segments are combined into an input feature sequence;

[0210] The third bidirectional interaction relationship extraction module calculates the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, and adds each feature in the input feature sequence according to the attention weight to obtain the target bidirectional interaction relationship feature between the object corresponding to each feature in the input feature sequence and the other objects corresponding to the other features.

[0211] The objects corresponding to each feature in the input feature sequence include the people in the long video segment composed of the current video segment and the neighboring video segments. The third bidirectional interaction relationship extraction module includes several stacked multi-layer multi-head self-attention layers.

[0212] Optionally, the process by which the third bidirectional interaction relationship extraction module combines the character information in the current video segment and the character information in the neighboring video segments into an input feature sequence may include:

[0213] The character information in the current video segment and the character information in the neighboring video segments are concatenated and expanded to a set uniform length input feature sequence using padding. Based on this, after calculating the attention weights of each feature in the input feature sequence and before weighted summation of the features according to the attention weights, the third bidirectional interaction relationship extraction module can also be used for:

[0214] Add a corresponding mask value to the attention weights of each feature in the input feature sequence and the other features in the input feature sequence to obtain the masked attention weights. The mask value added to the attention weights between non-padding element features and other non-padding element features in the input feature sequence is 0, and the mask value added to the attention weights between non-padding element features and padding element features in the input feature sequence is a set negative infinity constant value.

[0215] Optionally, the process by which the prediction module predicts the behavior category of the person in the current video segment based on the target two-way interaction relationship features and the person information in the long video segment may include:

[0216] The target two-way interaction relationship features and the character information on the long video segment are fused to obtain the fused features;

[0217] Based on the fusion features, the behavior category of the person in the current video segment is predicted.

[0218] The video behavior detection device provided in this application embodiment can be applied to video behavior detection devices, such as mobile phones, computers, servers, etc. Optionally, Figure 7 The hardware structure block diagram of the video behavior detection device is shown below. Figure 7 The hardware structure of a video behavior detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0219] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0220] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0221] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0222] The memory stores a program, which the processor can call. The program is used for:

[0223] Obtain the video features of the current video segment;

[0224] Based on the video features, obtain the information of the people in the current video segment, as well as the multi-scale context information representing different background information in the current video segment;

[0225] Based on the character information and the multi-scale context information, obtain the current bidirectional interaction relationship features between the character and the multi-scale context information in the current video segment;

[0226] Based on the current two-way interaction relationship characteristics, predict the behavior category of the person in the current video segment.

[0227] Optionally, the refined and extended functions of the program can be found in the description above.

[0228] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0229] Obtain the video features of the current video segment;

[0230] Based on the video features, obtain the information of the people in the current video segment, as well as the multi-scale context information representing different background information in the current video segment;

[0231] Based on the character information and the multi-scale context information, obtain the current bidirectional interaction relationship features between the character and the multi-scale context information in the current video segment;

[0232] Based on the current two-way interaction relationship characteristics, predict the behavior category of the person in the current video segment.

[0233] Optionally, the refined and extended functions of the program can be found in the description above.

[0234] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0235] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0236] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for video behavior detection, the method comprising: The method comprises: obtaining video features of a current video segment; based on the video features, obtaining character information in the current video segment and multi-scale context information representing different background information in the current video segment; based on the character information and the multi-scale context information, obtaining a current bidirectional interaction relationship feature between the character and the multi-scale context information in the current video segment; based on the current bidirectional interaction relationship feature, predicting the behavior category of the character in the current video segment; based on the character information and the multi-scale context information, obtaining a current bidirectional interaction relationship feature between the character and the multi-scale context information in the current video segment, comprising: combining the character information and the multi-scale context information into an input feature sequence; calculating, by a first bidirectional interaction relationship extraction module, the attention weight of each feature in the input feature sequence and the attention weight of each feature in the input feature sequence, and weighting and adding each feature in the input feature sequence according to the attention weight to obtain the current bidirectional interaction relationship feature between the object corresponding to each feature in the input feature sequence and the remaining objects corresponding to the remaining features; wherein the object corresponding to each feature in the input feature sequence comprises the character in the current video segment and the multi-scale context information.

2. The method of claim 1, wherein, based on the current bidirectional interaction relationship feature, predicting the behavior category of the character in the current video segment, comprising: based on the current bidirectional interaction relationship feature and the neighboring bidirectional interaction relationship feature, obtaining a target bidirectional interaction relationship feature between the character and the multi-scale context information in a long-time video segment; wherein the neighboring bidirectional interaction relationship feature is the bidirectional interaction relationship feature between the character and the multi-scale context information in the neighboring video segment before and / or after the current video segment in the original video; based on at least the target bidirectional interaction relationship feature, predicting the behavior category of the character in the current video segment.

3. The method of claim 2, wherein, Further comprising: based on the character information in the current video segment and the character information in the neighboring video segment, obtaining the character information in a long-time video segment; then, the at least based on the target bidirectional interaction relationship feature, predicting the behavior category of the character in the current video segment, comprising: based on the target bidirectional interaction relationship feature and the character information in the long-time video segment, predicting the behavior category of the character in the current video segment.

4. The method of claim 1, wherein, based on the video features, obtaining the character information in the current video segment, comprising: generating a position code corresponding to the video features; detecting, by a human body detection module, a target frame of the character in a key frame image in the current video segment; extracting, in the video features, a character feature corresponding to the position of the target frame of the character; extracting, in the position code corresponding to the video features, a character position code corresponding to the position of the target frame of the character, and composing the character information from the character feature and the character position code.

5. The method of claim 4, wherein, based on the video features, obtaining the multi-scale context information representing different background information in the current video segment, comprising: The video features and corresponding position encodings are processed by a multi-scale context information extraction module to obtain processed multi-scale context features and position encodings corresponding to each context feature, and the multi-scale context information is composed of the context features and the corresponding position encodings. The multi-scale context information extraction module includes a plurality of parallel adaptive pooling layers or convolution layers of different sizes.

6. The method of claim 5, wherein, Further comprising: The scale encodings corresponding to each context feature are obtained, and the multi-scale context information is composed of the context features, the corresponding position encodings, and the scale encodings.

7. The method of claim 1, wherein, The person information and the multi-scale context information are combined into an input feature sequence, including: The person information and the multi-scale context information are spliced and expanded to an input feature sequence of a set uniform length by padding; After calculating the attention weight of each feature in the input feature sequence with respect to each feature in the input feature sequence, before weighted addition of each feature in the input feature sequence according to the attention weight, further comprising: The attention weight of each feature in the input feature sequence with respect to each feature in the input feature sequence is added to the corresponding mask value, to obtain the masked attention weight, wherein the mask value added to the attention weight between the non-padding element feature and the remaining non-padding element feature in the input feature sequence is 0, and the mask value added to the attention weight between the non-padding element feature and the padding element feature in the input feature sequence is a set negative infinity constant value.

8. The method of claim 2, wherein, Based on the current bidirectional interaction relationship feature and the neighbor bidirectional interaction relationship feature, a target bidirectional interaction relationship feature between a person and multi-scale context information on a long-time video segment is obtained, including: The current bidirectional interaction relationship feature and the time position encoding of the current video segment in the original video are added to obtain the current bidirectional interaction relationship information; the neighbor bidirectional interaction relationship feature and the time position encoding of the neighbor video segment in the original video are added to obtain the neighbor bidirectional interaction relationship information; The current bidirectional interaction relationship information and the neighbor bidirectional interaction relationship information are combined into an input feature sequence; The attention weight of each feature in the input feature sequence with respect to each feature in the input feature sequence is calculated by a second bidirectional interaction relationship extraction module, and each feature in the input feature sequence is weighted and added according to the attention weight to obtain a target bidirectional interaction relationship feature between an object corresponding to each feature and the remaining objects corresponding to the remaining features; The objects corresponding to each feature in the input feature sequence include a person and multi-scale context information in a long-time video segment composed of the current video segment and the neighbor video segment.

9. The method of claim 8, wherein, The current bidirectional interaction relationship information and the neighbor bidirectional interaction relationship information are combined into an input feature sequence, including: The current bidirectional interaction relationship information and the neighbor bidirectional interaction relationship information are spliced and expanded to an input feature sequence of a set uniform length by padding; After calculating the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, before weighted addition of each feature in the input feature sequence according to the attention weight, further comprising: Adding a corresponding mask value to the attention weight of each feature in the input feature sequence and each feature in the input feature sequence to obtain the masked attention weight, wherein the mask value added to the attention weight between the non-padding element feature and the remaining non-padding element feature in the input feature sequence is 0, and the mask value added to the attention weight between the non-padding element feature and the padding element feature in the input feature sequence is a set negative infinity constant value.

10. The method of claim 3, wherein, Based on the character information in the current video segment and the character information in the adjacent video segment, the character information on the long-time video segment is obtained, comprising: Combining the character information in the current video segment and the character information in the adjacent video segment into an input feature sequence; Through the third bidirectional interaction relationship extraction module, the attention weight of each feature in the input feature sequence and each feature in the input feature sequence is calculated, and each feature in the input feature sequence is weighted and added according to the attention weight to obtain the character information on the long-time video segment composed of the current video segment and the adjacent video segment.

11. The method of claim 10, wherein, Combining the character information in the current video segment and the character information in the adjacent video segment into an input feature sequence, comprising: Splicing the character information in the current video segment and the character information in the adjacent video segment, and expanding to a set uniform length input feature sequence by padding; After calculating the attention weight of each feature in the input feature sequence and each feature in the input feature sequence, before weighted addition of each feature in the input feature sequence according to the attention weight, further comprising: Adding a corresponding mask value to the attention weight of each feature in the input feature sequence and each feature in the input feature sequence to obtain the masked attention weight, wherein the mask value added to the attention weight between the non-padding element feature and the remaining non-padding element feature in the input feature sequence is 0, and the mask value added to the attention weight between the non-padding element feature and the padding element feature in the input feature sequence is a set negative infinity constant value.

12. The method of claim 3, wherein, Based on the target bidirectional interaction relationship feature and the character information on the long-time video segment, the behavior category of the character in the current video segment is predicted, comprising: Fusing the target bidirectional interaction relationship feature and the character information on the long-time video segment to obtain a fusion feature; Based on the fusion feature, the behavior category of the character in the current video segment is predicted.

13. A video behavior detection apparatus, characterized by comprising: Comprising: A video feature extraction module for obtaining the video feature of the current video segment; A character information acquisition module for acquiring the character information in the current video segment based on the video feature; A multi-scale context information extraction module for acquiring multi-scale context information representing different background information in the current video segment based on the video feature; The first bidirectional interaction relationship extraction module is configured to acquire a current bidirectional interaction relationship feature between a person and multi-scale context information in the current video segment based on the person information and the multi-scale context information. The behavior category prediction module is configured to predict a behavior category of the person in the current video segment based on the current bidirectional interaction relationship feature. The first bidirectional interaction relationship extraction module is specifically configured to: combine the person information and the multi-scale context information into an input feature sequence; calculate, by the first bidirectional interaction relationship extraction module, an attention weight of each feature in the input feature sequence and each feature in the input feature sequence, and add each feature in the input feature sequence according to the attention weight to obtain a current bidirectional interaction relationship feature between an object corresponding to each feature in the input feature sequence and the remaining objects corresponding to the remaining features; wherein the objects corresponding to each feature in the input feature sequence include the person in the current video segment and the multi-scale context information.

14. A video behavior detection device, characterized by, The video behavior detection method comprises: a memory and a processor; the memory is configured to store a program; the processor is configured to execute the program to implement each step of the video behavior detection method according to any one of claims 1-12.

15. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement each step of the video behavior detection method according to any one of claims 1-12.

Images