Action detection method and device, computer device and storage medium

Abstract

The application relates to an action detection method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a common feature of a co-occurrence action in a to-be-detected video; the co-occurrence action is a same type action appearing in different video frames of the to-be-detected video; filtering video features satisfying a similarity close condition with the common feature from video features of video frames arranged in sequence in the to-be-detected video to obtain video features belonging to the co-occurrence action; generating an ordered similarity feature according to the similarity between the video features of the video frames and the video features belonging to the co-occurrence action; and identifying a video segment belonging to the co-occurrence action in the to-be-detected video according to the ordered similarity feature and the video features of the video frames. The method can effectively improve the accuracy of co-occurrence action detection of a video.

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 12, 2021, with application number 2021109240496 and invention name “Motion detection method, device, computer equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer vision technology, and in particular to a motion detection method, apparatus, computer device, and storage medium. Background Art

[0003] With the continuous development of computer vision technology, various video content processing technologies have emerged, such as video action detection, which can identify the movements of target objects in videos. Traditionally, if users want to extract video clips of various action categories from the original video, they usually need to manually edit the video clips of each action category, which is labor-intensive and time-consuming.

[0004] To improve video detection efficiency, related technologies have adopted methods that use full or weak supervision to train neural network models on multiple action categories. This allows the trained neural network models to identify action categories that have already been seen in videos. However, for action categories that the model has not seen before, this method has difficulty accurately locating the corresponding action segments, resulting in low accuracy in video action detection. Summary of the Invention

[0005] Based on this, it is necessary to provide a motion detection method, device, computer equipment and storage medium that can effectively improve the accuracy of video motion detection in order to address the above technical problems.

[0006] A motion detection method, comprising:

[0007] Obtaining common features of co-occurring actions in a video to be detected; the co-occurring actions are actions of the same type that appear in different video frames of the video to be detected;

[0008] Selecting video features that meet a close similarity condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtaining video features belonging to the co-occurring action;

[0009] generating ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action;

[0010] According to the ordered similarity features and the video features of each of the video frames, a video segment belonging to the co-occurring action in the video to be detected is identified.

[0011] A motion detection device, comprising:

[0012] A common feature generation module is used to obtain common features of co-occurring actions extracted from the video to be detected; the co-occurring actions are the same type of actions that appear in different video frames of the video to be detected;

[0013] A video feature screening module is used to screen video features that meet a similarity close condition with the common feature from the video features of each video frame arranged in time sequence in the video to be detected, and obtain video features belonging to the co-occurring action;

[0014] A similarity feature generation module, configured to generate ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action;

[0015] The co-occurrence action detection module is used to identify the video segments belonging to the co-occurrence action in the video to be detected based on the ordered similarity features and the video features of each of the video frames.

[0016] In one embodiment, the common feature generation module is also used to splice the video features corresponding to each video frame in the video to be detected in time sequence to obtain a video splicing feature; add position information to each video feature in the video splicing feature; encode the video splicing feature with added position information to generate a common feature of the co-occurring action in the video to be detected.

[0017] In one embodiment, the common feature generation module is also used to determine the attention weight corresponding to each video feature based on the correlation between the video features with added position information; the attention weight is positively correlated with the correlation; the corresponding video features are encoded based on the attention weight to generate common features corresponding to the co-occurring actions in the video to be detected.

[0018] In one embodiment, the video feature screening module is also used to fuse the common features with the video features of each video frame arranged in time sequence in the video to be detected to obtain fused video features; determine the similarity between each fused video feature and the common features; and screen video features that meet the similarity close conditions with the common features from the video features of each video frame according to the similarity to obtain video features belonging to co-occurring actions.

[0019] In one embodiment, the co-occurrence action detection module is also used to predict the video segment corresponding to the candidate action in the video to be detected based on the target similarity between the video features of each of the video frames and the ordered similarity features; and based on the classification results obtained by action classification of the video features in the video segment corresponding to the candidate action, identify the video segment belonging to the co-occurrence action from the video segment corresponding to the candidate action.

[0020] In one embodiment, the co-occurrence action detection module is further used to splice the ordered similarity features and the video features of each video frame to obtain spliced ​​video features; determine the target similarity between the video features of each video frame based on the spliced ​​video features; and predict the video clip corresponding to the candidate action in the video to be detected based on the position information corresponding to the video features whose target similarity reaches the similarity threshold.

[0021] In one embodiment, the co-occurrence action detection module is also used to query the target similarity between the video features of each spliced ​​video frame in the spliced ​​video features to obtain a query result; based on the query result, the starting position and ending position corresponding to the candidate action in the video to be detected are predicted to obtain a video clip corresponding to the candidate action.

[0022] In one embodiment, the action detection method is performed by an action detection model; the action detection model is obtained through training steps; the action detection device also includes a model training module for obtaining sample video features corresponding to each video frame in the sample video; the sample video carries a sample label, and the sample label includes the position and category of the sample co-occurring action in the sample video; in each round of iterative training, the marked features of this round are added to the sample video features after splicing in time sequence, and the sample video features after adding the marked features of this round are encoded through the encoding layer of the action detection model to be trained, and the predicted common features of the sample co-occurring action in the sample video are predicted and generated; through the similarity encoding generator of the action detection model, the sample video features are generated from each of the sample co-occurring actions. The sample video features are screened out from the sample video features, which meet the similarity close condition with the predicted common features, to obtain the sample video features belonging to the sample co-occurrence action; based on the similarity between the sample video features of each video frame and the sample video features belonging to the sample co-occurrence action, a predicted ordered similarity feature is generated; through the parallel decoder of the action detection model, based on the sample ordered similarity feature and each of the sample video features, a predicted video segment belonging to the sample co-occurrence action in the sample video is predicted; based on the difference between the predicted video segment and the sample label, a prediction result loss is determined; based on the prediction result loss, the model parameters of the action detection model are adjusted and training is continued until the training stop condition is met.

[0023] In one embodiment, the model training module is also used to splice the sample ordered similarity features with each of the sample video features to obtain spliced ​​sample video features; through the parallel decoder in the action detection model, predict the predicted video segment corresponding to the candidate action in the sample video based on the spliced ​​sample video features; perform action classification on the sample video features corresponding to the predicted video segment to obtain a predicted classification result.

[0024] In one embodiment, the prediction result loss includes regression loss and classification loss; the model training module is also used to determine the regression loss based on the difference between the position of the predicted video clip and the position belonging to the sample co-occurrence action in the sample label; and determine the classification loss based on the difference between the classification result of the predicted video clip and the category of the sample co-occurrence action of the sample label.

[0025] In one embodiment, the model training module is also used to extract positive sample video frames belonging to sample co-occurrence actions from the sample video according to the sample labels, and extract corresponding sample common features from the sample video features corresponding to the positive sample video frames; determine a first feature loss based on the similarity difference between the predicted common features and the sample video features of the positive sample video frames; determine a second feature loss based on the difference between the predicted common features and the sample common features; adjust the model parameters of the action detection model according to the prediction result loss, the first feature loss and the second feature loss and continue training until the training is terminated when the training stop condition is met.

[0026] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the action detection method in each embodiment of the present application when executing the computer program.

[0027] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the action detection method in each embodiment of the present application.

[0028] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium; a processor of the computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, implements the steps of the action detection method in each embodiment of the present application.

[0029] The above-mentioned action detection method, device, computer equipment and storage medium obtain the common features of co-occurring actions extracted from the video to be detected. Among them, co-occurring actions are the same type of actions that appear in different video frames of the video to be detected, thereby effectively extracting the features corresponding to the co-occurring actions that appear repeatedly in the video to be detected. Then, from the video features of each video frame arranged in time sequence in the video to be detected, video features that meet the similarity close condition with the common features are screened, thereby obtaining video features that can accurately screen out co-occurring actions based on the similarity with the common features. Then, based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action, ordered similarity features are generated, and then co-occurring action detection is performed based on the ordered similarity features and the video features of each video frame. Therefore, it does not rely on action categories for identification, but uses the common features of co-occurring actions to identify video clips belonging to co-occurring actions in the video to be detected, thereby avoiding the limitations brought by relying on action categories, improving applicability, and thus effectively improving the accuracy of co-occurring action detection in videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram illustrating an application environment of an action detection method according to an embodiment;

[0031] Figure 2 1 is a flow chart of an action detection method according to an embodiment;

[0032] Figure 3 1 is a flow chart of the training steps of an action detection model in one embodiment;

[0033] Figure 4 is a structural block diagram of an action detection model in one embodiment;

[0034] Figure 5 is a schematic diagram of an action detection model to be trained in one embodiment;

[0035] Figure 6 A schematic diagram of the results of co-occurrence action detection on a test video in one embodiment;

[0036] Figure 7 Schematic diagram of comparison between co-occurrence action detection results and sample labels in one embodiment;

[0037] Figure 8 is a schematic diagram of a video search result in one embodiment;

[0038] Figure 9 is a result of co-occurrence action detection on the searched video in one embodiment;

[0039] Figure 10 is a structural block diagram of a motion detection device in one embodiment;

[0040] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] The motion detection method provided in this application can be applied to a computer device. The computer device can be a terminal or a server. It is understood that the motion detection method provided in this application can be applied to a terminal or a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server.

[0043] The motion detection method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. Specifically, the terminal 102 can obtain the video to be detected from the server 104. After the terminal 102 obtains the video to be detected, it first obtains the common features of the co-occurring actions in the video to be detected, and then screens the video features that meet the similarity close conditions with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtains the video features belonging to the co-occurring actions. Then, based on the similarity between the video features of each video frame and the video features belonging to the co-occurring actions, an ordered similarity feature is generated. The terminal 102 further identifies the video clips belonging to the co-occurring actions in the video to be detected based on the ordered similarity feature and the video features of each video frame.

[0044] Among them, the server 104 server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0045] Cloud computing refers to the delivery and usage model of IT (Internet Technology) infrastructure, enabling on-demand, scalable access to resources over the internet. Broadly speaking, cloud computing refers to the delivery and usage model of services, enabling on-demand, scalable access to services over the internet. These services can be IT-related, software-related, internet-related, or other services. Cloud computing is the product of the convergence of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the growth of the internet, real-time data streams, and the diversification of connected devices, as well as the demands for search services, social networking, mobile commerce, and open collaboration, cloud computing has rapidly developed.

[0046] It can be understood that the video processing method in each embodiment of the present application adopts the machine learning technology in artificial intelligence technology to train an action detection model that accurately detects co-occurrence actions in videos. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0047] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by formula. It can be understood that the action detection model trained in some embodiments of the present application is trained using machine learning technology. Based on the machine learning technology, the action detection model can be trained to have a high accuracy in co-occurrence action detection in videos.

[0048] In one embodiment, Figure 2 As shown, a motion detection method is provided, and the method is described by taking the application of the method to a computer device as an example. The computer device can be Figure 1 The terminal or server in the embodiment includes the following steps:

[0049] Step S202: obtaining common features of co-occurring actions extracted from the video to be detected.

[0050] The video refers to a series of image frames, that is, a video includes multiple frames of images in a time sequence, and can be specifically a series of continuous static images generated by capturing dynamic images using photography. The video to be detected is the video that needs to be motion detected.

[0051] In one embodiment, the video to be detected may be one video. In another embodiment, the video to be detected may include multiple videos, wherein multiple refers to at least two.

[0052] It can be understood that co-occurring actions are the same type of actions that appear in different video frames of the video to be detected, and can represent the features corresponding to the repeated action types in the video to be detected. Common features are features that represent co-occurring actions.

[0053] After the computer device obtains the video to be detected, it can first perform feature extraction on the video to be detected to obtain video features corresponding to each video frame in the video to be detected. It can be understood that the video features extracted from each video frame include the action features in each video frame.

[0054] The computer device then further performs similarity coding on the video features corresponding to each video frame to generate common features of the co-occurring actions. Specifically, the computer device may also splice the video features corresponding to each video frame in the temporal and semantic dimensions, and then perform similarity coding on the spliced ​​video features to generate common features. Common features are features generated by encoding the video features, rather than being features of a single frame directly extracted from the video features of each video frame.

[0055] It is understandable that video features of the same type of action may appear frequently in the video to be detected. Usually, these video features are similar to each other. Therefore, by encoding these similar video features, common features that aggregate the same type of actions can be generated.

[0056] Step S204 , screening video features that meet a close similarity condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtaining video features belonging to the co-occurring action.

[0057] It is understood that the video to be detected includes a series of video frames arranged in a time sequence. Therefore, each video frame in the video to be detected has a corresponding time sequence. Among them, the similarity close condition refers to the condition for screening video features based on similarity.

[0058] After obtaining the common features of the co-occurring actions from the video to be detected, the computer device calculates the similarity between the video features of each video frame arranged in time sequence and the common features. It then further selects, from the video features of each video frame, video features that meet a close similarity condition with the common features. Specifically, the computer device may screen a preset number of video features with the greatest similarity values ​​based on the similarity values ​​between the video features of each video frame and the common features. A greater similarity value indicates a higher similarity between the video features and the common features.

[0059] Since the common features are common features of the same type of actions in the video to be detected, the screened video features are highly similar to the common features. Therefore, the video features of the screened video frames can be used as video features belonging to the co-occurring actions.

[0060] Step S206 : generating ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring actions.

[0061] The ordered similarity feature may be a feature matrix with a sequence related only to similarity, and the matrix includes similarity values ​​between the video features of each video frame and the video features of the co-occurring action.

[0062] Specifically, after the computer device filters out the video features belonging to the co-occurring actions from the video features of each video frame, it further calculates the similarity value between the video features of each video frame and the video features belonging to the co-occurring actions, and generates corresponding ordered similarity features according to the time sequence of each video frame.

[0063] Step S208 : identifying video segments belonging to co-occurring actions in the video to be detected based on the ordered similarity features and the video features of each video frame.

[0064] The video clip is a part of the video to be detected, and each video clip includes a plurality of continuous video frames.

[0065] After the computer device generates the ordered similarity features, it can concatenate the ordered similarity features with the video features of each video frame. Because the ordered similarity features include the similarity between the video features of each video frame and the video features of the co-occurring action, it is possible to determine, from the concatenated features, video features whose similarity to the video features of the co-occurring action meets a similarity threshold.

[0066] The computer device can then predict the starting and ending positions of possible co-occurring actions based on video features that meet a similarity threshold. Action type recognition is then performed on the video features of the video frames corresponding to the starting and ending positions to identify video segments that belong to the co-occurring actions. Because co-occurring actions are identified based on common features that appear frequently in the video features of each video frame, there is no need to rely on identifying the specific action category of each video frame to identify co-occurring actions. Based on the common features, video segments that belong to co-occurring actions, regardless of category, can be accurately identified.

[0067] In the above-mentioned action detection method, the computer device first obtains the common features of the co-occurring actions in the video to be detected. Among them, the co-occurring action is the same type of action that appears in different video frames of the video to be detected, thereby effectively extracting the features corresponding to the co-occurring actions that appear repeatedly in the video to be detected. Then, from the video features of each video frame arranged in time sequence in the video to be detected, the video features that meet the similarity close condition with the common features are screened, thereby obtaining the video features that can accurately screen out the co-occurring actions based on the similarity with the common features. Then, based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action, an ordered similarity feature is generated, and then the co-occurring action detection is performed based on the ordered similarity feature and the video features of each video frame, so that it does not rely on the action category for identification, but uses the common features of the co-occurring action to identify the video clips belonging to the co-occurring action in the video to be detected, thereby avoiding the limitations brought by relying on the action category, improving the applicability, and thus effectively improving the accuracy of co-occurring action detection in the video.

[0068] In one embodiment, the step of obtaining the common features of co-occurring actions in the video to be detected includes: splicing the video features corresponding to each video frame in the video to be detected in time sequence to obtain a video splicing feature; adding position information to each video feature in the video splicing feature; encoding the video splicing feature with added position information to generate the common features of the co-occurring actions in the video to be detected.

[0069] Position information represents the location of each video frame within the video being tested. It's understood that the order of video frames within the video being tested is crucial. It's not only a component of the video but also a crucial factor in conveying the semantics of the video's content. Different positions or orderings of video frames within the video can lead to discrepancies in the semantics of the video's content.

[0070] It is understood that each video frame in the video to be detected has a corresponding time sequence. Specifically, after the computer device extracts the video features corresponding to each video frame in the video to be detected, it first splices the video features corresponding to each video frame according to the time sequence corresponding to each video frame, thereby obtaining a video splicing feature after the time sequence is spliced.

[0071] The computer device then adds corresponding position information to each video feature in the time-series spliced ​​video features. Specifically, each video feature in the video splicing features can be position-encoded. For example, at least one position encoding method such as learning position vectors, relative position expression, and sinusoidal position encoding can be used to position-encode each video feature, thereby adding a position coder or a time sequence identifier to each video feature so that each video feature contains specific position information. This effectively prevents the accuracy of action detection from being affected by the disordered order of video frames during action detection based on each video feature.

[0072] The computer device then encodes the video splicing features with added position information. Specifically, the computer device can aggregate the central features corresponding to the recurring video features based on the correlation between the video features after adding the position information, so as to accurately generate common features that represent the co-occurring actions in the video to be detected.

[0073] In one embodiment, the steps of encoding the video splicing features with added position information to generate common features of co-occurring actions in the video to be detected include: determining the attention weight corresponding to each video feature based on the correlation between the video features with added position information; the attention weight is positively correlated with the correlation; and encoding the corresponding video features based on the attention weight to generate common features corresponding to the co-occurring actions in the video to be detected.

[0074] Correlation, also known as similarity, represents the degree of association and similarity between two objects. Similarity refers to the distance between the features of two objects; the smaller the distance, the greater the similarity; conversely, the smaller the similarity. In this embodiment, correlation refers to the correlation between video features.

[0075] The computer device may encode the corresponding video features based on the correlation between the video features to which the position information is added, thereby generating common features corresponding to the co-occurring actions in the video to be detected.

[0076] In a specific embodiment, when generating common features, the computer device may employ an attention mechanism to perform a weighted summation process on the video splicing features that incorporate location information. Specifically, the computer device may add a marker feature to the video splicing features and then, using the attention mechanism, calculate the similarity between the video features of each video frame in the video splicing features and the marker feature. For example, the similarity between the video features of two adjacent video frames and the marker feature may be calculated. Modeling is then performed based on the calculated similarities to obtain video features labeled with similarities.

[0077] Then, based on the video features with similarity marked, corresponding attention weights are assigned to each video feature. For example, the current video frame can be used as the center, and a sequence of 2n+1 video frames consisting of the first n video frames and the last n video frames can be used as the attention. The similarity between the context video frame and the central video frame can be determined, and the central video frame can be scored based on the various similarities. The scores are weighted averaged to obtain the importance of the central video frame to the entire video to be detected, that is, the attention weight. Similarly, the computer device can obtain the attention weight corresponding to each video feature.

[0078] The computer then performs a weighted average of the attention weights corresponding to each video feature to obtain a corresponding attention value. The computer then performs feature encoding on the video splicing features based on the attention value, thereby accurately generating a common feature with a large weight.

[0079] In this embodiment, an attention mechanism is used to obtain attention weights corresponding to each video feature, thereby assigning greater weight to video features containing rich action information in each video frame. By taking a weighted average of the attention weights corresponding to each video feature, a common feature with a high weight corresponding to the most frequent co-occurrences in the video to be detected can be generated more accurately.

[0080] In one embodiment, the steps of screening video features that meet a similarity close condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtaining video features belonging to co-occurring actions include: fusing the common features with the video features of each video frame arranged in time sequence in the video to be detected, respectively, to obtain fused video features; determining the similarity between each fused video feature and the common features; and screening video features that meet a similarity close condition with the common features from the video features of each video frame according to the similarity, to obtain video features belonging to the co-occurring actions.

[0081] Fusion refers to feature fusion, which optimizes the combination of extracted feature vectors. Its purpose is to merge video features into more discriminative features. Feature fusion can be achieved by directly concatenating features or by combining multiple features into composite features.

[0082] The computer device encodes the video features of each video frame in the video to be detected, generates common features representing co-occurring actions, and then fuses the common features with the video features of each video frame to obtain the video features of each video frame after fusion with the common features. In this way, the video features of each video frame after fusion can pay more attention to features related to the common features.

[0083] The computer device then calculates the similarity between the common features and the video features of each video frame. Based on the similarity, the computer device selects video features that meet a certain similarity condition from the video features of each video frame arranged in time sequence, for example, selecting a preset number of video features with the highest similarity values. Specifically, the computer device may sort the similarity values ​​between the video features of each video frame and the common features in descending order, and then select a preset number of video features with the highest similarity values. The computer device may then mark the selected video features as video features belonging to the co-occurring action.

[0084] In this embodiment, since the common features gather features corresponding to co-occurring actions that appear repeatedly in the video to be detected, by identifying video features that are more similar to the common features based on the similarity between the common features, video features belonging to co-occurring actions can be accurately screened out.

[0085] In one embodiment, the step of identifying video segments belonging to co-occurring actions in a video to be detected based on ordered similarity features and video features of each video frame includes: predicting video segments corresponding to candidate actions in the video to be detected based on target similarity between the video features of each video frame and the ordered similarity features; and identifying video segments belonging to co-occurring actions from video segments corresponding to the candidate actions based on classification results obtained by performing action classification on the video features in the video segments corresponding to the candidate actions.

[0086] Among them, candidate actions represent the type of action to be identified. Candidate actions may be co-occurring actions or other types of actions. Action classification refers to identifying the action category in the video clip.

[0087] After the computer device selects video features that meet the similarity close conditions with the common features from the video features of each video frame as video features belonging to the co-occurring action, the computer device calculates the similarity between each video feature belonging to the co-occurring action and the video features of each video frame respectively, and generates an ordered similarity value, that is, an ordered similarity feature.

[0088] The computer device then concatenates the video features of each video frame with the ordered similarity features to obtain a concatenated feature. It is understood that the target similarity between the video features of each video frame and the ordered similarity features can be directly determined based on the concatenated feature, that is, the target similarity between the video features of each video frame in the concatenated feature and the video features of the co-occurring action can be determined.

[0089] The computer device further predicts a video segment corresponding to the candidate action in the video to be detected based on the target similarity between the video features of each video frame and the video features of the co-occurring action. Specifically, the target similarity corresponding to each adjacent video frame can be determined. The computer device can then predict the starting and ending positions of the video segment in the video to be detected that may belong to the co-occurring action based on the target similarity values ​​corresponding to each adjacent video frame, thereby obtaining the video segment corresponding to the candidate action.

[0090] The computer then performs action classification on the video features in the video clips corresponding to the candidate actions, obtaining corresponding classification results, which serve as the action categories for each video clip. The computer then identifies video clips belonging to co-occurring actions from the video clips corresponding to the candidate actions. Specifically, the action category with the largest number of video clips among the action categories of each video clip can be identified as the co-occurring action. This allows accurate identification of video clips belonging to co-occurring actions in the video to be detected.

[0091] In one embodiment, the step of predicting a video segment corresponding to a candidate action in a video to be detected based on a target similarity between the video features of each video frame and the ordered similarity features includes: splicing the ordered similarity features and the video features of each video frame to obtain a spliced ​​video feature; determining the target similarity between the video features of each video frame based on the spliced ​​video feature; and predicting a video segment corresponding to the candidate action in the video to be detected based on position information corresponding to the video features whose target similarity reaches a similarity threshold.

[0092] The position information refers to the position of the video frame in the video to be detected, for example, it may be a temporal position in the video to be detected, and the temporal position may specifically include a start position and an end position.

[0093] Specifically, after the computer device generates the ordered similarity features, it splices the ordered similarity features with the video features of each video frame. Specifically, the computer device can directly connect the ordered similarity features with the video features of each video frame to obtain the spliced ​​video features. Alternatively, the computer device can fuse the ordered similarity features with the video features of each video frame to obtain the spliced ​​video features.

[0094] The computer device can then determine the target similarity between the video features of each video frame based on the spliced ​​video features. The target similarity between the video features of each adjacent video frame can also be determined.

[0095] The computer device then predicts the starting position and ending position of the video frame corresponding to each candidate action in the video to be detected based on the position information corresponding to the video features whose target similarity reaches the similarity threshold. Then, based on the starting position and ending position of the video frame corresponding to each candidate action, the computer device can accurately predict the video clip corresponding to each candidate action.

[0096] In this embodiment, ordered similarity features are generated based on the similarity between the video features of each video frame and the video features belonging to the co-occurring actions, and action detection is then performed based on the ordered similarity features and the video features of each video frame, so that the video clips corresponding to the candidate actions in the video to be detected can be accurately identified.

[0097] In one embodiment, the step of determining target similarity between video features of each video frame based on the spliced ​​video features includes querying the spliced ​​video features for target similarity between video features of each spliced ​​video frame to obtain a query result. The step of predicting a video segment corresponding to a candidate action in the video to be detected based on position information corresponding to video features whose target similarity reaches a similarity threshold includes predicting a starting position and an ending position corresponding to the candidate action in the video to be detected based on the query result to obtain a video segment corresponding to the candidate action.

[0098] Among them, the target similarity is finally determined to identify the video frames belonging to the co-occurring actions.

[0099] In the process of determining the target similarity between the video features of each video frame based on the spliced ​​video features, the computer device can specifically calculate the similarity between the video features of each video frame in the spliced ​​video features and the marker features through the attention mechanism.

[0100] For example, an attention mechanism can be used to employ at least one query embedding vector to query the target similarity between the video features of two adjacent video frames and the marker features in the spliced ​​video features to obtain a query result. Then, based on the target similarity between the adjacent video frames in the query result, the adjacent video features that meet a similarity threshold are used to determine the left and right boundaries of the candidate action interval, namely, the start and end frames of the candidate action. This allows accurate identification of video segments corresponding to candidate actions that may be co-occurring actions.

[0101] In a specific embodiment, a computer device can perform co-occurrence action detection processing on a video to be detected using a trained action detection model. The trained action detection model is a machine learning model that can extract video clips corresponding to the same type of co-occurring actions appearing in different video frames, enabling accurate co-occurrence action detection and classification in the video.

[0102] like Figure 4 Figure 2 shows a block diagram of the action detection model in a specific embodiment. The action detection model includes an encoder, a similarity code generator, and a parallel decoder. The encoder also includes a multi-head attention layer and a feature fusion layer.

[0103] After acquiring the video to be detected, the computer extracts the video features corresponding to each time-sequential frame in the video to be detected. These features are then fed into a trained action detection model. The computer first performs feature encoding on the video features corresponding to each frame using the multi-head attention layer in the action detection model's encoder, generating common features for co-occurring actions. The common features are then fused with the video features corresponding to each frame using the feature fusion layer in the encoder to generate fused video features.

[0104] The computer then uses the action detection model's similarity code generator to determine the similarity between the common features and the fused video features corresponding to each video frame. It then selects video features from the video features of each video frame in the video to be detected that meet the similarity criteria with the common features to obtain video features belonging to the co-occurring action. Then, based on the similarity between the fused video features corresponding to each video frame and the video features belonging to the co-occurring action, an ordered similarity feature is generated.

[0105] The computer then inputs the fused video features obtained by the encoder and the ordered similarity features generated by the similarity code generator into the parallel decoder of the action detection model. Specifically, the fused video features and the ordered similarity features are concatenated before being input into the parallel decoder. The parallel decoder then predicts video segments within the video to be detected that contain co-occurring actions based on the concatenated video features.

[0106] In this embodiment, action detection is performed on the video to be detected through a trained action detection model that has the ability to identify video clips belonging to co-occurring actions from a video, which can accurately generate common features corresponding to the co-occurring actions, and then based on the common features, the video clips belonging to the co-occurring actions can be accurately identified.

[0107] In one embodiment, the above-mentioned motion detection method is performed by a motion detection model; the motion detection model is obtained by training steps, such as Figure 3 As shown in Figure 2, the training steps of the action detection model include:

[0108] Step S302: Obtain sample video features corresponding to each video frame in the sample video.

[0109] Step S304: In each round of iterative training, the marker features of this round are added to the sample video features spliced ​​in time sequence. The sample video features after adding the marker features of this round are encoded through the encoding layer of the action detection model to be trained to predict and generate the predicted common features of the sample co-occurring actions in the sample video.

[0110] In step S306, the similarity coding generator of the action detection model is used to screen and predict the sample video features whose common features meet the similarity proximity conditions from each sample video feature to obtain the sample video features belonging to the sample co-occurring action; and based on the similarity between the sample video features of each video frame and the sample video features belonging to the sample co-occurring action, a predicted ordered similarity feature is generated.

[0111] Step S308 : predicting a video segment belonging to the sample co-occurring action in the sample video according to the sample ordered similarity feature and each sample video feature through the parallel decoder of the action detection model.

[0112] Step S310 : determining the prediction result loss based on the difference between the predicted video segment and the sample label.

[0113] Step S312: Adjust the model parameters of the action detection model according to the prediction result loss and continue training until the training stop condition is met.

[0114] It can be understood that the sample video refers to the video data used for training action detection. The sample video carries a sample label, which is a label marked according to the category of the sample video. The sample label includes the position and category of the sample co-occurring action in the sample video.

[0115] The prediction loss refers to the difference between the predicted video clips of the sample co-occurring actions obtained through action detection and the position and category of the co-occurring actions in the corresponding sample labels. The training stop condition refers to the end of model training. For example, the training stop condition can be reaching a preset number of iterations or the action recognition performance indicators of the action detection model after adjusting the parameters meet the preset indicators.

[0116] The computer device first obtains a sample video and then performs feature extraction on the sample video. Specifically, the computer device can perform feature extraction on the sample video using a trained video feature extraction network or a preset feature detection algorithm, thereby extracting sample video features corresponding to each video frame in the sample video.

[0117] The computer device then iteratively trains the action detection model to be trained based on the sample video features corresponding to each video frame in the sample video. Specifically, in each round of iterative training, the computer device splices the sample video features corresponding to each video frame in the sample video of that round in a temporal sequence and adds preset marker features to the sample video features after splicing them in a temporal sequence. The computer device then inputs the sample video features with the added marker features into the encoding layer of the action detection model to be trained. The encoding layer of the action detection model then encodes the sample video features with the added marker features of that round to predict and generate the predicted common features of the sample co-occurring actions in the sample video.

[0118] The computer device then inputs the common features and sample video features generated by the encoder into the similarity coding generator of the action detection model. The similarity coding generator calculates the similarity between the sample video features and the common features of each video frame, and then selects sample video features from each sample video feature based on the similarity that satisfies the similarity close condition with the predicted common features. The sample video features that meet the similarity close condition are used as sample video features belonging to the sample co-occurring action. The computer device then fuses the selected sample video features with each sample video feature through the similarity coding generator, and generates predicted ordered similarity features based on the similarity between the sample video features of each video frame and the sample video features belonging to the sample co-occurring action.

[0119] The computer device then splices the fused sample video features output by the encoder and the predicted ordered similarity features output by the similarity coding generator, and then inputs the spliced ​​sample video features into the parallel decoder of the action detection model to be trained. Then, through the parallel decoder, the predicted video clips belonging to the sample co-occurring actions in the sample video are predicted based on the sample ordered similarity features and each sample video feature.

[0120] After the computer device predicts a video segment for the sample video containing the co-occurring action using the action detection model in this round, it calculates the difference between the predicted video segment and the sample label and determines the corresponding prediction loss. Based on the prediction loss, the computer device then adjusts the model parameters of the action detection model in this round in a direction that reduces the prediction loss and continues training until the training stop condition is met.

[0121] If the training stop condition is not met in the current round, the computer device returns to the step of adding the marked features of the current round to the sample video features after splicing them in time sequence, thus entering the next round. The computer device then continues to encode the sample video features after adding the marked features of the current round through the encoding layer of the action detection model to be trained, and predicts and generates the predicted common features of the sample co-occurring actions in the sample video for iterative training. When the training stop condition is met, the iterative training stops, and a trained action detection model is obtained.

[0122] It can be understood that the trained action detection model is a machine learning model that has the ability to extract video clips corresponding to the same type of co-occurring actions appearing in different video frames from the video. It can accurately detect and classify co-occurring actions in the video to accurately identify video clips belonging to co-occurring actions in the video.

[0123] In one embodiment, the difference between the predicted video segments of a sample video and the corresponding sample labels can be measured using a loss function, such as cross entropy or mean squared error. During the iterative training of the action detection model, a backpropagation algorithm can be used to update the model parameters in the direction of gradient descent, adjust the weights and biases to minimize the overall error, and gradually adjust the model parameters of the action detection model to iteratively train the recognition model. For example, training can be terminated when the value of the loss function is less than a preset value, resulting in an action detection model that can accurately and effectively identify video segments of co-occurring actions in a video.

[0124] In this embodiment, by using sample videos to train an action detection model, the trained action detection model is able to accurately identify video clips containing co-occurring actions. When handling action detection tasks, even if the video to be detected contains action types that the trained action detection model has never seen, the action detection model can still accurately identify video clips containing co-occurring actions in the video.

[0125] In one embodiment, the steps of predicting a predicted video segment belonging to a sample co-occurring action in a sample video based on the sample ordered similarity features and each sample video feature through a parallel decoder of an action detection model include: splicing the sample ordered similarity features with each sample video feature to obtain a spliced ​​sample video feature; predicting a predicted video segment corresponding to a candidate action in the sample video based on the spliced ​​sample video feature through a parallel decoder in the action detection model; and performing action classification on the sample video features corresponding to the predicted video segment to obtain a predicted classification result.

[0126] It can be understood that after the computer device generates the sample co-occurrence features corresponding to the sample video through the encoding layer of the action detection model to be trained, the computer device also fuses the generated sample common features with the sample video features of each video frame through the encoder to optimize the sample video features of each video frame and obtain the fused sample video features.

[0127] Specifically, the computer device generates predicted ordered similarity features based on the similarity between the video features of each video frame and the video features of the co-occurring actions through the similarity coding generator of the action detection model to be trained. The computer device then inputs the fused sample video features generated by the encoder and the predicted ordered similarity features generated by the similarity coding generator into the parallel decoder of the action detection model to be trained. The parallel decoder first splices the fused sample video features with the predicted ordered similarity features, specifically in the temporal dimension and the semantic dimension, to obtain the spliced ​​sample video features.

[0128] The computer device then uses a preset number of query embedding vectors through a parallel decoder to query the similarity between the sample video features after splicing. Sample video features whose similarity meets a similarity threshold are used to obtain corresponding query results. The regression branch network in the parallel decoder then predicts the predicted video segments corresponding to the candidate actions in the sample videos based on the query results. The classification branch network in the parallel decoder further performs action classification on the sample video features corresponding to each predicted video segment, obtaining predicted classification results for each predicted video segment. Based on the predicted classification results, the predicted video segments corresponding to the candidate actions are then determined to be the co-occurring sample action.

[0129] In one embodiment, the prediction result loss includes regression loss and classification loss; the step of determining the prediction result loss based on the difference between the predicted video clip and the sample label includes: determining the regression loss based on the difference between the predicted position of the predicted video clip and the position belonging to the sample co-occurrence action in the sample label; determining the classification loss based on the difference between the predicted classification result of the predicted video clip and the category of the sample co-occurrence action in the sample label.

[0130] The regression loss refers to the position regression loss, which represents the difference between the position of the sample co-occurring action in the predicted video clip and the position of the sample co-occurring action in the corresponding sample label. The classification loss refers to the difference between the classification result of the predicted video clip and the category of the sample co-occurring action in the corresponding sample label.

[0131] After the computer device predicts a video segment of the sample video that contains the sample co-occurring action using the trained action detection model, it calculates the difference between the predicted position of the video segment and the position of the sample co-occurring action in the sample label, and determines a regression loss based on the difference. It also determines the difference between the predicted classification result of the predicted video segment and the category of the sample co-occurring action in the sample label, and determines a classification loss based on the difference.

[0132] The computer device then adjusts the model parameters of the current action detection model in the direction of reducing the regression loss and the classification loss, and continues training until the training stop conditions are met, so that the accuracy of the co-occurrence action detection of the trained action detection model is higher.

[0133] In one embodiment, the above-mentioned action detection method further includes: extracting positive sample video frames belonging to the sample co-occurring action from the sample video according to the sample label, extracting corresponding sample common features from the sample video features corresponding to the positive sample video frames; determining a first feature loss based on the similarity difference between the predicted common features and the sample video features of the positive sample video frames; and determining a second feature loss based on the difference between the predicted common features and the sample common features. The step of adjusting the model parameters of the action detection model according to the prediction result loss and continuing training until the training stop condition is met includes: adjusting the model parameters of the action detection model according to the prediction result loss, the first feature loss, and the second feature loss and continuing training until the training stop condition is met.

[0134] It can be understood that a positive sample video frame is a sample video frame that carries a sample label indicating that it belongs to the sample co-occurring action in the sample video, that is, a sample video frame belonging to the sample co-occurring action. Conversely, a sample video frame that does not carry a sample label indicating that it belongs to the sample co-occurring action in the sample video is a negative sample video frame.

[0135] The first feature loss refers to the difference loss in the similarity between the sample common features and the sample video features of the positive sample video frame. The second feature loss refers to the feature difference loss between the sample common features and the sample common features corresponding to the positive sample video frame.

[0136] Specifically, after obtaining the sample video, the computer device extracts positive sample video frames belonging to the sample co-occurring action from the sample video according to the corresponding sample label, and then extracts the corresponding sample common features from the sample video features corresponding to the positive sample video frames. Specifically, the computer device can perform weighted averaging processing on the sample video features corresponding to each positive sample video frame, such as calculating the mean of the sample video features corresponding to each positive sample video frame, to obtain a central feature representing the sample co-occurring action, that is, the sample common feature.

[0137] The computer device then generates predicted common features through the action detection model to be trained in this round, and predicts the predicted video segments belonging to the sample co-occurring actions in the sample video. The computer device also calculates the similarity difference between the predicted common features and the sample video features of the positive sample video frame. For example, the similarity difference between the predicted common features and the sample video features of the positive sample video frame can be calculated through a cross-entropy loss function, and the first feature loss is determined based on the difference.

[0138] The computer device also calculates the feature difference between the predicted common features and the sample common features and determines a second feature loss based on the feature difference. The computer device then adjusts model parameters of the action detection model based on the predicted loss, the first feature loss, and the second feature loss in a direction that reduces the predicted loss, the first feature loss, and the second feature loss, and continues training until a training stop condition is met.

[0139] In this embodiment, by calculating the first feature loss and the second feature loss, and combining the first feature loss and the second feature loss to adjust the model parameters of the action detection model, in the process of training the action detection model, when the action detection model generates predicted common features and predicts sample video frames belonging to sample co-occurring actions, the predicted common features are made more similar to the sample common features, and the predicted sample video frames belonging to sample co-occurring actions are made closer to the positive sample video frames and farther away from the negative sample video frames, thereby making the predicted common features generated by the action detection model more accurate, and thus effectively improving the recognition accuracy of the action detection model in identifying video clips belonging to co-occurring actions in the video.

[0140] In a specific embodiment, Figure 5 FIG2 is a schematic diagram of an action detection model to be trained in a specific embodiment. The action detection model includes an encoder 52, a similarity coding generator 54, and a parallel decoder 56. The encoder can be a co-token encoder (CTE), a similarity coding generator (SCG), and a parallel decoder.

[0141] The encoder 52 includes a multi-head attention layer 521 and a relation layer 522. The parallel decoder 56 includes a decoding layer 561, a regression branch network 562, and a classification branch network 563. The parallel decoder 56 also includes a preset number of query embedding vectors 56a.

[0142] Specifically, the computer device may first extract features from multiple sample videos to obtain corresponding sample video features. Figure 5 As shown, the computer device inputs the sample video features corresponding to sample video 5a, sample video 5b and sample video 5c into the action detection model to be trained, and then adds a tag feature 5f to the input sample video features. The tag feature 5f can specifically be a Co-token tag feature.

[0143] Among them, the Co-token encoder is configured with a token (marker) to gather the common features of co-occurring actions. This token is also encoded with certain knowledge during the training process, so part of the knowledge will be encoded into the weight corresponding to this token during pre-training. A dictionary is established by sorting the number of times the video features appear, and the video features of each video frame are mapped to a unique index number, and each index is mapped to a token. The multi-head attention layer 521 in the encoding layer 52 is used to generate predicted common features corresponding to each sample video feature, and the relationship layer 522 is used to fuse the predicted common features with the sample video features of each video frame respectively to obtain the fused sample video features corresponding to each video frame. The sample video features are encoded by the encoder 52 to gather the common features of the co-occurring actions corresponding to each sample video feature, and the common features of the co-occurring actions are encoded layer by layer into the Co-token tag features, so that the predicted common features try to be closer to the feature embedding of the sample video frames belonging to the co-occurring actions.

[0144] After encoder 52 collects the predicted common features corresponding to the common attributes of each sample video frame, it uses similarity code generator 54 to measure the relationship between each sample video frame and the predicted common features. Similarity code generator 54 is used to calculate the similarity between the predicted common features and the fused sample video features corresponding to each video frame, and select sample video features whose similarity meets the similarity proximity condition. For example, after sorting in descending order by similarity value, sample video features with similarity values ​​in the top-K are selected, i.e., the K sample video features with the highest similarity values. The similarity of the selected K sample video features is then calculated with the sample video features to generate predicted ordered similarity features.

[0145] The parallel decoder 56 jointly considers the features of each sample video and the ordered similarity features to determine the co-occurring action behaviors. The decoding layer 561 in the parallel decoder 56 decodes the features of each sample video and the ordered similarity features. Specifically, the parallel decoder 56 can use a preset number of query embedding vectors 56a to initiate a query to each sample video feature of each sample video segment, and use the regression branch 562 in the parallel decoder 56 to predict the start and end positions of the action based on the query results. The classification branch 563 in the parallel decoder 56 determines whether the time interval is a co-occurring action.

[0146] The computer first needs to obtain multiple sample videos to train the action detection model. The sample videos include multiple sample video clips that represent sample co-occurring actions. The sample videos carry sample labels, which include the location and category of the sample co-occurring actions in the sample videos.

[0147] The computer device may first perform feature extraction on the sample video to obtain sample video features corresponding to each video frame in the sample video.

[0148] Specifically, assume that the sample video feature representation of the input action detection model is:

[0149] Among them, F represents the video feature, F n represents the sample video features of the nth video; T and D are the temporal dimension and feature dimension for extracting sample video features. N represents the number of sample videos, all of which contain co-occurring actions.

[0150] The purpose of co-occurrence action detection is to detect video frames that belong to sample co-occurrence actions. Among them, M n is the number of co-occurring actions in the nth video. A represents the sample video clips of the co-occurring action, and M is the video clips in which the co-occurring action occurs M times in the Nth video.

[0151] First, all sample videos in a group are spliced ​​together in the time dimension through the encoder to obtain the video splicing feature F∈R TN×D .

[0152] During the training process, a learnable marker feature can be added to the side of the video splicing feature. It can be expressed as: [c, F] S ∈R (TN+1)×D .

[0153] Where c represents the marker feature; [.,.] S Indicates splicing in time sequence. T*N represents T videos, each with N frames, with TN+1 sample video features, and the dimension of each sample video feature includes D dimensions.

[0154] Then, the encoder adds a learnable position code to the sample video features corresponding to each video frame, and obtains the video splicing features after adding the position, which can be used as the final input representation E∈R (TN+1)×D .

[0155] First, the sample video features after adding the marked features of this round are encoded through the multi-head attention layer in the encoder. The specific expression can be:

[0156] MulAttn(X,Y)=[Attn(X,Y)1,...,Attn(X,Y) H ] E W head ;

[0157] Here, X is E. In the encoder, we can set X=Y=E, that is, X and Y both represent the sample video features of each video frame. E Represents splicing in the semantic dimension; W head ∈R (TN+1)×D , represents the learned fusion parameters. H represents the number of attention layers in the multi-head attention layer, Attn refers to the result obtained by each single-head attention layer; MulAttn represents the result obtained by multiple Attn.

[0158] Among them, the attention layer that makes up the multi-head attention can be expressed as:

[0159]

[0160] Among them, K represents the features of X after dimensionality reduction; V represents the features of Y after dimensionality reduction, and Q represents the query vector. key is the feature dimension of K. Specifically, it can be understood that the attention module mechanism can be regarded as a mapping from a query vector (Query) to a series of key-value pairs. In this embodiment, the token marking feature added in the encoder can be used as the query vector Q. The sample video features of the sample video are regarded as a series of<Key,Value> When calculating attention for data pairs, the Key and Value can be combined into one, pointing to the same sample video feature. Given a query vector Q for a target element, the similarity or correlation between the query vector Q and each Key is calculated to obtain the attention weight coefficient for each Key-to-Value pair. After normalization using the softmax function, the weighted sum of the attention weight coefficient and the corresponding Value is calculated to obtain the final attention value.

[0161] The attention mechanism essentially performs a weighted summation of the values ​​within a data pair. The query vector Q and the key are used to calculate the weight coefficient for the corresponding value. The attention mechanism selectively filters out a small amount of important information from a large amount of information and focuses on this important information, while ignoring the majority of less important information. This focusing process is reflected in the calculation of the weight coefficient. The larger the weight, the more focus is placed on the corresponding value. In other words, the weight represents the importance of the information, while the value represents the corresponding information.

[0162] Then, through the relation layer in the encoder, the predicted common features are fused with the sample video features corresponding to each video frame. Its expression can be:

[0163] Rela(c,F)=GELU([C,F] E W R1 +b R1 )WR2 +b R2 ;

[0164] Among them, GELU represents the GELU activation function, W and b are the parameters corresponding to the activation function, R1 and R2 represent the dimensions of W; Rela(c,F) represents the fused video splicing features.

[0165] The cyclic processing of the encoder in the action detection model can be expressed as follows:

[0166] E l-1 =[c l-1 ,F l-1 ] S ∈R (TN+1)×D (1);

[0167]

[0168]

[0169] Where l∈[1,2,...,L En ], L En Represents the number of layers of the encoder, and l represents the lth layer in the encoder. Through the continuous iterative processing of the model, c and F are continuously updated. The above formula (1) represents the position encoding of each sample video feature after splicing in time sequence and adding marker features to obtain the encoded sample video features. The above formula (2) represents the encoding processing of the encoded sample video features through the multi-head attention mechanism to generate predicted common features. The above formula (3) represents the fusion of the predicted common features with each sample video feature to obtain the fused sample video features. The encoder can effectively generate the predicted common features corresponding to the co-occurring actions, thereby obtaining the feature description of the co-occurring actions.

[0170] Next, a similarity code generator attempts to use the predicted common features to flag video frames that may represent co-occurring actions. First, the similarity between the sample video features of each video frame and the predicted common features is calculated, and the top K video frames with the highest similarity are selected. The ordered similarity features between these K video frames and each video frame are then calculated. For example, this can be a feature matrix corresponding to the ordered similarity values, which can range from 0 to 1. A K-dimensional similarity code is calculated for each frame.

[0171] Furthermore, the action detection model connects the ordered similarity features output by the similarity coding generator with the video splicing features output by the encoder and uses them as the input of the parallel decoder.

[0172] Specifically, we can first select the nearest K feature frames and record them as A∈RK×D A∈R K×D Where D is the feature dimension, and then compare the similarity between A and all other video frames The similarity code S can serve as a useful clue to support the decoder to find co-occurring actions. Then S is concatenated to the output of CET, that is, as input to the decoder.

[0173] The parallel decoder uses a fixed number of query embedding vectors (Co-action Query) to query each video. The query results are output through the classification branch to determine whether the output result is a co-occurring action, and the regression branch to obtain the left and right boundaries of the action interval. The specific method is as follows: First, the M output by the similarity code generator is converted to En ∈R TN×(D+K) , split into That is, the video features corresponding to all videos in a set are split into feature sets corresponding to each video, that is, split into N videos.

[0174] Then, through the parallel decoder, Q learnable query embedding vectors Q0∈R are introduced at each layer for query Q ×(D+K) According to the above description, the decoding process is finally expressed as a cycle of the following three steps:

[0175]

[0176]

[0177]

[0178] Where l∈[1,2,...,L De ]; L De represents the number of decoder layers; Q0 represents the initial query embedding vector; the above formula (4) represents the initialization of the query embedding vector in each layer through the self-attention mechanism of the decoding layer, M represents the query embedding vector for iteration in each layer. ln Represents the video features input in the parallel decoder; the above formula (5) represents the iterative query of the input video features in each layer using the query embedding vector through the cross attention mechanism of the decoding layer, Indicates the query results in each layer; D ln represents the predicted video clip of the sample co-occurrence action; FFN represents the fully connected layer; the above formula (6) represents the regression prediction of the query results obtained by querying each layer to predict the start and end positions of the action, and judges whether the predicted video clip in the time series interval is the sample co-occurrence action through the classification branch.

[0179] The fully connected layer can include a regression branch and a classification branch. The regression branch is used to predict the start and end locations of the action based on the query results, that is, the predicted video segments corresponding to the candidate actions. The classification branch classifies the predicted video segments corresponding to the candidate actions to identify the predicted video segments that belong to the sample co-occurring actions in the video segments corresponding to the candidate actions.

[0180] The computer device can splice the fused sample video features corresponding to each video frame and input the spliced ​​sample video features into the parallel decoder of the action detection model. Using the decoding layer and the corresponding query embedding vector in the parallel decoder, the similarity between the sample video features is queried within the spliced ​​sample video features. Sample video features whose similarity satisfies a similarity threshold are used to obtain the corresponding query results. A regression branch network is then used to predict the predicted video segments corresponding to the candidate actions in the sample videos based on the query results. The classification branch network then performs action classification on the sample video features corresponding to each predicted video segment, obtaining a predicted classification result for each predicted video segment.

[0181] During training, two losses are added to each encoder layer: a first feature loss and a second feature loss. The first feature loss can be a co-token loss, and the second feature loss can be a prototype loss. When calculating the first feature loss, the similarity between the predicted common features and each positive sample video frame is calculated, and a similarity loss is determined based on the similarity, resulting in the first feature loss. For true sample labels, during training, the similarity between the predicted common features and positive sample video frames belonging to the sample co-occurring action is kept close to 1, and the similarity with negative sample video frames is kept close to 0. The first feature loss is used to optimize the cross-entropy between the similarity and the true sample labels, striving to maximize the similarity between the predicted common features and the sample common features. When calculating the second feature loss, the computer device first obtains the center of the sample co-occurring action corresponding to each positive sample video frame, namely the sample common feature. Then, during training, the predicted common features belonging to the sample co-occurring action are positioned closer to the center of the sample co-occurring action and further away from the center of the negative sample video frames.

[0182] After the computer device further calculates the regression loss, classification loss, first feature loss, and second feature loss, it adjusts the model parameters of the encoder in the action detection model based on the first feature loss and the second feature loss, and adjusts the model parameters of the parallel decoder in the action detection model based on the regression loss and the classification loss, and continues training until the training stop condition is met. This can make the common features predicted by the trained action detection model more accurate, thereby effectively improving the action detection model's recognition accuracy of video clips belonging to co-occurring actions in the video.

[0183] In a specific test embodiment, the above-mentioned action detection method is used to perform co-occurrence action detection on videos in a preset data set. For example, the test can be performed on ActivityNet, Charades, and HACS data sets, and the start and end times of all co-occurrence actions are located in time sequence to obtain the corresponding test results. Figure 6 The figure shows the result of co-occurrence action detection on a test video in one embodiment. Figure 6 It can be seen that by performing action detection on the test video, the video clips corresponding to the co-occurring actions belonging to the "high jump" action category in the video are detected, that is, the time interval where the "high jump" action is located is detected. Figure 6 9 video clips detected as belonging to the co-occurring action "high jump" are shown in FIG. Among them, video frames 601 to 609, that is, video frames 1 to 9, are key frames corresponding to the 9 identified video clips.

[0184] Reference Figure 7 , Figure 7 The sample label 7a belonging to the sample co-occurrence action in the sample video is presented, and the Figure 6 The comparison diagram of the detection result 7b of the co-occurrence action detection of the test video in FIG. The sample label 7a and the detection result 7b respectively show the corresponding time intervals of the 9 video clips belonging to the co-occurrence action "high jump" in the test video corresponding to the video frames 1-9. Figure 7 As can be seen from the figure, the results of co-occurrence action detection for the test video are slightly different from the labeled sample labels. This shows that the above action detection method can accurately identify the video clips in the video that belong to co-occurrence actions by performing co-occurrence action detection on the video.

[0185] In a specific application scenario, the above action detection method can be applied to video search scenarios and video behavior recognition. Specifically, the computer device responds to the video search keywords input by the user, and obtains video search results that match the video search keywords from the corresponding database through the video search engine. Figure 8The figure shows a schematic diagram of a video search in one embodiment. A computer device can enter a video search keyword in the search box 802 of the search interface. The computer device then obtains a video set that matches the video search keyword and displays the video search results in the search interface. The video search results are a video set that includes at least one searched video. That is, the video search results can include videos from multiple scenes. Taking the video search keyword "back flip" as an example, video search results matching the video search keyword "back flip" are obtained and displayed in the search interface. The video search results include video 8a, video 8b, and video 8c.

[0186] In another specific embodiment, after obtaining a video collection that matches the video search keyword, the computer device can also perform co-occurrence action detection on the videos in the video collection to extract video clips that contain co-occurring actions. The extracted video clips that contain co-occurring actions are then displayed in the search interface.

[0187] The computer device then identifies video clips belonging to the co-occurring actions from the videos in the video collection. Specifically, the computer device first extracts the video features corresponding to each video frame in the video, and generates common features of the co-occurring actions based on the video features corresponding to each video frame. Then, from the video features of each video frame arranged in time sequence in the video, the computer device selects video features that meet the similarity close condition with the common features to obtain video features belonging to the co-occurring actions. Then, based on the similarity between the video features of each video frame and the video features belonging to the co-occurring actions, the computer device generates ordered similarity features. The computer device then identifies video clips belonging to the co-occurring actions in the video based on the ordered similarity features and the video features of each video frame.

[0188] like Figure 9 As shown, in one embodiment Figure 8 Results of co-occurrence action detection in videos. Figure 9 Figure 1 shows the co-occurring action detection results for Videos 8a, 8b, and 8c. The detection results are the video segments that are marked with detection boxes as belonging to the co-occurring action. Video segments 9T1 and 9T2 in Video 8a, 9T3 in Video 8b, and 9T4 in Video 8c are identified as belonging to the co-occurring action "backflip."

[0189] In one embodiment, the video clips identified as co-occurring actions can be further used to construct training datasets for downstream tasks such as action classification and action localization, thereby training task models for the corresponding tasks. This allows for rapid construction of training datasets, increases the responsiveness of downstream tasks, and effectively reduces the cost of manually labeling sample labels in the training datasets.

[0190] In another specific application scenario, the above-mentioned action detection method can be applied to a video behavior recognition scenario. Specifically, for the safety of the elderly and children, some families will spontaneously use equipment to monitor their own home scene areas. The computer device can obtain the collected monitoring video. The monitoring video may include some background frames and video frames with action. Among them, the background frame may refer to a video frame with a continuous static picture state. The video frame with action may refer to a video frame with a continuous action state. For example, the monitoring video may be a video collected for the corresponding scene area, and the monitoring video may include a static picture of the scene area, and may also include a dynamic picture with action.

[0191] After the computer device obtains the monitoring video, it identifies the video clips belonging to the co-occurring action from the video in the video set. Specifically, the computer device first extracts the video features corresponding to each video frame in the video, and generates the common features of the co-occurring action based on the video features corresponding to each video frame. Then, from the video features of each video frame arranged in time sequence in the video, the video features that meet the similarity close condition with the common features are screened to obtain the video features belonging to the co-occurring action, and then the ordered similarity features are generated based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action. The computer device then identifies the video clips belonging to the co-occurring action in the video based on the ordered similarity features and the video features of each video frame. For example, in a home scene area, if the elderly and children fall down, then the above-mentioned action detection method can accurately identify the video clips belonging to the falling action, so that the situation of the elderly and children falling down can be accurately monitored, thereby improving safety.

[0192] It should be understood that although Figure 2-Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2-Figure 3 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0193] In one embodiment, Figure 10As shown, an action detection device 1000 is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a common feature generation module 1002, a video feature screening module 1004, a similar feature generation module 1006 and a co-occurrence action detection module 1008, wherein:

[0194] The common feature generation module 1002 is used to obtain common features of co-occurring actions extracted from the video to be detected; co-occurring actions are the same type of actions that appear in different video frames of the video to be detected.

[0195] The video feature screening module 1004 is used to screen video features that meet a similarity close condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtain video features belonging to the co-occurring action.

[0196] The similarity feature generating module 1006 is configured to generate ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring actions.

[0197] The co-occurrence action detection module 1008 is configured to identify video segments belonging to co-occurrence actions in the video to be detected based on the ordered similarity features and the video features of each video frame.

[0198] In one embodiment, the common feature generation module 1002 is also used to splice the video features corresponding to each video frame in the video to be detected in time sequence to obtain a video splicing feature; add position information to each video feature in the video splicing feature; encode the video splicing feature with added position information to generate common features of co-occurring actions in the video to be detected.

[0199] In one embodiment, the common feature generation module 1002 is also used to determine the attention weight corresponding to each video feature based on the correlation between the video features with added position information; the attention weight is positively correlated with the correlation; the corresponding video features are encoded based on the attention weight to generate common features corresponding to the co-occurring actions in the video to be detected.

[0200] In one embodiment, the video feature screening module 1004 is further used to fuse the common features with the video features of each video frame arranged in time sequence in the video to be detected to obtain fused video features; determine the similarity between each fused video feature and the common features; and screen video features that meet the similarity close conditions with the common features from the video features of each video frame according to the similarity to obtain video features belonging to co-occurring actions.

[0201] In one embodiment, the co-occurrence action detection module 1008 is also used to predict the video segments corresponding to the candidate actions in the video to be detected based on the target similarity between the video features of each video frame and the ordered similarity features; and identify the video segments belonging to the co-occurrence action from the video segments corresponding to the candidate actions based on the classification results obtained by performing action classification on the video features in the video segments corresponding to the candidate actions.

[0202] In one embodiment, the co-occurrence action detection module 1008 is further used to splice the ordered similarity features and the video features of each video frame to obtain the spliced ​​video features; determine the target similarity between the video features of each video frame based on the spliced ​​video features; and predict the video clip corresponding to the candidate action in the video to be detected based on the position information corresponding to the video features whose target similarity reaches the similarity threshold.

[0203] In one embodiment, the co-occurrence action detection module 1008 is also used to query the target similarity between the video features of each spliced ​​video frame in the spliced ​​video features to obtain the query results; based on the query results, the starting position and ending position corresponding to the candidate action in the video to be detected are predicted to obtain the video clip corresponding to the candidate action.

[0204] In one embodiment, the action detection method is performed by an action detection model; the action detection model is obtained by training through a training step; the above-mentioned action detection device also includes a model training module for obtaining sample video features corresponding to each video frame in the sample video; the sample video carries a sample label, and the sample label includes the position and category of the sample co-occurring action in the sample video; in each round of iterative training, the marked features of this round are added to the sample video features after splicing in time sequence, and the sample video features after adding the marked features of this round are encoded through the encoding layer of the action detection model to be trained, and a prediction of the sample co-occurring action in the sample video is generated. common features are measured; through the similarity coding generator of the action detection model, the sample video features in each sample video feature that meet the similarity close conditions with the predicted common features are respectively fused into the features of each sample video to obtain the predicted ordered similarity features; through the parallel decoder of the action detection model, the predicted video clips belonging to the sample co-occurring actions in the sample video are predicted according to the sample ordered similarity features and the features of each sample video; based on the difference between the predicted video clips and the sample labels, the prediction result loss is determined; according to the prediction result loss, the model parameters of the action detection model are adjusted and training is continued until the training stop condition is met.

[0205] In one embodiment, the model training module is also used to splice the sample ordered similarity features with each sample video feature in the semantic dimension to obtain the spliced ​​sample video features; through the parallel decoder in the action detection model, the predicted video segment corresponding to the candidate action in the sample video is predicted based on the spliced ​​sample video features; the sample video features corresponding to the predicted video segment are action classified to obtain the predicted classification results.

[0206] In one embodiment, the prediction result loss includes regression loss and classification loss; the model training module is also used to determine the regression loss based on the difference between the predicted video clip and the position of the co-occurring action in the sample label; and determine the classification loss based on the difference between the classification result of the predicted video clip and the category of the sample co-occurring action of the sample label.

[0207] In one embodiment, the model training module is also used to extract positive sample video frames belonging to co-occurring actions from the sample video according to the sample labels, and extract corresponding sample common features from the sample video features corresponding to the positive sample video frames; determine the first feature loss based on the difference between the predicted common features and the sample video features of the positive sample video frames; determine the first feature loss based on the difference between the predicted common features and the sample common features; adjust the model parameters of the action detection model according to the prediction result loss, the first feature loss and the second feature loss and continue training until the training is terminated when the training stop condition is met.

[0208] For the specific definition of the motion detection device, please refer to the definition of the motion detection method above and will not be repeated here. Each module in the above-mentioned motion detection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0209] In one embodiment, a computer device is provided, which can be a server or a terminal. The internal structure diagram can be as follows: Figure 11 As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a motion detection method.

[0210] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0211] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0212] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0213] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0214] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0215] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0216] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A motion detection method, characterized in that: The method comprises: Obtaining common features of co-occurring actions in a video to be detected; the co-occurring actions are actions of the same type that appear in different video frames of the video to be detected; Selecting video features that meet a close similarity condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtaining video features belonging to the co-occurring action; generating ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action; According to the ordered similarity features and the video features of each of the video frames, a video segment belonging to the co-occurring action in the video to be detected is identified.

2. The method according to claim 1, characterized in that The obtaining of common features of co-occurring actions in the video to be detected includes: Splicing the video features corresponding to each video frame in the video to be detected in time sequence to obtain video splicing features; Adding position information to each video feature in the video splicing feature; The video splicing features with added position information are encoded to generate common features of the co-occurring actions in the video to be detected.

3. The method according to claim 2, characterized in that The encoding process of the video splicing features with added position information to generate common features of the co-occurring actions in the video to be detected includes: Determining an attention weight corresponding to each video feature according to the correlation between the video features with added position information; wherein the attention weight is positively correlated with the correlation; The corresponding video features are encoded based on the attention weights to generate common features corresponding to the co-occurring actions in the video to be detected.

4. The method according to claim 1, wherein The step of selecting video features that meet a close similarity condition with the common features from the video features of each video frame arranged in time sequence in the video to be detected, and obtaining video features belonging to the co-occurring action, includes: Fusing the common features with the video features of each video frame arranged in time sequence in the video to be detected to obtain fused video features; respectively determining the similarity between each fused video feature and the common feature; According to the similarity, video features that meet a similarity close condition with the common feature are screened from the video features of each of the video frames to obtain video features belonging to the co-occurring action.

5. The method according to claim 1, wherein The step of identifying the video segments belonging to the co-occurring action in the video to be detected based on the ordered similarity features and the video features of each of the video frames includes: Predicting a video segment corresponding to a candidate action in the video to be detected based on a target similarity between the video feature of each of the video frames and the ordered similarity feature; Based on a classification result obtained by performing action classification on the video features in the video segments corresponding to the candidate actions, video segments belonging to the co-occurring action are identified from the video segments corresponding to the candidate actions.

6. The method according to claim 5, characterized in that The predicting, based on the target similarity between the video features of each of the video frames and the ordered similarity features, a video segment corresponding to the candidate action in the video to be detected, includes: Splicing the ordered similarity features and the video features of each of the video frames to obtain spliced ​​video features; Determine the target similarity between the video features of each video frame according to the spliced ​​video features; Predicting a video segment corresponding to the candidate action in the video to be detected based on the position information corresponding to the video feature whose target similarity reaches a similarity threshold.

7. The method according to claim 6, characterized in that The determining of the target similarity between the video features of each video frame according to the spliced ​​video features includes: Querying the target similarity between the video features of each spliced ​​video frame in the spliced ​​video features to obtain a query result; The predicting, based on the position information corresponding to the video features whose target similarity reaches the similarity threshold, the video segment corresponding to the candidate action in the video to be detected includes: Based on the query result, the starting position and the ending position corresponding to the candidate action in the video to be detected are predicted to obtain a video clip corresponding to the candidate action.

8. The method according to any one of claims 1 to 7, characterized in that The motion detection method is performed by a motion detection model; the motion detection model is obtained by training in a training step; The training step includes: Obtaining sample video features corresponding to each video frame in the sample video; the sample video carries a sample label, and the sample label includes the position and category of the sample co-occurrence action in the sample video; In each round of iterative training, the marked features of this round are added to the sample video features after splicing in time sequence. The sample video features after adding the marked features of this round are encoded through the encoding layer of the action detection model to be trained to predict and generate the predicted common features of the sample co-occurring actions in the sample video; The similarity code generator of the action detection model selects sample video features that meet a similarity close condition with the predicted common features from each of the sample video features to obtain sample video features belonging to the sample co-occurring action; and generates a predicted ordered similarity feature based on the similarity between the sample video features of each video frame and the sample video features belonging to the sample co-occurring action; Predicting, by a parallel decoder of the action detection model, a predicted video segment belonging to a sample co-occurring action in the sample video based on the predicted ordered similarity features and each of the sample video features; Determining a prediction result loss based on a difference between the predicted video segment and the sample label; According to the prediction result loss, the model parameters of the action detection model are adjusted and training is continued until the training is terminated when a training stop condition is met.

9. The method according to claim 8, characterized in that The method of predicting a predicted video segment belonging to a sample co-occurring action in the sample video by using the parallel decoder of the action detection model according to the predicted ordered similarity feature and each of the sample video features includes: Splicing the predicted ordered similarity features with each of the sample video features to obtain spliced ​​sample video features; Predicting, by means of a parallel decoder in the action detection model, a predicted video segment corresponding to a candidate action in the sample video according to the features of the spliced ​​sample video; Action classification is performed on the sample video features corresponding to the predicted video clip to obtain a prediction classification result.

10. The method according to claim 9, characterized in that The prediction result loss includes regression loss and classification loss; and determining the prediction result loss based on the difference between the predicted video segment and the sample label includes: determining a regression loss based on a difference between a predicted position of the predicted video segment and a position in the sample label belonging to a sample co-occurring action; A classification loss is determined based on a difference between a predicted classification result of the predicted video segment and a category of a sample co-occurring action of the sample label.

11. The method according to claim 8, characterized in that The method further comprises: Extracting positive sample video frames belonging to sample co-occurrence actions from the sample video according to the sample labels, and extracting corresponding sample common features from sample video features corresponding to the positive sample video frames; determining a first feature loss based on a similarity difference between the predicted common feature and the sample video feature of the positive sample video frame; determining a second feature loss based on a difference between the predicted common feature and the sample common feature; The step of adjusting the model parameters of the action detection model according to the prediction result loss and continuing the training until the training stop condition is met includes: According to the prediction result loss, the first feature loss and the second feature loss, the model parameters of the action detection model are adjusted and training is continued until the training is terminated when a training stop condition is met.

12. A motion detection device, characterized in that: The device comprises: A common feature generation module is used to obtain common features of co-occurring actions in the video to be detected; the co-occurring actions are the same type of actions that appear in different video frames of the video to be detected; A video feature screening module is used to screen video features that meet a similarity close condition with the common feature from the video features of each video frame arranged in time sequence in the video to be detected, and obtain video features belonging to the co-occurring action; A similarity feature generation module, configured to generate ordered similarity features based on the similarity between the video features of each video frame and the video features belonging to the co-occurring action; The co-occurrence action detection module is used to identify the video segments belonging to the co-occurrence action in the video to be detected based on the ordered similarity features and the video features of each of the video frames.

13. The device according to claim 12, characterized in that The common feature generation module is also used to splice the video features corresponding to each video frame in the video to be detected in time sequence to obtain a video splicing feature; add position information to each video feature in the video splicing feature; encode the video splicing feature with added position information to generate a common feature of the co-occurring action in the video to be detected.

14. The device according to claim 13, characterized in that The common feature generation module is also used to determine the attention weight corresponding to each video feature based on the correlation between the video features with added position information; the attention weight is positively correlated with the correlation; based on the attention weight, the corresponding video features are encoded to generate common features corresponding to the co-occurring actions in the video to be detected.

15. The device according to claim 12, characterized in that The video feature screening module is further configured to fuse the common features with the video features of each video frame arranged in time sequence in the video to be detected to obtain fused video features; The similarity between each fused video feature and the common feature is determined respectively; and video features that meet a similarity close condition with the common feature are screened from the video features of each video frame according to the similarity to obtain video features belonging to the co-occurring action.

16. The device according to claim 12, characterized in that The co-occurrence action detection module is also used to predict the video segments corresponding to the candidate actions in the video to be detected based on the target similarity between the video features of each video frame and the ordered similarity features; and identify the video segments belonging to the co-occurrence action from the video segments corresponding to the candidate actions based on the classification results obtained by performing action classification on the video features in the video segments corresponding to the candidate actions.

17. The device according to claim 16, characterized in that The co-occurrence action detection module is also used to splice the ordered similarity features and the video features of each video frame to obtain spliced ​​video features; determine the target similarity between the video features of each video frame based on the spliced ​​video features; and predict the video clip corresponding to the candidate action in the video to be detected based on the position information corresponding to the video features whose target similarity reaches the similarity threshold.

18. The device according to claim 17, characterized in that The co-occurrence action detection module is also used to query the target similarity between the video features of each spliced ​​video frame in the spliced ​​video features to obtain a query result; based on the query result, the starting position and ending position corresponding to the candidate action in the video to be detected are predicted to obtain a video clip corresponding to the candidate action.

19. The device according to claim 12, characterized in that The device also includes a model training module, which is used to obtain sample video features corresponding to each video frame in the sample video; The sample videos carry sample labels, which include the positions and categories of sample co-occurring actions in the sample videos. In each round of iterative training, the marked features of this round are added to the sample video features spliced ​​in time sequence. The sample video features after adding the marked features of this round are encoded by the encoding layer of the action detection model to be trained to predict and generate predicted common features of the sample co-occurring actions in the sample videos. The similarity encoding generator of the action detection model is used to screen sample video features from each of the sample video features that meet a similarity close condition with the predicted common features to obtain sample video features belonging to the sample co-occurring actions. According to the similarity between the sample video features of each video frame and the sample video features belonging to the sample co-occurring action, a predicted ordered similarity feature is generated; through the parallel decoder of the action detection model, a predicted video segment belonging to the sample co-occurring action in the sample video is predicted based on the predicted ordered similarity feature and each of the sample video features; based on the difference between the predicted video segment and the sample label, the prediction result loss is determined; according to the prediction result loss, the model parameters of the action detection model are adjusted and training is continued until the training stop condition is met.

20. The device according to claim 19, characterized in that The model training module is also used to splice the predicted ordered similarity features with each of the sample video features to obtain spliced ​​sample video features; through the parallel decoder in the action detection model, predict the predicted video segment corresponding to the candidate action in the sample video based on the spliced ​​sample video features; perform action classification on the sample video features corresponding to the predicted video segment to obtain a predicted classification result.

21. The device according to claim 20, characterized in that The prediction result loss includes regression loss and classification loss; the model training module is also used to determine the regression loss based on the difference between the predicted position of the predicted video clip and the position belonging to the sample co-occurrence action in the sample label; and determine the classification loss based on the difference between the predicted classification result of the predicted video clip and the category of the sample co-occurrence action of the sample label.

22. The device according to claim 19, characterized in that The model training module is further configured to extract positive sample video frames belonging to sample co-occurrence actions from the sample video according to the sample labels, and extract corresponding sample common features from the sample video features corresponding to the positive sample video frames; Based on the similarity difference between the predicted common features and the sample video features of the positive sample video frame, a first feature loss is determined; based on the difference between the predicted common features and the sample common features, a second feature loss is determined; according to the prediction result loss, the first feature loss and the second feature loss, the model parameters of the action detection model are adjusted and training is continued until the training is terminated when the training stop condition is met.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images