Motion detection method and device, storage medium and electronic device

Abstract

The present application discloses an action detection method and device, a storage medium and an electronic device. The method includes: obtaining a target image sequence and target text information; inputting the target image sequence and target text information into a pre-trained target detection model to obtain a target detection result, wherein the target detection result includes a target detection frame, a target action category and a target confidence, and the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and fuse the target fusion feature and the target image sequence and the target text information through a cross-attention mechanism to obtain a target multimodal feature, and determine the target detection frame, the target action category and the target confidence according to the target multimodal feature. The present application solves the technical problem of low accuracy of action detection.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a motion detection method and device, a storage medium, and an electronic device. Background Art

[0002] At present, when detecting actions appearing in videos and other files, the related technology usually adopts multiple detection models to be used in combination in sequence to obtain action detection results. Furthermore, since different models need to use different feature extraction methods, etc., the generalization ability of the detection model is insufficient in the process of action positioning and recognition. That is, the detection model performs poorly on unknown data and cannot achieve the complete accuracy and stability of a single model, which makes action detection prone to misjudgment, missed judgment, etc. In summary, there is a technical problem of low accuracy of action detection in the related technology.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a motion detection method and device, a storage medium, and an electronic device to at least solve the technical problem of low accuracy of motion detection.

[0005] According to one aspect of an embodiment of the present application, a method for action detection is provided, comprising: obtaining a target image sequence and target text information, wherein the target image sequence comprises an image to be subjected to action detection, and the target text information is used to specify an action category to be detected from the target image sequence; inputting the target image sequence and the target text information into a pre-trained target detection model to obtain a target detection result, wherein the target detection result comprises a target detection frame, a target action category, and a target confidence, the target detection frame is used to indicate an image area detected from the target image sequence, the target action category is used to indicate an action category corresponding to the target detection frame, and the target confidence is used to indicate a degree of matching between the target action category and the target text information, the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and determine the target detection frame, the target action category, and the target confidence according to the target multimodal feature.

[0006] According to another aspect of an embodiment of the present application, there is also provided an action detection device, comprising: an acquisition module, for acquiring a target image sequence and target text information, wherein the target image sequence comprises an image to be subjected to action detection, and the target text information is used to specify an action category to be detected from the target image sequence; a detection module, for inputting the target image sequence and the target text information into a pre-trained target detection model to obtain a target detection result, wherein the target detection result comprises a target detection frame, a target action category, and a target confidence, wherein the target detection frame is used to indicate an image area detected from the target image sequence, the target action category is used to indicate an action category corresponding to the target detection frame, and the target confidence is used to indicate a degree of matching between the target action category and the target text information, and the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and to fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and to determine the target detection frame, the target action category, and the target confidence according to the target multimodal feature.

[0007] Optionally, the device is used to input the target image sequence and the target text information into a pre-trained target detection model to obtain a target detection result in the following manner: performing feature extraction on the target text information and the target image sequence respectively to obtain initial image sequence features and initial text features; performing feature fusion on the initial image sequence features and the initial text features based on the cross-attention mechanism to obtain target adaptive text features and target image sequence features; performing matrix multiplication operations on the target adaptive text features and the target image sequence features to obtain the target fusion features; performing sorting operations on the values ​​of each dimension in the target fusion features, and determining, based on the sorting results, a target query feature whose value meets a preset condition; performing feature fusion on the target query feature, the target image sequence features and the target adaptive text features in turn based on the cross-attention mechanism to obtain the target multimodal feature; determining the target detection box, the target action category and the target confidence based on the target multimodal feature to generate the target detection result.

[0008] Optionally, the device is used to perform feature fusion on the initial image sequence features and the initial text features based on the cross-attention mechanism to obtain target adaptive text features and target image sequence features in the following manner: processing the initial image sequence features based on the self-attention mechanism to obtain self-attention image sequence features; adding the values ​​of the self-attention image sequence features and the initial text features in each dimension to obtain initial adaptive text features; performing feature fusion on the initial image sequence features and the initial adaptive text features based on the cross-attention mechanism to obtain the target adaptive text features and the target image sequence features.

[0009] Optionally, the device is used to perform feature fusion on the initial image sequence features and the initial adaptive text features based on the cross-attention mechanism to obtain the target adaptive text features and the target image sequence features in the following manner: based on the self-attention mechanism, the initial image sequence features and the initial adaptive text features are respectively processed to obtain a first image sequence feature and a first adaptive text feature; the first image sequence feature is determined as a first query vector, the first adaptive text feature is determined as a first key vector and a first value vector, and a second image sequence feature and a second adaptive text feature are determined based on the cross-attention mechanism; the second adaptive text feature is determined as a second query vector, the second image sequence feature is determined as a second key vector and a second value vector, and a third image sequence feature and a third adaptive text feature are determined based on the cross-attention mechanism; the third image sequence feature and the third adaptive text feature are input into a fully connected feed-forward network module to obtain the target adaptive text feature and the target image sequence feature.

[0010] Optionally, the device is used to perform feature fusion on the target query feature and the target image sequence feature and the target adaptive text feature in sequence based on the cross-attention mechanism in the following manner to obtain the target multimodal feature: processing the target query feature based on the self-attention mechanism to obtain a self-attention query feature; determining the self-attention query feature as a third query vector, the target image sequence feature as a third key vector and a third value vector, and determining a first cross-attention query feature based on the cross-attention mechanism; determining the first cross-attention query feature as a fourth query vector, the target adaptive text feature as a fourth key vector and a fourth value vector, and determining a second cross-attention query feature based on the cross-attention mechanism; inputting the second cross-attention query feature into a fully connected feedforward network module to obtain the target multimodal feature.

[0011] Optionally, the device is used to perform a sorting operation on the values ​​of each dimension in the target fusion feature in the following manner, and determine the target query feature whose value meets a preset condition based on the sorting result: obtain the value of the target fusion feature in each dimension, wherein the value of the target fusion feature in each dimension represents the similarity between the initial image sequence feature and the initial text feature in each dimension; perform the sorting operation on the values ​​in each dimension, and determine the dimensions where the K maximum values ​​are located based on the sorting result, and determine the target query feature based on the feature values ​​of the dimensions where the K maximum values ​​are located, wherein K is a positive integer.

[0012] Optionally, the device is also used to: obtain a sample image sequence, sample text information and detection frame annotation information, wherein the sample image sequence includes a sample image to be predicted, the sample text information is used to specify an action category to be predicted from the sample image sequence, and the detection frame annotation information is used to indicate whether there is an image area of ​​a preset action category in the sample image sequence, and whether the sample image sequence matches the sample text information; input the sample image sequence and the sample text information into an initial detection model to obtain a sample detection result, wherein the sample detection result includes a sample detection frame, a sample action category and a sample confidence, the sample detection frame is used to indicate an image area detected from the sample image sequence, the sample action category is used to indicate the action category corresponding to the sample detection frame, and the sample confidence is used to indicate the degree of matching between the sample action category and the sample text information; calculate a target loss value based on the detection frame annotation information and the sample detection result, and train the initial detection model based on the target loss value to obtain the target detection model.

[0013] Optionally, the device is used to calculate a target loss value according to the detection box annotation information and the sample detection result in the following manner, and train the initial detection model according to the target loss value to obtain the target detection model: calculate a first loss value, a second loss value and a third loss value according to the detection box annotation information and the sample detection result, wherein the first loss value is determined by the predicted probability of the sample action category and the annotated action category indicated by the detection box annotation information, the second loss value is determined by the size and position information of the sample detection box and the size and position information of the annotated detection box indicated by the detection box annotation information, and the third loss value is determined by the sample confidence and whether the sample image sequence indicated by the detection box annotation information matches the sample text information; the first loss value, the second loss value and the third loss value are superimposed according to preset weights to obtain the target loss value, and the network parameters of the initial detection model are iteratively updated through a back propagation algorithm according to the target loss value to obtain the target detection model.

[0014] Optionally, the device is used to calculate a target loss value according to the detection box annotation information and the sample detection result in the following manner, and train the initial detection model according to the target loss value to obtain the target detection model: perform feature extraction on the sample text information and the sample image sequence respectively to obtain initial sample image sequence features and initial sample text features; perform feature fusion on the initial sample image sequence features and the initial sample text features based on the cross-attention mechanism to obtain sample adaptive text features and sample image sequence features; perform matrix multiplication operation on the sample adaptive text features and the sample image sequence features to obtain the sample fusion features; perform sorting operation on the values ​​of each dimension in the sample fusion features, and determine the sample query features whose values ​​meet the preset conditions according to the sorting results; perform feature fusion on the sample query features with the sample image sequence features and the sample adaptive text features in turn based on the cross-attention mechanism to obtain the sample multimodal features; determine the sample detection box, the sample action category and the sample confidence according to the sample multimodal features to generate the sample detection result.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned action detection method when running.

[0016] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above action detection method.

[0017] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the motion detection method through the computer program.

[0018] In an embodiment of the present application, a target image sequence and target text information are obtained, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used to specify the action category to be detected from the target image sequence; the target image sequence and the target text information are input into a pre-trained target detection model to obtain a target detection result, wherein the target detection result includes a target detection box, a target action category, and a target confidence, wherein the target detection box is used to indicate an image area detected from the target image sequence, the target action category is used to indicate the action category corresponding to the target detection box, and the target confidence is used to indicate the matching degree between the target action category and the target text information. The target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain the target fusion feature, and the target fusion feature and the target image sequence and the target text information are respectively fused through the cross-attention mechanism to obtain the target multimodal feature, and the target detection frame, target action category and target confidence are determined according to the target multimodal feature, thereby achieving the purpose of detecting actions using an end-to-end target detection model, improving the generalization ability of the target detection model, and achieving the technical effect of improving the accuracy and robustness of action detection, thereby solving the technical problem of low accuracy of action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of an application environment of an optional motion detection method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of an optional motion detection method according to an embodiment of the present application;

[0022] Figure 3is a schematic diagram of an optional motion detection method according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of another optional motion detection method according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of another optional motion detection method according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of another optional motion detection method according to an embodiment of the present application;

[0026] Figure 7 is a schematic structural diagram of an optional motion detection device according to an embodiment of the present application;

[0027] Figure 8 is a schematic structural diagram of an optional motion detection product according to an embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The present application is described below in conjunction with embodiments:

[0032] According to one aspect of an embodiment of the present application, a motion detection method is provided. Optionally, in this embodiment, the motion detection method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1 As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or the application installed on the terminal device. The application 107 can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The terminal device 103 may be a terminal configured with an application, and may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (Virtual Reality, VR for short) terminal, an augmented reality (Augmented Reality, AR for short) terminal, a mixed reality (Mixed Reality, MR for short) terminal and other computer devices. The above-mentioned server may be a single server, or a server cluster consisting of multiple servers, or a cloud server.

[0033] Combination Figure 1 As shown, the above-mentioned action detection method can be executed by an electronic device, which can be a terminal device or a server. The above-mentioned action detection method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0034] The above is only an example and is not specifically limited in this embodiment.

[0035] Optionally, as an optional implementation, as Figure 2 As shown, the above action detection method includes:

[0036] S202, obtaining a target image sequence and target text information, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used to specify an action category to be detected from the target image sequence;

[0037] Optionally, in an embodiment of the present application, the above-mentioned target image sequence can be understood as a group of sequence data composed of a number of consecutive image frames, which may include but is not limited to representing videos, animations, etc.; the above-mentioned target text information may include but is not limited to action categories, object categories, scene categories, character identities, etc. that can be recognized and expected to be recognized in the target image sequence.

[0038] Exemplarily, the above-mentioned target image sequence is a video, and each frame in the video can be a still image, that is, the above-mentioned image to be subjected to action detection. Taking the action category expected to be recognized in the target text information as an example, it can include but is not limited to walking, running, jumping, bending over, raising hands, etc.

[0039] S204, input the target image sequence and the target text information into the pre-trained target detection model to obtain the target detection result, wherein the target detection result includes a target detection frame, a target action category and a target confidence, the target detection frame is used to indicate the image area detected from the target image sequence, the target action category is used to indicate the action category corresponding to the target detection frame, the target confidence is used to indicate the degree of matching between the target action category and the target text information, the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and determine the target detection frame, the target action category and the target confidence according to the target multimodal feature.

[0040] It should be noted that in the above-mentioned target detection model, it may include but is not limited to a text-image cross-attention network module, a text cross-attention network module, an image-text cross-attention network module, an image cross-attention network module, etc. These modules may include but are not limited to a self-attention mechanism, a multi-head attention mechanism, an attention weight calculation, a cross-modal information fusion, an information transfer mechanism, a loss function design, etc., wherein the self-attention mechanism is used to calculate the correlation between each input element and other elements, and adjust the element representation according to these correlations; the multi-head attention mechanism divides the input into multiple parts, calculates the correlations of different attention heads respectively, and merges them together; the attention weight calculation obtains the correlation weight between each element and other elements through learning, which is used to adjust the element representation; the cross-modal information fusion fuses the information of text and image so as to effectively transfer and interact between different modalities; the information transfer mechanism transfers and interacts the information of different modalities through the cross-attention network module so as to realize cross-modal information fusion and representation learning; the loss function design designs a suitable loss function according to the requirements of the task to guide the training and optimization of the target detection model.

[0041] Optionally, in an embodiment of the present application, the above-mentioned target detection box refers to the area in the target image sequence where the action can be determined to occur; the above-mentioned target action category refers to the category of the action detected from the target image sequence, for example, walking, running, etc.; the above-mentioned target confidence refers to the degree of confidence of the target detection model in the above-mentioned target detection result, that is, the probability that the action identified by the target detection result is a real action can be obtained through the target confidence.

[0042] For example, Figure 3 is a schematic diagram of an optional action detection method according to an embodiment of the present application. The overall framework flow chart of the above target detection model can be as follows Figure 3 As shown, it includes: feature extraction module, feature fusion module, query feature generation module, feature decoding module, regression classification module, etc.

[0043] It should be noted that, based on the above-mentioned target detection model proposed in the present application, several action categories can be detected from the above-mentioned target image sequence. The action category can be an action category that has appeared in the training set during the training process of the above-mentioned target detection model, or an action category that has not appeared in the training set. That is, in an embodiment of the present application, only the above-mentioned target text information can be used to detect action categories that do not appear in the training set, so as to improve the scalability, robustness and portability of action detection.

[0044] In an exemplary embodiment, the present application does not limit the number of the above target action categories. For example, the above target detection result is expressed as:

[0045] {Target action category 1: A1, target detection frame 1: B1, target confidence 1: C1;

[0046] Target action category 2: A2, target detection box 2: B2, target confidence 2: C2}.

[0047] Furthermore, if the above target text information indicates that the action category to be detected from the target image sequence is "running", target action category 1 is "running", target detection box 1: B1, target confidence 1: 98%; target action category 2 is "walking", target detection box 2: B1, target confidence 2: 50%, it can be obtained that the matching degree between target action category 1 and the target text information is higher than the matching degree between target action category 2 and the target text information.

[0048] Exemplarily, after the target image sequence and target text information are input into the above target detection model, as shown in Figure 3As shown, the target image sequence and the target text information are first encoded respectively, and feature extraction is performed. Then, the encoded text features and image sequence features are used to generate adaptive text features. Furthermore, the image sequence features and the adaptive text features are fused as described above to obtain more comprehensive and detailed target fusion features. That is, the target fusion features contain information of the target image sequence and the target text information, and after feature extraction and fusion processing, they can better express the semantic and visual information of the target.

[0049] For example, after generating the above target fusion features, the values ​​of the target fusion features can be sorted. Figure 3 In the query feature generation module shown in FIG, the K maximum value features contained in the target fusion feature are determined as query features, and then, Figure 3 In the feature decoding module shown, the query feature is input into the self-attention network, and then the target multimodal feature is generated based on the output result of the self-attention network and the target fusion feature.

[0050] It should be noted that the value of the above-mentioned target fusion feature refers to the specific value of the target fusion feature in the feature space, and the purpose of sorting is to: contain the most important information contained in the target fusion feature, that is, the K features with the largest values, so as to use these K features to better express the semantic and visual information of the target image sequence and the target text information.

[0051] Further, in Figure 3 In the feature decoding module shown, the query feature is input into the self-attention network to generate the above-mentioned target multimodal feature. It should be noted that the value of the above-mentioned target fusion feature refers to the specific value of the target fusion feature in the feature space, and the purpose of sorting is to obtain the most important information contained in the target fusion feature, that is, the K features with the largest values ​​(query features), so as to use these K features to better express the semantics and visual information of the target image sequence and the target text information.

[0052] It should also be noted that the above-mentioned target multimodal features refer to the feature representation formed by fusion of features extracted from different modalities (the above-mentioned target image sequence and the above-mentioned target text information). Specifically, Figure 3 As shown, the target multimodal features may be obtained by processing in the feature decoding module in the target detection model in the following manner, including but not limited to:

[0053] S1, input the generated query features into the self-attention network in the feature decoding module to generate features ;

[0054] S2, the feature As Q, the image sequence features in the above target fusion features are used as K, V is input into the image cross attention network in the feature decoding module, and the generated features are , where Q represents the query vector (Query), K represents the key vector (Key), and V represents the value vector (Value). In the image cross-attention network, Q, K, and V are used to calculate the attention weight of the image in order to perform a weighted combination of image features of different parts;

[0055] S3, the feature As Q, the text information features in the above target fusion features are used as K and V to input the text cross attention network in the feature decoding module to generate features. , where Q refers to the query, K refers to the key, and V refers to the value. In the text cross-attention network, Q and K are used to calculate the attention weights, and V is used to obtain the final output based on the weighted sum of the attention weights;

[0056] S4, the characteristics The fully connected feed-forward network (FFN) in the feature decoding module is input to generate the above target multimodal features.

[0057] For example, after obtaining the above target multimodal features, Figure 3 As shown, the target multimodal features are input into a regression classification module to obtain the above target detection results, wherein the regression classification module may include but is not limited to various machine learning modules such as linear regression, logistic regression, decision tree, support vector machine, etc.

[0058] In an exemplary embodiment, Figure 4 is a schematic diagram of another optional action detection method according to an embodiment of the present application. The action detection method proposed in the present application can be applied to the field of video analysis and processing technology. Specifically, it can include but is not limited to multimodal feature fusion, action detection, language model field, etc. The action detection method proposed in the present application can include but is not limited to the following: Figure 4 As shown, including but not limited to:

[0059] S402, start;

[0060] S404, obtaining a target image sequence and target text information;

[0061] S406, inputting the target image sequence and target text information into the pre-trained target detection model for processing, specifically:

[0062] S408-1, fusing the target image sequence and the target text information through a cross attention mechanism to obtain a target fusion feature;

[0063] S408-2, fusing the target fusion feature, the target image sequence and the target text information respectively through a cross attention mechanism to obtain a target multimodal feature;

[0064] S408-3, after the target multimodal features are processed by the regression classification module, a target detection result is obtained, including but not limited to a target detection frame, a target action category, and a target confidence;

[0065] S410, end.

[0066] Through the embodiments of the present application, a target image sequence and target text information are obtained, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used to specify the action category to be detected from the target image sequence; the target image sequence and the target text information are input into a pre-trained target detection model to obtain a target detection result, wherein the target detection result includes a target detection box, a target action category, and a target confidence, wherein the target detection box is used to indicate the image area detected from the target image sequence, the target action category is used to indicate the action category corresponding to the target detection box, and the target confidence is used to indicate the matching degree between the target action category and the target text information. The target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain the target fusion feature, and the target fusion feature and the target image sequence and the target text information are respectively fused through the cross-attention mechanism to obtain the target multimodal feature, and the target detection frame, target action category and target confidence are determined according to the target multimodal feature, thereby achieving the purpose of detecting actions using an end-to-end target detection model, improving the generalization ability of the target detection model, and achieving the technical effect of improving the accuracy and robustness of action detection, thereby solving the technical problem of low accuracy of action detection.

[0067] As an optional solution, the target image sequence and the target text information are input into a pre-trained target detection model to obtain a target detection result, including: performing feature extraction on the target text information and the target image sequence respectively to obtain initial image sequence features and initial text features; performing feature fusion on the initial image sequence features and the initial text features based on the cross-attention mechanism to obtain target adaptive text features and target image sequence features; performing matrix multiplication operation on the target adaptive text features and the target image sequence features to obtain the target fusion features; performing sorting operation on the values ​​of each dimension in the target fusion features, and determining the target query features whose values ​​meet the preset conditions according to the sorting results; performing feature fusion on the target query features, the target image sequence features and the target adaptive text features respectively and in turn based on the cross-attention mechanism to obtain the target multimodal features; determining the target detection box, the target action category and the target confidence according to the target multimodal features to generate the target detection result.

[0068] Optionally, in an embodiment of the present application, the above-mentioned feature extraction refers to extracting representative and distinguishing features from the target text information and the target image sequence, respectively, to describe the respective properties and characteristics of the target text information and the target image sequence.

[0069] In an exemplary embodiment, for the above-mentioned target text information, it may include but is not limited to extracting text features through natural language processing technology, for example, using methods such as a text encoder to convert text into a vector representation, or using a pre-trained word embedding model to convert text into a dense vector representation to obtain the above-mentioned initial text features for describing the semantic and grammatical information of the target text information.

[0070] In another exemplary embodiment, for the above-mentioned target image sequence, it may include but is not limited to using computer vision technology to extract image features, for example, using an image encoder (SWIM Transformer) to perform feature extraction, converting the image into a high-dimensional feature vector representation, so as to obtain the above-mentioned initial image sequence features for describing the visual information and features of the image.

[0071] Specifically, it may include but is not limited to:

[0072] S1, a text encoder (BERT) can be used to extract features from the target text information to generate the initial text features.

[0073] S2, using the image encoder (SWIM Transformer) to extract features from the target image sequence to generate the initial image sequence features.

[0074] Furthermore, after obtaining the above-mentioned initial text features and the above-mentioned initial image features, the above-mentioned cross-attention mechanism can be further used to perform feature fusion operations on the two to generate the above-mentioned target adaptive text features and target image sequence features.

[0075] Optionally, in an embodiment of the present application, the above-mentioned cross-attention mechanism refers to an attention mechanism for processing sequence data, which introduces cross-information at different processing stages to better capture the global and local relationships of the sequence data.

[0076] Exemplarily, a multi-head attention network is used to further fuse the initial image sequence features and the above-mentioned initial text features, and then input them into the text self-attention network and the image self-attention network respectively to generate the above-mentioned target adaptive text features and target image sequence features.

[0077] In an exemplary embodiment, the above-mentioned matrix multiplication operation is performed on the above-mentioned target adaptive text feature and the above-mentioned target image sequence feature to obtain the above-mentioned target fusion feature. The present application does not limit the size of the target fusion feature, which can be expressed in the form of a matrix, wherein the number of matrix rows and columns can be flexibly set, and an element in the matrix represents the similarity between an image area and the corresponding text description.

[0078] In another exemplary embodiment, the above-mentioned sorting operation is performed on the values ​​of each dimension in the above-mentioned target fusion feature in turn to obtain the sorted target fusion feature, wherein the dimension of the target fusion feature refers to the classification or grouping of different attributes or feature values ​​of the action category, which is used to describe or distinguish the action category.

[0079] Furthermore, any feature values ​​with earlier sequence numbers of the sorted target fusion features can be determined as the target query features. For example, feature values ​​corresponding to the sequence numbers of the K maximum values ​​are taken out to generate the target query features, and the value of K is not limited.

[0080] Exemplarily, the cross-attention mechanism can be used to perform feature fusion on the target query feature and the target image sequence feature, respectively, and then perform feature fusion on the target adaptive text feature to generate the target multimodal feature, for example:

[0081] S1, input the target query feature into the self-attention network in the feature decoding module to generate features ;

[0082] S2, the feature As Q, the above target image sequence features are used as K, V is input into the image cross attention network in the feature decoding module, and the generated features are ;

[0083] S3, the feature As Q, the above target adaptive text features are used as K and V to input the text cross attention network in the feature decoding module to generate features ;

[0084] S4, the characteristics The fully connected feed-forward network (FFN) in the feature decoding module is input to generate the above target multimodal features.

[0085] As an optional scheme, the above-mentioned initial image sequence features and the above-mentioned initial text features are feature-fused based on the above-mentioned cross-attention mechanism to obtain target adaptive text features and target image sequence features, including: processing the above-mentioned initial image sequence features based on the self-attention mechanism to obtain self-attention image sequence features; adding the values ​​of the above-mentioned self-attention image sequence features and the above-mentioned initial text features in each dimension to obtain initial adaptive text features; and feature-fusing the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features based on the above-mentioned cross-attention mechanism to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features.

[0086] Optionally, in an embodiment of the present application, the self-attention image sequence features refer to image sequence features after the initial image sequence features are processed by the self-attention mechanism, and the initial adaptive text features refer to text features obtained by adding the feature values ​​of each dimension of the self-attention image sequence features and the initial text features. The self-attention image sequence features may include but are not limited to vectors, matrices, etc.

[0087] Specifically, the above-mentioned initial image sequence features can be input into a multi-head attention network module (Mulit-Head-Attention) for processing to generate self-attention image sequence features, wherein the number of heads in the multi-head attention network is N, and the value of N is not limited, and can be the total number of frames of the above-mentioned target image sequence; then, the initial text features and the self-attention image sequence features are added, and then passed through two fully connected layers to obtain the above-mentioned initial adaptive text features.

[0088] As an optional scheme, the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features are subjected to feature fusion based on the above-mentioned cross-attention mechanism to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features, including: processing the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features respectively based on the above-mentioned self-attention mechanism to obtain the first image sequence features and the first adaptive text features; determining the above-mentioned first image sequence features as the first query vector, determining the above-mentioned first adaptive text features as the first key vector and the first value vector, and determining the second image sequence features and the second adaptive text features based on the above-mentioned cross-attention mechanism; determining the above-mentioned second adaptive text features as the second query vector, determining the above-mentioned second image sequence features as the second key vector and the second value vector, and determining the third image sequence features and the third adaptive text features based on the above-mentioned cross-attention mechanism; inputting the above-mentioned third image sequence features and the third adaptive text features into the fully connected feedforward network module to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features.

[0089] Optionally, in an embodiment of the present application, the above-mentioned first image sequence features, first adaptive text features, second image sequence features, second adaptive text features, third image sequence features, third adaptive text features, target adaptive text features, and target image sequence features may all include but are not limited to being in the form of vectors, matrices, etc.

[0090] For example, Figure 5 is a schematic diagram of another optional action detection method according to an embodiment of the present application. The feature fusion process may include but is not limited to the following: Figure 5 As shown, including but not limited to:

[0091] S1, inputting the initial image sequence features into the image self-attention network module to generate the first adaptive text features, and inputting the initial adaptive text features into the text self-attention network module to generate the first adaptive text features;

[0092] S2, taking the first adaptive text feature as the first query vector Q1 input of the image-text cross attention network module, taking the first adaptive text feature as the first key vector K1 and the first value vector V1 input of the image-text cross attention network module, generating a second image sequence feature and a second adaptive text feature, wherein in the image-text cross attention network module, the first query vector is used to guide the degree of attention to the input, the first key vector represents the importance of the input, and the first value vector represents the characteristics of the input;

[0093] S3, taking the second adaptive text feature as the second query vector Q2 input of the text-image cross attention network module, taking the second image sequence feature as the second key vector K2 and the second value vector V2 input of the image-text cross attention network module, generating a third image sequence feature and a third adaptive text feature, wherein in the text-image cross attention network module, the second query vector is used to guide the degree of attention to the input, the second key vector represents the importance of the input, and the second value vector represents the characteristics of the input;

[0094] S4, the third image sequence feature and the third adaptive text feature are respectively passed through a fully connected feedforward network module (FFN) to generate the above-mentioned target adaptive text feature and the above-mentioned target image sequence feature.

[0095] As an optional scheme, the above-mentioned target query features are respectively and sequentially fused with the above-mentioned target image sequence features and the above-mentioned target adaptive text features based on the above-mentioned cross-attention mechanism to obtain the above-mentioned target multimodal features, including: processing the above-mentioned target query features based on the self-attention mechanism to obtain self-attention query features; determining the above-mentioned self-attention query features as the third query vector, the above-mentioned target image sequence features as the third key vector and the third value vector, and determining the first cross-attention query features based on the above-mentioned cross-attention mechanism; determining the above-mentioned first cross-attention query features as the fourth query vector, the above-mentioned target adaptive text features as the fourth key vector and the fourth value vector, and determining the second cross-attention query features based on the above-mentioned cross-attention mechanism; inputting the above-mentioned second cross-attention query features into the fully connected feedforward network module to obtain the above-mentioned target multimodal features.

[0096] Optionally, in an embodiment of the present application, the above-mentioned first cross-attention query feature and the second cross-attention query feature may include but are not limited to the form of vectors, matrices, etc.

[0097] In an exemplary embodiment, Figure 6 is a schematic diagram of another optional action detection method according to an embodiment of the present application, such as Figure 6 This may include, but is not limited to:

[0098] S1, passing the above target query features through the spatial self-attention network module to generate the above self-attention query features;

[0099] S2, using the self-attention query feature as the third query vector, the target image sequence feature as the third key vector and the third value vector, inputting them into the image cross-attention network module to generate a first cross-attention query feature, wherein in the image cross-attention network module, the third query vector is used to guide the degree of attention to the input, the third key vector represents the importance of the input, and the third value vector represents the characteristics of the input;

[0100] S3, taking the first cross-attention query feature as a fourth query vector, the target adaptive text feature as a fourth key vector and a fourth value vector, and inputting them into a text cross-attention network module to generate a second cross-attention query feature, wherein in the text cross-attention network module, the fourth query vector is used to guide the degree of attention to the input, the fourth key vector represents the importance of the input, and the fourth value vector represents the characteristics of the input;

[0101] S4, input the second cross-attention query feature into a fully connected feedforward network module (FFN) for processing to obtain the above-mentioned target multimodal feature.

[0102] As an optional scheme, the above-mentioned sorting operation is performed on the values ​​of each dimension in the above-mentioned target fusion feature, and the target query feature whose value meets the preset conditions is determined according to the sorting result, including: obtaining the value of the above-mentioned target fusion feature in each dimension, wherein the value of the above-mentioned target fusion feature in each dimension represents the similarity between the above-mentioned initial image sequence feature and the above-mentioned initial text feature in each dimension; performing the above-mentioned sorting operation on the values ​​in each dimension, and determining the dimensions where the K maximum values ​​are located according to the above-mentioned sorting results, and determining the above-mentioned target query feature according to the feature values ​​of the dimensions where the K maximum values ​​are located, wherein K is a positive integer.

[0103] Optionally, in an embodiment of the present application, the above-mentioned sorting operation can be understood as sorting in descending order according to the similarity values ​​on each dimension, that is, the dimension with the largest similarity value is arranged in front, and the dimension with the smallest similarity value is arranged in the back. According to the above-mentioned sorting results, the dimensions where the K maximum values ​​are located can be determined, and then, the above-mentioned target query features can be determined based on the feature values ​​of these K dimensions.

[0104] Exemplarily, the values ​​of the target fusion features in each dimension are first obtained, and the values ​​represent the similarities between the initial image sequence features and the initial text features in each dimension. Then, the values ​​in each dimension are sorted to determine the dimensions where the K maximum values ​​are located, and the target query features are determined based on the feature values ​​of these dimensions.

[0105] In an exemplary embodiment, assume that the target query feature contains five dimensions: dimension 1, dimension 2, dimension 3, dimension 4, and dimension 5, and the corresponding values ​​are: {dimension 1: 0.8; dimension 2: 0.6; dimension 3: 0.9; dimension 4: 0.7; dimension 5: 0.5}. Then, these values ​​are sorted and arranged from large to small as follows: {dimension 3: 0.9; dimension 1: 0.8; dimension 4: 0.7; dimension 2: 0.6; dimension 5: 0.5}. At the same time, the value of K is set to 3, then dimension 3, dimension 1 and dimension 4 can be selected as the above-mentioned target query features, and the target query features are expressed as: {dimension 3: 0.9; dimension 1: 0.8; dimension 4: 0.7}.

[0106] As an optional solution, the method further includes: obtaining a sample image sequence, sample text information and detection frame annotation information, wherein the sample image sequence includes a sample image to be predicted, the sample text information is used to specify the action category to be predicted from the sample image sequence, and the detection frame annotation information is used to indicate whether there is an image area of ​​a preset action category in the sample image sequence, and whether the sample image sequence matches the sample text information; inputting the sample image sequence and the sample text information into an initial detection model to obtain a sample detection result, wherein the sample detection result includes a sample detection frame, a sample action category and a sample confidence, the sample detection frame is used to indicate the image area detected from the sample image sequence, the sample action category is used to indicate the action category corresponding to the sample detection frame, and the sample confidence is used to indicate the degree of matching between the sample action category and the sample text information; calculating a target loss value based on the detection frame annotation information and the sample detection result, and training the initial detection model based on the target loss value to obtain the target detection model.

[0107] Optionally, in an embodiment of the present application, the above-mentioned sample image sequence can be understood as a group of sequence data composed of several consecutive image frames, which may include but is not limited to representing videos, animations, etc.; the above-mentioned sample text information may include but is not limited to action categories, object categories, scene categories, character identities, etc. that can be recognized and expected to be recognized in the sample image sequence, and the above-mentioned detection frame annotation information may include but is not limited to action categories that actually exist in the sample image sequence; detection frame position: including the coordinates of the upper left corner and the lower right corner of the detection frame, the center point coordinates, and the width and height of the detection frame, etc.; key point position: the key point position in each detection frame, for example, human body key points or object key points; action timing information: the time period in which the action corresponding to each detection frame occurs; action intensity information: the intensity of the action in each detection frame, for example, the running speed, the jumping height, etc.; action environment information: the environment information in which the action occurs in each detection frame, for example, indoors or outdoors, lighting conditions, etc.

[0108] It is understandable that the present application does not limit the time for obtaining the above-mentioned sample image sequence and sample text information.

[0109] Exemplarily, the above-mentioned sample image sequence is a video, and each frame in the video can be a still image, that is, the above-mentioned image to be subjected to action detection. Taking the action category expected to be identified in the sample text information as an example, it can include but is not limited to walking, running, jumping, bending over, raising hands, etc.

[0110] It should be noted that the above-mentioned initial detection model may include but is not limited to a text-image cross-attention network module, a text cross-attention network module, an image-text cross-attention network module, an image cross-attention network module, and the like.

[0111] Optionally, in an embodiment of the present application, the above-mentioned sample detection box refers to the area in the sample image sequence where the action can be determined to occur; the above-mentioned sample action category refers to the category of the action detected from the sample image sequence, for example, walking, running, etc.; the above-mentioned sample confidence refers to the degree of confidence of the initial detection model in the above-mentioned sample detection result, that is, the sample confidence can be used to obtain the probability that the action identified by the sample detection result is a real action.

[0112] Furthermore, after obtaining the above-mentioned sample detection results, the above-mentioned target loss value can be calculated based on the sample detection results and the row number detection box annotation information, and then, the above-mentioned initial detection model can be trained using the target loss value to obtain the above-mentioned target detection model. For example, the target loss value is calculated using a cross entropy loss function or a mean square error loss function, and the model parameters of the initial detection model are updated according to the loss value through optimization algorithms such as gradient descent, so that the initial detection model can better predict the position and category of the target detection box to obtain the above-mentioned target detection model.

[0113] As an optional solution, the target loss value is calculated according to the detection box annotation information and the sample detection results, and the initial detection model is trained according to the target loss value to obtain the target detection model, including: calculating a first loss value, a second loss value and a third loss value according to the detection box annotation information and the sample detection results, wherein the first loss value is determined by the predicted probability of the sample action category and the annotated action category indicated by the detection box annotation information, the second loss value is determined by the size and position information of the sample detection box and the size and position information of the annotated detection box indicated by the detection box annotation information, and the third loss value is determined by the sample confidence and whether the sample image sequence indicated by the detection box annotation information matches the sample text information; the first loss value, the second loss value and the third loss value are superimposed according to preset weights to obtain the target loss value, and the network parameters of the initial detection model are iteratively updated through a back propagation algorithm according to the target loss value to obtain the target detection model.

[0114] Optionally, in an embodiment of the present application, there is no limitation on the specific values ​​of the first loss value, the second loss value and the third loss value. The detection box annotation information may include but is not limited to the action category annotated by the detection box, the size and position information of the annotated detection box, and the corresponding sample image sequence, etc. The sample detection result may include but is not limited to the predicted probability of the sample action category, the size and position information of the detection box, the sample confidence, etc.

[0115] In an exemplary embodiment, the first loss value may include but is not limited to the focal loss function, that is, using the loss function term Sure:

[0116] ;

[0117] in, and are two hyperparameters, 2 and 4 respectively, represents the predicted probability of the above sample action category at the above detection box coordinates (x, y), represents the supervision signal of the real action category at the detection box coordinate (x, y), where and The value of can be set according to the actual business situation, and may include but is not limited to the value set in the embodiment of the present application. is 2, is 4, otherwise means that in the Focal Loss loss function, when it does not satisfy = 1, another operation or calculation will be performed.

[0118] In another exemplary embodiment, the second loss value may include but is not limited to L1 Loss (absolute value loss function), that is, using the loss function term Sure:

[0119] ;

[0120] in, is the predicted detection box of the action to be detected in the above sample image sequence, It is the supervision information of the real detection box in the sample image sequence, which can include but is not limited to:

[0121] ;

[0122] Among them, x center and center is the center point of the detection box, h is the width of the detection box, and w is the height of the detection box.

[0123] In another exemplary embodiment, the third loss value may include but is not limited to using CrossEntropy Loss, that is, using the loss function Sure:

[0124] ;

[0125] in, represents the true label, Represents the value of the above sample fusion feature after softmax (an activation function). For each sample text information, the image in the sample image sequence that matches it is classified as 1, and the image that does not match it is classified as 0; for each image in the sample image sequence, the category of each sample text information that matches it is 1, and the category of the sample text information that does not match it is 0, where the true label can indicate the corresponding area of ​​the above-mentioned annotation detection box.

[0126] Furthermore, the above-mentioned preset weights can be flexibly set, and the present application does not impose any limitation on this. Thus, the above-mentioned first loss value, the second loss value and the third loss value are superimposed according to the preset weights to obtain the above-mentioned target loss value, and then, the network parameters of the above-mentioned initial detection model are iteratively updated through the back propagation algorithm to obtain the target detection model.

[0127] As an optional solution, the target loss value is calculated according to the detection box annotation information and the sample detection result, and the initial detection model is trained according to the target loss value to obtain the target detection model, including: extracting features from the sample text information and the sample image sequence respectively to obtain initial sample image sequence features and initial sample text features; fusing the initial sample image sequence features and the initial sample text features based on the cross-attention mechanism to obtain sample adaptive text features and sample image sequence features; performing matrix multiplication operations on the sample adaptive text features and the sample image sequence features to obtain the sample fusion features; performing sorting operations on the values ​​of each dimension in the sample fusion features, and determining the sample query features whose values ​​meet the preset conditions according to the sorting results; fusing the sample query features with the sample image sequence features and the sample adaptive text features in turn based on the cross-attention mechanism to obtain the sample multimodal features; determining the sample detection box, the sample action category and the sample confidence according to the sample multimodal features to generate the sample detection result.

[0128] Optionally, in an embodiment of the present application, the above-mentioned initial sample image sequence features, initial sample text features, sample adaptive text features, sample image sequence features, sample fusion features, sample query features, and sample multimodal features may all include but are not limited to forms such as vectors and matrices.

[0129] Exemplarily, assuming that there is a sample data set containing a sample image sequence and sample text information, first, feature extraction is performed on the sample image sequence and the sample text information respectively to obtain the corresponding initial sample image sequence features and initial sample text features. Then, the initial sample image sequence features and the initial sample text features are fused using a cross-attention mechanism to obtain the above-mentioned sample adaptive text features and the above-mentioned sample image sequence features.

[0130] Furthermore, a matrix multiplication operation is performed on the sample adaptive text features and the sample image sequence features to obtain sample fusion features, and the values ​​of each dimension of the sample fusion features are sorted, and the sample query features that meet the preset conditions are determined according to the sorting results; then, the cross-attention mechanism is used to perform feature fusion on the sample query features with the sample image sequence features and the sample adaptive text features respectively to obtain sample multimodal features; finally, the sample detection frame, sample action category and sample confidence are determined according to the sample multimodal features to generate the above-mentioned sample detection results.

[0131] In an exemplary embodiment, the present application can be applied to the field of video analysis and processing technology, and by fusing text information and image information, the purpose of locating and detecting actions in a video (the above-mentioned target image sequence) can be achieved using only text information.

[0132] For example, the embodiments of the present application may include but are not limited to:

[0133] S1, generating a data set: selecting video action sequences in different time periods, converting them into images for storage and preprocessing, and using the preprocessed data as training and test samples, wherein each image sequence (the above sample image sequence) in the training and test samples generates a corresponding text description (the above sample text information) to train the above initial detection model and obtain the above target detection model;

[0134] S2, feature extraction module: The text description uses a text encoder (BERT) to extract features and generate text features (The above initial sample text features), the image sequence (a total of N frames) uses the image encoder (SWIM Transformer) to extract features and generate image sequence features (The above initial sample image sequence characteristics);

[0135] S3, Adaptive Video Description Generation Module: Image Sequence Features First, the self-attention sample image sequence features are generated through the multi-head attention network module (Mulit-Head-Attention) , where the number of heads in the multi-head attention network module is N. Then, the text features are and self-attention sample image sequence features Add them together and then pass through two fully connected layers to obtain the initial sample adaptive text features ;

[0136] S4, feature fusion module, adapts the initial sample to text features , image sequence features Input the text self-attention network module and the image self-attention network module respectively;

[0137] Next, the first adaptive sample text features are generated respectively , the first sample image sequence features ; After the image-text cross attention network module, the second adaptive sample text feature is generated and the second sample image sequence features , where the first sample image sequence features As the Q (query value) input of the image-text cross-attention network module, the first adaptive sample text feature It is used as the K (key) and V (value) input of the image-text cross attention network module, and then passes through the text-image cross attention network module to generate the third adaptive sample text feature. And the third sample image sequence features , where the second adaptive sample text feature As the Q input of the text-image cross attention network module, the second sample image sequence feature As the K and V inputs of the text-image cross attention network module, the generated third adaptive sample text features are finally And the third sample image sequence features The corresponding sample adaptive text features are generated through the fully connected feedforward network module (FFN) And the above sample image sequence features ;

[0138] S5, query feature generation module: sample adaptive text features And sample image sequence features Perform matrix multiplication to obtain the above sample fusion features , to fuse features of samples Sort the values ​​of each dimension of , take out the feature values ​​where the K maximum values ​​are located, and generate the above sample query features ;

[0139] S6, feature decoding module: sample query features First, the self-attention network module is used to generate the self-attention sample query feature , query the features of the self-attention sample As Q, the sample image sequence features As K, V input image cross attention network module, generate the first cross attention sample query feature , query the first cross-attention sample feature As Q, sample adaptive text features As K and V, the text cross-attention network module is input to generate the second cross-attention sample query feature , and finally the second cross-attention sample query feature Generate sample multimodal features through a fully connected feedforward network module (FFN) ;

[0140] S7, regression classification module: multimodal features of samples Input to a 1 1 convolution layer, through the ReLu layer (activation layer), and then through a 1 1 convolution layer to obtain the sample detection frame, sample action category and sample confidence;

[0141] S8, model training: CrossEntropy Loss, Focal Loss and L1 Loss are used to supervise the intermediate feature map, action category, confidence and coordinates of the detection box of the initial detection model respectively. The three loss functions are superimposed according to the set ratio. The SGD optimizer (stochastic gradient descent optimizer) is used to iteratively update the network parameters through the back propagation algorithm. The results on the test samples are used to select the best model. Specifically:

[0142] The Focal Loss function term is recorded as :

[0143] ,in, and are two hyperparameters, 2 and 4 respectively, represents the predicted probability of the above sample action category at the above detection box coordinates (x, y), represents the supervision signal of the real action category at the detection box coordinate (x, y), where and The value of can be set according to the actual business situation, and may include but is not limited to the value set in the embodiment of the present application. is 2, is 4, otherwise means that in the Focal Loss loss function, when it does not satisfy = 1, another operation or calculation will be performed.

[0144] The L1 Loss function term is recorded as , ,in, , are respectively the predicted detection box of the action to be detected in the above sample image sequence and the supervision information of the real detection box in the above sample image sequence, which may specifically include but are not limited to:

[0145] ;

[0146] Among them, x center and center is the center point of the detection box, h is the width of the detection box, and w is the height of the detection box.

[0147] The CrossEntropy Loss function is recorded as ,in, represents the true label, Represents the value of the above sample fusion feature after softmax (an activation function). For each sample text information, the image in the sample image sequence that matches it is classified as 1, and the image that does not match it is classified as 0; for each image in the sample image sequence, the category of each sample text information that matches it is 1, and the category of the sample text information that does not match it is 0, where the true label can indicate the corresponding area of ​​the above-mentioned annotation detection box.

[0148] In summary, the embodiments of the present application realize end-to-end multimodal video action detection. By fusing text information and image information, it is possible to locate action behaviors in the video through text. It also includes adaptive text description prompt generation: it can adaptively generate text description prompts corresponding to the image sequence according to the global image information and the action category; multimodal feature fusion strategy: the text information is fused with the image sequence information, so that the target detection model pays more attention to the image sequence area corresponding to the text information; video action detection: the action category that does not appear in the training set can be detected only through text information, which has stronger scalability, robustness and portability.

[0149] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0150] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0151] According to another aspect of the embodiments of the present application, a motion detection device for implementing the above motion detection method is also provided. Figure 7 As shown, the device comprises:

[0152] An acquisition module 702 is used to acquire a target image sequence and target text information, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used to specify an action category to be detected from the target image sequence;

[0153] The detection module 704 is used to input the target image sequence and the target text information into the pre-trained target detection model to obtain the target detection result, wherein the target detection result includes a target detection frame, a target action category and a target confidence. The target detection frame is used to indicate the image area detected from the target image sequence, the target action category is used to indicate the action category corresponding to the target detection frame, and the target confidence is used to indicate the degree of matching between the target action category and the target text information. The target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and to fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and to determine the target detection frame, the target action category and the target confidence according to the target multimodal feature.

[0154] As an optional solution, the above-mentioned device is used to input the above-mentioned target image sequence and the above-mentioned target text information into a pre-trained target detection model in the following manner to obtain a target detection result: feature extraction is performed on the above-mentioned target text information and the above-mentioned target image sequence respectively to obtain initial image sequence features and initial text features; based on the above-mentioned cross-attention mechanism, feature fusion is performed on the above-mentioned initial image sequence features and the above-mentioned initial text features to obtain target adaptive text features and target image sequence features; matrix multiplication operation is performed on the above-mentioned target adaptive text features and the above-mentioned target image sequence features to obtain the above-mentioned target fusion features; sorting operation is performed on the values ​​of each dimension in the above-mentioned target fusion features, and target query features whose values ​​meet preset conditions are determined according to the sorting results; based on the above-mentioned cross-attention mechanism, feature fusion is performed on the above-mentioned target query features in turn with the above-mentioned target image sequence features and the above-mentioned target adaptive text features to obtain the above-mentioned target multimodal features; the above-mentioned target detection box, the above-mentioned target action category and the above-mentioned target confidence are determined according to the above-mentioned target multimodal features to generate the above-mentioned target detection result.

[0155] As an optional solution, the above-mentioned device is used to perform feature fusion on the above-mentioned initial image sequence features and the above-mentioned initial text features based on the above-mentioned cross-attention mechanism to obtain target adaptive text features and target image sequence features in the following manner: processing the above-mentioned initial image sequence features based on the self-attention mechanism to obtain self-attention image sequence features; adding the values ​​of the above-mentioned self-attention image sequence features and the above-mentioned initial text features in each dimension to obtain initial adaptive text features; performing feature fusion on the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features based on the above-mentioned cross-attention mechanism to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features.

[0156] As an optional solution, the above-mentioned device is used to perform feature fusion on the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features based on the above-mentioned cross-attention mechanism in the following manner to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features: based on the above-mentioned self-attention mechanism, the above-mentioned initial image sequence features and the above-mentioned initial adaptive text features are respectively processed to obtain the first image sequence features and the first adaptive text features; the above-mentioned first image sequence features are determined as the first query vector, the above-mentioned first adaptive text features are determined as the first key vector and the first value vector, and the second image sequence features and the second adaptive text features are determined based on the above-mentioned cross-attention mechanism; the above-mentioned second adaptive text features are determined as the second query vector, the above-mentioned second image sequence features are determined as the second key vector and the second value vector, and the third image sequence features and the third adaptive text features are determined based on the above-mentioned cross-attention mechanism; the above-mentioned third image sequence features and the third adaptive text features are input into the fully connected feedforward network module to obtain the above-mentioned target adaptive text features and the above-mentioned target image sequence features.

[0157] As an optional solution, the above-mentioned device is used to perform feature fusion on the above-mentioned target query features and the above-mentioned target image sequence features and the above-mentioned target adaptive text features in sequence based on the above-mentioned cross-attention mechanism in the following manner to obtain the above-mentioned target multimodal features: processing the above-mentioned target query features based on the self-attention mechanism to obtain self-attention query features; determining the above-mentioned self-attention query features as the third query vector, the above-mentioned target image sequence features as the third key vector and the third value vector, and determining the first cross-attention query features based on the above-mentioned cross-attention mechanism; determining the above-mentioned first cross-attention query features as the fourth query vector, the above-mentioned target adaptive text features as the fourth key vector and the fourth value vector, and determining the second cross-attention query features based on the above-mentioned cross-attention mechanism; inputting the above-mentioned second cross-attention query features into a fully connected feedforward network module to obtain the above-mentioned target multimodal features.

[0158] As an optional scheme, the above-mentioned device is used to perform a sorting operation on the values ​​of each dimension in the above-mentioned target fusion feature in the following manner, and determine the target query feature whose values ​​meet the preset conditions according to the sorting results: obtain the values ​​of the above-mentioned target fusion feature in each dimension, wherein the values ​​of the above-mentioned target fusion feature in each dimension represent the similarity between the above-mentioned initial image sequence feature and the above-mentioned initial text feature in each dimension; perform the above-mentioned sorting operation on the values ​​in each dimension, and determine the dimensions where the K maximum values ​​are located according to the above-mentioned sorting results, and determine the above-mentioned target query feature according to the feature values ​​of the dimensions where the K maximum values ​​are located, wherein K is a positive integer.

[0159] As an optional solution, the above-mentioned device is also used to: obtain a sample image sequence, sample text information and detection frame annotation information, wherein the above-mentioned sample image sequence includes a sample image to be predicted, the above-mentioned sample text information is used to specify the action category that needs to be predicted from the above-mentioned sample image sequence, and the above-mentioned detection frame annotation information is used to indicate whether there is an image area of ​​a preset action category in the above-mentioned sample image sequence, and whether the above-mentioned sample image sequence matches the above-mentioned sample text information; input the above-mentioned sample image sequence and the above-mentioned sample text information into the initial detection model to obtain a sample detection result, wherein the above-mentioned sample detection result includes a sample detection frame, a sample action category and a sample confidence, the above-mentioned sample detection frame is used to indicate the image area detected from the above-mentioned sample image sequence, the above-mentioned sample action category is used to indicate the action category corresponding to the above-mentioned sample detection frame, and the above-mentioned sample confidence is used to indicate the degree of matching between the above-mentioned sample action category and the above-mentioned sample text information; calculate the target loss value according to the above-mentioned detection frame annotation information and the above-mentioned sample detection result, and train the above-mentioned initial detection model according to the above-mentioned target loss value to obtain the above-mentioned target detection model.

[0160] As an optional solution, the above-mentioned device is used to calculate the target loss value according to the above-mentioned detection box annotation information and the above-mentioned sample detection results in the following manner, and train the above-mentioned initial detection model according to the above-mentioned target loss value to obtain the above-mentioned target detection model: calculate the first loss value, the second loss value and the third loss value according to the above-mentioned detection box annotation information and the above-mentioned sample detection results, wherein the above-mentioned first loss value is determined by the predicted probability of the above-mentioned sample action category and the annotated action category indicated by the above-mentioned detection box annotation information, the above-mentioned second loss value is determined by the size and position information of the above-mentioned sample detection box and the size and position information of the annotated detection box indicated by the above-mentioned detection box annotation information, and the above-mentioned third loss value is determined by the above-mentioned sample confidence and whether the above-mentioned sample image sequence indicated by the above-mentioned detection box annotation information matches the above-mentioned sample text information; the above-mentioned first loss value, the above-mentioned second loss value and the above-mentioned third loss value are superimposed according to preset weights to obtain the above-mentioned target loss value, and the network parameters of the above-mentioned initial detection model are iteratively updated through the back propagation algorithm according to the above-mentioned target loss value to obtain the above-mentioned target detection model.

[0161] As an optional solution, the above-mentioned device is used to calculate the target loss value according to the above-mentioned detection box annotation information and the above-mentioned sample detection results in the following manner, and train the above-mentioned initial detection model according to the above-mentioned target loss value to obtain the above-mentioned target detection model: feature extraction is performed on the above-mentioned sample text information and the above-mentioned sample image sequence respectively to obtain initial sample image sequence features and initial sample text features; based on the above-mentioned cross-attention mechanism, feature fusion is performed on the above-mentioned initial sample image sequence features and the above-mentioned initial sample text features to obtain sample adaptive text features and sample image sequence features; matrix multiplication operation is performed on the above-mentioned sample adaptive text features and the above-mentioned sample image sequence features to obtain the above-mentioned sample fusion features; sorting operation is performed on the values ​​of each dimension in the above-mentioned sample fusion features, and sample query features whose values ​​meet the above-mentioned preset conditions are determined according to the sorting results; based on the above-mentioned cross-attention mechanism, feature fusion is performed on the above-mentioned sample query features in turn with the above-mentioned sample image sequence features and the above-mentioned sample adaptive text features to obtain the above-mentioned sample multimodal features; the above-mentioned sample detection box, the above-mentioned sample action category and the above-mentioned sample confidence are determined according to the above-mentioned sample multimodal features to generate the above-mentioned sample detection results.

[0162] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0163] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0164] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0165] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0166] Figure 8 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0167] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0168] like Figure 8 As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0169] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0170] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.

[0171] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions provided by the embodiment of the present application are performed.

[0172] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned action detection method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0173] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0174] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0175] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations shown.

[0176] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the motion detection method and device in the embodiments of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, to implement the above-mentioned motion detection method. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store information such as the above-mentioned target image sequence and target text information. As an example, such as Fig. 9 As shown, the memory 902 may include but is not limited to the acquisition module 702 and the detection module 704 in the motion detection device. In addition, it may also include but is not limited to other module units in the motion detection device, which will not be described in detail in this example.

[0177] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0178] In addition, the electronic device further comprises: a display 908 for displaying the target image sequence and target text information; and a connection bus 910 for connecting various module components in the electronic device.

[0179] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0180] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the action detection method provided in various optional implementations of the above-mentioned action detection aspect.

[0181] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0182] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0183] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0184] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0185] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A motion detection method, characterized in that: include: Acquire a target image sequence and target text information, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used to specify an action category to be detected from the target image sequence; The target image sequence and the target text information are input into a pre-trained target detection model to obtain a target detection result, wherein the target detection result includes a target detection frame, a target action category, and a target confidence, wherein the target detection frame is used to indicate an image area detected from the target image sequence, the target action category is used to indicate an action category corresponding to the target detection frame, and the target confidence is used to indicate a matching degree between the target action category and the target text information, wherein the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and to fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and to determine the target detection frame, the target action category, and the target confidence according to the target multimodal feature; The target image sequence and the target text information are input into a pre-trained target detection model to obtain a target detection result, including: extracting features from the target text information and the target image sequence respectively to obtain an initial image sequence feature and an initial text feature; fusing the initial image sequence feature and the initial text feature based on the cross attention mechanism to obtain a target adaptive text feature and a target image sequence feature; performing a matrix multiplication operation on the target adaptive text feature and the target image sequence feature to obtain the target fusion feature; sorting the values ​​of each dimension in the target fusion feature, and determining a target query feature whose values ​​of each dimension meet a preset condition according to the sorting result; fusing the target query feature with the target image sequence feature and the target adaptive text feature in turn based on the cross attention mechanism to obtain the target multimodal feature; determining the target detection frame, the target action category and the target confidence according to the target multimodal feature to generate the target detection result.

2. The method according to claim 1, characterized in that The step of fusing the initial image sequence features and the initial text features based on the cross attention mechanism to obtain target adaptive text features and target image sequence features includes: Processing the initial image sequence features based on a self-attention mechanism to obtain self-attention image sequence features; Adding the values ​​of the self-attention image sequence feature and the initial text feature in each dimension to obtain an initial adaptive text feature; Based on the cross attention mechanism, the initial image sequence features and the initial adaptive text features are fused to obtain the target adaptive text features and the target image sequence features.

3. The method according to claim 2, characterized in that The step of fusing the initial image sequence features and the initial adaptive text features based on the cross attention mechanism to obtain the target adaptive text features and the target image sequence features includes: Based on the self-attention mechanism, the initial image sequence feature and the initial adaptive text feature are processed respectively to obtain a first image sequence feature and a first adaptive text feature; Determine the first image sequence feature as a first query vector, determine the first adaptive text feature as a first key vector and a first value vector, and determine a second image sequence feature and a second adaptive text feature based on the cross attention mechanism; Determine the second adaptive text feature as a second query vector, determine the second image sequence feature as a second key vector and a second value vector, and determine a third image sequence feature and a third adaptive text feature based on the cross attention mechanism; The third image sequence feature and the third adaptive text feature are input into a fully connected feed-forward network module to obtain the target adaptive text feature and the target image sequence feature.

4. The method according to claim 1, characterized in that: The method of fusing the target query feature with the target image sequence feature and the target adaptive text feature in sequence based on the cross attention mechanism to obtain the target multimodal feature includes: Processing the target query feature based on a self-attention mechanism to obtain a self-attention query feature; Determine the self-attention query feature as a third query vector, determine the target image sequence feature as a third key vector and a third value vector, and determine a first cross-attention query feature based on the cross-attention mechanism; Determine the first cross-attention query feature as a fourth query vector, determine the target adaptive text feature as a fourth key vector and a fourth value vector, and determine a second cross-attention query feature based on the cross-attention mechanism; The second cross-attention query feature is input into a fully connected feed-forward network module to obtain the target multimodal feature.

5. The method according to claim 1, characterized in that The performing a sorting operation on the values ​​of each dimension in the target fusion feature, and determining the target query feature whose values ​​of each dimension meet the preset conditions according to the sorting result, includes: Acquire the value of the target fusion feature in each dimension, wherein the value of the target fusion feature in each dimension represents the similarity between the initial image sequence feature and the initial text feature in each dimension; The sorting operation is performed on the values ​​on each dimension, and the dimensions where the K maximum values ​​are located are determined according to the sorting results, and the target query features are determined according to the feature values ​​of the dimensions where the K maximum values ​​are located, where K is a positive integer.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a sample image sequence, sample text information, and detection frame annotation information, wherein the sample image sequence includes sample images to be predicted, the sample text information is used to specify an action category to be predicted from the sample image sequence, and the detection frame annotation information is used to indicate whether there is an image region of a preset action category in the sample image sequence, and whether the sample image sequence matches the sample text information; Inputting the sample image sequence and the sample text information into an initial detection model to obtain a sample detection result, wherein the sample detection result includes a sample detection frame, a sample action category, and a sample confidence, wherein the sample detection frame is used to indicate an image area detected from the sample image sequence, the sample action category is used to indicate an action category corresponding to the sample detection frame, and the sample confidence is used to indicate a matching degree between the sample action category and the sample text information; A target loss value is calculated according to the detection box annotation information and the sample detection result, and the initial detection model is trained according to the target loss value to obtain the target detection model.

7. The method according to claim 6, characterized in that The calculating the target loss value according to the detection box annotation information and the sample detection result, and training the initial detection model according to the target loss value to obtain the target detection model includes: Calculating a first loss value, a second loss value, and a third loss value according to the detection frame annotation information and the sample detection result, wherein the first loss value is determined by the predicted probability of the sample action category and the annotated action category indicated by the detection frame annotation information, the second loss value is determined by the size and position information of the sample detection frame and the size and position information of the annotated detection frame indicated by the detection frame annotation information, and the third loss value is determined by the sample confidence and whether the sample image sequence indicated by the detection frame annotation information matches the sample text information; The first loss value, the second loss value and the third loss value are superimposed according to preset weights to obtain the target loss value, and the network parameters of the initial detection model are iteratively updated through a back propagation algorithm according to the target loss value to obtain the target detection model.

8. The method according to claim 6, characterized in that The calculating the target loss value according to the detection box annotation information and the sample detection result, and training the initial detection model according to the target loss value to obtain the target detection model includes: Extracting features from the sample text information and the sample image sequence to obtain initial sample image sequence features and initial sample text features; Based on the cross attention mechanism, feature fusion is performed on the initial sample image sequence features and the initial sample text features to obtain sample adaptive text features and sample image sequence features; Performing a matrix multiplication operation on the sample adaptive text feature and the sample image sequence feature to obtain a sample fusion feature; Performing a sorting operation on the values ​​of each dimension in the sample fusion feature, and determining the sample query features whose values ​​of each dimension meet the preset conditions according to the sorting result; Based on the cross attention mechanism, the sample query feature is sequentially fused with the sample image sequence feature and the sample adaptive text feature to obtain the sample multimodal feature; The sample detection frame, the sample action category and the sample confidence are determined according to the sample multimodal features to generate the sample detection result.

9. A motion detection device, characterized in that: include: An acquisition module, used for acquiring a target image sequence and target text information, wherein the target image sequence includes images to be subjected to action detection, and the target text information is used for specifying an action category to be detected from the target image sequence; A detection module, used for inputting the target image sequence and the target text information into a pre-trained target detection model to obtain a target detection result, wherein the target detection result includes a target detection frame, a target action category and a target confidence, wherein the target detection frame is used to indicate an image area detected from the target image sequence, the target action category is used to indicate an action category corresponding to the target detection frame, and the target confidence is used to indicate a matching degree between the target action category and the target text information, wherein the target detection model is used to fuse the target image sequence and the target text information through a cross-attention mechanism to obtain a target fusion feature, and to fuse the target fusion feature and the target image sequence and the target text information respectively through the cross-attention mechanism to obtain a target multimodal feature, and to determine the target detection frame, the target action category and the target confidence according to the target multimodal feature; The device is used to input the target image sequence and the target text information into a pre-trained target detection model in the following manner to obtain a target detection result: feature extraction is performed on the target text information and the target image sequence respectively to obtain an initial image sequence feature and an initial text feature; feature fusion is performed on the initial image sequence feature and the initial text feature based on the cross attention mechanism to obtain a target adaptive text feature and a target image sequence feature; matrix multiplication is performed on the target adaptive text feature and the target image sequence feature to obtain the target fusion feature; sorting is performed on the values ​​of each dimension in the target fusion feature, and a target query feature whose values ​​of each dimension meet a preset condition is determined according to the sorting result; feature fusion is performed on the target query feature in turn with the target image sequence feature and the target adaptive text feature based on the cross attention mechanism to obtain the target multimodal feature; the target detection frame, the target action category and the target confidence are determined according to the target multimodal feature to generate the target detection result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images