Intelligent detection method and system for moving targets based on weakly supervised dynamic optimization

Abstract

The application provides a kind of weak supervision dynamic optimization-based moving target intelligent detection method and system, respectively extracts video feature and text feature to original video and text, and carries out fusion, generates frame-level semantic similarity score as pseudo label;Using learnable object query and fusion feature interaction, generate positive and negative proposal mask;Using pseudo label to guide the feature comparison learning of positive proposal, make positive proposal infinitely fit the text feature in semantic space, and make negative proposal infinitely deviate from the related area of text feature;Using positive and negative proposal mask, text reconstruction is carried out based on mask condition Transformer, and semantic consistency training is carried out on different proposals, to obtain video time domain positioning result;Dynamic optimization is carried out on video time domain positioning result, to generate final positioning result, to complete moving target intelligent detection.The application constructs learnable negative proposal and dynamic pseudo label constraint mechanism, which significantly improves the time domain positioning accuracy under weak supervision condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video temporal target positioning, in particular to a motion target intelligent detection method and system based on weakly supervised dynamic optimization, and relates to a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] With the continuous progress of artificial intelligence, video coding and multi-modal understanding technology, video has become one of the main media forms of information expression and dissemination in modern society. Compared with static images or text, video integrates visual, audio and semantic information, and has stronger scene restoration ability and time continuity, so it has been widely used in social media, content recommendation, intelligent monitoring and other fields. Compared with images and text, it contains more rich semantic information. There are many current video understanding related tasks, such as video action classification, video question answering, video text retrieval, etc. These tasks are all about understanding the content of the edited short video. For unedited long video understanding, it is still a great challenge to automatically locate a segment in the long video according to the text description proposed by the user.

[0003] The video temporal target positioning task aims to automatically locate the semantic related video time segment in the unedited long video according to a natural language description. The challenge of this task is that the model needs to have both temporal modeling ability and cross-modal semantic alignment ability, and there is often a lack of precise time boundary annotation in the training process. In order to reduce the cost of manual annotation, weakly supervised video temporal positioning has gradually become the focus of research. Its goal is to learn the alignment relationship between text and video segment under the condition of lacking start and end time labels. Existing weakly supervised positioning methods can be roughly divided into two categories: one is based on multiple instance learning (MIL) (such as WSLLN, etc.), which selects the most relevant proposal to the text as the positive sample by scoring multiple proposals; the other is based on the reconstruction idea, that is, a set of candidate proposals is constructed, and the text is reconstructed using them, and the model training process is optimized through the reconstruction error (such as WSTAN, etc.). Both of them are designed under the above framework. However, the method based on multiple instance learning has the problems of difficult to distinguish between action and background, incomplete positioning, etc.; while the method based on the reconstruction idea faces the challenges of inconsistency between reconstruction target and positioning task, insufficient negative sample modeling, etc., which affects the semantic alignment ability and discrimination performance of the model. Therefore, the existing methods have deficiencies in the accuracy of proposal generation, the discrimination ability of positive and negative samples, and the consistency of training target, etc.

[0004] In recent years, the success of DETR in the image object detection task has inspired some researchers to apply it to the time domain positioning scene. Inspired by it, some methods introduce a proposal generation module composed of a learnable object query and a Transformer decoder to realize an end-to-end learnable proposal prediction mechanism. This kind of method reduces the dependence on human design while improving the expression ability and structural flexibility of the proposal. However, how to generate discriminative negative proposals in a weakly supervised scene is still a key difficulty. On the one hand, static generated negative proposals cannot cover complex and diverse video interference areas, resulting in a lack of supervision signals in the training process; on the other hand, if "difficult negative samples" that are too close to the semantic boundary are introduced in the early training stage, the model may be misled, leading to error accumulation. Therefore, some research proposes to use a pseudo label mechanism as a soft supervision signal to guide the model to learn the semantic alignment relationship between the proposal and the text. However, most of these methods use pseudo labels as classification supervision signals or static scoring basis, without explicitly introducing the continuity information of the pseudo label into the feature space for structure optimization, lacking a feature comparison mechanism based on the pseudo label, resulting in limited structural distinguishability of positive and negative proposals in the semantic space. SUMMARY

[0005] The present application aims at the above-mentioned deficiencies in the prior art, and provides a weakly supervised dynamic optimization-based intelligent detection method and system for moving targets, as well as a corresponding computer terminal and computer readable storage medium.

[0006] According to one aspect of the present application, a weakly supervised dynamic optimization-based intelligent detection method for moving targets is provided, comprising:

[0007] Video features and text features of the original video and the text are extracted respectively, and the video features and the text features are fused to obtain cross-modal fusion features;

[0008] Based on the fusion features, frame-level semantic similarity scores are generated as pseudo labels;

[0009] Learnable object queries are used to interact with the fusion features to generate positive and negative proposal masks;

[0010] The pseudo labels are used to guide the feature comparison learning of the positive proposals, so that the positive proposals infinitely fit the text features in the semantic space, and the negative proposals infinitely deviate from the related areas of the text features;

[0011] The positive and negative proposal masks are used to perform text reconstruction based on a proposal mask conditional Transformer, and the different proposals are subjected to semantic consistency training to obtain a video time domain positioning result;

[0012] The video time domain positioning result is dynamically optimized to generate a final positioning result, and intelligent detection of the moving target is completed.

[0013] According to another aspect of the present application, a system for intelligent detection of a moving target based on weak supervision dynamic optimization is provided, comprising:

[0014] a feature extraction module configured to extract video features and text features of the original video and the text respectively, and to fuse the video features and the text features to obtain cross-modal fusion features;

[0015] a pseudo-label construction module configured to generate frame-level semantic similarity scores as pseudo-labels based on the fusion features;

[0016] a proposal generation module configured to generate positive and negative proposal masks by interacting a learnable object query with the fusion features;

[0017] a proposal optimization module configured to guide feature contrast learning of the positive proposal by using the pseudo-labels, so that the positive proposal infinitely fits the text features in a semantic space, and the negative proposal infinitely deviates from a relevant region of the text features;

[0018] a text reconstruction module configured to perform text reconstruction based on a proposal mask conditional Transformer by using the positive and negative proposal masks, and to perform semantic consistency training on different proposals to obtain a video time domain positioning result;

[0019] a positioning optimization module configured to dynamically optimize the video time domain positioning result to generate a final positioning result, and to complete intelligent detection of the moving target.

[0020] According to a third aspect of the present application, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program to perform the method described above, or to run the system described above.

[0021] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to perform the method described above, or to run the system described above.

[0022] Compared with the prior art, the present application has at least one of the following beneficial effects due to the adoption of the above technical solutions:

[0023] The application provides a weakly supervised dynamic optimization-based intelligent motion target detection method and system, which is used for weakly supervised video time domain target positioning tasks and is realized by dividing an alignment process between positive and negative proposals and text semantics into two stages of "pseudo label generation" and "proposal guidance". First, pseudo labels are obtained by modeling video-text feature correlation, and then the pseudo labels are used to guide the positive and negative proposals to be close to or far from semantic regions, so that stable and reliable supervision signals are provided, and the proposal quality and semantic matching accuracy are improved.

[0024] The weakly supervised dynamic optimization-based intelligent motion target detection method and system provided by the application realizes dynamic modeling of negative proposal difficulty and fine optimization of time sequence boundaries by using a weighted reconstruction and semantic comparison mechanism. First, the semantic modeling of each group of negative proposal masks is performed by a mask conditional Transformer, the alignment degree of the negative proposal masks with the text description is evaluated in combination with reconstruction error, and weights are allocated on the basis of the alignment degree to generate weighted negative proposal masks. Then, a progressive cross-entropy loss is designed, the model is guided to learn negative proposals from "semantically irrelevant" to "semantically similar" by a weight coefficient that increases with the training round, the discrimination ability of the model for negative samples is improved, and the semantic perception ability of the model for boundary regions of the text description is strengthened, so that more fine video time sequence positioning effects are realized under weak supervision. BRIEF DESCRIPTION OF DRAWINGS

[0025] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0026] Figure 1 A workflow diagram of the weakly supervised dynamic optimization-based intelligent motion target detection method in a preferred embodiment of the application.

[0027] Figure 2 A composition module schematic diagram of the weakly supervised dynamic optimization-based intelligent motion target detection system in a preferred embodiment of the application.

[0028] Figure 3 A flowchart of the weakly supervised dynamic optimization-based intelligent motion target detection method in a specific application example of the application.

[0029] Figure 4 A flowchart of weighted negative proposal mask generation and mask reconstruction in a specific application example of the application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation modes and specific operation processes are given. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0031] In the prior art, when facing the video time domain target positioning task, due to the lack of accurate timestamp labeling, the traditional method has great difficulty in constructing positive and negative proposals or realizing semantic alignment. On the one hand, the negative proposal generated statically cannot cover the complex and diverse video interference area, resulting in a lack of supervision signal in the training process; on the other hand, if difficult negative samples that are too close to the semantic boundary are introduced in the early training stage, the model may be misled, leading to error accumulation. In addition, the existing method lacks an effective semantic comparison mechanism in the proposal construction process, making it difficult to accurately evaluate the semantic consistency between the positive and negative proposals and the text, resulting in unstable proposal quality; at the same time, in weakly supervised training, the pseudo label is usually only used as a classification supervision signal or a static scoring basis, lacking deep combination and comparison modeling with the proposal structure, and failing to fully improve the discriminability of the positive and negative proposals in the semantic space, thereby limiting the model's ability in fine-grained semantic alignment and boundary discrimination.

[0032] To solve the above problems, an embodiment of the present application provides a weakly supervised dynamic optimization based intelligent detection method for moving targets, which designs a new paradigm for solving the video time domain target positioning task, obtains the semantic correlation information of the text and the video by introducing a pseudo label mechanism, and combines a learnable Gaussian proposal structure to realize the guidance and optimization of the positive and negative proposals through a dynamic course training strategy, thereby constructing a time sequence positioning technology with stronger semantic expression ability and discriminability.

[0033] Specifically, as shown in Figure 1 The weakly supervised dynamic optimization based intelligent detection method for moving targets provided by the embodiment can include the following steps:

[0034] S1, video features and text features of an original video and a text are respectively extracted, and the video features and the text features are fused to obtain cross-modal fusion features;

[0035] S2, based on the fusion features, a frame-level semantic similarity score is generated as a pseudo label;

[0036] S3, a positive and negative proposal mask is generated by using learnable object query and fusion feature interaction;

[0037] S4, the pseudo label is used to guide the feature contrast learning of the positive proposal, so that the positive proposal infinitely fits the text features in the semantic space, and the negative proposal infinitely deviates from the related area of the text features;

[0038] S5, text reconstruction is performed based on a proposal mask conditioned transformer (MCT) using positive and negative proposal masks, and semantic consistency training is performed on different proposals to obtain a video time domain positioning result;

[0039] S6, dynamic optimization is performed on the video time domain positioning result to generate a final positioning result, and intelligent detection of a moving target is completed.

[0040] In some preferred embodiments, S1 above, video features and text features of the original video and text are extracted respectively, and the video features and text features are fused to obtain cross-modal fusion features, which can further include:

[0041] S11, for a given original video and text, a pre-trained 3D convolutional network is used to extract time sequence features of the video frames, and a pre-trained GloVe model is used to extract word vector features of the text to obtain basic semantic representation;

[0042] S12, the time sequence features and the word vector features are mapped to a unified embedding space through a fully connected layer, and the context information of the time sequence features and the word vector features is encoded using a transformer to obtain cross-modal fusion features H containing semantic and time sequence relationships, represented as:

[0043] H=Dec(V,Enc(Q))

[0044] In the formula, Enc(·) represents an encoder; and Dec(·) represents a decoder.

[0045] In some preferred embodiments, S2 above, based on the fusion features, frame-level semantic similarity scores are generated as pseudo labels, which can further include:

[0046] In the pseudo label construction stage, a feedforward neural network (FFN) is used to process the cross-modal features H to predict a semantic correlation score S c .

[0047] S21, the fusion features H are input into the feedforward neural network FFN to obtain a correlation score between each frame of video and the text as a pseudo label, represented as:

[0048] S c =σ(FFN c (H))

[0049] In the formula, S c is a pseudo label; sigma is a Sigmoid activation function, representing the semantic correlation degree of each frame of video and the text; and FFN c is a feedforward neural network.

[0050] The higher the score, the stronger the relevance of the video segment to the semantic of the text. The score is used as a pseudo label in the training stage to assist in guiding the feature comparison between positive and negative proposals.

[0051] In some preferred embodiments, the S3, using the learnable object query to interact with the fusion feature, generates the positive and negative proposal masks, can further include:

[0052] S32, the generation of the proposal mask is based on the DETR decoder and the learnable query mechanism. K positive query vectors Q p ∈R K×D and mK negative query vectors Q n ∈R mK×D are input into a proposal generation model PFG (Proposal Feature Generator) together with the fusion feature H to obtain positive and negative proposal features, denoted as:

[0053] F p = PFG(Q p , H), F n = PFG(Q n , H)

[0054] In the formula, F p is the positive proposal feature; F n is the negative proposal feature; R represents the real number space; D is the dimension of each query vector.

[0055] S32, based on F p , F n predict the center position c k and the width ω k of the corresponding time period, and further construct the Gaussian mask function in the time dimension, defined as follows:

[0056]

[0057] In the formula, f (k) (t) is the value of the kth time proposal at the tth time position, which is used to construct the Gaussian mask corresponding to the kth time proposal; N is the total number of frames of the video; t is the index of the current time frame.

[0058] S33, the output of the Gaussian mask function generated based on the positive and negative proposal features at each time position is stacked in turn by row, and the positive proposal mask set M p ∈R K×N and the negative proposal mask set M n ∈R mK×N, for subsequent feature comparison and reconstruction training. In order to alleviate the problem of lack of accurate boundary label in the weakly supervised scene, a pseudo-label guided contrast loss is introduced, and the relative relationship between the positive and negative proposals and the text is constructed to realize the soft supervision learning of the semantic boundary.

[0059] In some preferred embodiments, S4 above utilizes pseudo-label guided positive proposal feature contrast learning to make the positive proposal infinitely fit the text features in the semantic space, and make the negative proposal infinitely deviate from the related area of the text features, which can further include:

[0060] S41, in order to improve the semantic discriminability of positive and negative proposals, a pseudo-label guided contrast loss L p As follows:

[0061]

[0062] In the formula, is the i-th negative proposal mask in the negative proposal mask set; is the i-th positive proposal mask in the positive proposal mask set; K is the number of positive query vectors; S c is the pseudo label; a p is a hyperparameter for controlling the contrast interval; T is a matrix transpose symbol;

[0063] S42, using the pseudo-label guided contrast loss L p guides the feature contrast learning of the positive proposal, so that in the semantic space, the positive proposal infinitely fits the text features, and the negative proposal infinitely deviates from the text features.

[0064] This loss encourages the weighted total score of the positive proposal on the pseudo label score to be higher than that of the negative proposal, which reflects that the positive proposal is more consistent with the text semantics, and the negative proposal is far away from the semantic related area, thereby improving the distinguishability of the positive and negative proposals, and providing high-quality candidates for the subsequent reconstruction module.

[0065] In some preferred embodiments, S5 above performs text reconstruction based on the proposal mask conditional Transformer, and performs semantic consistency training on different proposals, which can further include:

[0066] In order to further improve the discriminability of the model for easily confused fragments, a progressive negative proposal constraint mechanism is introduced. First, the mask conditional Transformer is used to extract the multi-modal reconstruction features corresponding to each negative proposal mask, and the reconstruction error of the text description is predicted; in order to realize the dynamic training strategy from easy to difficult, a weighting coefficient gradually increasing with the training round is introduced.

[0067] S51, in order to improve the semantic reconstruction ability, the mask conditional Transformer is used to perform text reconstruction on the positive proposal mask and the weighted negative proposal mask; wherein:

[0068] The weighted negative proposal is represented as:

[0069]

[0070] In the formula, is the weighted negative proposal mask corresponding to the kth positive proposal; is the Gaussian mask of the ith negative proposal associated with the kth positive proposal; m is the number of candidate negative proposals corresponding to each positive proposal; ω i is the importance weight of the ith negative proposal mask, obtained in the following way:

[0071]

[0072] In the formula, FFN w is a feedforward neural network; is the multi-modal feature of the ith set of negative proposal masks CLS representation of the

[0073] The positive proposal mask, the weighted negative proposal mask, and the mask corresponding to the original video are respectively input into the mask text Transformer to predict the masked text, reconstruct the original text, and calculate the text reconstruction loss L rec The semantic consistency of different proposals is evaluated; wherein the mask text refers to a mask input formed by randomly masking a set proportion of words from the original text

[0074] Further, the text reconstruction loss L rec is represented as:

[0075]

[0076] In the formula, respectively represent the positive proposal, the reference difficult negative proposal, and the progressive negative proposal cross-entropy loss, k * is the current loss minimum positive proposal number.

[0077] S52, select the sample with the largest reconstruction error in each set of negative proposals as the training target, and perform semantic consistency training on different proposals; let the current training round be e, and the maximum number of rounds be e max , define the dynamic adjustment weight as:

[0078]

[0079] In the formula, β and α are hyperparameters for controlling the training progress;

[0080] The final progressive negative proposal cross-entropy loss is represented as: ​

[0081]

[0082] wherein, is the cross-entropy loss of the weighted negative proposal; k * is the number of the positive proposal with the minimum current loss.

[0083] The loss mechanism prompts the model to focus on negative samples with a large semantic distance in the early stage, and gradually guides the model to distinguish "difficult negative samples" with close but irrelevant semantics in the later stage, effectively improving the semantic accuracy and generalization ability of the model in time domain positioning.

[0084] In some preferred embodiments, the above S6, dynamically optimizes the video time domain positioning result to generate the final positioning result, can further include:

[0085] The multi-loss joint optimization mechanism is used to realize the prediction output of the final time boundary; wherein:

[0086] The multi-loss includes: pseudo-label guided contrast loss L p , text reconstruction loss L rec , and an intra-video contrast loss L ivc and a proposal diversity loss L div ; the final optimization target of joint optimization is:

[0087] L total = λ p L p + λ rec L rec + λ ivc L ivc + λ div L div

[0088] wherein, λ p , λ rec , λ ivc and λ div are respectively.

[0089] Further, the intra-video contrast loss L ivc and the proposal diversity loss L div are obtained by the following ways respectively:

[0090]

[0091] wherein, respectively represent the cross-entropy loss of the positive proposal and the reference difficult negative proposal, k * is the number of the positive proposal with the minimum current loss, and β1 and β2 are hyperparameters of the contrast interval.

[0092] L div =‖Mp (M p ) T -λI‖ F

[0093] wherein M p is a set of positive proposal masks, T is a matrix transpose symbol, λ is a control coefficient for adjusting the degree of overlap, I is a unit matrix, ‖·‖ F is a Frobenius norm.

[0094] The motion target intelligent detection method based on weakly supervised dynamic optimization provided by the above-mentioned embodiments of the application comprises the following technical points:

[0095] For a given original video and text, first, a pre-trained 3D convolutional network is used to extract the time sequence features of the video frames, and a GloVe model is used to extract the word vector representation of the text. Then, the two features are mapped to a unified embedding space through a fully connected layer, and the context information of the video and the text is encoded using a Transformer to obtain a basic representation containing semantic and time sequence relationships, i.e., a cross-modal fusion feature is obtained.

[0096] A pseudo-label supervision mechanism is introduced, and combined with the DETR decoder structure, the centers and widths of positive and negative proposals are generated to construct a learnable Gaussian mask. The semantic alignment of the proposal and the text is guided by the pseudo-label, and the contrastive loss is used to optimize the discrimination of the positive and negative proposals. On the basis of inheriting the weak supervision idea, the structural expression capability and boundary discrimination effect are improved. By fusing the video and text features, a frame-level semantic similarity score is generated as a pseudo-label, and a contrastive loss is introduced to make the positive and negative proposals more discriminative in the semantic space.

[0097] To alleviate the problem that the negative proposal is too random in the early training stage and lacks supervision value in the later stage, a progressive negative proposal modeling mechanism is introduced to dynamically adjust the difficulty of the negative proposal according to the training stage, gradually transitioning from easy-to-negative proposals that are irrelevant in semantics to difficult-to-negative proposals that have semantic overlap with the text description, effectively improving the discrimination ability and training stability of the model at the boundary of positive and negative samples. The proposal mask is generated by the interaction of the learnable query and the fusion feature, and the progressive negative proposal modeling mechanism is designed to realize semantic modeling from easy to difficult, and to improve boundary discrimination and negative sample quality.

[0098] The masked conditioned transformer (MCT) is used to reconstruct the masked text using the proposal information, so as to evaluate the coverage of the proposal on semantics. In the training process, the positive proposal should be able to accurately reconstruct the text, representing a semantic segment highly consistent with the description; the negative proposal is far away from the text semantics at the beginning of the training, and it is difficult to restore the text. With the progress of the training, the content with a certain semantic overlap with the text description is gradually introduced, so that it produces greater prediction error in the reconstruction process, thereby strengthening the distinction between positive and negative proposals in semantic modeling, and improving the boundary judgment ability and the proposal expression effect. The semantic expression ability of various proposals is evaluated by reconstructing the original text, and the semantic separation effect of the proposal is enhanced.

[0099] The contrast loss constrains the semantic distinction between positive and negative proposals using frame-level semantic score; the diversity loss encourages the difference between positive proposals, avoiding convergence to the same position; the text reconstruction loss measures the coverage ability of the proposal on the description semantics; and the intra-video contrast loss further strengthens the relative advantages and disadvantages between positive and negative proposals and reference proposals. Multiple losses are optimized cooperatively to improve the positioning accuracy and stability from three aspects of semantic representation, time domain boundary and reconstruction ability. By fusing the pseudo-label guided contrast loss, reconstruction loss, diversity and contrast loss and other losses, the semantic alignment quality and boundary distinction ability are comprehensively optimized, and the positioning robustness of the model under weak supervision is improved.

[0100] In summary, the method provided by the above embodiments of the application designs a pseudo-label guided learnable proposal mechanism to realize the video time domain target positioning task under weak supervision. First, the features of the original input video and text are extracted to obtain semantic feature representations containing context information; second, the extracted video features and text features are cross-modally fused, and the semantic correlation score between each frame and the text is generated through a prediction network to construct a pseudo-label for subsequent training supervision; then, the pseudo-label guided learnable positive and negative proposals are generated, and the discrimination ability between the positive and negative proposals is optimized through the semantic contrast loss; further, the mask conditioned reconstruction technology is introduced to evaluate the semantic consistency of different proposals; finally, the training difficulty is dynamically adjusted through the progressive training strategy to guide the model to gradually learn more complex negative sample features, thereby improving the boundary discrimination ability and semantic perception ability. Without relying on artificial annotation of time boundaries, the method combines the pseudo-supervision signal and the learnable structure of the proposal mechanism to effectively alleviate the problem of insufficient training signals in the weak supervision scenario, improve the semantic alignment effect between the text and the video, and has good positioning accuracy and generalization ability.

[0101] Based on the same inventive concept, an embodiment of the application also provides a motion target intelligent detection system based on weak supervision dynamic optimization.

[0102] Specifically, as shown in Figure 2 The embodiment provides a weakly supervised dynamic optimization-based intelligent motion target detection system, which can include:

[0103] a feature extraction module, which is configured to extract video features and text features of the original video and the text respectively, and fuse the video features and the text features to obtain cross-modal fusion features;

[0104] a pseudo-label construction module, which is configured to generate a frame-level semantic similarity score as a pseudo-label based on the fusion features;

[0105] a proposal generation module, which is configured to generate positive and negative proposal masks by interacting a learnable object query with the fusion features;

[0106] a proposal optimization module, which is configured to guide feature contrast learning of the positive proposal by using the pseudo-label, so that the positive proposal infinitely fits the text features in a semantic space, and the negative proposal infinitely deviates from a relevant area of the text features;

[0107] a text reconstruction module, which is configured to perform text reconstruction based on a proposal mask conditional Transformer by using the positive and negative proposal masks, and perform semantic consistency training on different proposals to obtain a video time domain positioning result;

[0108] a positioning optimization module, which is configured to perform dynamic optimization on the video time domain positioning result to generate a final positioning result, and complete intelligent motion target detection.

[0109] The specific content of each functional module of the intelligent motion target detection system constituting the above embodiment of the application will be further described in detail below.

[0110] The feature extraction module is configured to extract features from the original video and the text respectively, obtain basic semantic representations, and fuse the extracted features to obtain cross-modal fusion features. Further comprising:

[0111] Given an uncut video sequence and the corresponding natural language description First, a pre-trained 3D convolutional neural network is used to extract video frame features, obtaining A pre-trained GloVe model is used to extract semantic embeddings of each word, obtaining text features

[0112] Subsequently, the text features are input into a Transformer encoder for context modeling, obtaining Enc(Q), and are interacted with the video features in a Transformer decoder to construct cross-modal fusion features, denoted as: H∈R X×D :

[0113] H = Dec(V, Enc(Q))

[0114] where X is the number of extracted video frames; D is the hidden representation dimension of cross-modal fusion features; Enc(·) represents an encoder, and Dec(·) represents a decoder.

[0115] The generated fusion feature H will be used as the basis feature for subsequent proposal generation and semantic supervision.

[0116] A pseudo-label construction module generates frame-level semantic similarity scores as pseudo-labels based on the fusion features. Further comprising:

[0117] The fusion feature H is input into a feedforward neural network FFN to obtain a correlation score between each frame of video and text:

[0118] S c = σ(FFN c (H))

[0119] where σ is a Sigmoid activation function, S c as a pseudo-label, represents the semantic correlation degree between each frame and text.

[0120] A proposal generation module introduces a contrastive loss to make positive and negative proposals more discriminative in semantic space. The process of constructing positive and negative proposals further comprises:

[0121] K positive query vectors Q p ∈ R K×D and mK negative query vectors Q n ∈ R mK×D are input into a proposal feature generator PFG together with H to obtain proposal features:

[0122] F p = PFG(Q p ,H), F n = PFG(Q n ,H)

[0123] Based on F p , F n , the center position c k and the width ω k of the corresponding time period are predicted, and a Gaussian mask function in the time dimension is further constructed, defined as follows:

[0124]

[0125] where f (k)(t) is the value of the kth time proposal at the tth time position, used to construct the Gaussian mask corresponding to the kth time proposal; N is the total number of frames of the video; t is the index of the current time frame;

[0126] The outputs of the Gaussian mask function generated based on the positive and negative proposal features at each time position are stacked in turn by row, and the positive proposal mask set M p ∈R K×N , and the negative proposal mask set M n ∈R mK×N

[0127] The proposal optimization module generates proposal masks by interacting with learnable queries and fusion features, designs a progressive negative proposal modeling mechanism, realizes semantic modeling from easy to difficult, and improves boundary discrimination and negative sample quality. Further comprising:

[0128] In order to improve the semantic discrimination ability of positive and negative proposals, the pseudo-label guided contrastive loss is constructed as follows:

[0129]

[0130] Wherein, α p is a hyperparameter for controlling the contrast interval.

[0131] The text reconstruction module evaluates the semantic expression ability of each type of proposal by reconstructing the original text, and enhances the semantic separation effect of the proposal. Further comprising:

[0132] In order to improve the semantic reconstruction ability of the model, the mask conditional Transformer module is used to reconstruct the text of the positive proposal and the weighted negative proposal. The negative proposal mask weighting method is as follows:

[0133]

[0134] Wherein, ω i is the importance weight of the ith negative proposal mask, which is obtained in the following way:

[0135]

[0136] The positive proposal mask, the weighted negative proposal mask and the reference proposal mask are respectively input into the Transformer with the mask text, which is used to reconstruct the original text, and the cross entropy loss L ce is calculated as the training signal.

[0137] In order to realize the progressive negative sample training strategy, the progressive cross entropy loss is introduced, and its weight is defined as:

[0138]

[0139] Among them, e is the current training round, e max is the maximum number of rounds, and β and α are hyperparameters that control the training progress.

[0140] The final progressive negative proposal constraint loss is expressed as:

[0141]

[0142] Among them, k * The number of the positive proposal with the smallest current loss.

[0143] The positioning optimization module integrates pseudo-label guided contrast loss, reconstruction loss, diversity and contrast loss, and other modules to comprehensively optimize the semantic alignment quality and boundary differentiation ability, and improve the positioning robustness of the model under weak supervision conditions. It further includes:

[0144] Integrate multiple loss functions to build a multi-objective optimization framework. First, the pseudo-label guided positive and negative proposal comparison loss L p Used to improve the semantic discrimination ability of proposals; text reconstruction loss L rec It consists of a positive proposal and a reference proposal, and is used to optimize the alignment between the proposal and the language semantics.

[0145] Furthermore, the intra-video contrast loss L is introduced ivc , encourages positive proposals to have advantages over other fragments in semantic reconstruction; at the same time, the proposal diversity loss L is added div , to prevent multiple proposal masks from highly overlapping and improve the structural expression ability. The final training goal is:

[0146] L total =λ p L p +λ rec L rec +λ ivc L ivc +λ div L div .

[0147] The following is a detailed description of the technical solution provided by the above embodiment of the present invention in conjunction with a specific application example. In this specific application example, the workflow diagram of the moving target intelligent detection method based on weakly supervised dynamic optimization is as follows: Figure 3 shown.

[0148] The first step is to use the feature extractor to extract relevant features from the original input video and text, and then perform feature fusion;

[0149] Given an uncut video sequence and the corresponding natural language description The first step is to use a pre-trained 3D convolutional neural network to extract video frame features, and obtain The semantic embedding of each word is extracted using GloVe to obtain text features Then the text features are input into the Transformer encoder for context modeling to obtain Enc(Q), and interact with the video features in the Transformer decoder to build cross-modal fusion feature representation H∈R X×D :

[0150] H=Dec(V,Enc(Q))

[0151] The fusion feature H will be used as the basis for subsequent proposal generation and semantic supervision.

[0152] The second step is to generate frame-level pseudo labels based on the fusion features, and combine the learnable query to build Gaussian mask proposals, to realize the guidance and differentiation of positive and negative proposals;

[0153] After obtaining the cross-modal fusion feature H∈R X×D After that, the first step is to predict the semantic relevance of each frame video segment through a feedforward neural network to obtain the pseudo label score sequence S c That is:

[0154] S c =σ(FFN c (H))

[0155] Where σ is the Sigmoid function, FFN c represents the feedforward prediction head, and the pseudo label is used to measure the semantic matching degree between each time step and the text. The pseudo label will provide a soft guidance signal at the frame level in the subsequent positive and negative proposal supervision.

[0156] The third step is to establish a learnable time sequence candidate segment under weak supervision. This step introduces the learnable query mechanism in the DETR decoder structure. K positive proposal queries Q p and mK negative proposal queries Q n are constructed respectively, and interact with the fusion feature H to generate corresponding proposal representations:

[0157] F p =PFG(Q p ,H),F n =PFG(Q n ,H)

[0158] Based on F p , F n predict the center position c k and the width ω k of the corresponding time period, and further construct the Gaussian mask function in the time dimension, defined as follows:

[0159]

[0160] where f (k) (t) is the value of the kth temporal proposal at the tth time position, used to construct the Gaussian mask corresponding to the kth temporal proposal; N is the total number of frames of the video; t is the index of the current time frame;

[0161] The output of the Gaussian mask function generated based on the positive and negative proposal features at each time position is stacked in sequence by row, and the positive proposal mask set M p ∈R K×N and the negative proposal mask set M n ∈R mK×N

[0162] In the fourth step, in order to improve the semantic discrimination ability between proposals, a pseudo-label guided contrastive loss function is introduced to constrain the difference in semantic scores between positive and negative proposals, which is defined as follows:

[0163]

[0164] where α p is a hyperparameter that controls the interval, ensuring that the positive proposal is closer to the text semantic region, while the negative proposal maintains a distance from it. Through this mechanism, the model can obtain stable semantic guidance in a weakly supervised scenario, thereby optimizing the proposal quality and matching accuracy.

[0165] In the fifth step, a mask-conditioned text reconstruction mechanism is introduced to guide the model to learn more refined semantic alignment and context restoration ability by weighted fusion of multiple negative proposal masks, as shown in Figure 4 .

[0166] To further improve the semantic alignment ability between the proposal and the text, a mask-conditioned Transformer module is introduced in this step to assist in optimizing the semantic representation of the proposal through the text reconstruction task. In particular, for the diversity of negative proposals, a weight-based mask fusion mechanism is designed to dynamically select representative masks from multiple negative proposals.

[0167] First, for each group of negative proposal masks extract the multi-modal features and take their CLS representation input the feedforward network to calculate the weight and perform Softmax normalization:

[0168]

[0169] Accordingly, the weighted fused negative proposal mask is obtained:

[0170]

[0171] The mask is input into a reconstruction module together with the positive proposal mask, the reference mask, and the mask text , a prediction of the occluded text is calculated, and a reconstruction loss L rec is calculated, so as to enhance the perception ability of the model to the semantic coverage of the proposal.

[0172] The loss weight dynamically changes with the training round, progressive difficulty modeling is performed on the negative proposal, and the gradual perception and discrimination ability of the model to the semantic boundary region is realized.

[0173] To improve the discrimination ability of the model to the semantic boundary, a progressive training strategy is introduced in the text reconstruction module in this step, and the difficulty of negative proposal supervision is dynamically controlled. By adjusting the influence coefficient of the negative proposal cross-entropy loss, the training process gradually transitions from the "semantically distinct" negative samples to the "boundary confused" high-difficulty negative samples.

[0174] Specifically, first, the sample with the maximum reconstruction error in each group of negative proposals is selected as the training target. Let the current training round be e, and the maximum round be e max , the dynamic adjustment weight is defined as:

[0175]

[0176] Where β and α are hyperparameters that control the intensity and rate of change. The final negative proposal cross-entropy loss is:

[0177]

[0178] Where k * is the current loss minimum positive proposal number. This mechanism ensures that the model focuses on learning easy-to-distinguish negative samples in the early stage, and gradually focuses on boundary ambiguous samples in the later stage, realizing dynamic refinement of semantic understanding granularity.

[0179] Step 6: Utilize the multi-item loss joint optimization mechanism and learnable proposal parameter regression to realize the final time boundary prediction output;

[0180] In the model training phase, this step integrates multiple loss functions to build a multi-objective optimization framework. First, the positive and negative proposal comparison loss L p guided by the pseudo label is used to improve the semantic discrimination ability of the proposal; the text reconstruction loss L rec is composed of the positive proposal and the reference proposal, and is used to optimize the alignment consistency between the proposal and the language semantics.

[0181] Further, the intra-video comparison loss L ivc is introduced to encourage the positive proposal to have an advantage in semantic reconstruction over other segments; at the same time, the proposal diversity loss L div, to prevent multiple proposal masks from being highly overlapped and improve the structural representation capability. The final training objective is:

[0182] L total = lambda p L p + lambda rec L rec + lambda ivc L ivc + lambda div L div

[0183] The technical effects of the technical solutions provided by the above embodiments of the application are evaluated below in combination with a specific verification example.

[0184] In the specific verification example:

[0185] I. Data composition, including:

[0186] The data sets used for evaluation are Charades-STA and ActivityNet Captions respectively. For the video part, Charades-STA uses a pre-trained I3D model to extract features, and ActivityNet Captions uses a C3D model. For the text part, GloVe embedding is used to obtain word vector representation, and a 300-dimensional word embedding vector is extracted for each word in the text description. The above features are used as the input of multi-modal fusion for subsequent cross-modal alignment and proposal modeling.

[0187] II. Evaluation criteria, including:

[0188] R@a,IoU>=b and mIoU are used as evaluation indexes. R@a,IoU>=b represents the percentage of test samples in which at least one prediction has an IoU value greater than or equal to b among the top-a predicted results with respect to the total test samples; mIoU represents the average IoU of all test samples. Here, a is set to be in {1, 5} and b is set to be in {0.1, 0.3, 0.5, 0.7}.

[0189] III. Comparison algorithm, including:

[0190] In the comparative experiment, the present application is compared with several algorithms which are the most advanced in recent years on Charades-STA and ActivityNet Captions data sets, including CTF, RTBPN, VCA, LCNet, CPL and CPL-R. It is worth noting that, due to the significant difference between the reproduction results of the CPL method in the experimental process and the performance results given in the original paper, this difference is particularly pronounced on the ActivityNet Captions data set, therefore, in order to compare more fairly, the reproduction results of the CPL method in the experimental process are named CPL-R and included in the comparison.

[0191] Table 1 shows the performance of the present application compared with various existing methods on the Charades-STA validation set. From the table, it can be seen that the time domain target positioning algorithm proposed by the present application improves by 4.59% compared with the current best baseline method CPL-R under the evaluation index R@1, IoU = 0.7, showing strong positioning accuracy advantage. At the same time, the method exceeds the CPL-R method by 3.17% and 1.01% respectively in R@1mIoU and R@5mIoU two indexes, verifying the advantage of the proposed mechanism in comprehensive positioning accuracy.

[0192] Table 2 shows the comparison results of the present application method and the existing representative method on the ActivityNet Captions data set. From the table, it can be seen that under the R@1 index, the overall performance of the method in this paper is slightly better than the baseline method CPL and CPL-R; while under the R@5 index, the method of the present application is significantly better than CPL in multiple settings, showing strong positioning recall ability. The method of the present application has shown good performance under the condition of complete weak supervision, verifying its feasibility in practical application.

[0193] Therefore, under the verification of the comparative experiment, the algorithm proposed by the present application, under the synergistic action of the learnable proposal mechanism and the pseudo-label supervision, can more effectively capture the time segment features highly related to the text semantics, strengthen the cross-modal alignment between the video and the text, and improve the accuracy and robustness of the positioning boundary.

[0194] Table 1 Performance comparison of the algorithm proposed by the present application on the Charades-STA validation set

[0195]

[0196] Table 2 Performance comparison of the algorithm proposed by the present application on the ActivityNet Captions validation set

[0197]

[0198]

[0199] The embodiment of the present application also provides a computer terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to execute the method of any one of the above-mentioned embodiments of the present application or run the system of any one of the above-mentioned embodiments of the present application.

[0200] Optionally, the memory is used for storing programs; the memory can comprise volatile memory (for example, random-access memory (RAM), such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM) and the like) and non-volatile memory (for example, flash memory). The memory is used for storing computer programs (for example, application programs, functional modules and the like for implementing the above-mentioned method), computer instructions and the like; the above-mentioned computer programs, computer instructions and the like can be stored in one or more memories in a partitioned manner. Furthermore, the above-mentioned computer programs, computer instructions and data can be invoked by the processor.

[0201] The processor is used for executing the computer program stored in the memory to implement each step in the method or each module of the system according to the above-mentioned embodiments. Details can be referred to the related description in the above-mentioned method and system embodiments.

[0202] The processor and the memory can be an independent structure or an integrated structure. When the processor and the memory are an independent structure, the memory and the processor can be coupled and connected through a bus to transmit data signals. The embodiment of the present application does not make redundant description hereon.

[0203] The execution subject of the above-mentioned processor and memory can be a computer, a single-chip microcomputer, a microcontroller and the like computer terminal devices with computing functions; in a specific implementation, the embodiment of the present application does not limit the execution subject, which is selected according to the actual application needs.

[0204] The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted from a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that is accessible by a computer or a data storage device, such as a server, data center, etc., integrated with one or more available media. The available medium can be a magnetic medium, a semiconductor medium, etc.

[0205] An embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to perform the method of any of the above embodiments of the present application, or to run the system of any of the above embodiments of the present application.

[0206] The computer readable medium includes a computer storage medium and a communication medium. The communication medium includes any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a component of the processor. Suitable processors include, by way of example, both general and special purpose microprocessors. Of course, the degree of processor capability can vary depending on the exact configuration of the processor. According to an example embodiment, the processor can be configured to employ a set of rules to change an element of a data stream on the storage medium. Among other things, the processor can be configured to implement various algorithms for processing the data stream. According to an example embodiment, the processor can be configured to execute instructions stored on the storage medium to perform desired operations.

[0207] The computer readable storage medium includes, but is not limited to, flash memory, hard disk, solid state disk, etc.

[0208] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated.

[0209] The motion target intelligent detection method and system based on weak supervision dynamic optimization provided by the above embodiments of the present application extract video and text features from the input uncut video and the corresponding natural language description respectively, realize multi-modal feature interaction through a cross-modal Transformer encoder-decoder, and construct a cross-modal semantic representation; a plurality of positive and negative Gaussian proposals are generated by using a learnable object query, and a pseudo-label guided contrast loss function is introduced to improve the discrimination ability of the positive and negative proposals; further, the positive and negative proposals are text reconstructed by combining a mask conditional Transformer module, the semantic alignment effect is evaluated, and a progressive negative proposal constraint strategy is introduced to dynamically evolve the negative proposals from easy to difficult; finally, a variety of supervision signals are fused to optimize the accurate positioning ability of the model to the semantic segments in the video segment. By constructing a learnable negative proposal and a dynamic pseudo-label constraint mechanism, the time domain positioning accuracy under weak supervision conditions is significantly improved.

[0210] The details not described in the above embodiments of the present application are known in the art.

[0211] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A method for intelligent detection of moving targets based on weakly supervised dynamic optimization, characterized in that: include: Extracting video features and text features of the original video and text respectively, and fusing the video features and the text features to obtain cross-modal fusion features; Based on the fused features, generating a frame-level semantic similarity score as a pseudo label; Generate positive and negative proposal masks by using learnable object queries and fused features; Using the pseudo labels to guide feature contrast learning of positive proposals, the positive proposals are made to infinitely fit the text features in the semantic space, and the negative proposals are made to infinitely deviate from the relevant areas of the text features; Using the positive and negative proposal masks, text reconstruction is performed based on the proposal mask conditional Transformer, and semantic consistency training is performed on different proposals to obtain the video time domain positioning result; Dynamically optimize the video time domain positioning results to generate final positioning results and complete intelligent detection of moving targets; in: The proposed mask-conditional Transformer performs text reconstruction and semantic consistency training on different proposals, including: Use mask-conditional Transformer to reconstruct text from positive proposal masks and weighted negative proposal masks; where: The weighted negative proposal mask is expressed as: Where, is the weighted negative proposal mask corresponding to the k-th positive proposal; is the Gaussian mask of the i-th negative proposal associated with the k-th positive proposal; m is the number of candidate negative proposals corresponding to each positive proposal; ω i is the importance weight of the i-th negative proposal mask, which is obtained as follows: Where FFN w It is a feedforward neural network; is the negative proposal mask of group i Multimodal features CLS characterization; The positive proposal mask, weighted negative proposal mask, and the mask corresponding to the entire original video are input into Transformer along with the masked text, the masked text is predicted, the original text is reconstructed, and the text reconstruction loss L is calculated. rec , perform semantic consistency evaluation on different proposals; wherein, the masked text refers to the masked input formed by randomly masking a set proportion of words from the original text Select the sample with the largest reconstruction error in each group of negative proposals as the training target, and perform semantic consistency training on different proposals; let the current training round be e, and the maximum number of rounds be e max , define the dynamic adjustment weight as: Where β and α are hyperparameters that control the training progress; Final progressive negative proposal cross entropy loss Expressed as: Where, is the cross entropy loss of weighted negative proposals; k * The number of the positive proposal with the smallest current loss.

2. The method for intelligent detection of moving targets based on weakly supervised dynamic optimization according to claim 1, characterized in that: The extracting of video features and text features from the original video and text respectively, and fusing the video features and text features to obtain cross-modal fusion features, includes: For a given raw video and text, a pre-trained 3D convolutional network is used to extract the temporal features of the video frames, and a pre-trained GloVe model is used to extract the word vector features of the text. The time series features and the word vector features are mapped to a unified embedding space through a fully connected layer, and the context information of the time series features and the word vector features are interactively encoded using the Transformer encoder and decoder to obtain a cross-modal fusion feature H that contains semantic and time series relationships, which is expressed as: H=Dec(V,Enc(Q)) Where Enc(·) represents the encoder; Dec(·) represents the decoder.

3. The method for intelligent detection of moving targets based on weakly supervised dynamic optimization according to claim 1, characterized in that: Generating a frame-level semantic similarity score as a pseudo label based on the fusion feature includes: The fusion feature H is input into the feedforward neural network to obtain the correlation score between each frame of video and text as a pseudo label, which is expressed as: S c =σ(FFN c (H)) Where S c is a pseudo label; σ is a Sigmoid activation function, which indicates the semantic relevance between each frame of video and text; FFN c It is a feed-forward neural network.

4. The method for intelligent detection of moving targets based on weakly supervised dynamic optimization according to claim 1, characterized in that: The method of using learnable object query and fusion features to generate positive and negative proposal masks includes: Introduce K positive query vectors Q p ∈R K×D and mK negative query vectors Q n ∈R mK×D , and input it into a proposal generation model PFG (Proposal Feature Generator) together with the fusion feature H to obtain positive and negative proposal features, expressed as: F p =PFG(Q p ,H),F n =PFG(Q n ,H) Where, F p is a positive proposal feature; F n is the negative proposal feature; R represents the real number space; D is the dimension of each query vector; Based on the F p 、F n Predict the center position c of the corresponding time period k and width ω k , construct the Gaussian mask function on the time dimension, which is expressed as follows: Where, f (k) (t) is the value of the k-th time proposal at the t-th time position, which is used to construct the Gaussian mask corresponding to the k-th time proposal; N is the total number of frames in the video; t is the index of the current time frame; The outputs of the Gaussian mask function generated based on the positive and negative proposal features at each time position are stacked row by row to construct the positive proposal mask set M p ∈R K×N and the negative proposal mask set M n ∈R mK×N .

5. The method for intelligent detection of moving targets based on weakly supervised dynamic optimization according to claim 1, characterized in that: The method of using the pseudo labels to guide the feature contrast learning of the positive proposal so that the positive proposal infinitely fits the text feature in the semantic space and the negative proposal infinitely deviates from the relevant area of ​​the text feature includes: Construct pseudo-label guided contrast loss L p As shown below: Where, is the i-th negative proposal mask in the negative proposal mask set; is the i-th positive proposal mask in the positive proposal mask set; K is the number of positive query vectors; S c is a pseudo label; p is a hyperparameter that controls the contrast interval; T is the symbol for matrix transposition; Use the pseudo labels to guide the contrast loss L p , guiding the feature contrast learning of the positive proposal, so that in the semantic space, the positive proposal infinitely fits the text feature, while the negative proposal infinitely deviates from the text feature.

6. The method for intelligent detection of moving targets based on weakly supervised dynamic optimization according to claim 1, characterized in that: The dynamically optimizing the video time domain positioning result to generate a final positioning result includes: The prediction output of the final time boundary is achieved by using a multi-loss joint optimization mechanism; among them: The multiple losses include: pseudo-label guided contrast loss L p , text reconstruction loss L rec And a video contrast loss L ivc and a proposal diversity loss L div ; The final optimization goal of the joint optimization is: L total =λ p L p +λ rec L rec +λ ivc L ivc +λ div L div Where λ p ,λ rec ,λ ivc and λ div The four coefficients are hyperparameters used to adjust the impact of each loss on model training.

7. An intelligent moving target detection system based on weakly supervised dynamic optimization, characterized in that: include: A feature extraction module is used to extract video features and text features of the original video and text respectively, and fuse the video features and the text features to obtain cross-modal fusion features; a pseudo-label construction module, which generates a frame-level semantic similarity score as a pseudo-label based on the fused features; A proposal generation module that uses learnable object queries and fused features to generate positive and negative proposal masks; A proposal optimization module, which uses the pseudo-labels to guide feature contrast learning of positive proposals, so that the positive proposals infinitely fit the text features in the semantic space and the negative proposals infinitely deviate from the relevant areas of the text features; A text reconstruction module, which uses the positive and negative proposal masks to reconstruct text based on the proposal mask conditional Transformer, and performs semantic consistency training on different proposals to obtain video temporal localization results; A positioning optimization module, which is used to dynamically optimize the video time domain positioning results, generate the final positioning results, and complete the intelligent detection of moving targets; in: The text reconstruction module reconstructs text based on the proposal mask conditional Transformer and performs semantic consistency training on different proposals, including: Use mask-conditional Transformer to reconstruct text from positive proposal masks and weighted negative proposal masks; where: The weighted negative proposal mask is expressed as: Where, is the weighted negative proposal mask corresponding to the k-th positive proposal; is the Gaussian mask of the i-th negative proposal associated with the k-th positive proposal; m is the number of candidate negative proposals corresponding to each positive proposal; ω i is the importance weight of the i-th negative proposal mask, which is obtained as follows: Where FFN w It is a feedforward neural network; is the negative proposal mask of group i Multimodal features CLS characterization; The positive proposal mask, weighted negative proposal mask, and the mask corresponding to the entire original video are input into Transformer along with the masked text, the masked text is predicted, the original text is reconstructed, and the text reconstruction loss L is calculated. rec , perform semantic consistency evaluation on different proposals; wherein, the masked text refers to the masked input formed by randomly masking a set proportion of words from the original text Select the sample with the largest reconstruction error in each group of negative proposals as the training target, and perform semantic consistency training on different proposals; let the current training round be e, and the maximum number of rounds be e max , define the dynamic adjustment weight as: Where β and α are hyperparameters that control the training progress; Final progressive negative proposal cross entropy loss Expressed as: Where, is the cross entropy loss of weighted negative proposals; k * The number of the positive proposal with the smallest current loss.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 6, or run the system according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 6, or to run the system according to claim 7.

Images