Video moment positioning method and device, equipment and medium

Abstract

The invention belongs to the technical field of video moment positioning, and discloses a video moment positioning method, device and equipment and a medium, and the method comprises the steps: training a VTG model through employing a comparison loss function, and obtaining a target VTG model; using the target VTG model to generate a plurality of candidate frames and a first coarseness score; constructing a target video-text model, and extracting video features and text features of each candidate box by using a target video-text pre-training model; calculating a fine-grained score of each candidate frame according to the video features and the text features, calculating a total score of the candidate frames according to the first coarse-grained score and the fine-grained score, and performing fine-grained sorting on each candidate frame by using the total score; and carrying out video moment positioning on the input video according to the fine-grained sorting. By implementing the method, the problem of imbalance of positive and negative samples is solved, and the robustness of the model is improved. The matching of fine-grained sorting to downstream tasks is improved, and the accurate sorting of candidate boxes is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of video moment positioning technology in financial technology, and in particular to a method, device, equipment and medium for video moment positioning. Background Art

[0002] In order to analyze and locate key financial events, market trends, and risk signs in videos to help financial practitioners make more accurate and timely decisions, the FinTech industry needs to process financial data and multimodal information on a large scale.

[0003] There are many excellent video temporal localization (VTG) models, which take the entire video as input, the video and query text as input, and output multiple candidate boxes and corresponding coarse-grained scores. However, the training of many VTG models only focuses on positive samples, while ignoring a large number of negative samples, resulting in an imbalance between positive and negative samples, which also reduces the robustness of the VTG model to a certain extent.

[0004] When performing video moment retrieval tasks, existing VTG models generally use the indicator R@K, IoU@m model for verification evaluation, which indicates the percentage of the retrieved top-K candidate boxes that have at least one candidate box with an IoU greater than m with the true interval. However, firstly, since the indicator R@K, IoU@m model treats the top-K as a set and does not consider their ranking, it is not sensitive to ranking. Secondly, it uses the threshold θ to binarize the IoU of each video moment, so it is not sensitive to positioning, that is, the exact position (starting point and end point) of the video moment does not affect the indicator value. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a method, device, equipment and medium for video moment positioning to solve the current problems of imbalance between positive and negative samples and low matching degree of video features of candidate frames for downstream tasks.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a method for video moment positioning, which adopts the following technical solution, including: using positive samples and negative samples to obtain a contrast loss function, and using the contrast loss function to train a VTG model to obtain a target VTG model;

[0007] Input the entire video into the target VTG model, and use the target VTG model to generate multiple candidate boxes and a first coarse-grained score;

[0008] Constructing a target video-text model, and using the target video-text model to extract video features and text features of each candidate frame;

[0009] Calculating a fine-grained score for each candidate box according to the video features and the text features, calculating a total score for the candidate box according to the first coarse-grained score and the fine-grained score, and performing fine-grained sorting on each candidate box using the total score;

[0010] The video time is located for the input video according to the fine-grained sorting.

[0011] Furthermore, the use of positive samples and negative samples to obtain a contrast loss function includes:

[0012] Using the VTG model to obtain multiple candidate frames of the input video, performing a coarse-grained scoring on each candidate frame, obtaining a second coarse-grained score corresponding to a positive sample of the input video, and taking a difference between the second coarse-grained score to obtain a third coarse-grained score corresponding to a negative sample of the input video;

[0013] The contrast loss function is obtained by using the second coarse-grained score and the third coarse-grained score. The calculation formula of the contrast loss function is:

[0014]

[0015] in, is a positive sample, is a random contrast negative sample from the same batch, is the second coarse-grained score corresponding to the positive sample, is the third coarse-grained score corresponding to the negative sample.

[0016] Furthermore, the step of inputting the entire video into the target VTG model and using the target VTG model to generate a plurality of candidate boxes and a first coarse-grained score includes:

[0017] The entire video and the query text are input into the target VTG model, the target VTG model extracts key frames from the entire video according to the query text, and generates candidate frames on the key frames using a window sliding method;

[0018] The image features of the candidate box are extracted, and the candidate box is scored using a classifier to obtain the first coarse-grained score.

[0019] Furthermore, the constructing of the target video-text model includes:

[0020] Configure the transformer module in the neural network of the video-to-text model;

[0021] A feedforward neural network layer and a ReLU layer are sequentially arranged at the output end of the multi-head attention layer of the converter module to construct the target video-text model. The feedforward neural network layer and the ReLU layer constitute the adapter module.

[0022] Furthermore, the extracting the video features and text features of each candidate frame by using the target video-text model includes:

[0023] Using the target video-text model to obtain preliminary video features and corresponding text features of each candidate box, and using the converter module to align the preliminary video features with the text features;

[0024] Using the adapter module to learn and acquire adaptive video features, summing the preliminary video features and the adaptive video features to obtain adapted video features of the video frame;

[0025] The adapted video features of all the video frames of the candidate frame are obtained, and the adapted video features of all the video frames are averaged to obtain the video features of each candidate frame.

[0026] Furthermore, the calculating the total score of the candidate box according to the first coarse-grained score and the fine-grained score of the candidate box, and using the total score to perform fine-grained sorting on each candidate box, includes:

[0027] The first coarse-grained score and the fine-grained score of each candidate box are respectively normalized to obtain a coarse-grained normalized score and a fine-grained normalized score;

[0028] The coarse-grained normalized score and the fine-grained normalized score are summed to obtain a total score of the candidate box;

[0029] The candidate boxes are sorted in fine granularity according to the total scores.

[0030] Furthermore, after performing video time location on the input video according to the fine-grained sorting, the method for video time location further includes:

[0031] Obtain fine-grained normalized scores of candidate boxes for the target video-text model;

[0032] Calculate the AxIoU@K value of each sample using the fine-grained normalized score, and calculate the average value based on the AxIoU@K values ​​of all samples;

[0033] The performance of the target video-text model is evaluated based on the average value.

[0034] In order to solve the above problem, a device for video time positioning is also provided, which adopts the method for video time positioning, including:

[0035] An acquisition module, used to obtain a contrast loss function using the positive samples and the negative samples, and train the VTG model using the contrast loss function to obtain a target VTG model;

[0036] A generation module, inputting the entire video into the target VTG model, and using the target VTG model to generate multiple candidate boxes and a first coarse-grained score;

[0037] An extraction module, used to construct a target video-text model and extract video features and text features of each candidate frame using the target video-text model;

[0038] A sorting module, used to calculate the fine-grained score of each candidate box according to the video features and the text features, calculate the total score of the candidate box according to the first coarse-grained score and the fine-grained score of the candidate box, and perform fine-grained sorting on each candidate box using the total score;

[0039] A positioning module is used to locate the video time of the input video according to the fine-grained sorting.

[0040] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for video moment positioning as described in the first aspect are implemented.

[0041] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the method for video moment positioning as described in the first aspect are implemented.

[0042] Compared with the prior art, the embodiments of the present application mainly have the following technical effects:

[0043] By performing positive and negative sample comparison learning to obtain the target VTG model, the problem of imbalance between positive and negative samples is solved, which helps to improve the robustness of the model and is applicable to various multimodal data processing tasks. By building a target video-text model, calculating the fine-grained score of each candidate frame and the total score of the candidate frame, and using the total score to perform fine-grained sorting of each candidate frame, the matching of fine-grained sorting for downstream tasks is improved, and the accurate sorting of candidate frames is enhanced, which helps to improve the performance of multi-scale frameworks in the field of multimodal data processing and provide more accurate information retrieval and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flow chart of an embodiment of a method for video moment positioning according to the present application;

[0046] Figure 2 yes Figure 1 A flowchart of a specific implementation of step 100;

[0047] Figure 3 yes Figure 1 A flowchart of a specific implementation of step 200;

[0048] Figure 4 yes Figure 1 A flowchart of a specific implementation of step 300 of constructing a target video-text model;

[0049] Figure 5 yes Figure 1 A flowchart of a specific implementation of extracting video features and text features of each candidate box using the target video-text model in step 300;

[0050] Figure 6 yes Figure 1 A flowchart of a specific implementation of step 400;

[0051] Figure 7 yes Figure 1 A flowchart of a specific implementation method after step 500;

[0052] Figure 8 A schematic diagram of the overall process of a method for video moment positioning according to the present application;

[0053] Fig. 9 is a structural schematic diagram of an embodiment of a video moment positioning device according to the present application;

[0054] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0058] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0059] The method for video moment positioning provided in the embodiment of the present application can be applied to a computer device with the ability to run instructions or programs. Among them, the computer device can be, but is not limited to, various servers, workstations, personal computers, laptops, etc. Running on different computing devices is only a difference in the execution subject of the scheme. People in this field can foresee that running on different computing devices can produce the same technical effect. The present invention is described in detail below through specific embodiments.

[0060] The purpose of the embodiments of the present application is to propose a method, device, equipment and medium for video moment positioning to solve the current problems of imbalance between positive and negative samples, low matching degree of video features of candidate frames for downstream tasks, and excessive reliance on threshold θ for verification indicators.

[0061] In order to solve the above technical problems, the embodiment of the present application provides a method for video time positioning, which adopts the following technical solutions: Figure 1 , Figure 1 This is a flowchart of an embodiment of a method for video moment positioning according to the present application, including:

[0062] S100, using positive samples and negative samples to obtain a contrast loss function, and using the contrast loss function to train the VTG model to obtain a target VTG model.

[0063] In this embodiment, the video features presented by the positive sample data of the input video meet the desired goals, including the target features, and the accuracy and robustness of the target video-text model can be improved by using the positive sample data for training and learning.

[0064] The video features presented by the negative sample data of the input video do not meet the expected goals and usually do not include the target features. However, training and learning the model through negative sample data can improve the generalization ability of the target video-text model and prevent overfitting.

[0065] like Figure 8 , Figure 8 A schematic diagram of the overall process of a method for video moment positioning according to the present application; the overall principle of the embodiment of the present application is:

[0066] At the coarse-grained level, contrastive learning is performed between positive and negative samples to construct a contrastive learning loss function, which is used to train the VTG model. Secondly, candidate boxes are generated based on the trained VTG model and a coarse-grained score is assigned to each candidate box.

[0067] At the fine-grained level, an adapter module for learning adaptive video features is set in the video-text pre-trained model, and the video-text pre-trained model is improved to obtain a target video-text model based on clip improvement to obtain candidate box features and calculate fine-grained matching scores, and further accurately sort the candidate boxes in a fine-grained manner.

[0068] S200, input the entire video into the target VTG model, and use the target VTG model to generate multiple candidate boxes and a first coarse-grained score.

[0069] In this embodiment, the coarse-grained candidate frame refers to a relatively broad area range that is initially determined in the video moment positioning task. For example, in a sports event video, to locate the video moment of a wonderful goal, a coarse-grained candidate frame can be generated based on the audience's voice, and then fine-grained sorting and positioning can be performed based on the player's action.

[0070] When the target VTG model generates multiple candidate boxes, it also generates corresponding coarse-grained candidate box scores, that is, the first coarse-grained scores.

[0071] In some optional embodiments, such as Figure 2 , Figure 2 yes Figure 1A flowchart of a specific implementation method of S100; S100 includes: S110, using the VTG model to obtain multiple candidate frames of the input video, performing coarse-grained scoring on each candidate frame, obtaining a second coarse-grained score corresponding to the positive sample of the input video, and taking the difference of the second coarse-grained score to obtain a third coarse-grained score corresponding to the negative sample of the input video; S120, using the second coarse-grained score and the third coarse-grained score to obtain a contrast loss function.

[0072] The training of existing VTG models often focuses only on positive samples, while ignoring a large number of negative samples, which leads to an imbalance between positive and negative samples and reduces the robustness of the VTG model to a certain extent.

[0073] In this embodiment, in order to solve the above problem, this embodiment constructs a contrastive learning loss function, such as formula (1), to distinguish positive samples from negative samples through contrastive learning, so that the score of positive samples is close to 1 and the score of negative samples is close to 0.

[0074]

[0075] in, is the contrast loss function, is a positive sample, is a random negative sample from the same batch, s i is the corresponding coarse-grained score.

[0076] In the above formula (1) is the second coarse-grained score corresponding to the positive sample, is the third coarse-grained score corresponding to the negative sample.

[0077] By constructing a contrastive loss function to train the VTG model, effective video features can be learned without supervision, reducing dependence on labeled video data and improving the robustness of the VTG model.

[0078] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific implementation of S200; S200 includes:

[0079] S210, input the entire video and query text into the target VTG model, the target VTG model extracts key frames from the entire video according to the query text, and generates candidate boxes on the key frames using the window sliding method; S220, extract the image features of the candidate boxes, use the classifier to score the candidate boxes, and obtain the first coarse-grained score.

[0080] The query text generally includes:

[0081] Descriptive keywords, such as object and subject, scene and environment, action and behavior, etc.

[0082] Events and plots, such as a specific event, a wedding.

[0083] Emotion and atmosphere, such as emotional tendencies, such as sad parting, etc.

[0084] Time and space information, such as a meeting at 3 p.m.

[0085] In this embodiment, the target VTG model is used to preprocess the input video data, including but not limited to denoising, standardization and other operations to ensure video quality and consistency. The target VTG model uses a target detection algorithm or other methods to generate candidate boxes, which may contain target objects or areas related to target events. Candidate boxes can be generated at different scales and ratios to improve coverage. The evaluation index for coarse-grained sorting is defined based on the query text, which can be based on the similarity between the candidate box and the target, the confidence score, etc.

[0086] In this embodiment, each candidate box is assigned a coarse-grained score.

[0087] By first generating coarse-grained candidate boxes and the corresponding first coarse-grained scores, we prepare for the next step of fine-grained sorting and fine positioning.

[0088] S300: construct a target video-text model, and use the target video-text model to extract video features and text features of each candidate frame.

[0089] In this embodiment, the target video-text model is constructed based on a neural network, mainly including a converter module, a multi-head attention layer, a feedforward neural network layer, and a ReLU layer, wherein the feedforward neural network layer and the ReLU layer constitute an adapter module. Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation of constructing a target video-text model in S300; comprising:

[0090] S310, configuring a converter module in the neural network of the video-text model; S320, setting a feedforward neural network layer and a ReLU layer in sequence at the output end of the multi-head attention layer of the converter module to construct a target video-text model, and the feedforward neural network layer and the ReLU layer constitute an adapter module.

[0091] In this embodiment, a lightweight adapter module (which can be regarded as a bottleneck layer) is constructed at the output end of the multi-head attention layer of the converter module using a feedforward neural network layer (FFN) and a ReLU layer. The adapter module is used for tuning so that the acquired video feature representation adapts to the data distribution of the current downstream task. The constructed adapter module (bottleneck layer) is used to learn task-adaptive video features and is mixed with the original pre-trained preliminary video features. Formula (2) The i-th frame adapted video feature is:

[0092]

[0093] According to formula (2), the adapted video features of each video frame can be calculated, so that the video feature h of the jth candidate frame is j It is calculated by the mean pooling of each video frame, as shown in formula (3):

[0094]

[0095] in, and They are the start and end frames of the video respectively.

[0096] In this embodiment, the training of the adapter module follows standard contrastive learning and uses NCE loss, as shown in equation (4), h pos Video features for positive samples:

[0097]

[0098] in, is the loss function of the adapter module, and q is the candidate box parameter.

[0099] The fine-grained attention between a video frame and a text query will be weakened by the perturbations of many other frames, resulting in the loss of contextual information.

[0100] In some optional embodiments, such as Figure 5 , Figure 5 yes Figure 1 In S300, a flowchart of a specific implementation of extracting video features and text features of each candidate box by using the target video-text model is shown. After S320, S300 includes:

[0101] S330, using the target video-text model to obtain the preliminary video features and corresponding text features of each candidate box, and using the converter module to align the preliminary video features with the text features; S340, using the adapter module to learn and obtain adaptive video features, sum the preliminary video features and adaptive video features, and obtain the adapted video features of the video frame; S350, obtaining the adapted video features of all video frames of the candidate box, and calculating the mean of the adapted video features of all video frames to obtain the video features of each candidate box.

[0102] In this embodiment, the target video-text model is used to obtain preliminary video features v and text features q, as shown in equations (5) and (6).

[0103]

[0104] We pre-train a large-scale video-text model via contrastive learning to align visual representations with their associated textual representations, improving the matching efficiency of the target video-text model.

[0105] S400 , calculating a fine-grained score of each candidate box according to the video feature and the text feature, calculating a total score of the candidate box according to the first coarse-grained score and the fine-grained score, and performing fine-grained sorting on each candidate box using the total score.

[0106] In a preferred embodiment, Figure 6 , Figure 6 yes Figure 1 A flowchart of a specific implementation of S400; S400 includes:

[0107] S410, respectively perform maximum and minimum normalization on the first coarse-grained score and the fine-grained score of each candidate box to obtain a coarse-grained normalized score and a fine-grained normalized score; S420, sum the coarse-grained normalized score and the fine-grained normalized score to obtain a total score of the candidate box; S430, perform fine-grained sorting on the candidate boxes according to the total score.

[0108] In this embodiment, the candidate boxes are sorted in a fine-grained manner. For the jth candidate box, the final ranking score combines two components, as shown in formula (9):

[0109] (1) Coarse-grained score s of the candidate box generated by the VTG module j , as shown in formula (7);

[0110] (2) The fine-grained matching score m calculated by the video-text pre-trained model j =h j ·q, as in formula (8).

[0111]

[0112] Among them, N p is the total number of candidate boxes.

[0113] The former captures the correlation between proposals through complex context model design, while the latter focuses on fine-grained content matching between candidate boxes and text queries. By performing min-max normalization on these two types of scores, a more stable score fusion can be obtained. The final ranking score r j is the sum of two normalized scores.

[0114] In this embodiment, a fine-grained sorting strategy is used to calculate the candidate box matching scores for accurate sorting using a visual-text pre-training model based on contrastive learning.

[0115] S500: Position the input video at a certain time according to the fine-grained sorting.

[0116] This application obtains the target VTG model by performing positive and negative sample comparison learning, which solves the problem of imbalance between positive and negative samples, helps improve the robustness of the model, and is suitable for various multimodal data processing tasks. By constructing a target video-text model, calculating the fine-grained score of each candidate box and the total score of the candidate box, and using the total score to perform fine-grained sorting of each candidate box, the matching of fine-grained sorting for downstream tasks is improved, and the accurate sorting of candidate boxes is enhanced, which helps to improve the performance of multi-scale frameworks in the field of multimodal data processing and provide more accurate information retrieval and analysis.

[0117] In some preferred embodiments, the method further comprises:

[0118] like Figure 7 , Figure 7 yes Figure 1 A flowchart of a specific implementation method after S500; S600, obtaining the fine-grained normalized score of the target video-text model for the candidate box; S700, calculating the AxIoU@K value of each sample using the fine-grained normalized score, and calculating the average value based on the AxIoU@K values ​​of all samples; S800, evaluating the performance of the target video-text model based on the average value.

[0119] Among them, AxIoU (Axis-aligned Intersection over Union) represents the axis-aligned intersection over union ratio.

[0120] When evaluating the performance of the target video-text model, the average value of AxIoU@K is calculated, and the ranking position of the candidate box is considered based on the average precision, avoiding the dependence on the IoU threshold, which can be used for performance evaluation of various multimodal data analysis tasks. For video moment retrieval tasks, the indicator R@K,IoU@m is generally used for verification evaluation, which indicates the percentage of at least one candidate box among the retrieved top-K candidate boxes whose IoU with the true interval is greater than m, as shown in formula (10):

[0121]

[0122] However, this indicator has two problems. First, it is not sensitive to ranking, treating the top-K as a set without considering their ranking. Second, it is not sensitive to positioning, that is, the exact location (starting point and end point) of the video moment does not affect the indicator value because it uses a threshold θ to binarize the IoU of each video moment.

[0123] In this embodiment, the verification indicator AxIoU@K is used. It is based on the average precision (AP) and can avoid dependence on the θ threshold while considering the ranking position of the best candidate box in the top-K, as shown in formula (11):

[0124]

[0125] In order to solve the above technical problems, the embodiment of the present application also provides a video time positioning device 900, such as Fig. 9 , Fig. 9 1 is a schematic diagram of a structure of an embodiment of a video moment positioning device 900 according to the present application; a video moment positioning method according to the first aspect is adopted, comprising:

[0126] An acquisition module 901 is used to acquire a contrast loss function using positive samples and negative samples, and train a VTG model using the contrast loss function to acquire a target VTG model;

[0127] A generation module 902 inputs the entire video into a target VTG model, and generates a plurality of candidate boxes and a first coarse-grained score using the target VTG model;

[0128] An extraction module 903 is used to construct a target video-text model and extract video features and text features of each candidate frame using the target video-text model;

[0129] A sorting module 904 is used to calculate the fine-grained score of each candidate box according to the video feature and the text feature, calculate the total score of the candidate box according to the first coarse-grained score and the fine-grained score of the candidate box, and perform fine-grained sorting on each candidate box using the total score;

[0130] The positioning module 905 is used to perform video time positioning on the input video according to the fine-grained sorting. Compared with the prior art, the embodiment of the present application has the following technical effects:

[0131] By performing positive and negative sample comparison learning to obtain the target VTG model, the problem of imbalance between positive and negative samples is solved, which helps to improve the robustness of the model and is applicable to various multimodal data processing tasks. By building a target video-text model, calculating the fine-grained score of each candidate frame and the total score of the candidate frame, and using the total score to perform fine-grained sorting of each candidate frame, the matching of fine-grained sorting for downstream tasks is improved, and the accurate sorting of candidate frames is enhanced, which helps to improve the performance of multi-scale frameworks in the field of multimodal data processing and provide more accurate information retrieval and analysis.

[0132] In order to solve the above problems, an embodiment of the present application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for video moment positioning as in the first aspect are implemented.

[0133] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig.10 As shown, Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a network module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, a function or step of a method for positioning a video moment is implemented. The network module of the computer device may include a network interface and / or a wireless network module, and the computer device may communicate with other devices or service platforms through the network module. In addition, the computer device may also include a display screen and an input device, etc.

[0134] The memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the processor implements the steps of the method based on video moment positioning as described above when executing the instructions or codes.

[0135] In order to solve the above problem, the embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the steps of a method for video moment positioning in the first aspect are implemented. Figures 1 to 7 For the method of video moment positioning provided in each step, please refer to the implementation method provided in the above steps, which will not be repeated here.

[0136] The computer-readable storage medium may be a video moment positioning device provided by any of the aforementioned embodiments or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., provided on the computer device.

[0137] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0138] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0139] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0140] Compared with the prior art, the embodiments of the present application mainly have the following technical effects:

[0141] By performing positive and negative sample comparison learning to obtain the target VTG model, the problem of imbalance between positive and negative samples is solved, which helps to improve the robustness of the model and is applicable to various multimodal data processing tasks. By building a target video-text model, calculating the fine-grained score of each candidate frame and the total score of the candidate frame, and using the total score to perform fine-grained sorting of each candidate frame, the matching of fine-grained sorting for downstream tasks is improved, and the accurate sorting of candidate frames is enhanced, which helps to improve the performance of multi-scale frameworks in the field of multimodal data processing and provide more accurate information retrieval and analysis.

[0142] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0143] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for video time positioning, characterized in that it includes: Using positive samples and negative samples to obtain a contrast loss function, and using the contrast loss function to train a VTG model to obtain a target VTG model; Input the entire video into the target VTG model, and use the target VTG model to generate multiple candidate boxes and a first coarse-grained score; Constructing a target video-text model, and using the target video-text model to extract video features and text features of each candidate frame; Calculating a fine-grained score for each candidate box according to the video features and the text features, calculating a total score for the candidate box according to the first coarse-grained score and the fine-grained score, and performing fine-grained sorting on each candidate box using the total score; The video time is located for the input video according to the fine-grained sorting.

2. The method for video time positioning according to claim 1, characterized in that: The method of obtaining a contrast loss function using positive samples and negative samples includes: Using the VTG model to obtain multiple candidate frames of the input video, performing a coarse-grained scoring on each candidate frame, obtaining a second coarse-grained score corresponding to the positive sample of the input video, and taking a difference between the second coarse-grained score to obtain a third coarse-grained score corresponding to the negative sample of the input video; The contrast loss function is obtained by using the second coarse-grained score and the third coarse-grained score. The calculation formula of the contrast loss function is: in, is a positive sample, is a random contrast negative sample from the same batch, is the second coarse-grained score corresponding to the positive sample, is the third coarse-grained score corresponding to the negative sample.

3. The method for video time positioning according to claim 1, characterized in that: The step of inputting the entire video into the target VTG model and using the target VTG model to generate a plurality of candidate frames and a first coarse-grained score includes: The entire video and the query text are input into the target VTG model, the target VTG model extracts key frames from the entire video according to the query text, and generates candidate frames on the key frames using a window sliding method; The image features of the candidate box are extracted, and the candidate box is scored using a classifier to obtain the first coarse-grained score.

4. The method for video time positioning according to claim 1, characterized in that: The step of constructing a target video-text model comprises: Configure the transformer module in the neural network of the video-to-text model; A feedforward neural network layer and a ReLU layer are sequentially arranged at the output end of the multi-head attention layer of the converter module to construct the target video-text model, and the feedforward neural network layer and the ReLU layer constitute an adapter module.

5. The method for video time positioning according to claim 4, characterized in that: The extracting of video features and text features of each candidate frame by using the target video-text model comprises: Using the target video-text model to obtain preliminary video features and corresponding text features of each candidate box, and using the converter module to align the preliminary video features with the text features; Using the adapter module to learn and acquire adaptive video features, summing the preliminary video features and the adaptive video features to obtain adapted video features of the video frame; The adapted video features of all the video frames of the candidate frame are obtained, and the adapted video features of all the video frames are averaged to obtain the video features of each candidate frame.

6. The method for video time positioning according to claim 1, characterized in that: Calculating the total score of the candidate boxes according to the first coarse-grained score and the fine-grained score, and using the total score to perform fine-grained sorting on each candidate box, including: The first coarse-grained score and the fine-grained score of each candidate box are respectively normalized to obtain a coarse-grained normalized score and a fine-grained normalized score; The coarse-grained normalized score and the fine-grained normalized score are summed to obtain a total score of the candidate box; The candidate boxes are sorted in fine granularity according to the total scores.

7. The method for video time positioning according to claim 6, characterized in that: After performing video time positioning on the input video according to the fine-grained sorting, the video time positioning method further includes: Obtain fine-grained normalized scores of candidate boxes for the target video-text model; Calculate the Ax IoU@K value of each sample using the fine-grained normalized score, and calculate the average value based on the Ax IoU@K values ​​of all samples; The performance of the target video-text model is evaluated based on the average value.

8. A device for video time positioning, using the method for video time positioning according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to obtain a contrast loss function using the positive samples and the negative samples, and train the VTG model using the contrast loss function to obtain a target VTG model; A generation module, inputting the entire video into the target VTG model, and using the target VTG model to generate multiple candidate boxes and a first coarse-grained score; An extraction module, used to construct a target video-text model and extract video features and text features of each candidate frame using the target video-text model; A sorting module, used to calculate the fine-grained score of each candidate box according to the video features and the text features, calculate the total score of the candidate box according to the first coarse-grained score and the fine-grained score of the candidate box, and perform fine-grained sorting on each candidate box using the total score; A positioning module is used to locate the video time of the input video according to the fine-grained sorting.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for video moment positioning as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for video moment positioning according to any one of claims 1 to 7 are implemented.

Images