Video positioning method and device based on multiple modes, electronic equipment and medium

Abstract

The embodiment of the invention provides a video positioning method and device based on multiple modes, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring and identifying a video query text of a target video and a task category to which the video query text belongs, extracting task category features and video query text features, and performing visual spatio-temporal feature extraction on continuous target video frames to obtain target visual features; and fusing the video query text feature, the task category feature and the target visual feature to obtain a target fusion feature, and positioning the target video based on the target fusion feature to obtain target positioning video information including the starting time and the ending time. According to the embodiment of the invention, video positioning is carried out on the target video based on the fused text, vision and task features, the starting and ending time of the positioning video can be directly predicted based on the video global information, and the accuracy of video positioning is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal video positioning method and device, electronic equipment, and medium. Background Art

[0002] Traditional video localization methods typically use deep learning models (such as convolutional neural networks) and preset sliding windows to locate the time of video clips. In video retrieval applications, the query text for locating pedestrian behavior trajectories and the target video containing pedestrian behavior trajectories are input into the convolutional neural network. The convolutional neural network extracts and fuses the query text features and the video content features of the target video to obtain fused features. The fused features are then input into the linear layer to find the time of the video clip to be located. However, this method only achieves feature fusion of text and video frame images through simple splicing, making it difficult to capture the finer-grained semantic associations between text and video, resulting in low video localization accuracy. Therefore, how to improve the accuracy of video localization has become an urgent problem to be solved. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a multimodal video positioning method and device, electronic equipment and medium, aiming to improve the accuracy of video positioning.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multimodal video positioning method, the method comprising:

[0005] Obtaining a video query text for indicating a query target video;

[0006] Identifying the task category to which the video query text belongs to obtain a video positioning task category, and performing task category feature extraction on the video positioning task category to obtain a task category feature;

[0007] Performing text feature extraction on the video query text to obtain video query text features;

[0008] Extracting continuous target video frame images with a preset duration from the target video, and performing visual spatiotemporal feature extraction on the target video frame images to obtain target visual features;

[0009] Fusing the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature;

[0010] The target video is positioned based on the target fusion feature to obtain target positioning video information; wherein the target positioning video information includes a start time and an end time; and the duration between the start time and the end time is less than or equal to the preset duration.

[0011] In some embodiments, performing video positioning on the target video based on the target fusion feature to obtain target positioning video information includes:

[0012] Calculating feature weights on the target fusion features to obtain a target feature weight matrix;

[0013] Decoding the target feature weight matrix to obtain the target query object position;

[0014] Performing video frame object classification detection on the target fusion feature according to the target query object position to obtain the query object category probability;

[0015] The target fusion feature is time-located in the video frame based on the query object category probability and the target query object position to obtain the start time and the end time.

[0016] In some embodiments, fusing the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature includes:

[0017] Performing attention-weighted processing on the video query text features and the target visual features to obtain visual text semantic features;

[0018] Attention distribution adjustment is performed on the visual text semantic feature according to the task category feature to obtain the target fusion feature.

[0019] In some embodiments, performing attention-weighted processing on the video query text features and the target visual features to obtain visual text semantic features includes:

[0020] Calculating semantic similarity between the video query text features and the target visual features to obtain a video text similarity matrix; the video text similarity matrix includes semantic similarities between multiple video query text features and the target visual features;

[0021] For the video query text feature, selecting the maximum semantic similarity between the video query text feature and the target visual feature from the semantic similarities as the target semantic similarity;

[0022] Performing masking on the video text similarity matrix based on the target semantic similarity and a preset similarity threshold to obtain a similarity mask matrix;

[0023] Attention weights are allocated based on the similarity mask matrix, the video query text features, and the target visual features to obtain the visual text semantic features.

[0024] In some embodiments, extracting visual out-of-control features from the target video frame image to obtain target visual features includes:

[0025] Adaptively dividing the target video frame image into blocks to obtain a block video frame image;

[0026] Performing linear projection processing on the block video frame image to obtain fixed-dimensional video frame features;

[0027] Performing temporal position encoding on the fixed-dimensional video frame features to obtain video frame temporal features;

[0028] Performing self-attention encoding on the fixed-dimensional video frame features to obtain video frame spatial features;

[0029] The video frame temporal features and the video frame spatial features are fused to obtain the target visual features.

[0030] In some embodiments, performing temporal position encoding on the fixed-dimensional video frame features to obtain video frame temporal features includes:

[0031] Obtaining a video frame timestamp that matches the target video frame image;

[0032] Normalizing the video frame timestamp to obtain a normalized timestamp;

[0033] Timestamp position encoding is performed on the fixed-dimensional video frame feature based on the normalized timestamp to obtain the video frame temporal feature.

[0034] In some embodiments, extracting text features from the video query text to obtain video query text features includes:

[0035] Performing word segmentation processing on the video query text to obtain a word segmentation query text;

[0036] The word segmentation query text is encoded to obtain the video query text features.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a multimodal video positioning device, the device comprising:

[0038] A query text acquisition module, used to acquire a video query text indicating a query target video;

[0039] A task category feature extraction module is used to identify the task category to which the video query text belongs, obtain a video positioning task category, and perform task category feature extraction on the video positioning task category to obtain a task category feature;

[0040] A text feature extraction module is used to extract text features from the video query text to obtain video query text features;

[0041] A visual spatiotemporal feature extraction module is used to extract target video frame images with a preset duration and continuous duration from the target video, and perform visual spatiotemporal feature extraction on the target video frame images to obtain target visual features;

[0042] A feature fusion module, configured to fuse the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature;

[0043] A video positioning module is used to perform video positioning on the target video based on the target fusion feature to obtain target positioning video information; wherein, the target positioning video information includes a start time and an end time; the duration between the start time and the end time is less than or equal to the preset duration.

[0044] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0045] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of the above-mentioned first aspect.

[0046] The multimodal video positioning method and device, electronic device and medium proposed in the present application first obtain and identify the video query text and the task category to which it belongs, and extract the task category features and text features, which can facilitate the subsequent triggering of different query vector strategies for different task categories, thereby helping to improve the accuracy of video positioning; secondly, a target video frame image with a preset duration and continuousness is extracted from the target video, and the visual spatiotemporal features of the target video frame image are extracted to obtain the target visual features, which can extract the video content containing time and space information; finally, the target video is positioned based on the target fusion features to obtain the target positioning video information, and the target positioning video information including the start time and end time is obtained, which can directly predict the start and end time of the positioned video based on the global information of the video, thereby improving the accuracy of extracting the time of each video frame image, thereby significantly improving the accuracy of video positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of a multi-modal video positioning method provided in an embodiment of the present application;

[0048] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0049] Figure 3 yes Figure 1 Flowchart of step S104 in FIG.

[0050] Figure 4 yes Figure 3 Flowchart of step S303 in FIG.

[0051] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0052] Figure 6 yes Figure 5 Flowchart of step S501 in FIG.

[0053] Figure 7 yes Figure 1 Flowchart of step S106 in FIG.

[0054] Figure 8 Schematic diagram of the structure of a multi-modal video positioning device provided in an embodiment of the present application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] First, let’s analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0061] The embodiments of the present application provide a multimodal video positioning method and device, electronic device, and medium, aiming to improve the accuracy of video positioning.

[0062] The multimodal video positioning method and device, electronic device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the multimodal video positioning method in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0064] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0065] The multimodal video positioning method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The multimodal video positioning method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a multimodal video positioning method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0067] Figure 1 This is an optional flowchart of the multimodal video positioning method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0068] Step S101: obtaining a video query text indicating a query target video.

[0069] Step S102 : identifying the task category to which the video query text belongs, obtaining a video positioning task category, and extracting task category features from the video positioning task category to obtain task category features.

[0070] Step S103: extract text features from the video query text to obtain video query text features.

[0071] Step S104 , extracting target video frame images with a preset duration and continuous duration from the target video, and performing visual spatiotemporal feature extraction on the target video frame images to obtain target visual features.

[0072] Step S105 , fusing the video query text features, task category features, and target visual features to obtain target fused features.

[0073] Step S106, performing video positioning on the target video based on the target fusion feature to obtain target positioning video information; wherein the target positioning video information includes a start time and an end time; and the duration between the start time and the end time is less than or equal to a preset duration.

[0074] In steps S101 to S106 shown in the embodiment of the present application, first, by acquiring and identifying the video query text and the task category to which it belongs, and extracting the task category features and text features, different query vector strategies can be triggered for different task categories in the subsequent process, which helps to improve the accuracy of video positioning; secondly, a target video frame image with a preset duration and continuousness is extracted from the target video, and the visual spatiotemporal features of the target video frame image are extracted to obtain the target visual features, which can extract the video content containing time and space information; finally, the target video is positioned based on the target fusion features to obtain the target positioning video information, which includes the start time and the end time. The start and end time of the positioned video can be directly predicted based on the global information of the video, thereby improving the accuracy of extracting the time of each video frame image, thereby significantly improving the accuracy of video positioning.

[0075] In step S101 of some embodiments, specifically, the target video refers to a related video to be positioned.

[0076] For example, in an insurance application scenario, the target video can be a vehicle accident video recorded by a dashcam provided by the car owner, a vehicle accident video recorded by a surveillance camera, and a vehicle damage video recorded by an insurance agent.

[0077] Video query text refers to the text information entered by the user to retrieve or locate relevant video content.

[0078] For example, in an insurance application scenario, the video query text can be used to retrieve the vehicle collision moment clip in the vehicle accident video.

[0079] In step S102 of some embodiments, specifically, the video positioning task category is a positioning task requirement category determined based on the video query text, including but not limited to video clip retrieval, video action positioning, and video action segmentation.

[0080] For example, in an insurance application scenario, if the video query text is to retrieve the instantaneous fragment of a vehicle collision in a vehicle accident video, then the video positioning task category to which the video query text belongs is the video fragment retrieval task category; if the video query text is to locate the complete action of a head-on collision between the left front wheel and the guardrail in a vehicle accident video, then the video positioning task category to which the video query text belongs is video action positioning; if the video query text is to perform video segmentation of the damaged part of the vehicle, then the video positioning task category to which the video query text belongs is video action segmentation.

[0081] Specifically, the task category feature refers to the conversion of the discretized video positioning task category into a task category vector in a high-dimensional space.

[0082] Specifically, the task category features are first segmented to obtain task tokens. These task tokens are then fed into the task feature extraction layer of the Transformer model for word embedding. This maps the discrete video localization task categories to continuous numerical vector representations in a high-dimensional space, and outputs the final task tokens. The task feature extraction layer can be a Task-Encoder, which extracts task category features.

[0083] In this embodiment, by identifying the task category to which the video query text belongs, the video positioning task category is obtained, which can enhance the pertinence of subsequent video positioning and avoid positioning errors caused by confusion of task categories.

[0084] See also Figure 2 In some embodiments, step S103 includes but is not limited to steps S201 to S202:

[0085] Step S201 : performing word segmentation processing on the video query text to obtain a word segmentation query text.

[0086] Step S202: Encode the segmented query text to obtain video query text features.

[0087] In step S201 of some embodiments, specifically, the video query text may be segmented using a BERT tokenizer according to a domain dictionary to obtain segmented text tokens.

[0088] For example, in an insurance application scenario, if the video query text is to retrieve the moment of vehicle collision in a vehicle accident video, after preliminary word segmentation, we can obtain vocabulary units such as "search", "vehicle", "accident", "video", "in", "of", "collision", "moment", and "fragment", and retain nouns, verbs, and adjectives (that is, "search", "vehicle", "collision", "accident", "video", and "fragment") and filter out stop words (that is, "in", "of").

[0089] In this embodiment, by performing word segmentation processing on the video query text, the original video query text can be converted into a structured vocabulary sequence, which facilitates a more accurate understanding of the query intent later.

[0090] In step S202 of some embodiments, specifically, the video query text feature refers to converting the segmented query text into a text vector in a high-dimensional space.

[0091] Specifically, the segmented text tokens can be input into the text feature extraction network layer of the Transformer model for word embedding processing, mapping the discrete segmented query text into a continuous numerical vector representation in a high-dimensional space. The resulting video query text features (text tokens) are then output. These vectors can capture the semantic relationships between different words. The text feature extraction network layer can be a Text-Encoder, which is used to extract text features.

[0092] In this embodiment, by encoding the word segmentation query text, the video query text features are obtained, which can convert the text information into a numerical form that can be efficiently processed by the model, and retain the semantic relationship between words, further ensuring that the query intent can be accurately understood during subsequent video positioning.

[0093] Through step S201 to step S202, through word segmentation and encoding processing, the video query text can be converted into structured numerical features with semantic meaning, which improves the model's ability to understand the query intent and provides clear requirements for subsequent insurance video positioning.

[0094] In step S104 of some embodiments, specifically, the target video frame image refers to a video frame extracted from the target video at a fixed frame interval or a fixed duration.

[0095] For example, if the preset duration is 30 seconds and the interval is 2 minutes, then 30 consecutive seconds of video frames are extracted starting from the 0th minute of the vehicle accident video, then 2 minutes are skipped and 30 consecutive seconds of video frames are extracted starting from the 2nd minute, and so on, until the extraction is completed and the target video frame is obtained. This ensures that the extracted video frame images can cover all parts of the vehicle accident video while avoiding repeated processing of too much similar content.

[0096] In this embodiment, by extracting continuous video frame images with a preset length, the key segments of the video can be quickly extracted without processing the entire video content, and the key information of the video is retained, which improves processing efficiency while ensuring the relevance of the extracted video content to the query text intent.

[0097] See also Figure 3 In some embodiments, step S104 includes but is not limited to steps S301 to S305:

[0098] Step S301 : Adaptively divide the target video frame image into blocks to obtain a block video frame image.

[0099] Step S302: perform linear projection processing on the block video frame image to obtain fixed-dimensional video frame features.

[0100] Step S303 , performing temporal position encoding on the fixed-dimensional video frame features to obtain video frame temporal features.

[0101] Step S304: perform self-attention encoding on the fixed-dimensional video frame features to obtain the video frame spatial features.

[0102] Step S305 , fusing the temporal features of the video frames and the spatial features of the video frames to obtain target visual features.

[0103] In step S301 of some embodiments, specifically, dividing the video frame image into blocks refers to dividing each complete video frame image into a number of non-overlapping or partially overlapping rectangular area image blocks.

[0104] Specifically, the image feature extraction network layer of the Transformer model can be used to identify the image resolution of the target video frame image and adaptively divide it into different numbers of image blocks for different image resolutions to obtain a visual feature sequence that is positively correlated with the resolution. The image feature extraction network layer can be a vision encoder, which is used to extract the visual features of the target video frame image.

[0105] For example, in insurance applications, when processing dashcam videos with a 4K resolution, each video frame can be divided into 16x16 pixel blocks, and 32,508 block image tokens can be output. When processing vehicle collision videos with a 1080P resolution, each video frame can be divided into 32x32 image blocks, and 14,637 block image tokens can be output.

[0106] In another embodiment of the present application, if a motion blur area is detected in a video frame image with a 4K resolution, the motion blur area of the video frame image can be adaptively adjusted from 16x16 to 32x32.

[0107] For example, in insurance applications, for motion-blurred frames caused by high-speed collisions, the blurred area is dynamically expanded to 32x32 pixels, while the vehicle's VIN code area is still divided into 16x16 fine blocks.

[0108] In this embodiment, by adaptively dividing the target video frame image into blocks to obtain a block video frame image, the key areas of the target video frame images with different resolutions can be highlighted, while reducing the interference of noise and image background information, avoiding the loss of image information due to resolution differences, and helping to subsequently improve the accuracy of video feature extraction.

[0109] In step S302 of some embodiments, specifically, the fixed-dimensional video frame feature refers to a video frame image feature having a fixed-dimensional vector.

[0110] Specifically, the block video frame image is input into the fully connected layer connected to the Vision-Encoder for linear transformation to map the block video frame image to a fixed-dimensional vector space.

[0111] For example, in an insurance application scenario, each 16x16 pixel and 32x32 pixel block video frame image is converted into a 1536-dimensional feature vector.

[0112] In this embodiment, fixed-dimensional video frame features are obtained by performing linear projection processing on the block video frame images, which can uniformly process video information from different sources and facilitate subsequent projection into the same vector space as the video query text features to achieve the fusion of text and image modalities.

[0113] See also Figure 4 In some embodiments, step S303 includes but is not limited to steps S401 to S403:

[0114] Step S401: Obtain a video frame timestamp that matches a target video frame image.

[0115] Step S402 : normalize the video frame timestamp to obtain a normalized timestamp.

[0116] Step S403 , performing timestamp position encoding on the fixed-dimensional video frame features based on the normalized timestamp to obtain video frame temporal features.

[0117] In step S401 of some embodiments, specifically, the video frame timestamp refers to the specific time point at which each frame of video appears in the target video, usually in seconds (s).

[0118] In step S402 of some embodiments, specifically, normalizing the timestamp refers to converting the video frame timestamp into a standardized timestamp with a uniform numerical interval.

[0119] Specifically, the video frame timestamp may be subtracted from the video start time, and then divided by the total video duration to achieve normalization.

[0120] For example, in insurance applications, a 30-second video frame can be compressed into the range of 0 to 1 through normalization, where 0 represents the start time of the video and 1 represents the end time of the video.

[0121] In this embodiment, by normalizing the video frame timestamps to obtain normalized timestamps, the scale differences caused by different video lengths can be eliminated, making it easier for subsequent models to uniformly process insurance videos of various lengths.

[0122] In step S403 of some embodiments, specifically, a video frame temporal feature having the same dimension as the fixed-dimensional video frame feature may be generated by performing sine-cosine position encoding on the normalized timestamp.

[0123] For example, in applications in the insurance field, sine-cosine position encoding can be performed on the video frame sequence before and after a vehicle collision to embed the timing relationship corresponding to each video frame into the fixed-dimensional video frame features, thereby obtaining the timing characteristics of the vehicle collision process before, during, and after the vehicle collision.

[0124] In this embodiment, by encoding the timestamp position of fixed-dimensional video frame features based on normalized timestamps, the model's ability to understand the video timing logic is enhanced, so that the visual features carry clear temporal context information, can better capture dynamic changes in the video, and facilitate improving the accuracy of subsequent video positioning.

[0125] Through steps S401 to S403, by performing temporal position encoding on the fixed-dimensional video frame features, the time information of each frame of video can be effectively integrated into the fixed-dimensional video frame features, thereby better capturing dynamic changes in the video.

[0126] In step S304 of some embodiments, specifically, the video frame spatial feature refers to the vectorized representation of the geometric relationship and semantic association between fixed-dimensional video frame features, which is used to reflect the visual information of the video frame image in the spatial dimension, such as the shape and texture of the object.

[0127] Specifically, the 1536-dimensional fixed-dimensional video frame features are first converted into a video frame sequence matrix. The query Q, key K and value V vectors of the video frame sequence matrix are generated respectively through the multi-head attention mechanism, and self-attention processing is performed on the query Q, key K and value V vectors to obtain the video frame spatial features.

[0128] For example, in insurance applications, when processing video frame images containing front bumper collisions, the attention weight matrix between the bumper block image and the adjacent headlight block image is calculated, and a relative position bias (1536-dimensional fixed-dimensional video frame features ^ number of bumper blocks x number of adjacent headlight blocks) is introduced as a bias term. This bias term represents the spatial distance between the encoded blocks. Self-attention processing is implemented based on the calculation of the attention weight matrix and the bias term, which can ensure that the damage on the left and right sides of the vehicle hood can maintain positional distinction even if the semantics are similar.

[0129] In this embodiment, by performing self-attention encoding on fixed-dimensional video frame features, video frame spatial features are obtained, which can capture the semantic associations of different regions in the video frame image, and can also distinguish image regions with similar semantics, thereby improving the accuracy of video frame image spatial feature extraction.

[0130] In step S305 of some embodiments, specifically, the target visual feature refers to a feature that integrates visual information in two dimensions, temporal and spatial, into a unified representation.

[0131] Specifically, the temporal features of the video frames and the spatial features of the video frames are spliced to obtain preliminary fused video frame features, and the temporal features of the video frames are used as bias items. Self-attention calculation is performed on the bias items and the preliminary fused video frame features to obtain temporally enhanced target visual features (vision tokens).

[0132] For example, in applications in the insurance field, self-attention calculations are performed on the moment of vehicle collision (temporal features) and the vehicle collision damage (spatial features), which can capture the specific moment of the vehicle collision and the vehicle visual features including the collision damage caused after the collision.

[0133] In this embodiment, by fusing the temporal features and spatial features of video frames, the target visual features are obtained, and the spatial and temporal features of video frame images are constructed. This can not only trace the development process of events in insurance videos, but also present the details of the events, further improving the accuracy of video frame image feature extraction.

[0134] Through steps S301 to S305, visual features containing time and space information are constructed through adaptive blocking, linear projection, temporal coding, spatial coding and feature fusion processing, which can capture dynamic video frame changes at different resolutions and improve the accuracy of visual feature extraction.

[0135] See also Figure 5 In some embodiments, step S105 includes but is not limited to steps S501 to S502:

[0136] Step S501 : Perform attention weighting processing on the video query text features and the target visual features to obtain visual text semantic features.

[0137] Step S502 , adjusting the attention distribution of the visual text semantic features according to the task category features to obtain the target fusion features.

[0138] See also Figure 6 In some embodiments, step S501 includes but is not limited to steps S601 to S604:

[0139] Step S601 , performing semantic similarity calculation on the video query text features and the target visual features to obtain a video text similarity matrix; the video text similarity matrix includes semantic similarities between multiple video query text features and the target visual features.

[0140] Step S602 : for the video query text feature, the maximum semantic similarity between the video query text feature and the target visual feature is selected from the semantic similarities as the target semantic similarity.

[0141] Step S603 : Masking the video-text similarity matrix based on the target semantic similarity and a preset similarity threshold to obtain a similarity mask matrix.

[0142] Step S604: Attention weights are allocated based on the similarity mask matrix, the video query text features, and the target visual features to obtain visual text semantic features.

[0143] In step S601 of some embodiments, specifically, the video-text similarity matrix refers to a two-dimensional matrix calculated by cross-modal semantic alignment, which is used to represent the semantic association strength between each word in the video query text and the target video frame image.

[0144] Specifically, semantic similarity can be calculated using the following formula:

[0145]

[0146] Among them, S represents the video text similarity matrix, and represents the size of the video text similarity matrix, L v The number of tokens representing the target visual features, L t The number of tokens representing the text features of the video query; S ij V represents the video text similarity between the i-th target visual feature and the j-th video query text feature; i represents the i-th target visual feature; T j Represents the j-th video query text feature.

[0147] In step S602 of some embodiments, specifically, the target semantic similarity refers to the semantic similarity corresponding to the target visual feature having the highest semantic relevance to the video query text feature.

[0148] For example, in the application of insurance field, the video query text features are the left bumper and the left front wheel. Then, the semantic similarity between the target visual features containing the partial features of the left bumper area and the video query text features of the left bumper is the greatest, that is, the semantic relevance is the highest.

[0149] In this embodiment, for the video query text features, the maximum semantic similarity between the video query text features and the target visual features is selected from the semantic similarity as the target semantic similarity, which can strengthen the core semantic association, ensure that each text element can lock the most representative visual evidence, avoid secondary matches interfering with subsequent video positioning results, and help improve the accuracy of subsequent video positioning.

[0150] In step S603 of some embodiments, specifically, the similarity mask matrix refers to a Boolean matrix obtained by binarizing and filtering the video text similarity matrix, wherein a position with a value of 1 indicates that there is a strong semantic association between the video query text feature and the target visual feature, and a position with a value of 0 indicates that the semantic association between the video query text feature and the target visual feature is weak, and the visual feature can be regarded as a noise influence.

[0151] Specifically, if the preset similarity threshold is 0.7, the target semantic similarity with a semantic similarity less than 0.7 is marked as irrelevant and set to 0 in the similarity mask matrix, and the target semantic similarity with a semantic similarity greater than or equal to 0.7 is marked as relevant and set to 1 in the similarity mask matrix.

[0152] For example, in an insurance application scenario, if the video query text feature is windshield crack, the target visual feature is the windshield crack image area, and the semantic similarity matching degree exceeds 0.7, the windshield crack image area feature is retained.

[0153] In this embodiment, the video-text similarity matrix is masked based on the target semantic similarity and a preset similarity threshold to obtain a similarity mask matrix, which can remove irrelevant image features with low semantic relevance and retain the image features most relevant to the text query intent, thereby improving the accuracy and efficiency of subsequent text and visual feature fusion.

[0154] In step S604 of some embodiments, specifically, the visual text semantic feature refers to a preliminary fusion feature between the target visual feature and the video query text feature.

[0155] Specifically, attention weight distribution can be achieved through the cross-attention mechanism, that is, the video query text features are used as query values (Q), the target visual features are used as key (K) value (V) pairs, and the similarity mask matrix is used as the bias item. The attention weights of the video query text features and the target visual features are distributed through the activation function (such as the Softmax function). That is, the higher the semantic similarity, the greater the attention weight corresponding to the semantic similarity.

[0156] In this embodiment, by allocating attention weights to the video query text features and target visual features, an attention weight matrix is obtained, which can retain the global features of cross-modal interaction while highlighting the key target visual features, thereby improving the accuracy of the fusion of visual and text semantic features.

[0157] Through steps S601 to S604, by performing semantic similarity calculation, mask processing, and attention weight allocation on the video query text features and the target visual features, a comprehensive visual text semantic feature can be generated. This not only generates global cross-modal information including text and vision, but also highlights the visual features most relevant to the query text intent through the attention mechanism, which can facilitate subsequent more accurate positioning of the video clips most relevant to the query text intent.

[0158] In step S502 of some embodiments, specifically, the target fusion feature refers to the final fusion feature between the video query text feature and the target visual feature.

[0159] Specifically, the attention distribution adjustment can be further achieved through the cross-attention mechanism, that is, the video query text features are used as the query value (Q), the target visual features are still used as the key (K) value (V) pair, and the task category features are used as the bias item. The attention distribution of the video query text features and the target visual features is adjusted through the activation function (such as the Softmax function) to obtain the final target fusion features.

[0160] Specifically, in applications in the insurance field, the video query text feature is the left front wheel collision, and the target visual features include the vehicle's left front wheel, vehicle bumper and railing area features. If the task category feature is to locate vehicle collision damage, the weight of the vehicle's left front wheel and vehicle bumper area features is increased, and the weight of the railing area features is reduced to achieve attention distribution adjustment.

[0161] In this embodiment, the attention distribution of the visual text semantic features is adjusted according to the task category features to obtain the target fusion features, which can further suppress the image areas irrelevant to the insurance query text and help improve the accuracy of subsequent video positioning.

[0162] Through steps S501 to S502, the influence of irrelevant areas in the target visual features is suppressed by using the video query text features, and the visual areas irrelevant to the insurance query text are further suppressed in combination with the task category features, rather than simply achieving the fusion of text and visual features through simple splicing. The global information of text and visual features is effectively extracted, and the visual area most relevant to the query text intent is accurately located, which helps to improve the accuracy of video positioning in the future.

[0163] See also Figure 7 In some embodiments, step S106 includes but is not limited to steps S701 to S703:

[0164] Step S701: decoding the target fusion feature into a feature query object to obtain the target query object position.

[0165] Step S702 : performing video frame object classification detection on the target fusion features according to the target query object position to obtain the query object category probability.

[0166] Step S703 , based on the query object category probability and the target query object position, the target fusion feature is time-located in the video frame to obtain the start time and the end time.

[0167] In step S701 of some embodiments, specifically, the target query object position refers to the visual feature position of the target visual feature that is most relevant to the query text intent.

[0168] Specifically, since the target fusion feature only represents the semantic association between visual features and text features, but cannot further locate the specific time boundary of the video, Object Queries (query object vectors) can be inserted into the decoder (Decoder) of the Transformer model. Through the Object Queries, the Object Queries are used as the query value (Q) in the cross-attention mechanism, and the target fusion feature is used as the key (K) value (V) pair. The Object Queries and the target fusion feature are cross-attention processed through the activation function (such as the Softmax function) to obtain the target feature weight matrix. The target feature weight matrix obtained after the interaction between each Object Query and the target fusion feature is mapped to the normalized detection box coordinates through the multi-layer perceptron network.

[0169] For example, in insurance applications, if the video query text feature is a left front wheel collision, the target fusion feature is decoded into the position coordinates (x, y, w, h) of the vehicle's left front wheel area through Object Query, where x and y represent the coordinates of the center point of the detection frame, w represents the width of the detection frame, and h represents the height of the detection frame.

[0170] Furthermore, Object Queries is a set of parameterized vectors that can be learned by the Transformer model. It is used to dynamically aggregate visual features, text features, and task semantic features through a cross-attention mechanism in the Transformer's multimodal video localization task to dynamically locate the target video area and time interval.

[0171] Specifically, the mathematical representation of Object Query is Among them, N q Represents the number of queries, and d represents the vector dimension of queries, which is consistent with the dimension of the target fusion feature.

[0172] Specifically, when initializing Object Queries, the number of Object Queries can determine the task query strategy according to the video localization task category, and the number of query vectors can be determined based on the task query strategy.

[0173] For example, in the insurance field, if the video localization task category is video clip retrieval, which indicates that the video output granularity is coarse, and the query strategy for this clip retrieval is global sparse query, then the number of Object Queries can be between 5 and 10; if the video localization task category is video action localization, which means that more detection frames are required to capture video action details, then the video output granularity is fine-grained, and the query strategy for this action localization is medium-density query, then the number of Object Queries can be between 10 and 20; if the video localization task category is video action segmentation, which means that the video may have multiple overlapping areas and more detection frames are needed to further capture video action details, then the video output granularity is finer-grained, and the query strategy for this action localization is dense density query, then the number of Object Queries can be between 20 and 50.

[0174] In this embodiment, by calculating the feature weights of the target fusion features and obtaining the target feature weight matrix, we can further focus on the responses of the spatial features and temporal features of the image region, so that the model can adaptively focus on the feature region most relevant to the query text intent and the temporal features corresponding to the region, ensuring the accuracy of the target query object location recognition.

[0175] In step S702 of some embodiments, specifically, the query object category probability refers to the category probability to which the target query object belongs.

[0176] Specifically, the FFN (Feed-Forward Network) and ReLU activation function connected to the Decoder can be used to perform classification calculations on the target query object position to obtain the query object category probability.

[0177] For example, in insurance applications, the location of the left front wheel damage area is (x=0.35, y=0.42, w=0.58, h=0.4). The fully connected layer outputs a collision damage probability of 98%, a previous damage probability of 78%, and a natural aging damage probability of 70%. The query object category probability corresponding to the left front wheel damage area is determined to be a collision damage probability of 98%.

[0178] In this embodiment, the target fusion features are subjected to video frame object classification detection according to the target query object position to obtain the query object category probability, which can further confirm the degree of relevance between the target visual features of the video frame and the text query intent, and help improve the accuracy of subsequent video positioning.

[0179] In step S703 of some embodiments, specifically, the target positioning video information includes a start time and an end time; and the duration between the start time and the end time is less than or equal to a preset duration.

[0180] Specifically, by performing cross-attention processing between Object Queries and target fusion features, Object Queries can learn the temporal features of the target visual features and the feature weights corresponding to the temporal features of each frame. The temporal features and feature weights can be used to determine whether the identified objects are relevant to the text query intent through the query object category probability and the target query object position, thereby locating multiple insurance video positioning segments related to the text query intent and outputting the start time and end time of the insurance video positioning segments.

[0181] For example, in an insurance application scenario, if the target video frame image is 15 seconds, different time periods are assigned to different Object Queries (such as Query 1-5 is responsible for 0-3 seconds, Query 6-10 is responsible for 4-5 seconds, Query 11-20 is responsible for 6-10 seconds, and Query 20-35 is responsible for 11-15 seconds). After cross-attention processing, it can be learned that Query 3 is the moment when the vehicle collision starts, and is responsible for the 2nd second of the target video frame. The corresponding timestamp is 12:30:05 seconds, and the feature weight is 0.92. Therefore, 12:30:05 seconds is determined to be the start time of the vehicle collision. If Query 12 is the moment when the vehicle collision ends, and is responsible for the 7th second of the target video frame, the corresponding timestamp is 12:30:10 seconds, and the feature weight is 0.88, then 12:30:10 seconds is determined to be the end time of the vehicle collision, and the corresponding collision duration is 5 seconds.

[0182] In this embodiment, the target fusion feature is used to time locate the video frame based on the query object category probability and the target query object position to obtain the start time and end time. The start and end time of the positioning video can be directly predicted through the query object vector, avoiding the use of a sliding window to gradually traverse and calculate the time information of each video frame, thereby improving the efficiency of insurance video positioning, and also avoiding the situation of sliding and cutting global information through the sliding window. The start and end time of the positioning video is further directly predicted based on the video global information, thereby improving the accuracy of extracting the image time of each video frame, thereby significantly improving the accuracy of insurance video positioning.

[0183] Through steps S701 to S703, by performing query object decoding, video frame object classification detection, and video frame time positioning on the target fusion features, it is possible to locate the insurance video segment in the target video that is most relevant to the query text intent through Object Queries. The start and end times of the located video are directly predicted based on the global video information, which improves the accuracy of insurance video positioning and avoids the use of a sliding window to gradually traverse and calculate the time information of each video frame, thereby improving the efficiency of insurance video positioning.

[0184] The embodiment of the present application first obtains and identifies the video query text and the task category to which it belongs, and extracts the task category features and text features, which can facilitate the subsequent triggering of different query vector strategies for different task categories, thereby helping to improve the accuracy of video positioning; secondly, a target video frame image with a preset duration and continuousness is extracted from the target video, and the visual spatiotemporal features of the target video frame image are extracted to obtain the target visual features, which can extract the video content containing time and space information; finally, the target video is positioned based on the target fusion features to obtain the target positioning video information, and the target positioning video information including the start time and end time is obtained, which can directly predict the start and end time of the positioned video based on the global information of the video, thereby improving the accuracy of extracting the time of each video frame image, thereby significantly improving the accuracy of video positioning.

[0185] See also Figure 8 The embodiment of the present application further provides a multimodal video positioning device, which can implement the multimodal video positioning method described above, and the device includes:

[0186] A query text acquisition module, used to acquire a video query text indicating a query target video;

[0187] The task category feature extraction module is used to identify the task category to which the video query text belongs, obtain the video positioning task category, and perform task category feature extraction on the video positioning task category to obtain task category features;

[0188] A text feature extraction module is used to extract text features from the video query text to obtain video query text features;

[0189] The visual spatiotemporal feature extraction module is used to extract target video frame images with a preset duration and continuous duration from the target video, and perform visual spatiotemporal feature extraction on the target video frame images to obtain target visual features;

[0190] The feature fusion module is used to fuse the video query text features, task category features and target visual features to obtain the target fusion features;

[0191] The video positioning module is used to locate the target video based on the target fusion feature to obtain target positioning video information; wherein the target positioning video information includes a start time and an end time; the duration between the start time and the end time is less than or equal to a preset duration.

[0192] The specific implementation of the multimodal video positioning device is substantially the same as the specific embodiment of the multimodal video positioning method described above, and will not be described in detail here.

[0193] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the multimodal video positioning method. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0194] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0195] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0196] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store processing systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the multi-modal video positioning method of the embodiments of this application.

[0197] Input / output interface 903, used to implement information input and output;

[0198] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0199] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0200] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0201] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-modal video positioning method.

[0202] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0204] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multimodal video positioning method, characterized in that: The method comprises: Obtaining a video query text for indicating a query target video; Identifying the task category to which the video query text belongs to obtain a video positioning task category, and performing task category feature extraction on the video positioning task category to obtain a task category feature; Performing text feature extraction on the video query text to obtain video query text features; Extracting continuous target video frame images with a preset duration from the target video, and performing visual spatiotemporal feature extraction on the target video frame images to obtain target visual features; Fusing the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature; The target video is positioned based on the target fusion feature to obtain target positioning video information; wherein the target positioning video information includes a start time and an end time; and the duration between the start time and the end time is less than or equal to the preset duration.

2. The method according to claim 1, characterized in that The performing video positioning on the target video based on the target fusion feature to obtain target positioning video information includes: Performing feature query object decoding on the target fusion feature to obtain the target query object position; Performing video frame object classification detection on the target fusion feature according to the target query object position to obtain the query object category probability; The target fusion feature is time-located in the video frame based on the query object category probability and the target query object position to obtain the start time and the end time.

3. The method according to claim 1, characterized in that The step of fusing the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature includes: Performing attention-weighted processing on the video query text features and the target visual features to obtain visual text semantic features; Attention distribution adjustment is performed on the visual text semantic feature according to the task category feature to obtain the target fusion feature.

4. The method according to claim 3, characterized in that The performing attention weighted processing on the video query text features and the target visual features to obtain visual text semantic features includes: Calculating semantic similarity between the video query text features and the target visual features to obtain a video text similarity matrix; the video text similarity matrix includes semantic similarities between multiple video query text features and the target visual features; For the video query text feature, selecting the maximum semantic similarity between the video query text feature and the target visual feature from the semantic similarities as the target semantic similarity; Performing masking on the video text similarity matrix based on the target semantic similarity and a preset similarity threshold to obtain a similarity mask matrix; Attention weights are allocated based on the similarity mask matrix, the video query text features, and the target visual features to obtain the visual text semantic features.

5. The method according to claim 1, wherein The extracting visual out-of-control features from the target video frame image to obtain target visual features includes: Adaptively dividing the target video frame image into blocks to obtain a block video frame image; Performing linear projection processing on the block video frame image to obtain fixed-dimensional video frame features; Performing temporal position encoding on the fixed-dimensional video frame features to obtain video frame temporal features; Performing self-attention encoding on the fixed-dimensional video frame features to obtain video frame spatial features; The video frame temporal features and the video frame spatial features are fused to obtain the target visual features.

6. The method according to claim 5, characterized in that The performing temporal position encoding on the fixed-dimensional video frame features to obtain video frame temporal features includes: Obtaining a video frame timestamp that matches the target video frame image; Normalizing the video frame timestamp to obtain a normalized timestamp; Timestamp position encoding is performed on the fixed-dimensional video frame feature based on the normalized timestamp to obtain the video frame temporal feature.

7. The method according to any one of claims 1 to 6, characterized in that The extracting text features of the video query text to obtain video query text features includes: Performing word segmentation processing on the video query text to obtain a word segmentation query text; The word segmentation query text is encoded to obtain the video query text features.

8. A multi-modal video positioning device, characterized in that: The device comprises: A query text acquisition module, used to acquire a video query text indicating a query target video; A task category feature extraction module is used to identify the task category to which the video query text belongs, obtain a video positioning task category, and perform task category feature extraction on the video positioning task category to obtain a task category feature; A text feature extraction module is used to extract text features from the video query text to obtain video query text features; A visual spatiotemporal feature extraction module is used to extract target video frame images with a preset duration and continuous duration from the target video, and perform visual spatiotemporal feature extraction on the target video frame images to obtain target visual features; A feature fusion module, configured to fuse the video query text feature, the task category feature, and the target visual feature to obtain a target fusion feature; A video positioning module is used to perform video positioning on the target video based on the target fusion feature to obtain target positioning video information; wherein, the target positioning video information includes a start time and an end time; the duration between the start time and the end time is less than or equal to the preset duration.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multimodal video positioning method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal video positioning method according to any one of claims 1 to 7 is implemented.

Images