A method and system for locating target video segments based on multi-machine collaboration

Abstract

The present invention belongs to the field of target video segment positioning, and provides a method and system for positioning a target video segment based on multi-machine collaboration, the method comprising acquiring time-synchronized single-frame images of multiple perspectives; performing feature matching of each image, and establishing a correspondence between images of different perspectives based on the feature matching relationship; performing fusion of multi-perspective images based on the established correspondence relationship to obtain a complete panoramic video feature; and performing target video segment positioning based on the panoramic video feature in response to a query text. The present invention establishes a correspondence between different perspectives through feature matching, eliminates repeated redundant information through perspective fusion, generates a complete panoramic view, realizes the complementarity of different perspectives, and realizes efficient positioning of the target video segment based on the video after perspective fusion; and overcomes the defects of difficult multi-perspective video matching and poor fusion in the prior art.

Description

Technical Field

[0001] The present invention belongs to the field of target video segment positioning, and in particular relates to a target video segment positioning method and system based on multi-machine collaboration. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The Target Video Segment Localization task is to identify and extract short clips related to a specific query by analyzing a long video. This task requires the comprehensive use of the visual information of the video and possible text and audio clues to accurately locate the clips that meet the query requirements. The target video segment localization task has important application value in many scenarios. For example, in video retrieval and recommendation, it can quickly find the content that users are interested in and improve the user experience; in film and television production, it can quickly locate the required clips in a large amount of material to save editing time; in security monitoring, it can accurately locate key events in surveillance videos to improve monitoring efficiency.

[0004] In view of this, the task has attracted widespread attention from academia and industry, and many researchers have proposed a series of solutions based on deep neural networks. For example, some methods use convolutional neural networks (CNN) to extract visual features of video frames, and combine recurrent neural networks (RNN) or long short-term memory networks (LSTM) to capture temporal dynamic information in videos. In addition, some researchers use the attention mechanism to enhance the model's attention to key frames and important information, thereby improving positioning accuracy. In recent years, the successful application of the Transformer model in natural language processing and computer vision has also aroused widespread interest. Many of the latest works have begun to explore the application of Transformer to the task of positioning target video clips to further improve performance and efficiency.

[0005] The above work focuses on the localization of a single video shot from a single angle. However, due to the single viewing angle and limited field of view, independent cameras cannot fully cover the entire shooting or monitoring area, resulting in some key areas being ignored or blocked by other objects, which in turn affects the accuracy of video segment localization. In this case, even the most advanced target detection algorithms find it difficult to guarantee high-precision object recognition and tracking.

[0006] With the prevalence and maturity of acquisition equipment, it has become possible to obtain time-synchronized multi-view videos of the same scene, which has promoted the research on using multi-view collaboration to improve the accuracy of target video segment positioning. Compared with the target video segment positioning task of a single view, it mainly has the following challenges:

[0007] 1) Multi-view video matching is difficult. In videos from different perspectives, the target object may show different appearance characteristics due to factors such as shooting angle, lighting, and occlusion. Although these differences can be matched through advanced feature extraction algorithms, feature inconsistencies between perspectives still lead to matching difficulties. For example, even if the same target is shot from multiple angles, it may still be difficult to identify in some perspectives due to lighting problems or partial occlusion. Therefore, how to design a more robust feature matching algorithm to ensure consistency across perspectives has become a major problem in existing technologies.

[0008] 2) Poor fusion of multi-view videos. Although multi-view videos provide rich complementary information, the effective fusion of this information remains a complex problem. Existing methods usually rely on simple weighted averages, which have limited effectiveness when dealing with redundant information and noise. For example, videos from different perspectives may contain repeated or irrelevant information, which will affect the fusion effect. Therefore, how to design a more effective multi-view information fusion algorithm to fully utilize the advantages of multiple perspectives while avoiding the interference of information redundancy and noise has become the key to improving the positioning accuracy of multi-view video clips. Summary of the invention

[0009] In order to solve the above problems, the present invention proposes a method and system for locating a target video segment based on multi-machine collaboration. The present invention establishes a correspondence between different perspectives through feature matching, eliminates repeated redundant information through perspective fusion, generates a complete panoramic view, realizes the complementarity of different perspectives, and realizes efficient positioning of the target video segment based on the video after perspective fusion; the defects of difficult multi-perspective video matching and poor fusion in the prior art are overcome.

[0010] According to some embodiments, the present invention adopts the following technical solutions:

[0011] A method for locating a target video segment based on multi-machine collaboration includes the following steps:

[0012] Acquire time-synchronized single-frame images from multiple perspectives;

[0013] Perform feature matching on each image, and establish a correspondence between images of different viewing angles based on the feature matching relationship;

[0014] Based on the established correspondence, multi-view images are fused to obtain complete panoramic video features;

[0015] In response to the query text, the target video segment is located based on the panoramic video features.

[0016] As an optional implementation, the process of performing feature matching of each image includes: extracting time-synchronized single-frame images from continuous video data to ensure that the time difference between frames is constant;

[0017] Perform data preprocessing on the extracted video frames to ensure data consistency;

[0018] Use convolutional neural networks to extract feature points and descriptors of images;

[0019] Obtain the two-dimensional coordinate information of the feature points and map it to a high-dimensional vector space to generate a position feature vector;

[0020] The position feature vector is fused with the corresponding visual descriptor to form the final feature vector.

[0021] As a further step, in the process of fusing the position feature vector with the corresponding visual descriptor, splicing is used or the fusion weight is optimized through an attention mechanism.

[0022] As an optional implementation, the process of establishing the correspondence between images of different viewing angles based on the feature matching relationship includes:

[0023] Calculate the cosine similarity of the feature vectors corresponding to the feature points in images from different viewpoints to characterize the similarity of the feature points;

[0024] Construct the cost matrix based on the cosine similarity calculation results;

[0025] The Hungarian algorithm is introduced to perform optimization operations on the cost matrix, and the feature correspondence between multi-view images is determined by finding the matching pair with the minimum total cost;

[0026] Introduce a consistency check mechanism to verify the stability and accuracy of matching pairs;

[0027] Use local optimization algorithms or introduce domain knowledge to refine and adjust the initial matching results.

[0028] As an optional implementation, based on the established correspondence, the process of fusing the multi-view images includes:

[0029] Select key feature points of multi-view images as nodes, and determine the positions and feature information of the nodes;

[0030] Determine the connection edges between nodes based on the matching relationship, connect the nodes of adjacent frames according to the node timing, and establish a graph network structure;

[0031] Apply graph attention mechanism, assign attention weight to each node, and adopt multi-head attention mechanism to integrate information from different perspectives;

[0032] Through multiple rounds of iterative information diffusion, effective dissemination of all node information is achieved;

[0033] Aggregate nodes with connection relationships to obtain panoramic video features.

[0034] As a further implementation method, the process of aggregating nodes with connection relationships to obtain panoramic video features also includes aggregating connected feature point information through Q-Former based on feature point matching relationships, eliminating redundant information while retaining isolated feature points.

[0035] As an optional implementation, the process of locating the target video segment by using panoramic video features includes:

[0036] Build a vocabulary, convert the user query text into an ID sequence, and convert it into a text feature vector through the embedding layer;

[0037] Use Transformer to encode text feature vectors and panoramic video features for cross-modal interaction;

[0038] The target video segment positioning task is completed by outputting the time window of the positioning segment through the multi-layer perceptron network.

[0039] A target video segment positioning system based on multi-machine collaboration, comprising:

[0040] An image acquisition module, used to acquire time-synchronized multi-view single-frame images;

[0041] A feature matching module is used to perform feature matching of each image and establish a correspondence between images of different viewing angles based on the feature matching relationship;

[0042] The image fusion module is used to fuse multi-view images based on the established correspondence to obtain complete panoramic video features;

[0043] The target positioning module is used to locate the target video segment based on the panoramic video features in response to the query text.

[0044] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0045] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a method and system for locating a target video segment based on multi-machine collaboration, which is automatically implemented using a multi-view image feature matching network and a panoramic feature generation network, taking into account multiple key factors, including the difficulty of multi-view video matching and poor feature fusion.

[0048] The present invention uses convolutional neural networks to extract image feature points and descriptors, which can capture more image details and features and improve the accuracy of matching. It also enriches the feature representation by incorporating spatial position information, which helps to improve the accuracy of multi-view image matching.

[0049] The present invention ensures the optimality of feature point matching through the optimized matching of the Hungarian algorithm, effectively reduces mismatching, and improves matching accuracy and stability through consistency verification and optimization.

[0050] The present invention uses graph neural networks to propagate and fuse node information, which can better integrate multi-view information, reduce redundancy, and retain important image features. By aggregating nodes and retaining isolated feature points, the integrity and accuracy of panoramic video features are ensured to improve the efficiency and accuracy of locating target video clips.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0053] Figure 1 A flow chart of a method for locating a target video segment based on multi-machine collaboration according to an embodiment;

[0054] Figure 2 A flowchart of a process for establishing a multi-view matching relationship in one embodiment;

[0055] Figure 3 A schematic diagram of generating panoramic features by fusing multi-view information in an embodiment;

[0056] Figure 4 A schematic diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0061] Embodiment 1

[0062] A method for locating target video clips based on multi-machine collaboration, such as Figure 1 As shown in the figure, a convolutional neural network is used to extract feature points and descriptors of a single frame image, and spatial position information is introduced through a position encoder to more accurately express the image content. Then, the Hungarian matching algorithm is used to establish the correspondence between multi-view images to complete the feature matching process. At the same time, a consistency check mechanism is introduced to ensure the stability and accuracy of the matching pairs.

[0063] In order to achieve perspective fusion and reduce redundancy, a graph neural network is constructed based on the obtained matching relationship, and adjacent frame nodes are connected to supplement the temporal information. The graph attention algorithm is used to propagate node information. By aggregating the information of connected nodes and retaining isolated nodes, a complete panoramic video feature can be obtained. Based on the panoramic video feature, the target video segment can be efficiently and accurately located in combination with the user's query intent.

[0064] The specific contents are introduced below.

[0065] The specific steps include:

[0066] Step 1: perform feature matching on the acquired time-synchronized multi-view images, and establish corresponding relationships between images of different viewpoints accordingly.

[0067] Step 1.1, obtain the feature points and descriptors of a single frame image in each video, and integrate the spatial position information to enrich the feature expression.

[0068] Step 1.1.1, extract time-synchronized single-frame images from continuous video data to ensure that the time difference between frames is constant to reduce fluctuations in the time series.

[0069] Step 1.1.2, preprocess the extracted video frames to ensure data consistency.

[0070] Step 1.1.3, use convolutional neural network to extract feature points and descriptors of the image.

[0071] In a specific embodiment, there is The images captured by the cameras are For each image , through convolutional neural network CNN To extract feature points and descriptors:

[0072] (1)

[0073] in, and Respectively represent images No. feature points and descriptors, is the number of feature points.

[0074] Step 1.1.4, incorporate spatial location information to enhance feature representation.

[0075] In a specific implementation, the two-dimensional spatial position of the feature point is mapped to a high-dimensional vector, and a multi-layer perceptron MLP is used to fuse it with the visual information. The formula is as follows:

[0076] (2)

[0077] in, is an image No. Comprehensive representation of feature points, and Respectively represent images No. feature points and descriptors.

[0078] Step 1.2, establish the correspondence between multi-view images.

[0079] Step 1.2.1: Comprehensive representation of multi-view images , and construct the cost matrix by calculating the similarity.

[0080] In a specific implementation, the cosine similarity between each pair of feature points is calculated to construct a cost matrix , the formula is as follows:

[0081] (3)

[0082] in, is an image No. Comprehensive representation of feature points, is an image No. Comprehensive representation of feature points, is the cosine similarity between two feature points.

[0083] Step 1.2.2: Use the Hungarian algorithm to establish the correspondence between multi-view images.

[0084] In a specific implementation, the Hungarian algorithm is used to calculate the cost matrix Find the best matching pair on the network to minimize the total cost. The Hungarian algorithm finds the best match by optimizing the objective function:

[0085] (4)

[0086] in, is an indicator matrix, if the image The feature points and images The feature points match, then ,otherwise .

[0087] Step 1.2.3, introduce a consistency check mechanism to further verify the stability and accuracy of the matching pairs;

[0088] Step 1.2.4, use local optimization algorithms or introduce domain knowledge to refine and adjust the preliminary matching results to improve matching accuracy.

[0089] In a specific implementation, an existing local search algorithm is used to find a better solution from the neighborhood of the current matching feature point (i.e., feature points that are adjacent in the image space) to gradually optimize the initial matching result. In addition, by introducing domain knowledge of spatial continuity, matching results that do not meet the specific constraint of spatial continuity (i.e., the matching result of a feature point is not adjacent to the matching results of surrounding feature points) can be corrected.

[0090] Step 2: Realize efficient fusion of multi-view images and obtain complete panoramic video features.

[0091] Step 2.1: Improve the visual representation based on the matching relationship and reduce redundant information. Specifically include:

[0092] Step 2.1.1, select the key feature points of the multi-view image as nodes, and determine the location and feature information of the nodes.

[0093] Step 2.1.2, determine the connection edges between nodes based on the matching relationship, and connect the nodes of adjacent frames according to the node timing to establish a graph network structure.

[0094] Step 2.1.3, based on the graph network structure, realize the effective interaction of node information.

[0095] In a specific implementation, a graph network is constructed based on feature points and their matching relationships, and information is propagated. In this process, the feature points of each image will regard the matching relationship as a connecting edge, that is, if the feature points of two images are matched, there is an edge connecting them, and finally the information is diffused through the following formula:

[0096] (5)

[0097] in, Indicates The first image The feature point is The updated features of the layer, Indicates connection operation, information It is related to The first image The aggregation result of other feature points associated with feature points.

[0098] In step 2.1.4, a multi-head attention mechanism is used to fuse information from different perspectives and improve the expressiveness of node features.

[0099] In a specific implementation, the information of other connected feature points is aggregated through a multi-head attention mechanism, and the formula is as follows:

[0100] (6)

[0101] in, is the scaling factor, , , and is the trainable weight matrix, is a multi-head attention mechanism, For the connection operation, is the dot product attention mechanism, is the activation function, are query, key, and value matrices.

[0102] Step 2.1.5: Through multiple rounds of iterative information diffusion, effective dissemination of all node information is achieved.

[0103] In a specific embodiment, After the secondary image propagation, cross-view representation can be obtained .

[0104] Step 2.2: Eliminate visual redundancy and obtain panoramic video features.

[0105] Step 2.2.1, aggregate nodes with connection relationships, remove redundant information while retaining isolated nodes to ensure the integrity of information.

[0106] In a specific implementation, the connected feature point information is aggregated through Q-Former according to the feature point matching relationship, while retaining isolated feature points, so as to reduce redundancy while ensuring the integrity of the information.

[0107] Step 3: Achieve accurate positioning of the target video clip.

[0108] Step 3.1, use the embedding layer to convert the user query text into a text feature vector;

[0109] Step 3.2: Use Transformer to encode text feature vectors and panoramic video features to perform cross-modal interaction.

[0110] In this embodiment, cross-modal interaction can be implemented by a traditional Transformer model.

[0111] Step 3.3, the time window of the positioning segment is output by the multi-layer perceptron network to complete the target video segment positioning task.

[0112] Embodiment 2:

[0113] A target video segment positioning system based on multi-machine collaboration, comprising:

[0114] An image acquisition module, used to acquire time-synchronized multi-view single-frame images;

[0115] A feature matching module is used to perform feature matching of each image and establish a correspondence between images of different viewing angles based on the feature matching relationship;

[0116] The execution process of this module can be specifically referred to the process of step 1 of the first embodiment, which will not be repeated here;

[0117] The image fusion module is used to fuse multi-view images based on the established correspondence to obtain complete panoramic video features;

[0118] The execution process of this module can be specifically referred to the process of step 2 of the first embodiment, which will not be repeated here;

[0119] The target positioning module is used to locate the target video segment based on the panoramic video features in response to the query text.

[0120] The execution process of this module can be specifically referred to the process of step 3 of the first embodiment, which will not be repeated here.

[0121] The steps involved in the above embodiment 2 correspond to those in the method embodiment 1. For the specific implementation method, please refer to the relevant description part of the embodiment 1.

[0122] It can be understood that the above-mentioned units / modules can be separately or completely combined into one or several other units / modules, or one (some) of the units can be further divided into multiple functionally smaller units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application.

[0123] The above modules of the system are divided based on logical functions. In practical applications, the function of one module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. For example, the feature matching module of this embodiment may include:

[0124] A similarity calculation module is used to calculate the cosine similarity of feature vectors corresponding to feature points in images of different viewing angles to characterize the similarity of the feature points;

[0125] The cost matrix construction module is used to construct the cost matrix according to the cosine similarity calculation results;

[0126] The optimal operation module is used to introduce the Hungarian algorithm to perform optimization operations on the cost matrix, and determine the feature correspondence between multi-view images by finding the matching pair with the minimum total cost;

[0127] The consistency check module is used to introduce a consistency check mechanism to verify the stability and accuracy of the matching pair;

[0128] The matching optimization module is used to refine and adjust the preliminary matching results by adopting local optimization algorithms or introducing domain knowledge.

[0129] Similarly, in other embodiments of the present application, the system may also include other units / modules. In actual applications, these functions may also be implemented with the assistance of other units, and may be implemented by collaboration of multiple units.

[0130] According to another embodiment of the present application, the system described in this embodiment can be constructed and the method of Embodiment 1 can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Embodiment 1 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0131] Embodiment three:

[0132] This implementation provides an electronic device, such as Figure 4 As shown, the electronic device includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0133] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0134] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0135] The processor 1001 is configured to execute the following process:

[0136] Acquire time-synchronized single-frame images from multiple perspectives;

[0137] Perform feature matching on each image, and establish a correspondence between images of different viewing angles based on the feature matching relationship;

[0138] Based on the established correspondence, multi-view images are fused to obtain complete panoramic video features;

[0139] In response to the query text, the target video segment is located based on the panoramic video features.

[0140] Or the process of steps 1-3 in Example 1 will not be repeated here.

[0141] Embodiment 4:

[0142] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0143] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0144] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0145] Acquire time-synchronized single-frame images from multiple perspectives;

[0146] Perform feature matching on each image, and establish a correspondence between images of different viewing angles based on the feature matching relationship;

[0147] Based on the established correspondence, multi-view images are fused to obtain complete panoramic video features;

[0148] In response to the query text, the target video segment is located based on the panoramic video features.

[0149] Or the process of steps 1-3 in Example 1 will not be repeated here.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A method for locating a target video segment based on multi-machine collaboration, characterized in that: The following steps are involved: Acquire time-synchronized single-frame images from multiple perspectives; Perform feature matching on each image, and establish a correspondence between images of different viewing angles based on the feature matching relationship; The process of establishing the correspondence between images of different viewpoints based on the feature matching relationship includes: Calculate the cosine similarity of the feature vectors corresponding to the feature points in images from different viewpoints to characterize the similarity of the feature points; Construct the cost matrix based on the cosine similarity calculation results; The Hungarian algorithm is introduced to perform optimization operations on the cost matrix, and the feature correspondence between multi-view images is determined by finding the matching pair with the minimum total cost; Introduce a consistency check mechanism to verify the stability and accuracy of matching pairs; Use local optimization algorithms or introduce domain knowledge to refine and adjust the initial matching results; Based on the established correspondence, multi-view images are fused to obtain complete panoramic video features; Based on the established correspondence, the process of fusing multi-view images includes: Select key feature points of multi-view images as nodes, and determine the positions and feature information of the nodes; Determine the connection edges between nodes based on the matching relationship, connect the nodes of adjacent frames according to the node timing, and establish a graph network structure; Apply graph attention mechanism, assign attention weight to each node, and adopt multi-head attention mechanism to integrate information from different perspectives; Through multiple rounds of iterative information diffusion, effective dissemination of all node information is achieved; Aggregate nodes with connection relationships to obtain panoramic video features; In response to the query text, the target video segment is located based on the panoramic video features.

2. The method for locating a target video segment based on multi-machine collaboration as claimed in claim 1, characterized in that: The process of performing feature matching of each image includes: extracting time-synchronized single-frame images from continuous video data to ensure that the time difference between frames is constant; Perform data preprocessing on the extracted video frames to ensure data consistency; Use convolutional neural networks to extract feature points and descriptors of images; Obtain the two-dimensional coordinate information of the feature points and map it to a high-dimensional vector space to generate a position feature vector; The position feature vector is fused with the corresponding visual descriptor to form the final feature vector.

3. The method for locating a target video segment based on multi-machine collaboration as claimed in claim 2, characterized in that: In the process of fusing the position feature vector with the corresponding visual descriptor, splicing is used or the fusion weight is optimized through the attention mechanism.

4. The method for locating a target video segment based on multi-machine collaboration as claimed in claim 1, characterized in that: The process of aggregating nodes with connection relationships to obtain panoramic video features also includes aggregating connected feature point information through Q-Former based on the feature point matching relationship, eliminating redundant information while retaining isolated feature points.

5. The method for locating a target video segment based on multi-machine collaboration as claimed in claim 1, characterized in that: The process of locating the target video segment using panoramic video features includes: Build a vocabulary, convert the user query text into an ID sequence, and convert it into a text feature vector through the embedding layer; Use Transformer to encode text feature vectors and panoramic video features for cross-modal interaction; The target video segment positioning task is completed by outputting the time window of the positioning segment through the multi-layer perceptron network.

6. A target video segment positioning system based on multi-machine collaboration, characterized in that: include: An image acquisition module, used to acquire time-synchronized multi-view single-frame images; A feature matching module is used to perform feature matching of each image and establish a correspondence between images of different viewing angles based on the feature matching relationship; The process of establishing the correspondence between images of different viewpoints based on the feature matching relationship includes: Calculate the cosine similarity of the feature vectors corresponding to the feature points in images from different viewpoints to characterize the similarity of the feature points; Construct the cost matrix based on the cosine similarity calculation results; The Hungarian algorithm is introduced to perform optimization operations on the cost matrix, and the feature correspondence between multi-view images is determined by finding the matching pair with the minimum total cost; Introduce a consistency check mechanism to verify the stability and accuracy of matching pairs; Use local optimization algorithms or introduce domain knowledge to refine and adjust the initial matching results; The image fusion module is used to fuse multi-view images based on the established correspondence to obtain complete panoramic video features; Based on the established correspondence, the process of fusing multi-view images includes: Select key feature points of multi-view images as nodes, and determine the positions and feature information of the nodes; Determine the connection edges between nodes based on the matching relationship, connect the nodes of adjacent frames according to the node timing, and establish a graph network structure; Apply graph attention mechanism, assign attention weight to each node, and adopt multi-head attention mechanism to integrate information from different perspectives; Through multiple rounds of iterative information diffusion, effective dissemination of all node information is achieved; Aggregate nodes with connection relationships to obtain panoramic video features; The target positioning module is used to locate the target video segment based on the panoramic video features in response to the query text.

7. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 5 are completed.

Images