A Video Retrieval Method and Server Based on Semantic Embedding and Video Memory Coding

Abstract

This application provides a video retrieval method and server based on semantic embedding and video memory coding. The method performs visual content parsing on consecutive frames of the input video using a pre-trained visual language model, mapping the visual content of each frame to a semantic embedding vector. A video memory coding module is constructed based on the temporal series features of the semantic embedding vectors, dynamically encoding the semantic embedding vectors of consecutive frames in chronological order to generate a video memory feature sequence containing temporal correlation information. Semantic parsing is performed on the query text based on this visual language model, transforming the query text into a query semantic embedding vector in the same spatial dimension as the semantic embedding vector. Using this query semantic embedding vector as the retrieval basis, similarity matching is performed in the video memory feature sequence, and a candidate set of related video segments is determined through feature correlation analysis within a temporal window. Temporal coherence verification is performed on the candidate set, and combined with the temporal correlation degree of the semantic embedding vectors, the matching video segments are output.

Description

Technical Field

[0001] This application belongs to the field of video retrieval technology, specifically relating to a video retrieval method and server based on semantic embedding and video memory coding. Background Technology

[0002] With the rapid development of digital media technology, videos, as an important information carrier, are growing exponentially in quantity. How to quickly and accurately retrieve the desired content from this massive amount of video has become a pressing issue. Video retrieval technology has evolved from text-based tagging to visual feature-based retrieval, and then to semantic feature-based retrieval. Text-based tagging relies on manual annotation of video content, which is not only time-consuming and labor-intensive, but also highly susceptible to subjective factors, making it difficult to comprehensively and accurately reflect the semantic content of the video. Visual feature-based retrieval extracts and matches low-level visual features such as color, texture, and shape from video frames. While this avoids the shortcomings of manual annotation, a "semantic gap" exists between low-level visual features and high-level semantics, making it difficult to meet users' needs for retrieving semantic content from videos.

[0003] With the rise of deep learning technology, some existing video retrieval solutions use deep learning models to process video frames, extract semantic feature vectors from the video frames, and then achieve video retrieval by calculating the similarity between the semantic feature vectors and the query text feature vectors. However, these solutions still struggle to further improve the efficiency and accuracy of video retrieval. Summary of the Invention

[0004] This application provides a video retrieval method and server based on semantic embedding and video memory coding to improve the efficiency and accuracy of video retrieval.

[0005] In a first aspect, embodiments of this application provide a video retrieval method based on semantic embedding and video memory coding, applied to a video retrieval server, the method comprising:

[0006] Visual content parsing is performed on consecutive frames of the input video using a pre-trained visual language large model, mapping the visual content of each frame of the input video into a semantic embedding vector.

[0007] Based on the temporal sequence features of the semantic embedding vector, a video memory coding module is constructed to dynamically encode the semantic embedding vectors of consecutive frames in temporal order, generating a video memory feature sequence containing temporal correlation information.

[0008] Based on the pre-trained visual language model, semantic parsing is performed on the query text, transforming the query text into a query semantic embedding vector; wherein, the query semantic embedding vector and the semantic embedding vector are in the same spatial dimension;

[0009] Based on the query semantic embedding vector, similarity matching is performed in the video memory feature sequence, and a candidate set of video segments associated with the query semantic embedding vector is determined through feature association analysis within the time window;

[0010] Perform temporal coherence verification on the candidate video segments, and combine the temporal correlation of the semantic embedding vectors of each video segment in the candidate video segments to output the video segment that matches the query semantic embedding vector.

[0011] Secondly, embodiments of this application provide a video retrieval server, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described method.

[0012] Thirdly, embodiments of this application provide a computer-readable storage medium including a computer program that, when run on a video retrieval server, causes the video retrieval server to perform the steps of the above-described method.

[0013] In this application, a pre-trained visual language model is used to map the visual content of consecutive frames of input video to semantic embedding vectors. Combined with the video memory encoding module, the semantic embedding vectors of consecutive frames are dynamically encoded to generate a video memory feature sequence containing temporal correlation information. At the same time, the visual language model is used to transform the query text into a query semantic embedding vector of the same spatial dimension. Based on the query semantic embedding vector, similarity matching is performed in the video memory feature sequence, and a candidate set of video segments is determined by feature correlation analysis within the temporal window. Finally, the temporal coherence of the candidate set is verified, and the matching video segment is output by combining the temporal correlation of the semantic embedding vector.

[0014] Overall, this application's embodiments construct a complete technical chain from semantic representation of video visual content and temporal correlation feature encoding to cross-modal semantic matching and segment verification. This effectively breaks down the modal barriers between video visual features and text semantic features, achieving precise semantic alignment between video content and text queries. Through the dynamic encoding of semantic embedding vectors for consecutive frames using a video memory encoding module, the temporal correlation information of video content is fully captured, avoiding the isolation of semantic representation in a single frame. The design of query semantic embedding vectors in the same spatial dimension ensures the effectiveness and accuracy of similarity matching. The combination of feature correlation analysis within the temporal window and temporal coherence verification improves the completeness and coherence of video segment retrieval, solving problems such as inaccurate semantic matching, insufficient utilization of temporal information, and poor segment coherence in traditional video retrieval. Overall, this improves the efficiency and accuracy of video retrieval, enabling efficient retrieval and utilization of video content. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a video retrieval method based on semantic embedding and video memory coding, provided in an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of the structure of a video retrieval server provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0018] See Figure 1 This is a video retrieval method based on semantic embedding and video memory coding provided in the embodiments of this application. This method can be applied to a video retrieval server, and the specific process is as follows: steps 110-150.

[0019] Step 110: Perform visual content parsing on consecutive frames of the input video using a pre-trained visual language large model, mapping the visual content of each frame of the input video into a semantic embedding vector.

[0020] In this embodiment, the input video is a meteorological observation video recording a typhoon weather process in a certain region. The video contains multiple consecutive frames, each showing a different meteorological scene at a different time, such as cloud changes, wind speed and direction indicators, and precipitation. The video retrieval server first obtains all consecutive frames of the input video and inputs the image data of each frame into a pre-trained visual language model. This pre-trained visual language model includes an image feature extraction module, a text semantic mapping module, and a cross-modal alignment module. The image feature extraction module adopts a convolutional neural network structure, extracting visual features from each frame of the image through multiple convolutional layers and pooling layers, such as cloud morphology (cumulonimbus clouds, stratus clouds), anemometer pointer angle, and rain gauge readings. The text semantic mapping module converts the extracted visual features into corresponding text semantic descriptions. The cross-modal alignment module maps the text semantic descriptions to a preset semantic space, generating corresponding semantic embedding vectors. For example, one frame shows cumulonimbus clouds covering the observation area, an anemometer pointer pointing to 150 degrees, and a rain gauge reading of moderate. The visual language big data model extracts the morphological features of the cumulonimbus clouds, the angular features of the anemometer, and the rain gauge reading features in that frame. These features are then transformed into a textual semantic description of "cumulonimbus clouds covering the observation area, wind direction is southwest, and precipitation is moderate." This textual semantic description is then mapped into a multi-dimensional semantic embedding vector, which contains multi-dimensional information such as weather phenomenon category, meteorological element values, and spatial coverage.

[0021] Step 120: Based on the temporal sequence features of the semantic embedding vectors, construct a video memory coding module to dynamically encode the semantic embedding vectors of consecutive frames in chronological order, generating a video memory feature sequence containing temporal correlation information.

[0022] In this embodiment, the video retrieval server constructs a video memory coding module based on the temporal sequence features of the semantic embedding vectors of all consecutive frames generated in step 110. This video memory coding module includes a temporal feature extraction submodule, a dynamic coding submodule, and a feature fusion submodule. The temporal feature extraction submodule adopts a recurrent neural network structure, sequentially inputting the semantic embedding vectors of consecutive frames in chronological order to capture the temporal correlation between the semantic embedding vectors of adjacent frames. For example, if the cumulonimbus cloud coverage area in the previous frame is one-third of the observation area, and the cumulonimbus cloud coverage area in the next frame expands to one-half of the observation area, the temporal feature extraction submodule captures the temporal correlation of "expanded cumulonimbus cloud coverage area". The dynamic coding submodule encodes the captured temporal correlation information to generate a temporal correlation coding vector. The feature fusion submodule concatenates and fuses the semantic embedding vector of each frame with the temporal correlation coding vector to generate a video memory feature sequence. For example, the semantic embedding vectors of consecutive frames are, in chronological order, vector V1 (corresponding to cumulonimbus cloud covering one-third of the area, southwest wind speed, and moderate precipitation), vector V2 (corresponding to cumulonimbus cloud covering half of the area, southwest wind speed, and moderate precipitation), and vector V3 (corresponding to cumulonimbus cloud covering the entire area, southwest wind speed, and heavy precipitation). The temporal feature extraction submodule captures the temporal association of "cumulonimbus cloud coverage expanding" from V1 to V2 and the temporal association of "cumulonimbus cloud coverage further expanding and precipitation increasing" from V2 to V3. The dynamic encoding submodule encodes the above temporal associations into temporal association encoding vectors T1 and T2. The feature fusion submodule concatenates V1 with T1, V2 with T1 and T2, and V3 with T2 to generate a video memory feature sequence containing temporal association information.

[0023] Step 130: Perform semantic parsing on the query text based on the pre-trained visual language large model, and transform the query text into a query semantic embedding vector; wherein the query semantic embedding vector and the semantic embedding vector are in the same spatial dimension.

[0024] In this embodiment, the user inputs a query text of "a segment showing the expansion of cumulonimbus cloud coverage and increased precipitation during a typhoon". The video retrieval server performs semantic parsing on the query text. First, the query text is input into the text processing module of a pre-trained visual language model. The text processing module includes a word segmentation unit, a semantic extraction unit, and a spatial mapping unit. The word segmentation unit breaks down the query text into semantic units such as "typhoon weather process", "expansion of cumulonimbus cloud coverage", and "increased precipitation". The semantic extraction unit extracts the semantic information of each semantic unit, such as scene information for "typhoon weather process", meteorological phenomenon change information for "expansion of cumulonimbus cloud coverage", and meteorological element change information for "increased precipitation". The spatial mapping unit maps the extracted semantic information to the same semantic space as the semantic embedding vector in step 110, generating a query semantic embedding vector. The dimension of this query semantic embedding vector is consistent with the dimension of the semantic embedding vector generated in step 110, ensuring that the two can be matched for similarity in the future.

[0025] Step 131: Perform word segmentation on the query text, splitting the query text into several independent semantic units, each semantic unit corresponding to a core concept in the query text.

[0026] In this embodiment, the query text is "a segment showing the expansion of cumulonimbus cloud coverage and increased precipitation during a typhoon". The video retrieval server calls the word segmentation module to perform word segmentation on the query text. The word segmentation module uses a word segmentation algorithm that combines rules and statistics. First, it matches the words in the query text according to a preset meteorological terminology dictionary, identifying "typhoon weather process", "expansion of cumulonimbus cloud coverage", and "increased precipitation" as candidate semantic units. Then, it calculates the collocation probability between candidate semantic units through a statistical language model, determining the final semantic units as "typhoon weather process", "expansion of cumulonimbus cloud coverage", and "increased precipitation". Each semantic unit corresponds to a core concept in the query text: "typhoon weather process" corresponds to the core concept of the scene, "expansion of cumulonimbus cloud coverage" corresponds to the core concept of meteorological phenomenon change, and "increased precipitation" corresponds to the core concept of meteorological element change.

[0027] Step 132: Input each semantic unit into the pre-trained base semantic embedding model to generate the base semantic embedding vector for each semantic unit.

[0028] In this embodiment, the video retrieval server inputs the three semantic units obtained in step 131—"typhoon weather process," "cumulonimbus cloud coverage expansion," and "precipitation enhancement"—into a pre-trained basic semantic embedding model. This basic semantic embedding model employs a Transformer structure, containing multiple self-attention layers and feedforward neural network layers. The self-attention layers capture the semantic relationships within the semantic units, while the feedforward neural network layers perform nonlinear transformations on the semantic relationship information to generate corresponding basic semantic embedding vectors. For example, inputting "typhoon weather process" into the basic semantic embedding model allows the model to capture the semantic relationships between "typhoon," "weather," and "process," generating a basic semantic embedding vector containing information such as typhoon definition, weather characteristics, and process attributes. Similarly, inputting "cumulonimbus cloud coverage expansion" into the basic semantic embedding model allows the model to capture the semantic relationships between "cumulonimbus cloud," "coverage," and "expansion," generating a basic semantic embedding vector containing information such as cumulonimbus cloud type, coverage index, and expansion trend. Finally, inputting "precipitation enhancement" into the basic semantic embedding model allows the model to capture the semantic relationships between "precipitation" and "enhancement," generating a basic semantic embedding vector containing information such as precipitation type and enhancement level.

[0029] Step 133: Call the pre-trained context semantic association model, perform semantic association processing on the basic semantic embedding vector according to the original word order of the query text, capture the direct and indirect contextual relationships between semantic units through the semantic dependency layer of the context semantic association model, and generate an enhanced semantic embedding vector containing multi-level contextual dependency information.

[0030] In this embodiment, the video retrieval server calls a pre-trained contextual semantic association model and inputs the three basic semantic embedding vectors generated in step 132 into the model according to the original word order of the query text: "Typhoon weather process → Expansion of cumulonimbus cloud coverage → Increased precipitation". The contextual semantic association model includes a semantic dependency layer and a feature fusion layer. The semantic dependency layer adopts a graph neural network structure, taking each basic semantic embedding vector as a node and constructing the connection relationship between nodes according to the original word order of the query text. It captures the direct contextual relationship between semantic units (such as the direct association between "Typhoon weather process" and "Expansion of cumulonimbus cloud coverage", and the direct association between "Expansion of cumulonimbus cloud coverage" and "Increased precipitation") and the indirect contextual relationship (such as the indirect association between "Typhoon weather process" and "Increased precipitation" established through "Expansion of cumulonimbus cloud coverage"). The feature fusion layer concatenates and fuses the captured direct and indirect contextual relationship information with the basic semantic embedding vectors to generate an enhanced semantic embedding vector. For example, the semantic dependency layer captures that "typhoon weather process" is a prerequisite for "expansion of cumulonimbus cloud coverage" and "increased precipitation", and "expansion of cumulonimbus cloud coverage" is the cause of "increased precipitation". The feature fusion layer concatenates the above contextual relationship information with the three basic semantic embedding vectors to generate an enhanced semantic embedding vector containing multi-level contextual dependency information.

[0031] Step 134: Call the pre-trained query semantic temporal inference model, input the enhanced semantic embedding vector into the query semantic temporal inference model to perform temporal evolution prediction processing of the query semantic embedding vector, and generate the potential evolution path of query semantics in the video temporal dimension.

[0032] In this embodiment, the video retrieval server invokes a pre-trained query semantic temporal inference model, inputting the enhanced semantic embedding vector generated in step 133 into the model. This query semantic temporal inference model adopts a long short-term memory network structure, including an input layer, a hidden layer, and an output layer. The input layer receives the enhanced semantic embedding vector, the hidden layer captures the temporal features in the enhanced semantic embedding vector, and the output layer predicts the potential evolution path of the query semantics in the video temporal dimension based on the temporal features. For example, based on the enhanced semantic embedding vector of "the coverage of cumulonimbus clouds expands and precipitation intensifies during typhoon weather," the model predicts the potential evolution path as "cumulonimbus cloud coverage continues to expand → precipitation intensity further increases → cumulonimbus cloud coverage reaches its maximum value → precipitation intensity reaches its peak → cumulonimbus cloud coverage begins to shrink → precipitation intensity weakens." This path contains multiple temporal nodes, each node corresponding to a potential state of the query semantics.

[0033] Step 135: Call the pre-trained query semantic space mapping model, input the enhanced semantic embedding vector into the query semantic space mapping model to perform spatial semantic transformation processing of the query semantic embedding vector, map the query semantics to the dimension consistent with the spatial semantics of the video frame, and generate a spatial enhanced semantic embedding vector containing spatial semantic information.

[0034] In this embodiment, the video retrieval server invokes a pre-trained query semantic space mapping model, inputting the enhanced semantic embedding vector generated in step 133 into the model. This query semantic space mapping model includes a spatial semantic extraction layer and a dimension transformation layer. The spatial semantic extraction layer extracts spatially relevant semantic information from the enhanced semantic embedding vector, such as the spatial extent information corresponding to "cumulonimbus cloud coverage area." The dimension transformation layer employs a fully connected neural network structure to transform the extracted spatial semantic information to a dimension consistent with the spatial semantics of the video frame, generating a spatial enhanced semantic embedding vector containing spatial semantic information. For example, the model extracts the spatial extent information of "cumulonimbus cloud coverage area" from the enhanced semantic embedding vector, transforms it into spatial semantic information consistent with the dimension representing the cumulonimbus cloud coverage area in the video frame, and generates a spatial enhanced semantic embedding vector containing information such as spatial extent and spatial location.

[0035] Step 136: Call the pre-trained temporal-spatial fusion model, perform structural alignment and fusion processing on the spatially enhanced semantic embedding vector and the potential evolution path, and generate a composite semantic embedding vector that integrates temporal and spatial information.

[0036] In this embodiment, the video retrieval server invokes a pre-trained temporal-spatial fusion model, inputting the spatially enhanced semantic embedding vector generated in step 135 and the potential evolution path generated in step 134 into the model. This temporal-spatial fusion model includes a structure alignment layer and a feature fusion layer. The structure alignment layer aligns the spatial dimension in the spatially enhanced semantic embedding vector with the temporal dimension in the potential evolution path, ensuring structural matching. The feature fusion layer employs an attention mechanism to weightedly fuse the aligned spatially enhanced semantic embedding vector and the potential evolution path, generating a composite semantic embedding vector that integrates temporal and spatial information. For example, the model aligns the spatial information of "cumulonimbus cloud coverage" in the spatially enhanced semantic embedding vector with the temporal information of "cumulonimbus cloud coverage continuously expanding" in the potential evolution path, and weightedly fuses the two through an attention mechanism to generate a composite semantic embedding vector containing dynamic changes in spatial range.

[0037] Step 137: Input the composite semantic embedding vector into the pre-trained visual language big model, which is the same as the video frame processing, to perform semantic space mapping processing. The dimension of the query semantic embedding vector is automatically adjusted by the dimension adaptation layer of the pre-trained visual language big model so that the dimension of the query semantic embedding vector is consistent with that of the semantic embedding vector.

[0038] In this embodiment, the video retrieval server inputs the composite semantic embedding vector generated in step 136 into the same pre-trained visual language model as in step 110. The dimension adaptation layer of this visual language model adopts a fully connected neural network structure to adjust the dimension of the composite semantic embedding vector to match the dimension of the semantic embedding vector generated in step 110. For example, if the dimension of the composite semantic embedding vector is D1 and the dimension of the semantic embedding vector generated in step 110 is D2, the dimension adaptation layer adjusts the dimension of the composite semantic embedding vector from D1 to D2 through a linear transformation to generate the final query semantic embedding vector.

[0039] Step 140: Based on the query semantic embedding vector, perform similarity matching in the video memory feature sequence, and determine the candidate set of video segments associated with the query semantic embedding vector through feature association analysis within the time window.

[0040] In this embodiment, the video retrieval server uses the query semantic embedding vector generated in step 137 as the retrieval basis and performs similarity matching on the video memory feature sequence generated in step 120. First, the similarity score between the query semantic embedding vector and each video memory feature in the video memory feature sequence is calculated. The similarity score is obtained by calculating the cosine similarity of the angle between the two vectors. Then, the temporal window size is set, dividing the video memory feature sequence into multiple temporal windows, and the association between features is analyzed within each temporal window. Finally, based on the similarity score and the feature association analysis results within the temporal window, a candidate set of video segments associated with the query semantic embedding vector is determined. For example, if the query semantic embedding vector has a high similarity score with multiple video memory features within one temporal window of the video memory feature sequence, and the feature association within that temporal window conforms to the temporal evolution path of the query semantics, the video retrieval server determines the video segment corresponding to that temporal window as a candidate segment.

[0041] Step 141: Using the query semantic embedding vector as the core retrieval basis, construct a cross-dimensional semantic retrieval framework that integrates temporal inference and spatial semantic transformation.

[0042] In this embodiment, the video retrieval server uses the query semantic embedding vector generated in step 137 as the core retrieval basis to construct a cross-dimensional semantic retrieval framework. This framework includes a temporal deduction module, a spatial semantic transformation module, and a similarity matching module. The temporal deduction module predicts the possible matching temporal range in the video memory feature sequence based on the temporal evolution path of the query semantic embedding vector. The spatial semantic transformation module transforms the spatial feature representation in the video memory feature sequence based on the spatial semantic information of the query semantic embedding vector. The similarity matching module combines the temporal deduction results and the spatial semantic transformation results to calculate the similarity score between the query semantic embedding vector and the video memory features.

[0043] Step 142: Perform temporal segmentation processing on the video memory feature sequence. Identify semantic mutation points in the video memory feature sequence through the temporal segmentation layer of the cross-dimensional semantic retrieval framework, and divide the continuous video memory features into several temporal segments with complete semantics.

[0044] In this embodiment, the video retrieval server performs temporal segmentation processing on the video memory feature sequence generated in step 120 through the temporal segmentation layer of the cross-dimensional semantic retrieval framework. The temporal segmentation layer employs a sliding window method, setting a fixed-size sliding window that slides across the video memory feature sequence. It calculates the semantic change rate of the video memory features within each window. When the semantic change rate exceeds a preset threshold, the starting position of the window is identified as a semantic abrupt change point. Based on these abrupt changes, the video memory feature sequence is divided into several temporal segments with complete semantics. For example, if the semantic change rate at one position in the video memory feature sequence exceeds the preset threshold, the temporal segmentation layer identifies this position as a semantic abrupt change point, dividing the video memory features before the abrupt change point into one temporal segment and the video memory features after the abrupt change point into another temporal segment.

[0045] Step 143: Perform semantic aggregation processing on the video memory features within each time segment. The feature fusion layer of the cross-dimensional semantic retrieval framework integrates multiple video memory features within the time segment into a segment-level semantic embedding vector.

[0046] In this embodiment, the video retrieval server performs semantic aggregation processing on the video memory features within each temporal segment divided in step 142 through the feature fusion layer of a cross-dimensional semantic retrieval framework. The feature fusion layer employs an attention mechanism, assigning attention weights to each video memory feature within a temporal segment. These attention weights are determined based on the similarity score between the video memory feature and the query semantic embedding vector; the higher the similarity score, the greater the attention weight. Then, each video memory feature is multiplied by its corresponding attention weight, and all product results are concatenated and fused to generate a segment-level semantic embedding vector. For example, a temporal segment may contain three video memory features V1, V2, and V3, with similarity scores of S1, S2, and S3 respectively, and attention weights of W1, W2, and W3 respectively (W1 = S1 / (S1+S2+S3), W2 = S2 / (S1+S2+S3), W3 = S3 / (S1+S2+S3)). The feature fusion layer will... , , The segments are spliced ​​and fused to generate fragment-level semantic embedding vectors.

[0047] Step 144: Call the pre-trained video segment temporal inference model, input the segment-level semantic embedding vector into the video segment temporal inference model to perform temporal evolution prediction processing of the segment-level semantic embedding vector, and generate the subsequent evolution path of the segment-level semantic embedding vector.

[0048] In this embodiment, the video retrieval server invokes a pre-trained video segment temporal inference model, inputting the segment-level semantic embedding vectors generated in step 143 into the model. This model employs a long short-term memory network structure, comprising an input layer, hidden layers, and an output layer. The input layer receives the segment-level semantic embedding vectors, the hidden layer captures the temporal features within the vectors, and the output layer predicts the subsequent evolution path of the segment-level semantic embedding vectors based on these temporal features. For example, based on a segment-level semantic embedding vector (corresponding to a segment where cumulonimbus cloud coverage expands and precipitation intensifies), the model predicts the subsequent evolution path as "cumulonimbus cloud coverage continues to expand → precipitation intensity further increases → cumulonimbus cloud coverage reaches its maximum value → precipitation intensity reaches its peak value."

[0049] Step 1441: Input the segment-level semantic embedding vector into the pre-trained video segment temporal inference model, capture the temporal features of the segment-level semantic embedding vector through the temporal convolutional layer, and perform temporal dependency capture processing of the segment-level semantic embedding vector through the long short-term memory layer to generate the temporal dependency features of the segment-level semantic embedding vector.

[0050] In this embodiment, the video retrieval server inputs the segment-level semantic embedding vector generated in step 143 into a pre-trained video segment temporal inference model. First, a temporal convolutional layer is used to convolve the segment-level semantic embedding vector to capture local temporal features in the vector. Then, the result of the convolution operation is input into a long short-term memory layer. The long short-term memory layer captures long-term temporal dependencies in the segment-level semantic embedding vector through a gating mechanism, generating temporal dependency features. For example, the temporal convolutional layer captures the semantic change features of adjacent time steps in the segment-level semantic embedding vector, and the long short-term memory layer captures the long-term dependencies between these change features, generating temporal dependency features containing long-term temporal dependency information.

[0051] Step 1442: Based on the temporal dependency features of fragment-level semantic embedding vectors, extract the temporal feature vectors of multiple consecutive sub-fragment semantic embedding vectors.

[0052] In this embodiment, the video retrieval server extracts temporal feature vectors from multiple consecutive sub-segment semantic embedding vectors based on the temporal dependency features generated in step 1441. First, the segment-level semantic embedding vectors are divided into multiple consecutive sub-segment semantic embedding vectors. Then, for each sub-segment semantic embedding vector, the corresponding temporal feature vector is extracted based on the temporal dependency features. For example, the segment-level semantic embedding vectors are divided into sub-segments A, B, and C, and the temporal feature vectors T_A, T_B, and T_C for sub-segment A, B, and C are extracted respectively based on the temporal dependency features.

[0053] Step 1443: Capture the temporal correlation between the semantic embedding vectors of sub-segments through the sub-segment association layer, and generate the association feature vector of the semantic embedding vector of the sub-segment.

[0054] In this embodiment, the video retrieval server performs association analysis on the temporal feature vectors of the multiple sub-segment semantic embedding vectors extracted in step 1442 through a sub-segment association layer. The sub-segment association layer employs an attention mechanism to calculate the association weights between the temporal feature vectors of each sub-segment semantic embedding vector and the temporal feature vectors of other sub-segments, and generates associated feature vectors based on these association weights. For example, the sub-segment association layer calculates the association weight W_AB between the temporal feature vector T_A of sub-segment A and the temporal feature vector T_B of sub-segment B, and the association weight W_BC between the temporal feature vector T_B of sub-segment B and the temporal feature vector T_C of sub-segment C, and generates associated feature vectors based on these association weights.

[0055] Step 1444: Perform fusion processing on the temporal dependent features and associated feature vectors to generate a temporal fusion feature vector.

[0056] In this embodiment, the video retrieval server fuses the temporal dependency features generated in step 1441 with the associated feature vector generated in step 1443. The fusion process employs a concatenation method, concatenating the temporal dependency features and the associated feature vector according to their dimensions to generate a temporal fusion feature vector. For example, if the temporal dependency features have a dimension of D3 and the associated feature vector has a dimension of D4, the concatenated temporal fusion feature vector will have a dimension of D3+D4.

[0057] Step 1445: Perform post-evolution prediction processing of fragment-level semantic embedding vectors through the evolution prediction layer to generate multiple potential evolution directions of fragment-level semantic embedding vectors within several subsequent temporal units; transform the multiple potential evolution directions into corresponding semantic evolution paths through the path generation layer, with each semantic evolution path containing the predicted semantic embedding vector for each temporal unit; and fuse the multiple semantic evolution paths into a post-evolution path through the path fusion layer based on the evolution confidence of each semantic evolution path.

[0058] In this embodiment, the video retrieval server first performs subsequent evolution prediction processing on the temporal fusion feature vector generated in step 1444 through an evolution prediction layer. The evolution prediction layer adopts a fully connected neural network structure to perform nonlinear transformation on the temporal fusion feature vector, generating multiple potential evolution directions of the segment-level semantic embedding vector within several subsequent temporal units. Then, the path generation layer transforms each potential evolution direction into a corresponding semantic evolution path. The path generation layer determines the predicted semantic embedding vector for each temporal unit based on the potential evolution direction, constructing a complete semantic evolution path. Finally, the path fusion layer calculates the evolution confidence of each semantic evolution path, and performs weighted fusion of multiple semantic evolution paths based on the confidence, generating the final subsequent evolution path. For example, the evolution prediction layer generates potential evolution direction 1, potential evolution direction 2, and potential evolution direction 3. The path generation layer transforms these three directions into semantic evolution path 1, semantic evolution path 2, and semantic evolution path 3, respectively. The path fusion layer calculates the evolution confidence of these three paths as C1, C2, and C3, respectively, and performs weighted fusion of the paths based on the confidence, generating the subsequent evolution path.

[0059] Step 145: Call the pre-trained video segment spatial semantic transformation model, input the segment-level semantic embedding vector into the video segment spatial semantic transformation model to perform segment spatial semantic extraction and transformation processing, and generate spatial semantic features of segment-level semantic embedding vector.

[0060] In this embodiment, the video retrieval server calls a pre-trained video segment spatial semantic transformation model, inputting the segment-level semantic embedding vector generated in step 143 into the model. This model includes a spatial semantic extraction layer and a feature transformation layer. The spatial semantic extraction layer extracts spatially relevant semantic information from the segment-level semantic embedding vector, such as the coverage area and location of cumulonimbus clouds. The feature transformation layer uses a fully connected neural network structure to convert the extracted spatial semantic information into a preset spatial feature representation, generating spatial semantic features. For example, the model extracts from the segment-level semantic embedding vector that the coverage area of ​​cumulonimbus clouds is 50% of the observation area and its location is the center of the observation area, converting this information into spatial semantic features containing coverage indicators, location coordinates, etc.

[0061] Step 1451: Input the fragment-level semantic embedding vector into the pre-trained video fragment spatial semantic transformation model, and capture the local spatial semantic features of each object in the fragment-level semantic embedding vector through the local spatial semantic extraction layer.

[0062] In this embodiment, the video retrieval server inputs the segment-level semantic embedding vector generated in step 143 into a pre-trained video segment spatial semantic transformation model. First, the segment-level semantic embedding vector is processed by a local spatial semantic extraction layer. This layer employs a convolutional neural network structure, performing convolution operations on the semantic information corresponding to each object in the vector to capture the local spatial semantic features of the objects. For example, the segment-level semantic embedding vector contains semantic information about objects such as cumulonimbus clouds, anemometers, and rain gauges. The local spatial semantic extraction layer performs convolution operations on the semantic information of these objects respectively, capturing local spatial semantic features such as the local shape features of cumulonimbus clouds, the local position features of anemometers, and the local reading features of rain gauges.

[0063] Step 1452: Integrate local spatial semantic features into global scene spatial semantic features through the global spatial semantic integration layer, infer the spatial positional relationships between objects in the global scene through the spatial relationship inference layer, generate spatial relationship feature vectors, and perform fusion processing on the global scene spatial semantic features and spatial relationship feature vectors to generate fused spatial semantic feature vectors.

[0064] In this embodiment, the video retrieval server first integrates the local spatial semantic features generated in step 1451 through a global spatial semantic integration layer. The global spatial semantic integration layer uses pooling to globally pool the local spatial semantic features, generating spatial semantic features for the global scene. Then, a spatial relationship reasoning layer infers the spatial positional relationships of objects in the global scene. This layer uses a graph neural network structure, treating each object as a node and constructing connections between nodes based on the object's position information, inferring the spatial positional relationships between objects and generating a spatial relationship feature vector. Finally, the spatial semantic features of the global scene and the spatial relationship feature vector are concatenated and fused to generate a fused spatial semantic feature vector. For example, the global spatial semantic integration layer generates global spatial semantic features including the coverage area of ​​cumulonimbus clouds, the location of anemometers, and the location of rain gauges. The spatial relationship reasoning layer infers spatial positional relationships such as cumulonimbus clouds being above the anemometer and rain gauges being below the cumulonimbus clouds, generating a spatial relationship feature vector. These two features are then concatenated and fused to generate the fused spatial semantic feature vector.

[0065] Step 1453: The fused spatial semantic feature vector is converted into a spatial semantic embedding vector with the same spatial dimension as the query semantic embedding vector through the spatial semantic transformation layer. The temporal association layer of spatial semantic and temporal semantic is used to capture the temporal association between the spatial semantic embedding vector and the fragment-level semantic embedding vector.

[0066] In this embodiment, the video retrieval server first processes the fused spatial semantic feature vector generated in step 1452 through a spatial semantic transformation layer. This layer employs a fully connected neural network structure to convert the fused spatial semantic feature vector into a spatial semantic embedding vector with the same spatial dimension as the query semantic embedding vector. Then, a spatial semantic and temporal semantic association layer performs association analysis between the spatial semantic embedding vector and the fragment-level semantic embedding vector. This layer uses an attention mechanism to calculate the temporal association weights between the spatial semantic embedding vector and the fragment-level semantic embedding vector, capturing the temporal association relationship between them. For example, the spatial semantic transformation layer converts the dimension of the fused spatial semantic feature vector from D5 to D6 (D6 being consistent with the spatial dimension of the query semantic embedding vector), generating a spatial semantic embedding vector. The spatial semantic and temporal semantic association layer calculates the association weights between the spatial semantic embedding vector and the fragment-level semantic embedding vector at different time steps, capturing the temporal association relationship between them.

[0067] Step 1454: Perform fusion processing on the spatial semantic embedding vector and the temporal association to generate fragment spatial semantic features containing spatial scene features, object spatial relationships and temporal association information.

[0068] In this embodiment, the video retrieval server fuses the spatial semantic embedding vector generated in step 1453 with the temporal association relationship. The fusion process employs a concatenation method, concatenating the spatial semantic embedding vector and the feature vector corresponding to the temporal association relationship according to their dimensions to generate fragment spatial semantic features. For example, if the spatial semantic embedding vector has a dimension of D6, the feature vector corresponding to the temporal association relationship has a dimension of D7, and the concatenated fragment spatial semantic feature has a dimension of D6+D7.

[0069] Step 146: Perform semantic similarity association analysis on the query semantic embedding vector and the fragment-level semantic embedding vector, and capture the semantic overlap between the two through the semantic comparison layer of the cross-dimensional semantic retrieval framework.

[0070] In this embodiment, the video retrieval server performs semantic similarity association analysis on the query semantic embedding vector generated in step 137 and the fragment-level semantic embedding vector generated in step 143 through the semantic comparison layer of the cross-dimensional semantic retrieval framework. The semantic comparison layer uses the cosine similarity calculation method to calculate the similarity score between the two vectors. The higher the similarity score, the more semantic overlap there is. At the same time, the semantic components of the two vectors are decomposed through vector decomposition to determine the semantic overlap. For example, if the semantic components of the query semantic embedding vector are A1, A2, and A3, and the semantic components of the fragment-level semantic embedding vector are B1, B2, and B3, the semantic comparison layer calculates a similarity score of S and decomposes the semantic overlap into A1∩B1 and A2∩B2.

[0071] Step 147: Perform a comparison process between the temporal evolution path of the query semantic embedding vector and the subsequent evolution path of the fragment-level semantic embedding vector to determine the temporal evolution comparison result.

[0072] In this embodiment, the video retrieval server performs a comparison process between the temporal evolution path of the query semantic embedding vector generated in step 134 and the subsequent evolution path of the fragment-level semantic embedding vector generated in step 144. The comparison process employs a dynamic time warping method to align the temporal nodes in the two paths, calculate the similarity score between the aligned nodes, and determine the temporal evolution comparison result based on the similarity score. For example, the temporal evolution path of the query semantic includes nodes Q1, Q2, and Q3, and the subsequent evolution path of the fragment-level semantic includes nodes P1, P2, and P3. The dynamic time warping method aligns Q1 with P1, Q2 with P2, and Q3 with P3, calculates the similarity score for each aligned node pair, and generates the temporal evolution comparison result.

[0073] Step 148: Perform a comparison process between the spatial semantic information of the query semantic embedding vector and the spatial semantic features of the fragment-level semantic embedding vector to determine the spatial semantic comparison result.

[0074] In this embodiment, the video retrieval server performs a comparison process between the spatial semantic information of the query semantic embedding vector generated in step 135 and the spatial semantic features of the fragment-level semantic embedding vector generated in step 145. The comparison process uses the Euclidean distance calculation method to calculate the distance between the two spatial semantic representations; the smaller the distance, the more similar their spatial semantics. Simultaneously, key spatial semantic indicators, such as coverage and location, are analyzed to determine the spatial semantic comparison result. For example, if the query semantic spatial information has a coverage of 50% and a centered location, and the fragment-level semantic spatial features have a coverage of 45% and a location near the center, the Euclidean distance between them is calculated as D, and the spatial semantic comparison result is determined to be similar.

[0075] Step 149: Perform weighted fusion processing on the semantic overlap, temporal evolution comparison results, and spatial semantic comparison results to generate a comprehensive similarity score between the fragment-level semantic embedding vector and the query semantic embedding vector.

[0076] In this embodiment, the video retrieval server performs a weighted fusion process on the semantic overlap portion generated in step 146, the temporal evolution comparison result generated in step 147, and the spatial semantic comparison result generated in step 148. First, each result is normalized, converting it to a value between 0 and 1. Then, a weight is assigned to each result, determined based on its importance. Finally, a weighted sum is calculated to generate a comprehensive similarity score. For example, the normalized value of the semantic overlap portion is S1 with a weight of W1, the normalized value of the temporal evolution comparison result is S2 with a weight of W2, and the normalized value of the spatial semantic comparison result is S3 with a weight of W3. .

[0077] Step 1410: Mark the time segments with a comprehensive similarity score that exceeds the preset score as candidate segments, and integrate all candidate segments into a video segment candidate set.

[0078] In this embodiment, the video retrieval server sets a preset score, compares the comprehensive similarity score generated in step 149 with the preset score, and marks time segments with scores exceeding the preset score as candidate segments; then, all candidate segments are integrated into a video segment candidate set. For example, if the preset score is T, and the comprehensive similarity score of a time segment is S>T, the video retrieval server marks that segment as a candidate segment and integrates all candidate segments to generate a video segment candidate set.

[0079] Step 150: Perform temporal coherence verification on the video segment candidate set, and combine the temporal correlation of the semantic embedding vectors of each video segment in the video segment candidate set to output the video segment that matches the query semantic embedding vector.

[0080] In the embodiment of the present application, the video retrieval server first performs temporal coherence verification on the candidate set of video segments generated in step 1410, verifying the temporal continuity and semantic coherence of each candidate segment; then, calculates the temporal correlation degree of the semantic embedding vectors of each video segment in the candidate set, and the temporal correlation degree is calculated using the cosine similarity calculation method; finally, according to the temporal coherence verification result and the temporal correlation degree, outputs the video segments that match the query semantic embedding vector. For example, the temporal coherence between segment A and segment B in the candidate set is good, and the temporal correlation degree of their semantic embedding vectors is high, and the video retrieval server outputs after merging segment A and segment B.

[0081] Step 151: Extract timestamps for the candidate segments in the candidate set of video segments, obtain the start timestamp and end timestamp of each candidate segment, sort the candidate segments in the order of the start timestamp, perform a time interval check on adjacent candidate segments after sorting, identify adjacent segments with a time interval less than a preset value, perform semantic cohesion analysis on adjacent candidate segments with a time interval less than the preset value, and determine whether the semantics between adjacent candidate segments are coherent through the cohesion evaluation layer of the cohesion evaluation model, and perform a merging process on the adjacent candidate segments with coherent semantics to generate a merged video segment.

[0082] In the embodiment of the present application, the video retrieval server first extracts timestamps for each candidate segment in the candidate set of video segments to obtain the start timestamp and end timestamp; then, sorts the candidate segments in the order of the start timestamp; next, performs a time interval check on adjacent candidate segments after sorting, calculates the difference between the end timestamp and the start timestamp of adjacent segments, and when the difference is less than the preset value, identifies them as adjacent segments; then performs semantic cohesion analysis on the above adjacent segments, and the semantic cohesion analysis uses the cosine similarity calculation method to calculate the similarity score of the semantic embedding vectors of adjacent segments; finally, determines whether the semantics are coherent through the cohesion evaluation layer of the cohesion evaluation model, and performs a merging process on the coherent segments. For example, the end timestamp of candidate segment A is t1, the start timestamp of candidate segment B is t2, t2 - t1 < T (preset value), calculates the similarity score S of the semantic embedding vectors of segment A and segment B, and the cohesion evaluation layer determines that S exceeds the preset threshold, and merges the two to generate a merged video segment.

[0083] Step 152: Call the pre-trained merged segment temporal evolution model, input the segment-level semantic embedding vector of the merged video segment into the merged segment temporal evolution model to perform temporal evolution coherence check, and generate the temporal evolution coherence score of the merged segment.

[0084] In this embodiment, the video retrieval server invokes a pre-trained merged segment temporal inference model, inputting the segment-level semantic embedding vectors of the merged video segments generated in step 151 into the model. This model includes a temporal segmentation layer, a semantic evolution layer, and a coherence evaluation layer. The temporal segmentation layer divides the merged segment into multiple sub-segments, the semantic evolution layer captures the semantic evolution relationships between sub-segments, and the coherence evaluation layer calculates the temporal evolution coherence score based on these semantic evolution relationships. For example, if the merged segment is divided into sub-segment 1, sub-segment 2, and sub-segment 3, the semantic evolution layer captures the semantic evolution relationships from sub-segment 1 to sub-segment 2 and from sub-segment 2 to sub-segment 3, and the coherence evaluation layer calculates the coherence score of these relationships to generate the temporal evolution coherence score of the merged segment.

[0085] Step 1521: Input the segment-level semantic embedding vector of the merged video segment into the pre-trained merged segment temporal inference model, and divide the merged segment into several continuous sub-segments through the temporal segmentation layer.

[0086] In this embodiment, the video retrieval server inputs the segment-level semantic embedding vector of the merged video segment generated in step 151 into a pre-trained merged segment temporal inference model. First, the merged segment is divided by a temporal segmentation layer. The temporal segmentation layer uses a sliding window method, setting a fixed-size sliding window to divide the merged segment into several consecutive sub-segments. For example, the sliding window size is W, the length of the merged segment is L, and it is divided into L / W consecutive sub-segments.

[0087] Step 1522: Perform semantic embedding vector extraction on each sub-segment to generate the semantic embedding vector of the sub-segment.

[0088] In this embodiment, the video retrieval server performs semantic embedding vector extraction on each sub-segment divided in step 1521. The extraction method uses a pre-trained semantic embedding model, inputting the video data of the sub-segment into the model to generate the corresponding semantic embedding vector. For example, inputting the video data of sub-segment 1 into the model generates semantic embedding vector V1; inputting the video data of sub-segment 2 into the model generates semantic embedding vector V2.

[0089] Step 1523: Capture the semantic evolution relationship between adjacent sub-segments through the temporal evolution layer, and generate the semantic evolution path of adjacent sub-segments.

[0090] In this embodiment, the video retrieval server processes the semantic embedding vectors of adjacent sub-segments generated in step 1522 through a temporal evolution layer. The temporal evolution layer employs a long short-term memory network structure to capture the semantic evolution relationship between adjacent sub-segments and generate a semantic evolution path. For example, if the semantic embedding vector of sub-segment 1 is V1 and the semantic embedding vector of sub-segment 2 is V2, the temporal evolution layer captures the semantic evolution relationship from V1 to V2 and generates a semantic evolution path containing information such as the evolution direction and degree.

[0091] Step 1524: Perform evolution gradient calculation on the semantic evolution path of each adjacent sub-segment to generate an evolution gradient vector, perform semantic overlap analysis on adjacent sub-segments to calculate the semantic overlap ratio of adjacent sub-segments, perform smoothness evaluation on the semantic evolution path to calculate the rate of change of the semantic embedding vector in the semantic evolution path, and perform normalized weighted fusion processing on the evolution gradient vector, semantic overlap ratio, and rate of change to generate a comprehensive coherence score for adjacent sub-segments.

[0092] In this embodiment, the video retrieval server first performs evolution gradient calculation on the semantic evolution path of each adjacent sub-segment generated in step 1523. The evolution gradient calculation uses a difference method to calculate the semantic change of adjacent nodes in the path and generate an evolution gradient vector. Then, it performs semantic overlap analysis on the adjacent sub-segments. The semantic overlap analysis uses an intersection calculation method to calculate the intersection ratio of the semantic embedding vectors of the two sub-segments and generate a semantic overlap ratio. Next, it performs smoothness evaluation on the semantic evolution path. The smoothness evaluation uses a variance calculation method to calculate the variance of the rate of change of the semantic embedding vectors in the path and generate a rate of change. Finally, it normalizes the evolution gradient vector, semantic overlap ratio, and rate of change, assigns weights, and performs weighted fusion to generate a comprehensive coherence score for adjacent sub-segments. For example, the normalized value of the evolution gradient vector is G, the semantic overlap ratio is O, the normalized value of the rate of change is R, and the weights are Wg, Wo, and Wr, respectively. .

[0093] Step 1525: Perform averaging on the overall coherence scores between all adjacent sub-segments to generate the temporal evolution coherence score of the merged segment.

[0094] In this embodiment, the video retrieval server performs an averaging process on the overall coherence scores between all adjacent sub-segments generated in step 1524, calculates the average score, and generates the temporal evolution coherence score of the merged segment. For example, if the overall coherence scores between adjacent sub-segments are S1, S2, and S3, the average score = (S1 + S2 + S3) / 3, and this average score is the temporal evolution coherence score of the merged segment.

[0095] Step 153: Call the pre-trained merged segment spatial transformation model, input the segment-level semantic embedding vector of the merged video segment into the merged segment spatial transformation model to perform a spatial semantic transformation rationality check on the merged segment, and generate a spatial transformation rationality score for the merged segment.

[0096] In this embodiment, the video retrieval server invokes a pre-trained merged segment spatial transformation model, inputting the segment-level semantic embedding vectors of the merged video segments generated in step 151 into the model. This model includes a spatial segmentation layer, a spatial transformation layer, and a rationality evaluation layer. The spatial segmentation layer divides the merged segment into multiple spatial scene segments. The spatial transformation layer captures the spatial transformation relationships between scene segments. The rationality evaluation layer calculates a spatial transformation rationality score based on these relationships. For example, if the merged segment is divided into spatial scene segment 1, spatial scene segment 2, and spatial scene segment 3, the spatial transformation layer captures the spatial transformation relationships from scene segment 1 to scene segment 2 and from scene segment 2 to scene segment 3. The rationality evaluation layer calculates the rationality scores for these relationships, generating a spatial transformation rationality score for the merged segment.

[0097] Step 1531: Input the segment-level semantic embedding vector of the merged video segment into the pre-trained merged segment spatial transformation model, and divide the merged segment into several continuous spatial scene segments through the spatial segmentation layer.

[0098] In this embodiment, the video retrieval server inputs the segment-level semantic embedding vectors of the merged video segments generated in step 151 into a pre-trained merged segment spatial transformation model. First, the merged segments are divided using a spatial segmentation layer. This layer employs a clustering method, clustering the merged segments based on the spatial semantic information in the segment-level semantic embedding vectors, thus dividing them into several continuous spatial scene segments. For example, based on the coverage and location information of cumulonimbus clouds, the merged segments are divided into spatial scene segment A, spatial scene segment B, and spatial scene segment C.

[0099] Step 1532: Perform spatial semantic feature extraction on each spatial scene fragment to generate spatial semantic features of the spatial scene fragment.

[0100] In this embodiment, the video retrieval server performs spatial semantic feature extraction on each spatial scene segment divided in step 1531. The extraction method uses a pre-trained spatial semantic extraction model, inputting the video data of the spatial scene segment into the model to generate corresponding spatial semantic features. For example, inputting the video data of spatial scene segment A into the model generates spatial semantic features containing information such as the coverage and location of cumulonimbus clouds.

[0101] Step 1533: Capture the spatial transformation relationship between adjacent spatial scene fragments through the spatial transformation layer to generate the spatial transformation path of the spatial scene fragments, perform spatial transition naturalness evaluation on adjacent spatial scene fragments to obtain the semantic matching degree of the transition region, and perform semantic correlation calculation on the spatial transformation path to obtain the semantic correlation degree of the scene before and after the transformation.

[0102] In this embodiment, the video retrieval server first processes the spatial semantic features of adjacent spatial scene segments generated in step 1532 through a spatial transformation layer. The spatial transformation layer adopts a graph neural network structure, treating each spatial scene segment as a node, constructing connections between nodes based on spatial location relationships, capturing spatial transformation relationships between adjacent spatial scene segments, and generating spatial transformation paths. Then, it performs a spatial transition naturalness assessment on adjacent spatial scene segments, using a cosine similarity calculation method to calculate the similarity score of spatial semantic features in the transition region, obtaining the semantic matching degree. Next, it performs semantic association calculation on the spatial transformation paths, using a path analysis method to analyze the semantic association between scenes before and after the transformation, obtaining the degree of semantic association. For example, if the spatial transformation layer generates a spatial transformation path as scene A → scene B → scene C, the transition naturalness assessment yields a semantic matching degree of M1 between scene A and scene B, and a semantic matching degree of M2 between scene B and scene C. The semantic association calculation yields a semantic association degree of R1 between scene A and scene B, and a semantic association degree of R2 between scene B and scene C.

[0103] Step 1534: Determine the comprehensive rationality score between spatial scene segments based on spatial transformation path, semantic matching degree, and semantic association degree.

[0104] In this embodiment, the video retrieval server determines the comprehensive reasonableness score between spatial scene segments based on the spatial transformation path, semantic matching degree, and semantic association degree generated in step 1533. First, the reasonableness of the spatial transformation path is evaluated using an analysis method based on path length and the number of nodes. Then, a weighted fusion method is used to generate the comprehensive reasonableness score by combining the semantic matching degree and semantic association degree. For example, the reasonableness evaluation value of the spatial transformation path is P, the semantic matching degree is M, the semantic association degree is R, and the weights are Wp, Wm, and Wr, respectively. .

[0105] Step 1535: Perform an average processing on the overall reasonableness scores among all spatial scene segments to generate a reasonableness score for the spatial transformation of the merged segments.

[0106] In this embodiment of the application, the video retrieval server performs an average processing on the comprehensive reasonableness scores among all spatial scene segments generated in step 1534, calculates the average score, and generates a spatial transformation reasonableness score for the merged segments. For example, if the comprehensive reasonableness scores among the spatial scene segments are S1, S2, and S3, the average score = (S1 + S2 + S3) / 3, and this average score is the spatial transformation reasonableness score for the merged segments.

[0107] Step 154: Perform weighted fusion processing on the temporal evolution coherence score and the spatial transformation rationality score to generate a comprehensive coherence score for the merged segments. Merged segments with a comprehensive coherence score exceeding a preset threshold are marked as coherent segments.

[0108] In this embodiment, the video retrieval server first performs a weighted fusion process on the temporal evolution coherence score generated in step 152 and the spatial transformation rationality score generated in step 153. The weighted fusion adopts a linear weighting method, and after allocating weights, a weighted sum is calculated to generate a comprehensive coherence score. Then, a preset threshold is set, and the comprehensive coherence score is compared with the preset threshold. Merged segments with scores exceeding the preset threshold are marked as coherent segments. For example, the temporal evolution coherence score is Tc, the spatial transformation rationality score is Sc, and the weights are Wt and Ws, respectively. The preset threshold is Th. If the overall coherence score is greater than Th, it is marked as a coherent segment.

[0109] Step 155: Extract features that match the query semantics from coherent segments through the semantic coverage layer of the semantic coverage model and perform semantic integrity analysis to obtain complete semantic segments. Mark the complete semantic segments as valid segments and use them as video segments that match the query semantic embedding vector.

[0110] In this embodiment, the video retrieval server processes the coherent segments marked in step 154 ​​using the semantic coverage layer of the semantic coverage model. First, features matching the query semantics are extracted from the coherent segments using a feature matching method. Then, semantic integrity analysis is performed to determine whether the coherent segments cover all core concepts of the query semantics. Finally, a complete semantic segment is obtained based on the analysis results and marked as a valid segment. For example, if the features extracted from the coherent segment that match the query semantics are "expanded cumulonimbus cloud coverage" and "increased precipitation," and the semantic integrity analysis finds that it covers all core concepts of the query semantics, then the coherent segment is determined to be a complete semantic segment.

[0111] Step 1551: Perform semantic alignment processing on the fragment-level semantic embedding vector of the coherent segment and the query semantic embedding vector. Extract the surface features of the coherent segment that directly match the query semantic embedding vector through the feature mapping layer of the feature mapping model, and establish the surface semantic matching relationship.

[0112] In this embodiment, the video retrieval server performs semantic alignment processing on the segment-level semantic embedding vectors of the coherent segments marked in step 154 ​​and the query semantic embedding vectors generated in step 137. First, a vector alignment algorithm is used to align the dimensions and semantic spaces of the two vectors. Then, the feature mapping layer of the feature mapping model extracts surface features that directly match the query semantic embedding vectors in the coherent segments. These surface features include obvious semantic components, such as "expanded cumulonimbus cloud coverage" and "increased precipitation." Finally, a surface semantic matching relationship is established, recording the matched features and their positions. For example, after semantic alignment processing, the feature mapping layer extracts the feature "expanded cumulonimbus cloud coverage" from the coherent segments and matches it with the corresponding feature in the query semantics, establishing a surface semantic matching relationship.

[0113] Step 1552: Input the semantic sequence of the coherent segment into the pre-trained latent semantic evolution model, combine it with the latent evolution logic of the query semantic embedding vector, analyze whether the semantic advancement direction of the coherent segment matches the evolution path of the query semantic embedding vector, and obtain the matching weight in semantic development trend.

[0114] In this embodiment, the video retrieval server inputs the semantic sequence of coherent segments into a pre-trained latent semantic evolution model, and simultaneously inputs the latent evolution logic of the query semantic embedding vector. This model employs a recurrent neural network structure to analyze the degree of matching between the semantic progression direction of the coherent segments and the query semantic evolution path, generating a matching weight. For example, if the semantic progression direction of the coherent segments is "cumulonimbus cloud coverage expands → precipitation intensifies → cumulonimbus cloud coverage reaches its maximum value," and the evolution path of the query semantics is "cumulonimbus cloud coverage expands → precipitation intensifies → cumulonimbus cloud coverage reaches its maximum value → precipitation intensity reaches its peak," the model analyzes the degree of matching between the two and obtains a matching weight of W.

[0115] Step 1553: Perform semantic back-inference from the current semantic state of the coherent segment through the back-inference layer of the back-inference model, determine the pre-associative semantics and post-associative semantics of the query semantic embedding vector implied by the current semantic state, and generate a complete back-inference semantic chain.

[0116] In this embodiment, the video retrieval server performs semantic back-inference on the current semantic state of a coherent segment through the back-inference layer of the back-inference model. The back-inference layer uses a reverse temporal analysis method to infer the preceding and subsequent related semantics from the current semantic state, generating a back-inference semantic chain. For example, if the current semantic state is "cumulonimbus cloud coverage expands, precipitation intensifies", the back-inference layer infers the preceding related semantic as "cumulonimbus clouds begin to form, precipitation occurs", and the subsequent related semantic as "cumulonimbus cloud coverage reaches its maximum value, precipitation intensity reaches its peak", generating a complete back-inference semantic chain.

[0117] Step 1554: Perform multi-level semantic connection analysis on the surface semantics, deep semantics, and implicit semantics of the coherent fragments and their corresponding levels with the query semantic embedding vector. If there are uncovered semantic levels, prioritize the missing semantic levels.

[0118] In this embodiment, the video retrieval server performs multi-level semantic connection analysis on the surface semantics, deep semantics, and implicit semantics of coherent segments and their corresponding levels in the query semantic embedding vector. First, it extracts the surface semantics (directly visible semantics), deep semantics (implicit semantic relationships), and implicit semantics (potential semantic trends) of the coherent segments. Then, it performs matching analysis with the corresponding levels in the query semantic embedding vector. Finally, if there are uncovered semantic levels, they are prioritized according to their semantic importance. For example, if the surface semantics of the coherent segments cover the surface level of the query semantics, the deep semantics do not cover the deep level of the query semantics, and the implicit semantics cover the implicit level of the query semantics, the missing deep semantic levels are prioritized.

[0119] Step 1555: Call the pre-trained semantic completion model to select video sub-segments from the video memory feature sequence that match the missing semantic level and conform to the semantic evolution logic of coherent segments.

[0120] In this embodiment, the video retrieval server invokes a pre-trained semantic completion model to filter video segments from the video memory feature sequence generated in step 120. First, filtering conditions are set based on the missing semantic level determined in step 1554; then, combined with the semantic evolution logic of coherent segments, video segments that meet the conditions are filtered. For example, if the missing semantic level is "precipitation intensity reaches its peak", the filtering condition is video segments that contain this semantic level and conform to the evolution logic of "cumulonimbus cloud coverage expands → precipitation intensifies → precipitation intensity reaches its peak".

[0121] Step 1556: Through the temporal integration layer of the pre-trained semantic completion model, the selected video sub-segments are temporally aligned and semantically connected with the coherent segments to obtain complete semantic segments.

[0122] In this embodiment, the video retrieval server processes the video sub-segments selected in step 1555 and the coherent segments marked in step 154 ​​using the temporal integration layer of a pre-trained semantic completion model. First, temporal alignment is performed, concatenating the video sub-segments and coherent segments temporally based on their timestamps. Then, semantic concatenation is performed, using a semantic fusion method to connect the semantics of the sub-segments with the semantics of the coherent segments. Finally, a complete semantic segment is obtained. For example, if the timestamps of the video sub-segments are t3-t4 and the timestamps of the coherent segments are t1-t2, the temporal alignment concatenates the sub-segments after the coherent segments, and the semantic concatenation fuses the semantics of the sub-segments with the semantics of the coherent segments to obtain a complete semantic segment.

[0123] In one embodiment, the method further includes:

[0124] Step 210: Perform semantic update processing on the video memory feature sequence through the update layer of the semantic update model to identify outdated semantic segments in the video memory feature sequence.

[0125] In this embodiment, the video retrieval server performs semantic update processing on the video memory feature sequence generated in step 120 through the update layer of the semantic update model. The update layer uses a semantic timeliness analysis method to analyze the semantic timeliness of each video memory feature and identify outdated semantic segments. For example, if the semantics of one segment in the video memory feature sequence is "precipitation level classification under the old meteorological observation standard", the update layer identifies this segment as an outdated semantic segment.

[0126] Step 220: Re-execute visual content parsing on the video frames corresponding to the outdated semantic segments, and map the visual content of the video frames into new semantic embedding vectors through a pre-trained visual language big model.

[0127] In this embodiment, the video retrieval server re-inputs the video frames corresponding to the outdated semantic segments identified in step 210 into the pre-trained visual language large model to perform visual content parsing. Following the processing flow in step 110, the visual language large model extracts the visual features of the video frames, converts them into textual semantic descriptions, and maps them to new semantic embedding vectors. For example, if the video frame corresponding to the outdated semantic segment displays precipitation levels under the old standard, re-parsing it will generate a new semantic embedding vector containing precipitation levels under the new standard.

[0128] Step 230: Replace the outdated semantic segments in the video memory feature sequence with the new semantic embedding vector.

[0129] In this embodiment, the video retrieval server replaces the outdated semantic segments identified in step 210 with the new semantic embedding vector generated in step 220, thus updating the video memory feature sequence. For example, the new semantic embedding vector V_new replaces the outdated semantic segment V_old in the video memory feature sequence.

[0130] Step 240: Call the pre-trained video memory temporal inference model to perform semantic evolution prediction processing on the updated video memory feature sequence and generate the subsequent evolution path of the video memory features.

[0131] In this embodiment, the video retrieval server invokes a pre-trained video memory temporal inference model to perform semantic evolution prediction processing on the updated video memory feature sequence in step 230. This model employs a long short-term memory network structure to capture the temporal features of the updated video memory feature sequence and predict the subsequent evolution path. For example, the model predicts the subsequent evolution path of the updated video memory feature sequence as "precipitation level maintains the new standard → cumulonimbus cloud coverage shrinks → precipitation intensity weakens".

[0132] Step 250: Perform temporal expansion processing on the video memory feature sequence according to the subsequent evolution path, and re-execute the similarity matching step on the temporally expanded video memory feature sequence to generate a new set of video segment candidates.

[0133] In this embodiment, the video retrieval server performs temporal expansion processing on the updated video memory feature sequence according to the subsequent evolution path generated in step 240, extending the length of the video memory feature sequence; then, it re-executes the similarity matching step in step 140 to generate a new set of video segment candidates. For example, by expanding the video memory feature sequence according to the subsequent evolution path, adding several new video memory features, and re-executing similarity matching to generate a new candidate set.

[0134] Step 260: Generate new search results based on the new video segment candidate set, compare the semantic similarity of the new search results with the original search results, and if the semantic similarity of the new search results is greater than that of the original search results, replace the original search results with the new search results, and present the updated search results as the final results.

[0135] In this embodiment, the video retrieval server generates new retrieval results based on the new video segment candidate set generated in step 250. Then, it calculates the semantic similarity between the new retrieval results and the original retrieval results using cosine similarity. Finally, if the new retrieval results have a higher semantic similarity, the original retrieval results are replaced, and the updated results are presented. For example, if the semantic similarity of the new retrieval results is S_new and the semantic similarity of the original retrieval results is S_old, and S_new > S_old, the original retrieval results are replaced.

[0136] In one embodiment, the method further includes:

[0137] Step 310: Perform multiple rounds of semantic expansion processing on the query text through the extension layer of the semantic expansion model to generate multiple related expanded query texts.

[0138] In this embodiment, the video retrieval server performs multiple rounds of semantic expansion processing on the user-input query text through the expansion layer of the semantic expansion model. The expansion layer uses methods such as synonym replacement and semantic association to generate multiple related expanded query texts. For example, if the query text is "a segment showing the expansion of cumulonimbus cloud coverage and increased precipitation during typhoon weather," the expanded query texts generated might be "a segment showing the expansion of cumulonimbus cloud coverage and increased precipitation during typhoons," or "a segment showing the expansion of cumulonimbus clouds and intensified precipitation during tropical cyclones," etc.

[0139] Step 320: Perform semantic parsing on each extended query text to generate the corresponding extended query semantic embedding vector.

[0140] In this embodiment, the video retrieval server performs semantic parsing on each extended query text generated in step 310. The parsing process is the same as in step 130, generating a corresponding extended query semantic embedding vector. For example, semantic parsing is performed on the extended query text "fragments of increased precipitation due to expanded cumulonimbus cloud range during typhoons" to generate an extended query semantic embedding vector E1.

[0141] Step 330: Input the extended query semantic embedding vector into the pre-trained query semantic spatial transformation model to perform spatial semantic transformation processing of extended query semantics, and generate a spatial extended query semantic embedding vector containing spatial semantic information.

[0142] In this embodiment, the video retrieval server inputs the extended query semantic embedding vector generated in step 320 into a pre-trained query semantic spatial transformation model to perform spatial semantic transformation processing. The processing flow is the same as in step 135, generating a spatial extended query semantic embedding vector. For example, inputting the extended query semantic embedding vector E1 into the model generates a spatial extended query semantic embedding vector SE1.

[0143] Step 340: Perform a fusion process on all spatially expanded query semantic embedding vectors and the original query semantic embedding vectors to generate enhanced query semantic embedding vectors.

[0144] In this embodiment, the video retrieval server performs a fusion process on all spatially expanded query semantic embedding vectors generated in step 330 and the original query semantic embedding vectors generated in step 137. The fusion process uses a weighted average method to generate enhanced query semantic embedding vectors. For example, the spatially expanded query semantic embedding vectors are SE1 and SE2, and the original query semantic embedding vector is Q, with weights We1, We2, and Wq, respectively. .

[0145] Step 350: Using the enhanced query semantic embedding vector as the retrieval basis, perform a similarity matching step in the video memory feature sequence to generate an enhanced video segment candidate set.

[0146] In this embodiment, the video retrieval server uses the enhanced query semantic embedding vector generated in step 340 as the retrieval basis, and performs the similarity matching step in step 140 on the video memory feature sequence generated in step 120 to generate an enhanced video segment candidate set. For example, the enhanced query semantic embedding vector is matched with multiple segments in the video memory feature sequence to generate an enhanced candidate set.

[0147] Step 360: Generate enhanced search results based on the enhanced video segment candidate set; perform semantic similarity comparison between the enhanced search results and the original search results. If the similarity of the enhanced search results is greater than that of the original search results, then replace the original search results with the enhanced search results and finally output the search results.

[0148] In this embodiment, the video retrieval server generates enhanced retrieval results based on the enhanced video segment candidate set generated in step 350. Then, it calculates the semantic similarity between the enhanced retrieval results and the original retrieval results using cosine similarity. Finally, if the enhanced retrieval results have a higher semantic similarity, the original retrieval results are replaced and output. For example, if the semantic similarity of the enhanced retrieval results is S_enhanced and the semantic similarity of the original retrieval results is S_original, and S_enhanced > S_original, the original retrieval results are replaced.

[0149] This application embodiment uses a pre-trained visual language model to map the visual content of continuous frames of input video to semantic embedding vectors. Combined with the video memory encoding module, the dynamic encoding of the semantic embedding vectors of continuous frames generates a video memory feature sequence containing temporal correlation information. At the same time, relying on the visual language model, the query text is transformed into a query semantic embedding vector of the same spatial dimension. Using the query semantic embedding vector as the retrieval basis, similarity matching is performed in the video memory feature sequence, and a candidate set of video segments is determined through feature correlation analysis within the temporal window. Finally, the temporal coherence of the candidate set is verified, and the matching video segment is output by combining the temporal correlation of the semantic embedding vector.

[0150] Overall, this application's embodiments construct a complete technical chain from semantic representation of video visual content and temporal correlation feature encoding to cross-modal semantic matching and segment verification. This effectively breaks down the modal barriers between video visual features and text semantic features, achieving precise semantic alignment between video content and text queries. Through the dynamic encoding of semantic embedding vectors for consecutive frames using a video memory encoding module, the temporal correlation information of video content is fully captured, avoiding the isolation of semantic representation in a single frame. The design of query semantic embedding vectors in the same spatial dimension ensures the effectiveness and accuracy of similarity matching. The combination of feature correlation analysis within the temporal window and temporal coherence verification improves the completeness and coherence of video segment retrieval, solving problems such as inaccurate semantic matching, insufficient utilization of temporal information, and poor segment coherence in traditional video retrieval. Overall, this improves the efficiency and accuracy of video retrieval, enabling efficient retrieval and utilization of video content.

[0151] Based on the same inventive concept, embodiments of this application also provide a video retrieval server. See also... Figure 2 The diagram shown is a possible structural schematic of a video retrieval server provided in an embodiment of this application. Figure 2 In this context, the video retrieval server 200 includes a processor 210 and a memory 220. The memory 220 stores computer programs executable by the processor 210. By executing the instructions stored in the memory 220, the processor 210 can perform the steps of the video retrieval method based on semantic embedding and video memory coding described above.

[0152] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium including a computer program. When the computer program is run on a video retrieval server, it causes the video retrieval server to perform the steps of the video retrieval method based on semantic embedding and video memory coding described above. In some possible implementations, various aspects of the video retrieval method based on semantic embedding and video memory coding provided in this application can also be implemented as a program product including a computer program. When the program product is run on a video retrieval server, the computer program causes the video retrieval server to perform the steps of the video retrieval method based on semantic embedding and video memory coding described above. For example, the video retrieval server can perform actions such as... Figure 1 The steps are shown in the figure.

[0153] The above content is only a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application.

Claims

1. A video retrieval method based on semantic embedding and video memory coding, characterized in that, The method includes: Visual content parsing is performed on consecutive frames of the input video using a pre-trained visual language large model, mapping the visual content of each frame of the input video into a semantic embedding vector. Based on the temporal sequence features of the semantic embedding vector, a video memory coding module is constructed to dynamically encode the semantic embedding vectors of consecutive frames in temporal order, generating a video memory feature sequence containing temporal correlation information. Based on the pre-trained visual language model, semantic parsing is performed on the query text, transforming the query text into a query semantic embedding vector; wherein, the query semantic embedding vector and the semantic embedding vector are in the same spatial dimension; Using the query semantic embedding vector as the retrieval basis, similarity matching is performed in the video memory feature sequence, and a candidate set of video segments associated with the query semantic embedding vector is determined through feature association analysis within a time window; wherein, the feature association analysis within a time window includes: setting the time window size, dividing the video memory feature sequence into multiple time windows, and analyzing the association between features within each time window; Temporal coherence verification is performed on the candidate video segments, and the temporal correlation of the semantic embedding vectors of each video segment in the candidate video segments is combined to output the video segment that matches the query semantic embedding vector; wherein, the temporal coherence verification includes verifying the temporal continuity and semantic coherence of each candidate segment. The pre-trained visual language model performs semantic parsing on the query text, transforming the query text into a query semantic embedding vector, including: The query text is segmented into several independent semantic units, each of which corresponds to a core concept in the query text. Each semantic unit is input into a pre-trained basic semantic embedding model to generate a basic semantic embedding vector for each semantic unit; The pre-trained context semantic association model is invoked, and the basic semantic embedding vector is processed according to the original word order of the query text. The direct and indirect contextual relationships between the semantic units are captured through the semantic dependency layer of the context semantic association model, and an enhanced semantic embedding vector containing multi-level contextual dependency information is generated. The pre-trained query semantic temporal inference model is invoked, and the enhanced semantic embedding vector is input into the query semantic temporal inference model to perform temporal evolution prediction processing of the query semantic embedding vector, thereby generating the potential evolution path of the query semantic in the video temporal dimension. Invoke the pre-trained query semantic space mapping model, input the enhanced semantic embedding vector into the query semantic space mapping model to perform spatial semantic transformation processing of the query semantic embedding vector, map the query semantics to a dimension consistent with the spatial semantics of the continuous frames, and generate a spatial enhanced semantic embedding vector containing spatial semantic information; The pre-trained temporal-spatial fusion model is invoked to perform structural alignment and fusion processing on the spatially enhanced semantic embedding vector and the potential evolution path, thereby generating a composite semantic embedding vector that integrates temporal and spatial information. The composite semantic embedding vector is input into the pre-trained visual language big model, which is the same as the continuous frame processing, to perform semantic space mapping processing. The dimension of the query semantic embedding vector is automatically adjusted by the dimension adaptation layer of the pre-trained visual language big model so that the dimension of the query semantic embedding vector is consistent with that of the semantic embedding vector.

2. The method as described in claim 1, characterized in that, The step of using the query semantic embedding vector as the retrieval basis, performing similarity matching in the video memory feature sequence, and determining a candidate set of video segments associated with the query semantic embedding vector through feature association analysis within a time window includes: Using the query semantic embedding vector as the core retrieval basis, a cross-dimensional semantic retrieval framework integrating temporal inference and spatial semantic transformation is constructed. The video memory feature sequence is subjected to temporal segmentation processing. The temporal segmentation layer of the cross-dimensional semantic retrieval framework identifies semantic mutation points in the video memory feature sequence, and divides the continuous video memory features into several temporal segments with complete semantics. Semantic aggregation processing is performed on the video memory features within each time segment, and the feature fusion layer of the cross-dimensional semantic retrieval framework integrates multiple video memory features within the time segment into a segment-level semantic embedding vector. The pre-trained video segment temporal inference model is invoked, and the segment-level semantic embedding vector is input into the video segment temporal inference model to perform temporal evolution prediction processing of the segment-level semantic embedding vector, thereby generating the subsequent evolution path of the segment-level semantic embedding vector. A pre-trained video segment spatial semantic transformation model is invoked, and the segment-level semantic embedding vector is input into the video segment spatial semantic transformation model to perform the extraction and transformation of the segment spatial semantics, thereby generating the spatial semantic features of the segment-level semantic embedding vector; Semantic similarity association analysis is performed on the query semantic embedding vector and the fragment-level semantic embedding vector, and the semantic overlap between the two is captured through the semantic comparison layer of the cross-dimensional semantic retrieval framework; The temporal evolution path of the query semantic embedding vector is compared with the subsequent evolution path of the fragment-level semantic embedding vector to determine the temporal evolution comparison result. The spatial semantic information of the query semantic embedding vector is compared with the spatial semantic features of the fragment-level semantic embedding vector to determine the spatial semantic comparison result. The semantic overlap, the temporal evolution comparison result, and the spatial semantic comparison result are subjected to weighted fusion processing to generate a comprehensive similarity score between the fragment-level semantic embedding vector and the query semantic embedding vector. The time segments whose comprehensive similarity scores exceed the preset scores are marked as candidate segments, and all the candidate segments are integrated into the video segment candidate set.

3. The method as described in claim 2, characterized in that, The step of calling the pre-trained video segment temporal inference model involves inputting the segment-level semantic embedding vector into the model to perform temporal evolution prediction processing of the segment-level semantic embedding vector, generating the subsequent evolution path of the segment-level semantic embedding vector, including: The segment-level semantic embedding vector is input into the pre-trained video segment temporal inference model. The temporal features of the segment-level semantic embedding vector are captured by the temporal convolutional layer, and the temporal dependency capture processing of the segment-level semantic embedding vector is performed by the long short-term memory layer to generate the temporal dependency features of the segment-level semantic embedding vector. Based on the temporal dependency features of the segment-level semantic embedding vectors, temporal feature vectors of multiple consecutive sub-segment semantic embedding vectors are extracted. The temporal correlation between the semantic embedding vectors of the sub-segments is captured by the sub-segment association layer, and the association feature vector of the semantic embedding vector of the sub-segment is generated. The temporal dependency features and the associated feature vector are fused together to generate a temporal fusion feature vector. The evolution prediction layer performs subsequent evolution prediction processing on the fragment-level semantic embedding vector to generate multiple potential evolution directions of the fragment-level semantic embedding vector within several subsequent time units; the path generation layer transforms the multiple potential evolution directions into corresponding semantic evolution paths, each of which contains the predicted semantic embedding vector for each time unit; based on the evolution confidence of each semantic evolution path, the path fusion layer merges the multiple semantic evolution paths into the subsequent evolution path.

4. The method as described in claim 2, characterized in that, The process involves calling a pre-trained video segment spatial semantic transformation model, inputting the segment-level semantic embedding vector into the model to perform extraction and transformation of the segment's spatial semantics, and generating spatial semantic features of the segment-level semantic embedding vector, including: The segment-level semantic embedding vector is input into the pre-trained video segment spatial semantic transformation model, and the local spatial semantic features of each object in the segment-level semantic embedding vector are captured by the local spatial semantic extraction layer. The local spatial semantic features are integrated into the spatial semantic features of the global scene through the global spatial semantic integration layer. The spatial relationship reasoning layer infers the spatial positional relationship between objects in the global scene and generates a spatial relationship feature vector. The spatial semantic features of the global scene and the spatial relationship feature vector are fused to generate a fused spatial semantic feature vector. The fused spatial semantic feature vector is converted into a spatial semantic embedding vector with the same spatial dimension as the query semantic embedding vector through a spatial semantic transformation layer, and the temporal association layer between the spatial semantic embedding vector and the fragment-level semantic embedding vector is captured through a spatial semantic and temporal semantic association layer. The spatial semantic embedding vector and the temporal association are fused to generate fragment spatial semantic features that include spatial scene features, object spatial relationships, and temporal association information.

5. The method as described in claim 1, characterized in that, The step of performing temporal coherence verification on the candidate video segments and, in conjunction with the temporal correlation of the semantic embedding vectors of each video segment within the candidate video segments, outputting the video segment that matches the query semantic embedding vector, includes: Timestamp extraction is performed on candidate segments within the video segment candidate set to obtain the start and end timestamps of each candidate segment. The candidate segments are sorted according to the order of their start timestamps. Time interval checks are performed on adjacent candidate segments after sorting to identify adjacent segments with time intervals less than a preset value. Semantic coherence analysis is performed on the adjacent candidate segments with time intervals less than the preset value. The coherence evaluation layer of the coherence evaluation model is used to determine whether the semantics between the adjacent candidate segments are coherent. Semantically coherent adjacent candidate segments are merged to generate a merged video segment. The pre-trained merged segment temporal inference model is invoked, and the segment-level semantic embedding vector of the merged video segment is input into the merged segment temporal inference model to perform temporal evolution coherence check and generate the temporal evolution coherence score of the merged segment. The pre-trained merged segment spatial transformation model is invoked, and the segment-level semantic embedding vector of the merged video segment is input into the merged segment spatial transformation model to perform a spatial semantic transformation rationality check on the merged segment and generate a spatial transformation rationality score for the merged segment. The temporal evolution coherence score and the spatial transformation rationality score are weighted and fused to generate a comprehensive coherence score for the merged segments. Merged segments whose comprehensive coherence scores exceed a preset threshold are marked as coherent segments. The semantic coverage layer of the semantic coverage model extracts features that match the query semantics in the coherent segment and performs semantic integrity analysis to obtain a complete semantic segment. The complete semantic segment is marked as a valid segment and used as a video segment that matches the query semantic embedding vector.

6. The method as described in claim 5, characterized in that, The process involves calling a pre-trained merged segment temporal inference model, inputting the segment-level semantic embedding vectors of the merged video segments into the merged segment temporal inference model to perform a temporal evolution coherence check, and generating a temporal evolution coherence score for the merged segments, including: The segment-level semantic embedding vector of the merged video segment is input into the pre-trained merged segment temporal inference model, and the merged segment is divided into several continuous sub-segments through the temporal segmentation layer. Semantic embedding vector extraction is performed on each sub-segment to generate the semantic embedding vector of the sub-segment; The semantic evolution relationship between adjacent sub-segments is captured by the temporal evolution layer, and the semantic evolution path of the adjacent sub-segments is generated. Evolution gradient calculation is performed on the semantic evolution path of each adjacent sub-segment to generate an evolution gradient vector. Semantic overlap analysis is performed on the adjacent sub-segments to calculate the semantic overlap ratio of the adjacent sub-segments. Smoothness evaluation is performed on the semantic evolution path to calculate the rate of change of the semantic embedding vector in the semantic evolution path. The evolution gradient vector, the semantic overlap ratio, and the rate of change are subjected to normalized weighted fusion processing to generate a comprehensive coherence score of adjacent sub-segments. The combined coherence scores between all adjacent sub-segments are averaged to generate the temporal evolution coherence score of the merged segment.

7. The method as described in claim 5, characterized in that, The process involves calling a pre-trained merged segment spatial transformation model, inputting the segment-level semantic embedding vectors of the merged video segments into the merged segment spatial transformation model to perform a spatial semantic transformation rationality check on the merged segments, and generating a spatial transformation rationality score for the merged segments, including: The segment-level semantic embedding vector of the merged video segment is input into the pre-trained merged segment spatial transformation model, and the merged segment is divided into several continuous spatial scene segments through the spatial segmentation layer. Perform spatial semantic feature extraction on each of the spatial scene segments to generate spatial semantic features for the spatial scene segments; The spatial transformation path of the scene is generated by capturing the spatial transformation relationship between adjacent scene fragments through the spatial transformation layer. The naturalness of the spatial transition is evaluated on the adjacent scene fragments to obtain the semantic matching degree of the transition region. The semantic relevance is calculated on the spatial transformation path to obtain the semantic relevance of the scene before and after the transformation. Based on the spatial transformation path, the semantic matching degree, and the semantic association degree, a comprehensive reasonableness score is determined between the spatial scene segments; The overall rationality scores among all the aforementioned spatial scene segments are averaged to generate the spatial transformation rationality score of the merged segments.

8. The method as described in claim 5, characterized in that, The semantic coverage layer of the semantic coverage model extracts features that match the query semantics from the coherent fragment and performs semantic integrity analysis to obtain a complete semantic fragment, including: The segment-level semantic embedding vector of the coherent segment is semantically aligned with the query semantic embedding vector. The surface features that directly match the query semantic embedding vector in the coherent segment are extracted through the feature mapping layer of the feature mapping model to establish a surface semantic matching relationship. The semantic sequence of the coherent segment is input into a pre-trained latent semantic evolution model. Combined with the latent evolution logic of the query semantic embedding vector, the semantic advancement direction of the coherent segment is analyzed to determine whether it matches the evolution path of the query semantic embedding vector, thereby obtaining the fit weight in semantic development trend. The back-inference layer of the back-inference model performs semantic back-inference starting from the current semantic state of the coherent segment, determines the pre-associative semantics and post-associative semantics of the query semantic embedding vector implied by the current semantic state, and generates a complete back-inference semantic chain. The surface semantics, deep semantics, and implicit semantics of the coherent segment are respectively subjected to multi-level semantic connection analysis with the corresponding levels of the query semantic embedding vector. If there are uncovered semantic levels, priority sorting is performed on the missing semantic levels. The pre-trained semantic completion model is invoked to select video sub-segments from the video memory feature sequence that match the missing semantic level and conform to the semantic evolution logic of the coherent segment; The selected video segments are temporally aligned and semantically connected with the coherent segments through the temporal integration layer of the pre-trained semantic completion model to obtain the complete semantic segment.

9. A video retrieval server, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any one of the methods described in claims 1 to 8.

Images