A video annotation system and review method based on a multi-modal recognition model

Abstract

The application relates to the field of computer vision and artificial intelligence application, and particularly relates to a video annotation system and a reviewing method based on a multi-modal recognition model. The method comprises the following steps: obtaining a target video and annotation information, performing multi-modal feature extraction and time alignment, generating a candidate recognition result set; performing semantic feature space projection on the candidate result, constructing a feature semantic graph and calculating node vector representation to obtain a global confidence score; performing intelligent selection of the candidate result based on the confidence score, generating a structured multi-modal interaction instruction containing voice, action and expression identification, forming an annotation record; finally determining a display order, synchronously presenting the annotation on a shared screen and generating a meeting minutes. Through semantic graph reasoning and multi-modal instruction packaging, the application realizes deep correlation and context understanding of cross-modal information, significantly improves the accuracy and automation degree of video annotation, and enhances the naturalness and real-time performance of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of computer vision and artificial intelligence applications, and in particular to a video annotation system and review method based on a multimodal recognition model. Background Technology

[0002] In the field of computer vision and artificial intelligence applications, existing solutions for video annotation and review typically include keyframe extraction, single-modal image recognition or text matching, and annotation information recording. These solutions suffer from limitations such as insufficient reliability of candidate recognition results, inability to unify the analysis of visual and textual information, and a lack of real-time and automated annotation generation. Existing methods largely rely on manually set rules, fixed threshold filtering, and single-channel data processing.

[0003] In video annotation and review scenarios, misalignment of annotation information and asynchrony of multimodal information are prone to occur, making it difficult to meet the requirement of automatically generating annotation records with stable candidate recognition result sets and confidence score sets.

[0004] For the joint processing of the candidate recognition result set and the corresponding confidence score set, existing technologies generally suffer from common shortcomings such as inconsistent semantic feature mapping, lack of structure in the generation of multimodal interaction instructions, and missing node information propagation. It is difficult to form a continuous link of acquisition-alignment-judgment-control-recording in video annotation and review scenarios, resulting in delayed annotation record generation, insufficient accuracy, and incomplete fusion of multimodal information. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a video annotation system and review method based on a multimodal recognition model, comprising:

[0006] S100: Obtain the target video and annotation information, perform keyframe image extraction, first modality feature extraction, second modality feature extraction and instruction time dataset generation processing to obtain the first modality feature set, the second modality feature set and the instruction time dataset;

[0007] S200. Based on the first modal feature set, the second modal feature set, and the instruction time dataset, perform candidate recognition result generation processing to obtain a candidate recognition result set;

[0008] S300. Based on the candidate recognition result set, perform semantic feature space projection, feature semantic graph construction and node vector representation calculation to obtain the confidence score set corresponding to the candidate recognition result set;

[0009] S400. Based on the confidence score set corresponding to the candidate recognition result set, perform candidate recognition result selection and structured multimodal interaction instruction generation processing to obtain annotation records;

[0010] S500: Based on the annotation records, determine the display order, share the screen display, and generate meeting minutes to obtain the annotation results.

[0011] Furthermore, the processes of keyframe image extraction, first modality feature extraction, second modality feature extraction, and instruction time dataset generation include:

[0012] The keyframe image extraction process includes: extracting keyframes from the target video according to the timestamp, and recording the image data corresponding to each keyframe;

[0013] The first modal feature extraction process includes: extracting visual feature vectors from the keyframe images, wherein the visual feature vectors are used to characterize the image regions associated with the annotations;

[0014] The second modality feature extraction process includes: extracting audio feature vectors from the speech data in the annotation information, and extracting text feature vectors from the text data in the annotation information;

[0015] The instruction time dataset generation process includes: aligning and marking the timeline of the target video to generate a dataset containing timestamp indices of each modality;

[0016] After performing the above extraction and generation processes, the storage module saves the first modality feature set, the second modality feature set, and the instruction time dataset.

[0017] Furthermore, the process of generating candidate recognition results includes:

[0018] Based on the timestamp index in the instruction time dataset, the visual features in the first modal feature set are aligned across modal time with the audio features or text features in the second modal feature set;

[0019] Single-modal recognition and multimodal fusion recognition are performed on the aligned multimodal features to generate multiple candidate recognition results. Each candidate recognition result contains a semantic label of a modality and its original confidence score.

[0020] The candidate recognition results are organized into an initial list in chronological order to obtain a set of candidate recognition results.

[0021] Furthermore, the semantic feature space projection processing includes:

[0022] Each candidate recognition result in the candidate recognition result set is projected onto a trainable shared semantic feature space through a multimodal fusion encoder to obtain the semantic feature vector of the candidate recognition result;

[0023] The multimodal fusion encoder includes a visual encoder, an audio encoder, and a text encoder set in parallel, as well as a shared fully connected layer connected after each encoder. The shared fully connected layer projects features from different modalities onto a semantic feature space of the same dimension.

[0024] Furthermore, the process of constructing the feature semantic graph includes:

[0025] Taking each candidate in the candidate recognition result set as a node, calculate the cosine similarity between every two nodes in the shared semantic feature space;

[0026] Based on the cosine similarity, a differentiable k-nearest neighbor strategy is used to retain the top K neighbor nodes with the highest similarity for each node, and edges are constructed using the retained neighbor relationships to generate a dynamic feature semantic graph.

[0027] Furthermore, the node vector representation computation process includes:

[0028] Multiple rounds of information propagation are performed on the feature semantic graph. In each round of propagation, a multi-head graph attention layer is used to calculate the attention coefficient of each neighbor node relative to the center node, and the attention weight is obtained by softmax normalization.

[0029] Based on the attention weights, the semantic feature vectors of neighboring nodes are aggregated to obtain the aggregated context information;

[0030] The aggregated context information and the current feature vector of the central node are input into a gated recursive unit. The gated recursive unit controls the fusion ratio of the information by updating the gate and resetting the gate, and outputs the updated node vector representation.

[0031] After each round of propagation, residual connections and layer normalization are performed on the updated node vector representation to keep the network stable and preserve the initial features;

[0032] After a preset number of rounds of information propagation, the vector representation of each node obtained at the end is used as the refined semantic feature of the candidate recognition result, and the corresponding global confidence score is calculated based on the refined semantic feature.

[0033] The confidence score set is composed of the global confidence scores of all candidate identification results.

[0034] Furthermore, the process of selecting and processing candidate identification results includes:

[0035] Obtain the node vector representation and global confidence score of each candidate recognition result;

[0036] The node vector representation is input into a lightweight global rescoring network, which outputs a global calibration confidence score, which is used to characterize the probability that the candidate recognition result is adopted in the current semantic context.

[0037] Next, in the shared semantic feature space, a similarity-based non-maximum suppression operation is performed on the candidate recognition results: if the distance between the semantic feature vectors of two candidate recognition results is less than a preset threshold, only the one with the highest global calibration confidence is retained, thereby obtaining a subset of candidate recognition results after deduplication.

[0038] Furthermore, the process of generating structured multimodal interaction instructions includes:

[0039] For each selected result in the subset of deduplicated candidate recognition results, load a multimodal instruction template from the predefined instruction template library;

[0040] The semantic labels, global calibration confidence scores, and contextual information from the associated node vector representations of the selected results are filled into the instruction template to generate a structured interactive instruction in JSON format.

[0041] The structured interaction instructions include a voice interaction text field, an action identifier field, and a visual expression identifier field. The voice interaction text field is used to synthesize voice output, the action identifier field is used to trigger highlighting or selection actions on the screen, and the visual expression identifier field is used to generate emoticon icons displayed on the shared screen.

[0042] Arrange the structured interactive instructions corresponding to multiple selected results in a time sequence to generate an instruction sequence with time delay parameters or synchronization trigger markers;

[0043] The instruction sequence and its associated metadata are used together as annotation records.

[0044] Furthermore, the process of determining the display sequence includes:

[0045] Based on the time delay parameter or synchronization trigger flag carried by the instruction sequence in the annotation record, determine the appearance order and duration of each structured interactive instruction on the shared screen.

[0046] Furthermore, the process of sharing screen displays and generating meeting minutes includes:

[0047] The shared screen display processing includes: executing the action identifiers and visual expression identifiers in the structured interaction instructions according to a determined display order, sequentially displaying the highlighted areas of the annotations and expression icons on the shared screen, and simultaneously synthesizing and playing speech based on the voice interaction text field;

[0048] The meeting minutes generation process includes: combining the voice interaction text field, action identifier field, and corresponding timestamp in the structured interaction instructions into a text-formatted meeting minutes and annotation summary;

[0049] Finally, the annotation results are output to the storage module or display terminal.

[0050] The key innovations of this invention include:

[0051] (1) Based on the candidate recognition result set, semantic feature space projection and feature semantic graph are constructed, each candidate recognition result is used as a node, edge connection relationship is established according to the feature similarity between nodes, and node vector representation is calculated through multi-round information propagation and gating mechanism.

[0052] (2) Based on the set of confidence scores corresponding to the set of candidate recognition results, the candidate recognition results are dynamically filtered and sorted, and a structured multimodal interaction instruction in JSON format is generated, including voice interaction text content, action identifiers and visual expression identifiers.

[0053] (3) In the process of candidate recognition result selection and structured instruction generation, time series alignment and multimodal feature fusion are combined to realize the closed-loop processing of recognition-selection-generation-display, and support dynamic updates and multi-topic order management.

[0054] The following are its main beneficial effects:

[0055] (1) The candidate recognition result set is associated with the semantic feature space mapping and the feature semantic graph, and the unified processing of multimodal information is realized in the node vector representation, which improves the credibility and cross-modal consistency of the candidate recognition results and solves the problem that visual and text information cannot be analyzed in a unified manner.

[0056] (2) Based on dynamic filtering with high confidence scores and JSON structured instruction generation, the annotation records can achieve simultaneous output of visual, action and text three modalities during the generation process, which solves the problem of lack of real-time and automation in the annotation generation of traditional annotation systems.

[0057] (3) By combining time series alignment and multimodal feature fusion to form a closed-loop processing, the annotation records are stably generated under different topic sequences and time misalignments, which solves the problems of annotation information misalignment and multimodal asynchrony.

[0058] (4) The above innovations enable the processing, selection and interaction instruction generation of candidate recognition results in video annotation and review scenarios to form a complete link, improving the system's ability to continuously process multimodal data.

[0059] (5) Throughout the entire acquisition-alignment-judgment-control-recording chain, each link maintains consistency through the combined application of node vectors and structured instructions, realizing the automatic generation and traceability of annotation records. Attached Figure Description

[0060] Figure 1 A flowchart illustrating a video annotation system and review method based on a multimodal recognition model, provided for embodiments of this application;

[0061] Figure 2 This is a structural block diagram of a video annotation system and review method based on a multimodal recognition model, provided in an embodiment of this application. Detailed Implementation

[0062] Example 1: Refer to Figure 1 This is a flowchart illustrating a video annotation system and review method based on a multimodal recognition model provided in an embodiment of the present invention. The flowchart may include at least steps S100-S500:

[0063] S100: Obtain the target video and annotation information, perform keyframe image extraction, first modality feature extraction, second modality feature extraction and instruction time dataset generation processing to obtain the first modality feature set, the second modality feature set and the instruction time dataset;

[0064] S200. Based on the first modal feature set, the second modal feature set, and the instruction time dataset, perform candidate recognition result generation processing to obtain a candidate recognition result set;

[0065] S300. Based on the candidate recognition result set, perform semantic feature space projection, feature semantic graph construction and node vector representation calculation to obtain the confidence score set corresponding to the candidate recognition result set;

[0066] S400. Based on the confidence score set corresponding to the candidate recognition result set, perform candidate recognition result selection and structured multimodal interaction instruction generation processing to obtain annotation records;

[0067] S500: Based on the annotation records, determine the display order, share the screen display, and generate meeting minutes to obtain the annotation results.

[0068] S100: Obtain the target video and annotation information, perform keyframe image extraction, first modality feature extraction, second modality feature extraction and instruction time dataset generation processing to obtain the first modality feature set, the second modality feature set and the instruction time dataset;

[0069] Specifically, step S100 is initiated by the server after receiving the target video and annotation information sent by the user terminal. The target video refers to a sequence of video frames played or to be played by a video playback device. The target video includes consecutive video frames, video content corresponding to the operation time, and video content containing the annotation object. The annotation information refers to the annotation content, operation time, and dynamic input process corresponding to the annotation content input by the user terminal during the playback of the target video. The operation time is taken from the time information recorded when the user terminal receives the operation command and is synchronized with the playback time of the target video. After receiving the target video, the server first performs video frame sequence parsing, splitting the target video into video frames arranged by time. Then, according to the operation time, it locates the video frame interval corresponding to the annotation information in the video frame sequence. A fixed number of video frame sampling ranges are set before and after the video frame interval to form a keyframe image extraction range, thereby ensuring that the keyframe image extraction maintains a correspondence with the annotation information. The input comes from the target video and the annotation information. The resulting keyframe images, first modality feature set, second modality feature set and instruction time dataset are written to the server-side record at the end of this step and are called by the candidate recognition result generation and processing in S200.

[0070] The keyframe image extraction refers to selecting image frames from the video frame sequence that can represent the video content corresponding to the annotation information. The server first locates the center video frame according to the operation time, and then extracts video frames from the consecutive video frames before and after the center video frame that show significant changes in the image scene, significant differences in target detection information, and significant changes in motion analysis information, forming a keyframe image set. Understandably, when the duration of the dynamic input process corresponding to the annotation information is short, the server selects the center video frame and its adjacent video frames before and after it to form the keyframe image set; when the duration of the dynamic input process corresponding to the annotation information is long, the server selects video frames at multiple key time positions according to the display order to form the keyframe image set. The server simultaneously stores the keyframe images to a preset image storage location and records the image storage path. The image storage path is stored in the server's internal record for subsequent reference during sample label data generation. If the target video has missing video frames, an out-of-bounds operation time, or insufficient image scene changes, the server calls the adjacent video frames corresponding to the operation time to complete the keyframe image set and writes an anomaly flag in the internal record. The anomaly flag does not change the output field name of this step.

[0071] The first modality feature extraction refers to inputting the keyframe images into an image mapping module and an image vector embedding module to obtain the first modality feature set. The image mapping module is a processing unit that represents image scene information, target detection information, and motion analysis information from the keyframe images. Its input is the keyframe images, and its output is image features arranged in the order of the keyframe images. The image vector embedding module is a processing unit that represents the image features with a uniform length. Its input is the image features output by the image mapping module, and its output is image vectors corresponding to each keyframe image. The server sequentially calls the image mapping module and the image vector embedding module according to the display order of the keyframe images, and aggregates the image vectors corresponding to each keyframe image in chronological order to generate the first modality feature set. Each item in the first modality feature set carries the operation time location and image storage path association information corresponding to the keyframe image. Therefore, the first modality feature set is not an isolated image result, but rather an image feature result bound to the annotation information.

[0072] The second modality feature extraction refers to inputting the text data from the annotation information into a text mapping module and a text vector embedding module to obtain the second modality feature set. The text data refers to the annotation content, annotation object description, annotation summary draft, or text input records corresponding to the annotation content. The text mapping module is a processing unit that performs semantic segmentation and tag text template alignment on the text data. Its input is the text data, and its output is text features arranged by text segments. The text vector embedding module is a processing unit that represents the text features with a uniform length. Its input is the text features output by the text mapping module, and its output is text vectors corresponding to each text segment. After receiving the annotation information, the server first segments the annotation content into several text segments according to the operation time, then sends the text segments into the text mapping module to obtain the text features corresponding to the annotation content, and then sends them into the text vector embedding module to obtain the second modality feature set. Each item in the second modality feature set retains the dynamic input order corresponding to the operation time. This order is then cross-validated with the instruction time dataset to avoid misalignment between the annotation content and the operation time.

[0073] The instruction time dataset generation process refers to generating a dataset that records the dynamic input process based on the annotation information and the operation time. When the server receives the annotation information, it records the input start time, input end time, continuous input interval, pause interval, and input order of the annotation content, and aligns the records with the playback time of the target video to generate the instruction time dataset. The instruction time dataset records the dynamic input process of the annotation information over the operation time, including the input order, input length changes, and input interval changes of the annotation content at different operation times. In an engineering embodiment, when the user terminal inputs "subtitle too small" at the position corresponding to the operation time in the target video playback and continuously supplements the annotation content, the server first records the initial input time, then records the subsequent supplementary input times, and associates multiple time records with the same annotation content to generate the instruction time dataset; subsequently, the server writes the instruction time dataset together with the first modality feature set and the second modality feature set into the output record of this step, wherein the first modality feature set, the second modality feature set, and the instruction time dataset are used as direct inputs to S200 and enter the candidate recognition result generation process. If a user interrupts input or submits annotations repeatedly, the server retains the time of the first submission and records the time of subsequent submissions, forming a continuous time record for the same annotation information.

[0074] After the above processing, the field names output by S100 are fixed as the first modality feature set, the second modality feature set, and the instruction time dataset. The first modality feature set corresponds to the keyframe image extraction and the first modality feature extraction result, the second modality feature set corresponds to the second modality feature extraction result, and the instruction time dataset corresponds to the dynamic input process record of the annotation information over time. The server writes these three output fields into the same annotation record context, and in the step call relationship, sends the first modality feature set, the second modality feature set, and the instruction time dataset into the candidate recognition result generation processing in S200, forming a hierarchical input relationship across main steps.

[0075] In summary, this step integrates the target video, annotation information, and operation time into a single processing chain, forming three direct output fields: the first modality feature set, the second modality feature set, and the instruction time dataset. The keyframe image extraction, image mapping, image vector embedding, text mapping, and text vector embedding modules all revolve around the same annotation record, and the output fields have a fixed correspondence. These output fields directly enter step S200, forming a continuous processing path around the annotation information.

[0076] S200. Based on the first modal feature set, the second modal feature set, and the instruction time dataset, perform candidate recognition result generation processing to obtain a candidate recognition result set;

[0077] Specifically, S200 is automatically triggered by the server after completing S100. The input sources for S200 are the first modality feature set, the second modality feature set, and the instruction time dataset generated by S100. The first modality feature set corresponds to the image feature results of the keyframe image after passing through the image mapping module and the image vector embedding module. The second modality feature set corresponds to the text feature results of the text data in the annotation information after passing through the text mapping module and the text vector embedding module. The instruction time dataset records the dynamic input process of the annotation information over time. After reading the above three inputs, the server first synchronizes and aligns the first modality feature set and the second modality feature set according to the operation time order in the instruction time dataset, forming a candidate recognition result generation interval corresponding to the same annotation information. The candidate recognition result generation interval refers to the processing range of keyframe image features and text features aggregated around the same operation time. Subsequent image storage path generation, sample label data generation, and candidate recognition result output are all completed within the candidate recognition result generation interval, and the server writes the processing record corresponding to the current annotation information.

[0078] The first stage of the candidate recognition result generation process is image storage path generation. The server retrieves the keyframe images corresponding to each image feature from the first modal feature set, re-verifies the keyframe images, and writes them to a preset image storage location to generate an image storage path. The image storage path refers to the path record of the keyframe images in the server's storage space, and this path record is associated one-to-one with the operation time, annotation information, and the corresponding image feature in the first modal feature set. Understandably, when a usable image storage path already exists in S100, the server directly reads the image storage path and performs a validity check; when the image storage path is missing, invalid, or the keyframe image is updated, the server rewrites the keyframe image and updates the image storage path. After completing the image storage path generation, the server records the image storage path in the internal record of the current annotation information for use in sample label data generation.

[0079] The second stage of the candidate recognition result generation process is sample label data generation. The label text template refers to a pre-set text description format, which includes an image storage path field and an annotation information field. The server calls the image storage path and the annotation information, filling the image storage path into the image storage path field and the text content and operation time from the annotation information into the annotation information field to generate sample label data. The sample label data is not simply a text record, but rather associated data formed around the same keyframe image and the same annotation information, maintaining the correspondence between the image storage path, text content, and operation time internally. When generating the sample label data, the server also reads the text features corresponding to the annotation information from the second modality feature set and binds the text features to the sample label data. If the order of the text features in the second modality feature set is inconsistent with the order of the operation times in the instruction time dataset, the server rearranges the text features according to the dynamic input process of the instruction time dataset before performing the binding, thereby maintaining consistency between the input order of the sample label data and the annotation information.

[0080] The third stage of the candidate recognition result generation process is the generation of a candidate recognition result set. The server reads image features from the first modality feature set, text features from the second modality feature set, dynamic input processes from the instruction time dataset, and sample label data, and performs joint judgment on keyframe images and annotation information within the same candidate recognition result generation interval. The candidate recognition result set here includes image scene information, target detection information, and motion analysis information. The image scene information refers to the video scene description result corresponding to the keyframe image. The server generates multiple image scene information candidates based on the order of image features in the first modality feature set and the text content in the sample label data. The target detection information refers to the target content record in the keyframe image corresponding to the annotation information. The server generates multiple target detection information candidates based on the target position change in the image features and the annotation content matching relationship in the text features. The motion analysis information refers to the motion change record in the keyframe image within the candidate recognition result generation interval. The server generates multiple motion analysis information candidates based on the image feature changes and operation time sequence corresponding to adjacent keyframe images. The above three parts are combined to form the candidate recognition result set. Each candidate recognition result in the candidate recognition result set retains the corresponding image storage path, annotation information and operation time record.

[0081] In one engineering embodiment, a video playback device plays a target video. During screen sharing, a user terminal inputs annotation information, the corresponding text content of which is a record of annotations to the current video content. After receiving the annotation information, the server first reads the keyframe image features corresponding to the current operation time from the first modal feature set generated in S100, then reads the text features within the same operation time range from the second modal feature set, and reads the dynamic input process from the instruction time dataset. The server writes the corresponding keyframe image to a preset image storage location, generating an image storage path; then, it fills the image storage path and annotation information into a label text template to obtain sample label data; finally, based on the correspondence between image features and text features, it outputs image scene information, target detection information, and motion analysis information, ultimately forming a candidate recognition result set. If the current operation time corresponds to multiple keyframe images, the server performs candidate recognition result generation processing frame by frame according to the order recorded in the instruction time dataset, and writes the multiple output candidate recognition results into the same candidate recognition result set in display order.

[0082] After the above processing, the field name output by S200 is candidate recognition result set. The candidate recognition result set includes image scene information, target detection information, and action analysis information, as well as the image storage path, annotation information, and operation time association records corresponding to each candidate recognition result. The server sends the candidate recognition result set to the semantic feature space projection, feature semantic graph construction, and node vector representation calculation processing in S300. The candidate recognition result set serves as the direct input location of S300, maintaining traceability of its source relationship with the first modality feature set, the second modality feature set, and the instruction time dataset.

[0083] In summary, this step constructs a candidate recognition result generation processing chain based on the first modality feature set, the second modality feature set, and the instruction time dataset. The image storage path, sample label data, and candidate recognition result set are formed continuously within the same processing record. The candidate recognition result set is not an independent output but maintains a correspondence with annotation information, operation time, and keyframe images. The candidate recognition result set directly enters S300, forming the input basis for subsequent semantic feature space projection and feature semantic map construction.

[0084] S300. Based on the candidate recognition result set, perform semantic feature space projection, feature semantic graph construction and node vector representation calculation to obtain the confidence score set corresponding to the candidate recognition result set;

[0085] Specifically, step S300 is automatically triggered by the server after step S200 outputs the candidate recognition result set. The candidate recognition result set originates from the candidate recognition result generation process jointly participated in by the first modality feature set, the second modality feature set, and the instruction time dataset. The candidate recognition result set includes image scene information, target detection information, and action analysis information, and each candidate recognition result carries a corresponding image storage path, annotation information, and operation time association record. After reading the candidate recognition result set, the server first groups the results according to the same processing record belonging to the operation time and the annotation information, and then performs semantic feature space projection on the candidate recognition results within the same group. The semantic feature space refers to a unified representation space that carries the semantic feature representation of the candidate recognition results. The input objects entering the semantic feature space are only the candidate recognition results in the candidate recognition result set; no new external input records are introduced, thereby maintaining the continuity of the data chain between steps S200 and S300.

[0086] The semantic feature space projection refers to converting each candidate recognition result in the candidate recognition result set into a semantic feature representation. Specifically, the server reads the scene description content and corresponding image storage path for image scene information, reads the target content and corresponding keyframe image association records for target detection information, and reads the action change content and corresponding operation time sequence records for action analysis information. Then, the above content is sent to the same projection processing unit to obtain semantic feature representations of consistent length and order. The minimum set of core parameters here includes the candidate recognition result category, image storage path, annotation information, operation time, and the display order of the candidate recognition results in the candidate recognition result set. When the projection is completed, the server writes the semantic feature representation corresponding to each candidate recognition result into the current processing record. When a candidate recognition result has missing content, an invalid image storage path, or an operation time conflict, the server calls adjacent candidate recognition results in the same processing record to perform completion verification, retaining the original candidate recognition result position without changing the order structure of the candidate recognition result set.

[0087] The construction of the feature semantic graph refers to treating each candidate recognition result as a node within the semantic feature space, and constructing edge connections based on the feature similarity between nodes to form the feature semantic graph. This feature semantic graph is not an ordinary relationship graph, but rather a local graph structure built around the same batch of annotation information and the same operation time interval. The server first registers the semantic feature representations corresponding to image scene information, target detection information, and action analysis information as nodes, and then compares the feature similarity between nodes. Nodes with adjacent image storage paths, adjacent operation times, and overlapping annotation information content are all written into the feature semantic graph as edge connections. Understandably, when there is no effective edge connection between a candidate recognition result and other candidate recognition results, the server retains the independent node state of that candidate recognition result and marks it as an isolated node in the internal record; when the same batch of annotation information generates multiple candidate recognition results with similar content within a continuous operation time, the server retains multiple nodes, only adds edge connections, and does not perform node deletion, thus ensuring that subsequent node vector representation calculations retain the complete candidate path.

[0088] The node vector representation calculation process refers to the server calling a graph neural network to perform multiple rounds of information propagation on the feature semantic graph to obtain the node vector representation corresponding to each node. This graph neural network consists of a node reading unit, a neighborhood aggregation unit, a nonlinear fusion unit, and a gating mechanism unit. The node reading unit reads the current semantic feature representation of each node. The neighborhood aggregation unit reads the information of the neighboring nodes connected to that node. The nonlinear fusion unit nonlinearly fuses the aggregated neighboring node information with the information of the central node. The gating mechanism unit controls the information update ratio in each round of propagation. In specific operation, the server first calculates the importance weight of each node's neighborhood, then aggregates the neighboring node information according to the importance weight, then nonlinearly fuses the aggregated neighboring node information with the information of the central node, and controls the information update ratio through the gating mechanism to complete one round of node state update. The server repeats multiple rounds of information propagation until the states of all nodes in the same feature semantic graph reach the preset convergence condition, obtaining the final node vector representation. In an engineering embodiment, when the candidate recognition result set corresponding to the same batch of annotation information contains three image scene information, two target detection information and two action analysis information, the server first constructs seven nodes in the display order, then constructs edge connection relationships according to adjacent operation times, and then performs multiple rounds of information propagation, finally outputting node vector representations for the seven nodes respectively.

[0089] After obtaining the node vector representation, the server calculates the confidence score of each candidate recognition result based on the node vector representation corresponding to each node, and aggregates all confidence scores in the original order of the candidate recognition result set to generate a confidence score set corresponding to the candidate recognition result set. The confidence score set is the output field name of S300. This output field maintains a one-to-one correspondence with the candidate recognition result set; each score in the confidence score set corresponds to a candidate recognition result, and retains the association index of the image storage path, the annotation information, and the operation time. The server writes the confidence score set into the current annotation record context and directly sends it to the candidate recognition result selection and structured multimodal interaction instruction generation processing in S400, serving as the direct input location of S400. Simultaneously, the candidate recognition result set remains in the server for synchronous invocation when S400 performs candidate recognition result selection.

[0090] In summary, this step establishes a continuous processing chain around the candidate recognition result set, from semantic feature space projection to feature semantic graph construction, and then to node vector representation calculation. The confidence score set is not an independent score record, but rather a continuous output field corresponding to each item in the candidate recognition result set. The confidence score set directly enters S400, maintaining the continuity of the processing relationship between S200, S300, and S400.

[0091] In one specific embodiment:

[0092] In S300, based on the candidate recognition result set, semantic feature space projection, feature semantic graph construction, and node vector representation calculation are performed to obtain the confidence score set corresponding to the candidate recognition result set.

[0093] The server reads the candidate recognition result set. Each candidate recognition result in the set contains image scene information, target detection information, action analysis information, as well as the corresponding image storage path, annotation information, operation time association record, and display order in the set. The server first groups the candidate results according to the same processing record belonging to the operation time and annotation information, ensuring that all candidate results within the same video annotation context are processed together. Next, the server performs semantic feature space projection on each candidate recognition result within the group: the scene description content and image storage path in the image scene information, the target content and keyframe image association record in the target detection information, and the action change content and operation time order record in the action analysis information are all fed into a multimodal fusion projection unit. This projection unit contains parallel visual encoders, audio encoders, and text encoders, followed by a shared fully connected layer, projecting features from different modalities onto a trainable semantic feature space of the same dimension. This projection process solves the "heterogeneous gap" problem where original heterogeneous features (such as CLIP image features and BERT text features) cannot be directly compared, enabling nodes in the subsequent graph structure to measure semantic similarity in a unified space. To achieve a numerical description of the projection, formula ① is defined to map the original multimodal features of each candidate recognition result into a semantic feature vector.

[0094] Formula①

[0095]

[0096] in:

[0097] : No. The initial semantic feature vector after projecting the candidate recognition results has a fixed dimension of 1. (e.g., 512);

[0098] : Multimodal fusion projection function, representing the overall operation of mapping three input vectors to an output vector;

[0099] The raw visual feature vector extracted from the image scene information originates from the "Image Scene Information" field in the "Candidate Recognition Result Set," and is processed by a visual encoder. From image data Calculated;

[0100] The target feature vector extracted from the target detection information comes from the "target detection information" field and is processed by the same visual encoder. Extract from records associated with keyframe images;

[0101] The motion feature vectors extracted from the motion analysis information are derived from the "motion analysis information" field and are extracted by the motion encoder from the content of motion changes and the order of operation time.

[0102] : Linear rectified activation function, defined as , used to introduce nonlinearity;

[0103] The weight matrix of the shared fully connected layer has dimensions of [dimensional value missing]. Trainable parameters;

[0104] : Vector concatenation function, which concatenates multiple vectors within parentheses along the dimensional direction into a single long vector;

[0105] A visual encoder converts image data... Mapped to visual feature vectors;

[0106] : No. The image data pointed to by the image storage path of each candidate recognition result;

[0107] Audio encoder, which converts speech recognition data into text data. Mapped to audio feature vectors;

[0108] : Speech recognition text data in the annotation information;

[0109] A text encoder that processes handwritten or keyboard-input text data. Mapped to text feature vectors;

[0110] : Handwritten or keyboard-input text data in the annotation information;

[0111] Bias vector, dimension Trainable parameters.

[0112] Data Source → Metrics → Variable Mapping: Extracting image data from "Image Scene Information" in the "Candidate Recognition Result Set". The keyframe association records are extracted from "target detection information," and the motion change content is extracted from "motion analysis information." These three are then processed by their respective encoders to form... By inputting formula ① together, we can obtain... .

[0113] Simple numerical example: Let simplify, Encoding , , spliced ​​together , for The matrix is ​​all 1s. Linear transformation yields ReLU remains unchanged. (Floating-point numbers are used in actual training).

[0114] The practical problem solved by this formula is to map data from three heterogeneous modalities—images, targets, and actions—to the same numerical space, enabling subsequent graph neural networks to calculate semantic similarity across modalities. This avoids the information fragmentation caused by simply recognizing and fusing each modality separately in traditional methods.

[0115] After the server completes formula ①, it generates an initial semantic feature vector for each candidate recognition result within the group. These vectors are then bound to the corresponding candidate recognition result index, image storage path, annotation information, and operation time association record, and written into the current processing record. All initial vectors constitute the initial feature set, which serves as the direct input for the next step of feature semantic map construction.

[0116] To establish the relationships between candidate results within the semantic feature space, the server performs feature semantic graph construction. This graph uses each candidate recognition result as a node, and the node features are the output of formula ①. Edge connections are based on the similarity of features between nodes and auxiliary spatiotemporal proximity conditions (adjacent image storage paths, adjacent operation times, and overlapping annotation information). Since direct fully connected edges are computationally expensive and introduce noise, the server uses a differentiable k-nearest neighbor strategy to dynamically determine the neighbor set of each node and calculate edge weights. Formula ② is defined to calculate node weights. With nodes The initial similarity between them is used in formula ③ to convert the similarity into an adjacency matrix and perform row normalization.

[0117] Formula②

[0118]

[0119] in:

[0120] :node With nodes Cosine similarity between them, range of values ;

[0121] :vector transpose;

[0122] Nodes obtained from formula ① and nodes The initial semantic feature vector;

[0123] :vector The Euclidean norm (L2 norm) is defined as follows: ;

[0124] : Represents a node Belongs to node candidate neighbor set ;

[0125] :node The candidate neighbor set is determined by the server based on spatiotemporal conditions (operation time difference less than a threshold). The images were pre-filtered based on factors such as adjacent image storage paths and overlapping words in annotation information.

[0126] Data source → Metrics → Variable mapping: Based on the initial semantic feature vector in the currently processed record. and Substitute directly into formula ②; candidate neighbor set The selection criteria are based on the continuity of the timestamps in the "Operation Time Related Records" and the "Image Storage Path", as well as the text overlap of the "Annotation Information".

[0127] Simple numerical example: Let , dot product = , , The product = 10.75. .

[0128] Formula ② addresses the practical problem of quantifying the intrinsic similarity of two candidate recognition results in a unified semantic space, providing a basic metric for edge weights in graph structures, and avoiding simple matching that relies solely on time or text keywords.

[0129] The server calculates all Then, only the nodes with the highest similarity are retained. Neighbors (e.g.) The edges of the remaining neighbors are disconnected. Then, formula ③ is defined to perform row normalization on the adjacency matrix to balance the impact of nodes with higher degrees on information propagation.

[0130] Formula③

[0131]

[0132] in:

[0133] Unnormalized attention-based adjacency weights;

[0134] The similarity value obtained from formula ②, with the superscript "1" indicating that it comes from formula ②;

[0135] : Indicator function, when node Belongs to node top- The value is 1 when selecting a set of similar neighbors, and 0 otherwise.

[0136] :node top- The set of similar neighbors is determined by the server based on... Sort the values ​​in descending order and take the first few. One received;

[0137] : For sets all elements Summation;

[0138] :node top- Neighbors gather, ;

[0139] :node With nodes Cosine similarity between them (as defined in formula ②);

[0140] Minimal positive numbers (e.g.) (), used to prevent the denominator from being zero;

[0141] The normalized adjacency matrix elements satisfy... (For nodes with neighbors).

[0142] Data source → Indicator → Variable mapping: obtained from formula ② As input; top-K neighbor filtering is based on The numerical sorting is derived from the similarity matrix in the server's memory.

[0143] Simple numerical example: Node There are three neighbors with similarity scores of 1 / 3 and 1 / 4 respectively. ,Pick Keep the first three, and sum the denominators. , ,but Approximately .

[0144] Formula ③ addresses the practical problem of converting similarity into adjacency weights of a probability distribution, ensuring that the information aggregation of each node has a stable scale during subsequent propagation of the graph neural network, thus preventing nodes with high degrees from dominating the propagation.

[0145] The server obtains the adjacency matrix according to formula ③. Combining node features Complete the construction of the feature semantic graph. This graph retains metadata such as the original node index, image storage path, and operation time association. For isolated nodes (without any neighbors), And all others are 0. The output of this section is a feature semantic graph data structure, containing a node feature matrix. and adjacency matrix This output will be directly fed into the node vector representation calculation and processing.

[0146] The server then performs node vector representation computation, that is, runs a graph neural network on the aforementioned feature semantic graph for multiple rounds of information propagation. Here, a graph attention network combined with a gated recursive unit is used to adaptively aggregate neighbor information and mitigate oversmoothing. Formula ④ is defined to calculate the... Nodes in round propagation To his neighbors The attention weights are calculated using formula ⑤, which is used to perform neighborhood aggregation and gating updates, and formula ⑥, which is used to calculate the final confidence score.

[0147] Formula④

[0148]

[0149]

[0150] in:

[0151] : No. Nodes in round propagation To the neighbor Unnormalized attention energy;

[0152] : Current propagation round index, value range , The maximum number of rounds is preset;

[0153] : The activation function of a leaky linear rectified circuit is defined as follows: (Negative slope is usually set to 0.2);

[0154] Attention mechanisms can learn weight vectors transpose, dimension ;

[0155] Shared linear transformation weight matrix, dimension This is used to map node features to a high-dimensional attention space;

[0156] : No. Node at the start of round propagation eigenvectors. When hour, ;

[0157] : Vector concatenation operator, concatenates two vectors along the dimension direction;

[0158] : No. Normalized attention weights in round propagation satisfy ;

[0159] : Natural exponential function;

[0160] : To top- All nodes in the neighbor set Summation;

[0161] Same as the definition in formula ③.

[0162] Data source → Metrics → Variable mapping: Based on the node feature matrix of the current round (Source: previous propagation results or initial features) and adjacency matrix (The output of Formula ③) collectively determines the neighbor set; and The parameters are pre-trained for the server and stored in the model library.

[0163] Simple numerical example: Let , , , for A matrix consisting entirely of 0.5, then , spliced ​​together , dot product = LeakyReLU(3.2) = 3.2. Suppose that a neighboring node receives 2.5 energy. ,but .

[0164] Formula ④ addresses the practical problem of adaptively learning the importance of each neighbor to the central node, rather than relying solely on fixed similarity, enabling the system to distinguish between candidate results that are "relevant but minor" and those that are "core associations".

[0165] After obtaining the attention weights, the server performs neighborhood aggregation and updates the node state through a gated recursive unit. Equation ⑤ defines the complete process of aggregation information and node updates.

[0166] Formula⑤

[0167]

[0168] in:

[0169] : No. The neighbor information vector obtained by round aggregation, dimension ;

[0170] : To top- All nodes in the neighbor set Summation;

[0171] Attention weights obtained from formula ④;

[0172] Value transformation weight matrix, dimension This is used to map neighbor features to the output dimension;

[0173] : No. Neighbor nodes at the start of round propagation eigenvectors;

[0174] Update the gate vector, dimension Each element takes a value in between;

[0175] Reset gate vector, dimension Each element takes a value in between;

[0176] The Sigmoid activation function is defined as follows: Compress the input to interval;

[0177] Learnable weight matrix, with dimensions as follows: , , ;

[0178] : to vector and splicing, dimension ;

[0179] The candidate's next-round state vector, dimension ;

[0180] The hyperbolic tangent activation function is defined as follows: Output range ;

[0181] Element-wise multiplication (Hadamard product);

[0182] Updated version Node at the start of round propagation eigenvectors;

[0183] : Represents a vector whose elements are all 1s, with dimension 1. .

[0184] Data source → Indicator → Variable mapping: From formula ④ Current round node features and The calculated weight matrices are all pre-trained parameters from the server.

[0185] Simple numerical example: Continuing with the previous example, let... simplify, , , , , , splicing , for All 0.2, get , have to (Update door). The same applies to resetting the door. , spliced ​​together , 0.3 all , have to ,final .

[0186] Formula ⑤ addresses the practical problem of dynamically controlling the fusion ratio of historical information and newly aggregated information through a gating mechanism, avoiding excessive feature smoothing (i.e., convergence of all node vectors) during multiple rounds of propagation, while preserving long-term memory of the initial semantics.

[0187] The server repeatedly executes formulas ④ and ⑤. Wheel (e.g.) ), thus obtaining the final node vector representation. After multiple rounds of information propagation, the server calculates the global confidence score for each candidate recognition result based on the final node vector. Formula ⑥ is defined as a lightweight prediction head (single-layer fully connected plus Sigmoid) that outputs the confidence score.

[0188] Formula⑥

[0189]

[0190] in:

[0191] : No. The global confidence score of each candidate identification result, with a value ranging from... interval;

[0192] : Sigmoid activation function (defined in formula ⑤);

[0193] Learnable weight vector transpose, dimension ;

[0194] :go through After round propagation, the node The final eigenvector (obtained by iterating from formula ⑤);

[0195] Learnable bias scalar;

[0196] : The preset maximum number of propagation rounds (e.g. ), which is a positive integer.

[0197] Data source → Indicator → Variable mapping: The final node vector output by formula ⑤ As input; the weights and biases are obtained by the server from historical annotation data.

[0198] Simple numerical example: Let , , dot product = Sigmoid(1.07) = 0.745, meaning the confidence level of the candidate identification result is 0.745.

[0199] Formula ⑥ addresses the practical problem of transforming the high-dimensional semantic features fused from graph neural networks into a comparable normalized confidence level, providing a unified decision-making basis for the next step of selecting candidate results for S400.

[0200] Engineering Example: In a video annotation task, the candidate recognition result set contains 3 image scene information (corresponding to keyframes at seconds 10, 12, and 15 of the video, respectively), 2 object detection information (detecting "chart" and "hand pointer"), and 2 action analysis information ("waving" and "pointing"), for a total of 7 candidates. The server sorts them into nodes 1 to 7 according to the operation time. Formula ① generates a 512-dimensional semantic vector for each node. Formula ② calculates pairwise cosine similarity, and Formula ③ uses K=3 to construct an adjacency matrix. For example, node 2 (chart) has the highest similarity with nodes 4 (pointer) and 6 (pointing). Subsequently, 5 rounds of updates using formulas ④ and ⑤ are performed, and finally, formula ⑥ outputs 7 confidence scores: for example, node 6 (pointing action) gets 0.92, node 2 (chart) gets 0.88, node 1 (scene) gets 0.45, etc. The server aggregates these scores into a confidence score set in their original order and binds them to the image storage path, annotation information, and operation time association index in the candidate recognition result set. The set of confidence scores, as the output field of S300, is directly fed into the candidate recognition result selection and structured multimodal interaction instruction generation processing in S400 as input. At the same time, the set of candidate recognition results continues to be retained in the server for synchronous use by S400.

[0201] This section summarizes the technical effects: This step achieves closed-loop refinement from multimodal candidate results to high-confidence semantic vectors through semantic feature space projection, dynamic feature semantic graph construction, and multi-round propagation of graph neural networks based on attention and gating mechanisms. The generated set of confidence scores maintains a one-to-one correspondence with the candidate results, providing verifiable and traceable context-aware input for subsequent intelligent decision-making.

[0202] S400. Based on the confidence score set corresponding to the candidate recognition result set, perform candidate recognition result selection and structured multimodal interaction instruction generation processing to obtain annotation records;

[0203] Specifically, step S400 is triggered by the server after step S300 is completed. The input sources for S400 include the confidence score set output by S300, and a set of candidate recognition results corresponding to each item in the confidence score set. Each candidate recognition result in the set carries image scene information, target detection information, or action analysis information, as well as image storage path, annotation information, and operation time association records. The server first reads the confidence score set, then establishes a correspondence according to the original order of the candidate recognition result set, forming a candidate recognition result selection queue. The candidate recognition result selection queue is the direct processing object of this step; subsequent candidate recognition result selection and structured multimodal interaction instruction generation all revolve around the candidate recognition result selection queue.

[0204] The candidate recognition result selection refers to determining the target candidate recognition result corresponding to the current annotation information from the candidate recognition result selection queue. The server first compares the scores in the confidence score set and filters out the candidate recognition result with the highest score. When two candidate recognition results have similar scores, the server retrieves the annotation content and operation time from the annotation information, then compares the image storage path and display order corresponding to these two candidate recognition results, selecting the candidate recognition result that has a higher degree of matching with the annotation content and a closer correspondence with the operation time as the current target candidate recognition result. Understandably, if similar image scene information is consecutively arranged in the candidate recognition result set, the server retains the image scene information with the highest score and writes the remaining image scene information into an internal retention record. If the target detection information and action analysis information correspond to the same operation time interval, the server first retains the one with the higher score and then attaches the other to the supplementary record of the current target candidate recognition result. After this processing, the server outputs a unique current target candidate recognition result and writes this result into the processing intermediate record of the current annotation information.

[0205] The structured multimodal interaction command generation process refers to generating interaction commands with complete structured fields and fixed field relationships based on the current target candidate recognition result. The structured multimodal interaction commands here adopt JSON (JavaScript Object Notation) format. The JSON format refers to a text format organized by field names and field values. The server sets the text content field, action identifier field, and visual expression identifier field for voice interaction in the JSON format. The text content field for voice interaction is derived from the merged result of the annotation content in the annotation information and the image scene information, target detection information, or action analysis information in the current target candidate recognition result; the action identifier field is derived from the action analysis information or target detection information in the current target candidate recognition result; the visual expression identifier field is derived from the image scene information or target detection information in the current target candidate recognition result. When generating the structured multimodal interaction commands, the server also writes the image storage path and operation time into the internal index record, so that the JSON format commands are associated with the corresponding content in the target video.

[0206] In an engineering embodiment, the user terminal submits an annotation for the target video. After the annotation enters steps S100, S200, and S300, the server has obtained a set of candidate recognition results and a set of confidence scores. In step S400, the server first selects the candidate recognition result with the highest score from the set of confidence scores. If the candidate recognition result belongs to target detection information, the server reads the annotation content and time associated with the target detection information, and then generates a text content field for voice interaction; subsequently, it generates an action identifier field based on the action content in the target detection information, and then generates a visual expression identifier field based on the image scene information, ultimately forming the structured multimodal interaction command. If the candidate recognition result belongs to image scene information, the server first generates a visual expression identifier field based on the image scene information, then combines it with the annotation content to generate a text content field for voice interaction, and generates an action identifier field if there is supplementary action analysis information. The entire process is executed continuously by the server without manual intervention. If a field is missing, the server calls the supplementary record of the current target candidate recognition result to fill it in; if the supplementary record also does not exist, the server retains an empty field and writes an exception record.

[0207] The annotation record generation process refers to writing the structured multimodal interaction instructions and annotation information together into an annotation record. During writing, the server organizes the annotation record according to a fixed field structure. The annotation record includes the topic, annotation content, participants, and time. Here, the topic comes from the topic information to which the current annotation information belongs; the annotation content comes from the text content in the annotation information and the text content of the voice interaction in the structured multimodal interaction instructions; the participants come from the user terminal login authentication record; and the time comes from the operation time in the instruction time dataset. The server attaches the structured multimodal interaction instructions to the same annotation record and marks this annotation record as a formal record for shared screen display and meeting minutes generation. Thus, the output field name of S400 is the annotation record, and after this step, the annotation record is directly sent to the display order determination, shared screen display, and meeting minutes generation process in S500.

[0208] In summary, this step establishes a one-to-one correspondence between the confidence score set and the candidate recognition result set, and determines the target candidate recognition result. The structured multimodal interaction instructions and the annotation records are continuously generated in the same processing record, with fixed field relationships. The annotation records directly enter S500, forming a continuous processing path from candidate recognition results to shared screen display and meeting minutes generation.

[0209] S500: Based on the annotation records, determine the display order, share the screen display, and generate meeting minutes to obtain the annotation results.

[0210] Specifically, step S500 is triggered by the server after step S400 outputs the annotation records. The annotation records originate from the candidate recognition result selection and structured multimodal interaction instruction generation process, and include the topic, annotation content, participants, and time, as well as the structured multimodal interaction instructions corresponding to the topic. After receiving the annotation records, the server first merges them by topic, then establishes a sequence table for annotation records under the same topic by time, and simultaneously reads participant information and the current shared screen status to form the input record for the display order determination process. Here, the display order refers to the sequential relationship of multiple annotation records appearing on the shared screen. The shared screen is the display area where the video playback device or conference terminal publicly presents the annotation content to the participants. The annotation result is the final confirmation record after the annotation records have completed the display order determination, shared screen display, and meeting minutes generation.

[0211] The display order determination process refers to the server establishing a stable display order relationship for the annotation records based on the time, topic, and participant information in the annotation records. Specifically, the server first compares the time of each annotation record under the same topic, and the annotation record with the earlier time is displayed first. When two annotation records have the same time, the server continues to compare the participant submission order, and the annotation record with the earlier submission order is displayed first. When both time and submission order are the same, the server then compares the completeness of the association between the annotation content and the structured multimodal interaction instructions, and the annotation record with higher completeness of association is displayed first. Understandably, in one implementation, the server determines the display order of a single annotation record; in another implementation, the server determines the batch display order of multiple annotation records. If an annotation record contains missing topics, missing times, or missing participant records, the server retains the annotation record, places it at the end of the current topic order list, and writes an exception record, which does not change the original content of the annotation record.

[0212] The shared screen display processing refers to the server sending one or more annotation records to the shared screen according to the display order, or sending the annotation records to the shared screen after sorting them according to the display order. Specifically, the server reads the sequence list corresponding to the current topic, selects the first annotation record from the sequence list, packages the topic, annotation content, participants, and time in the annotation record into shared screen display data, and then attaches the text content of the voice interaction, action identifiers, and visual expression identifiers from the structured multimodal interaction instructions to the same shared screen display data before sending it to the shared screen. If there are multiple annotation records to be displayed under the current topic, the server sends them one by one according to the display order; if the conference terminal is in batch display mode, the server sorts the multiple annotation records according to the display order and then combines them before sending. The sending action here is actively executed by the server, and the triggering condition is that the shared screen is in the open state and there are valid annotation records for the current topic. If the shared screen is in the closed state, the server writes the shared screen display data into the record to be displayed, and continues to send when the shared screen is reopened.

[0213] The meeting minutes generation process refers to the server generating meeting minutes based on the official annotation records under the current topic after the annotation records are displayed on the shared screen. Specifically, when the annotation records under the same topic receive final confirmation from the participants, the server categorizes the annotation records into the final annotation version. The final annotation version is a fixed annotation version record corresponding to the current topic, which internally stores the annotation records that have been confirmed. The server reads all annotation records in the final annotation version, extracts the topic, annotation content, participants, and time, and writes them into the meeting minutes in the display order; at the same time, it reads the text content, action identifiers, and visual expression identifiers of the voice interaction in the structured multimodal interaction instructions and writes them into the annotation minutes in the display order of the annotation records. If the current topic has not yet entered the final confirmation state, the server does not write to the final annotation version, but retains the current annotation records and sequence list, and executes the meeting minutes generation process after receiving final confirmation from the participants.

[0214] In one engineering embodiment, a video playback device plays a target video, and multiple participants submit annotations via user terminals. Based on the annotation records output by S400, the server first determines the display order for the same topic, then sends the first annotation record to the shared screen. After the shared screen displays the annotation record, the server continues to send subsequent annotation records. Once all annotation records for the topic have been displayed and final confirmation has been received from the participants, the server categorizes all annotation records into the final annotation version, generates meeting minutes and annotation minutes, and associates and saves the meeting minutes, annotation minutes, and final annotation version together. If new annotation records are added during the process, the server re-executes the determination of the display order for the current topic, and then updates the shared screen display data and the meeting minutes generation order.

[0215] After the above processing, the field name output by S500 is "Annotation Result". The Annotation Result consists of annotation records after the completion of display order determination, shared screen display, and meeting minutes generation processing, and internally maintains the correspondence with the final annotation version, meeting minutes, and annotation minutes. The Annotation Result is written to the server as the final output record of this method and can be used for subsequent querying, retrieval, and reuse of the final annotation version.

[0216] In summary, this step integrates the annotation records into a continuous processing chain involving display order determination, shared screen display, and meeting minutes generation. The annotation results maintain a fixed correspondence with the meeting minutes, annotation minutes, and finalized annotation versions. The annotation records are publicly presented and formally archived in this step.

[0217] Example 2: Figure 2 This diagram illustrates a structural block diagram of a video annotation system and review method based on a multimodal recognition model according to an embodiment of the present invention. Figure 2 As shown, the structure may include:

[0218] Installation and calibration unit 01 is used for accessing, initializing, configuring, and calibrating the target video and annotation information;

[0219] Specifically, the system receives target video and annotation information from external video interfaces and annotation input terminals. Under the configured initialization parameters and system calibration constraints, it completes video stream access, annotation information registration, and device calibration data generation to form a system configuration state. The system configuration state is recorded as installation calibration information and maintains a consistent association with the initial timestamp. The installation calibration information is transmitted to the threshold and strategy configuration unit as an input field, while log data is retained for subsequent traceability and updates.

[0220] Threshold and strategy configuration unit 02 is used to set the candidate recognition result selection criteria and the multimodal interaction instruction generation strategy;

[0221] Specifically, the system receives installation calibration information and external policy templates output from the installation and calibration unit, performs threshold setting, policy mapping, and policy verification processing on the installation calibration information, and forms a policy configuration object based on the policy table. The policy configuration object is then passed as a field to the data acquisition unit for invocation, and the corresponding version information is registered in the policy buffer for the feature extraction and alignment unit to read.

[0222] Data acquisition unit 03 is used to acquire keyframe images, text annotation information, and instruction time datasets;

[0223] Specifically, the system receives policy configuration objects and installation calibration information from the threshold and policy configuration unit, and performs keyframe capture, annotation text reading and timestamp recording processing through the video stream parsing module and text interface acquisition module to form a data acquisition set. The data acquisition set is output to the feature extraction and alignment unit as an input field, while the original data is retained in the buffer storage for log recording and anomaly tracking.

[0224] The feature extraction and alignment unit 04 is used to perform first modality feature extraction and second modality feature extraction on the keyframe image and annotation information, and to perform time series alignment processing.

[0225] Specifically, based on the data acquisition set provided by the data acquisition unit, the keyframe image is input into the image mapping module and the image vector embedding module to perform first modality feature extraction, while the annotation text is input into the text mapping module and the text vector embedding module to perform second modality feature extraction. Combined with the instruction time dataset, time series alignment and synchronization processing are performed to form a first modality feature set, a second modality feature set, and aligned time data. The set is provided to the candidate recognition generation unit as an input field and a synchronization index is recorded in the intermediate storage.

[0226] The candidate recognition generation unit 05 is used to generate a candidate recognition result set based on the feature set and time data, and to record the image storage path and sample label data.

[0227] Specifically, the system receives a first modality feature set, a second modality feature set, and alignment time data from the feature extraction and alignment unit. It then performs image path generation, sample label mapping, and candidate result construction processing through the candidate generation engine to form a candidate recognition result set. The candidate recognition result set is output to the semantic projection and node representation unit as an input field and the image path and label index are recorded in the storage system for subsequent annotation generation and tracking.

[0228] The semantic projection and node representation unit 06 is used to project the candidate recognition result set onto the semantic feature space, construct a feature semantic graph, and calculate the node vector representation and confidence score set.

[0229] Specifically, the system receives a set of candidate recognition results from the candidate recognition generation unit, maps the candidate recognition results to a multi-dimensional semantic feature space through the semantic projection module, establishes a relationship network between nodes and edges in the feature semantic graph construction module, and performs vector calculation and confidence score generation in the node representation calculation module to form a set of node vector representations and confidence scores. The set is provided to the annotation generation unit as an input field, and the node mapping relationship and confidence value are registered in the index storage.

[0230] The annotation generation unit 07 is used to select candidate recognition results based on the confidence score set and generate structured multimodal interaction instructions in JSON format to form annotation records;

[0231] Specifically, the system receives node vector representations and confidence score sets from the semantic projection and node representation unit. In the candidate selection module, candidate recognition results are determined based on thresholds and priorities. Subsequently, in the interaction instruction generation module, a structured multimodal interaction instruction in JSON format is constructed to generate annotation records. The annotation records are output to the display and meeting recording unit as input fields and the annotation sequence and participant information are registered in the storage.

[0232] Display and meeting recording unit 08 is used to determine the display order of the annotation records, share the screen display, and generate meeting minutes.

[0233] Specifically, the system receives annotation records from the annotation generation unit, determines the display order through the display sorting module, synchronously displays the annotation content on the shared screen interface, and simultaneously organizes the annotation records, time information, and participant list in the meeting minutes generation module to form the annotation result. The annotation result is saved as the final output of the system, and the display order and minutes index are recorded in the internal storage for historical tracking and backtracking queries.

Claims

1. A video annotation system and review method based on a multimodal recognition model, characterized in that, include: S100: Obtain the target video and annotation information, perform keyframe image extraction, first modality feature extraction, second modality feature extraction and instruction time dataset generation processing to obtain the first modality feature set, the second modality feature set and the instruction time dataset; S200. Based on the first modal feature set, the second modal feature set, and the instruction time dataset, perform candidate recognition result generation processing to obtain a candidate recognition result set; S300. Based on the candidate recognition result set, perform semantic feature space projection, feature semantic graph construction and node vector representation calculation to obtain the confidence score set corresponding to the candidate recognition result set; S400. Based on the confidence score set corresponding to the candidate recognition result set, perform candidate recognition result selection and structured multimodal interaction instruction generation processing to obtain annotation records; S500: Based on the annotation records, determine the display order, share the screen display, and generate meeting minutes to obtain the annotation results.

2. The method according to claim 1, characterized in that, The process of keyframe image extraction, first modality feature extraction, second modality feature extraction, and instruction-time dataset generation includes: The keyframe image extraction process includes: extracting keyframes from the target video according to the timestamp, and recording the image data corresponding to each keyframe; The first modal feature extraction process includes: extracting visual feature vectors from the keyframe images, wherein the visual feature vectors are used to characterize the image regions associated with the annotations; The second modality feature extraction process includes: extracting audio feature vectors from the speech data in the annotation information, and extracting text feature vectors from the text data in the annotation information; The instruction time dataset generation process includes: aligning and marking the timeline of the target video to generate a dataset containing timestamp indices of each modality; After performing the above extraction and generation processes, the storage module saves the first modality feature set, the second modality feature set, and the instruction time dataset.

3. The method according to claim 2, characterized in that, The process of generating candidate recognition results includes: Based on the timestamp index in the instruction time dataset, the visual features in the first modal feature set are aligned across modal time with the audio features or text features in the second modal feature set; Single-modal recognition and multimodal fusion recognition are performed on the aligned multimodal features to generate multiple candidate recognition results. Each candidate recognition result contains a semantic label of a modality and its original confidence score. The candidate recognition results are organized into an initial list in chronological order to obtain a set of candidate recognition results.

4. The method according to claim 3, characterized in that, The process of semantic feature space projection processing includes: Each candidate recognition result in the candidate recognition result set is projected onto a trainable shared semantic feature space through a multimodal fusion encoder to obtain the semantic feature vector of the candidate recognition result; The multimodal fusion encoder includes a visual encoder, an audio encoder, and a text encoder set in parallel, as well as a shared fully connected layer connected after each encoder. The shared fully connected layer projects features from different modalities onto a semantic feature space of the same dimension.

5. The method according to claim 4, characterized in that, The process of constructing a feature semantic graph includes: Taking each candidate in the candidate recognition result set as a node, calculate the cosine similarity between every two nodes in the shared semantic feature space; Based on the cosine similarity, a differentiable k-nearest neighbor strategy is used to retain the top K neighbor nodes with the highest similarity for each node, and edges are constructed using the retained neighbor relationships to generate a dynamic feature semantic graph.

6. The method according to claim 5, characterized in that, The process of calculating and processing node vector representations includes: Multiple rounds of information propagation are performed on the feature semantic graph. In each round of propagation, a multi-head graph attention layer is used to calculate the attention coefficient of each neighbor node relative to the center node, and the attention weight is obtained by softmax normalization. Based on the attention weights, the semantic feature vectors of neighboring nodes are aggregated to obtain the aggregated context information; The aggregated context information and the current feature vector of the central node are input into a gated recursive unit. The gated recursive unit controls the fusion ratio of the information by updating the gate and resetting the gate, and outputs the updated node vector representation. After each round of propagation, residual connections and layer normalization are performed on the updated node vector representation to keep the network stable and preserve the initial features; After a preset number of rounds of information propagation, the vector representation of each node obtained at the end is used as the refined semantic feature of the candidate recognition result, and the corresponding global confidence score is calculated based on the refined semantic feature. The confidence score set is composed of the global confidence scores of all candidate identification results.

7. The method according to claim 6, characterized in that, The process of selecting and processing candidate identification results includes: Obtain the node vector representation and global confidence score of each candidate recognition result; The node vector representation is input into a lightweight global rescoring network, which outputs a global calibration confidence score, which is used to characterize the probability that the candidate recognition result is adopted in the current semantic context. Next, in the shared semantic feature space, a similarity-based non-maximum suppression operation is performed on the candidate recognition results: if the distance between the semantic feature vectors of two candidate recognition results is less than a preset threshold, only the one with the highest global calibration confidence is retained, thereby obtaining a subset of candidate recognition results after deduplication.

8. The method according to claim 7, characterized in that, The process of generating structured multimodal interaction instructions includes: For each selected result in the subset of deduplicated candidate recognition results, load a multimodal instruction template from the predefined instruction template library; The semantic labels, global calibration confidence scores, and contextual information from the associated node vector representations of the selected results are filled into the instruction template to generate a structured interactive instruction in JSON format. The structured interaction instructions include a voice interaction text field, an action identifier field, and a visual expression identifier field. The voice interaction text field is used to synthesize voice output, the action identifier field is used to trigger highlighting or selection actions on the screen, and the visual expression identifier field is used to generate emoticon icons displayed on the shared screen. Arrange the structured interactive instructions corresponding to multiple selected results in a time sequence to generate an instruction sequence with time delay parameters or synchronization trigger markers; The instruction sequence and its associated metadata are used together as annotation records.

9. The method according to claim 8, characterized in that, The process of determining the display order includes: Based on the time delay parameter or synchronization trigger flag carried by the instruction sequence in the annotation record, determine the appearance order and duration of each structured interactive instruction on the shared screen.

10. The method according to claim 9, characterized in that, The process of sharing screen display and generating meeting minutes includes: The shared screen display processing includes: executing the action identifiers and visual expression identifiers in the structured interaction instructions according to a determined display order, sequentially displaying the highlighted areas of the annotations and expression icons on the shared screen, and simultaneously synthesizing and playing speech based on the voice interaction text field; The meeting minutes generation process includes: combining the voice interaction text field, action identifier field, and corresponding timestamp in the structured interaction instructions into a text-formatted meeting minutes and annotation summary; Finally, the annotation results are output to the storage module or display terminal.

Images