A script-based record generation method and system based on cross-modal semantic reasoning

Abstract

The application discloses a script-style record generation method and system based on cross-modal semantic reasoning, comprising the following steps: S1, synchronously collecting and preprocessing audio stream, video stream and auxiliary text data in an interrogation scene, and outputting audio stream A'(t), video frame sequence V'(x, y, t) and formatted text T(n); S2, extracting audio stream A'(t) and video frame sequence V'(x, y, t) features in parallel, and outputting text sequence Text_Seq and visual feature sequence; S3, performing semantic conversion on the visual feature sequence to generate visual description sequence Vis_Seq described in natural language; and S4, performing time sequence alignment and semantic fusion on the text sequence Text_Seq and the visual description sequence Vis_Seq to generate a comprehensive record sequence T_comprehensive. The application generates script-style records with narrative logic and easy-to-understand, breaks the stereotyped mode of template generation, can clearly show the strategy adjustment of interrogation, the emotional change of the parties and the key turning point of the case, and makes the records more in line with the reading and cognitive habits of human beings.

Description

Technical Field

[0001] This invention belongs to the field of intelligent judicial assistance technology, and relates to a script-based transcript generation method and system based on cross-modal semantic reasoning. Background Technology

[0002] Intelligent record-keeping products are digital recording tools that integrate AI speech-to-text and natural language processing technologies. They can significantly improve the efficiency and standardization of record-keeping. In recent years, the application of intelligent record-keeping products in law enforcement and judicial fields has been continuously deepened and has made remarkable progress.

[0003] In-depth analysis of existing products and practical applications reveals that existing intelligent recording technologies lack core capabilities in areas such as cross-modal deep fusion, scripted narrative generation, deep semantic reasoning, and dynamic intelligent interaction. As a result, the recorded information they generate is dimensional, rigid, and lacks logical depth, making it difficult to meet the urgent needs of modern judicial and law enforcement activities for accurate case reconstruction, in-depth analysis, and strategic support. Summary of the Invention

[0004] In order to overcome at least one deficiency of the prior art, the present invention provides a script-based transcript generation method and system based on cross-modal semantic reasoning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a script-based transcript generation method based on cross-modal semantic reasoning, comprising the following steps: Step S1: Synchronously collect and preprocess audio streams, video streams, and auxiliary text data in the interrogation scenario, and output audio stream A'(t), video frame sequence V'(x,y,t), and formatted text T(n); Step S2: Extract features from audio stream A'(t) and video frame sequence V'(x,y,t) in parallel using dual tracks, and output text sequence Text_Seq and visual feature sequence; Step S3: Perform semantic transformation on the visual feature sequence to generate a visual description sequence Vis_Seq in natural language; Step S4: Perform temporal alignment and semantic fusion between the text sequence Text_Seq and the visual description sequence Vis_Seq to generate a fused transcript sequence T_comprehensive; Step S5: Based on the predefined script structure Tscript, map the content in the fusion transcript sequence T_comprehensive to the role, declaration, and action fields to generate a script-style transcript; Step S6: Based on the interactive time window function, responding to the user's operation on the semantic units in the script-style transcript, play back the corresponding original audio and video segments.

[0006] Furthermore, the synchronous acquisition and preprocessing method includes: Step S11: The system detects scene triggers and collects audio, video, and auxiliary text. Step S12: Synchronize the audio, video, and auxiliary text through time. The audio data is denoted as A(t), the video as V(x,y,t), and the auxiliary text as T(n), where n is the text segment index, and the collection duration is dynamically adjusted according to the scenario. Step S13: Separate the audio and video streams into independent tracks: audio stream A(t) and video frame sequence V(x,y,t); Step S14: Audio stream A(t) preprocessing: Analyze the spectrum through short-time Fourier transform, filter out non-verbal audio segments, and generate audio A'(t); Step S15: Video V(x,y,t) preprocessing: Perform video preprocessing using optical flow or Kalman filtering motion compensation to generate video V'(x,y,t); Step S16: Auxiliary text T(n) preprocessing: Format the auxiliary text data T(n) and align it with the audio and video; Step S17: Set the standard audio signal-to-noise ratio and video PSNR, compare the audio A'(t) and video V'(x,y,t) with the standard audio signal-to-noise ratio and video PSNR to verify data integrity. If the data is qualified, input it into step S2; otherwise, re-acquire or discard the segment.

[0007] Furthermore, the feature extraction method for the independent audio and video dual-track processing in step S2 is as follows: Step S21: Perform speaker separation and speech recognition on the audio stream A'(t) to generate a text sequence Text_Seq with time sequence and speaker labels; Step S22: Perform pose estimation, facial analysis, and environmental background feature extraction on the video frame sequence V'(x,y,t) to generate a visual feature sequence with temporal sequence and confidence level.

[0008] Furthermore, the voiceprint separation and speech recognition method in step S21 includes: Step S211: Voiceprint separation: Speaker separation is performed on audio A'(t) to identify multiple speakers; input audio segments, output audio data segments with speaker labels and time; Step S212: Speech Recognition: Transcribe the audio data segments of each speaker after separation, input the audio into the end-to-end ASR model, generate preliminary text, use audio-visual fusion ASR to enhance recognition in noisy scenes, add absolute timestamps: label each text segment with t_start and t_end, the timestamps are located by calling the VAD model to locate the start and end times of the speech segment, and the discrete timestamps are forcibly matched with the temporal information of the actual speech content; Step S213: Integrate audio stream features; The integrated result is the sequence Text_Seq = {(t_start, t_end, speaker, content)}, where speaker is the label and content is the transcribed text; Step S214: Post-process the audio stream speech recognition content using natural language processing (NLP) tools to remove filler words and correct errors.

[0009] Furthermore, the method for video frame sampling and key behavioral feature extraction of the video frame sequence in step S22 is as follows: Step S221: Video frame sampling: Sampling is based on Nyquist's theorem. Using OpenCV's frame extraction function, V'(x,y,t) is uniformly or adaptively sampled, and a frame sequence is generated based on motion detection to set the frame. Step S222: Pose estimation: Capture body and hand movements; The open-source multi-person pose estimation model is used to process the associated key points of the sampled frames, capture dynamic poses, extract human key points, and output the 2D / 3D joint coordinates of human pose with timestamps. Step S223: Facial analysis: Identify expression units and micro-expressions; The sampled frames were processed using an open-source face analysis model to extract feature vectors for facial action unit activation state, facial action unit AU intensity, basic expression, head pose estimation, and eye state detection. Step S224: Environmental Analysis: Identify key objects in the scene; The object detection model is used to detect and identify objects in the scene, and the detected and identified objects are classified, outputting the bounding boxes and label vectors of all detected and identified objects; Step S225: Integrate video stream features: Vectorize the extracted results into a key behavioral feature sequence, {(t_vis,pose_type&value&confidence, face_type&value&confidence, background_type&value&confidence)}; Use confidence filtering to remove noise and synchronize the feature sequence in time.

[0010] Furthermore, step S3 includes: Step S31: Filtering decision: Step S311: Apply a predefined threshold to each visual feature to filter out invalid jitter or noise, and use a rule engine to determine the validity of the feature; Step S312: Multi-feature aggregation: Cluster the relevant features using a clustering algorithm to group the events, calculate the composite threshold, and reduce redundancy; Step S313: Output intermediate sequence: Generate filtered video feature vector with timestamp t_vis; Step S32: Convert the video feature vector into a visual description sequence Vis_Seq based on the language-visual model.

[0011] Furthermore, the timing alignment and fusion method in step S4 includes: Step S41: Adaptive temporal mapping of time windows based on speech activity detection results; Step S42: Establish a semantic-level insertion mechanism. When the confidence of a description in the visual description sequence Vis_Seq is higher than the threshold X, and its semantic similarity with the current text window is higher than the threshold Y, insert the description in parenthetical notes after the most semantically relevant word segment in the text sequence. Step S43: Establish a rule base. When a negative semantic keyword appears in Text_Seq and a high-confidence description appears in the corresponding Vis_Seq, a contradiction marker is triggered. At the same time, LLM is used to perform deep semantic analysis on the context to assist in the judgment and detection of cross-modal contradictions. Step S44: Timing alignment and fusion output.

[0012] A script-based transcript generation system based on cross-modal semantic reasoning, including The transcript implementation module includes audio and video acquisition and preprocessing, which simultaneously acquires audio streams, video streams, and auxiliary text data from the interrogation scene, and supports start, stop, and pause operations for transcript recording. The dual-track parallel feature extraction module is connected to the audio and video acquisition and preprocessing module for recording. It is used to perform voiceprint separation and speech recognition on the audio stream to generate a text sequence with time sequence and speaker labels. It is also used to perform pose estimation, facial analysis and environmental background feature extraction on the video stream, integrate the feature extraction results of the video stream and quantize them into a key behavioral feature sequence. The video feature semanticization module, connected to the dual-track parallel feature extraction module, is used to apply a predefined threshold to the visual feature sequence to filter out invalid jitter or noise, use a rule engine to judge the validity of the features, and use a clustering algorithm to group the relevant features into events to generate a filtered list of timestamped t_vis vectors, which are then input into the VLM and output a sequence with timestamps and confidence levels in natural language description. The temporal alignment and fusion module, connected to the video feature semanticization module and the dual-track parallel feature extraction module, is used to perform temporal alignment and fusion of Vis_Seq = {(t_vis, description, confidence)} output by the video feature semanticization module and Text_Seq = {(t_start, t_end, speaker, content} output by the dual-track parallel feature extraction module, and output the fused transcript sequence T_comprehensive. The script-style structured definition and generation output module, connected to the time sequence alignment and fusion module, is used to process the fused transcript sequence through the LLM large language model and format it into a script-style transcript Tscript containing scenes, characters, content text, actions, or backgrounds. The script-based transcription operation module, based on an interactive time window function, along with the semantically defined W_callback callback function and the script-based transcription Tscript, is used to respond to user operations on the script-based transcription. It dynamically accesses video content through timestamp mapping and media API, enabling the playback of the corresponding original audio and video segments by calling the time window function based on the transcription.

[0013] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the script-based transcript generation system based on cross-modal semantic reasoning.

[0014] A computer program, characterized in that, when the computer program is executed by a processor, it is used to execute the script-based transcript generation system based on cross-modal semantic reasoning.

[0015] In summary, the advantages of this invention are: The transcript generated by this invention not only records "what was said" but also "the behavior and expression while speaking," achieving complete capture of multi-dimensional information from the interrogation scene. This significantly improves the semantic depth and fidelity of the transcript, overcomes the shortcomings of existing technologies with their single information dimension, enhances the completeness and semantic depth of the transcript information, and achieves "high-fidelity" recording of the interrogation process.

[0016] This invention generates script-style transcripts that have narrative logic and are easy to understand, breaking through the rigid pattern of template-based generation. It can clearly show the adjustment of questioning strategies, the emotional changes of the parties involved, and the key turning points of the case, making the transcripts more in line with human reading and cognitive habits.

[0017] This invention not only passively records information but also proactively performs preliminary logical consistency analysis, highlighting potential contradictions to interrogators or examiners, providing crucial technical support for judging the veracity of statements. Its intelligent analysis capabilities enhance the objectivity and credibility of the transcripts, providing intelligent assistance for review and judgment.

[0018] This invention upgrades static transcripts into dynamic, verifiable "hypertext," enabling any transcript conclusion to be traced back to the most original objective evidence. This greatly enhances the transparency of the judicial process and the strength of the evidence chain, thereby achieving traceability and verifiability of transcript content and forming a closed-loop audit capability.

[0019] This invention indirectly achieves intelligent assistance and dynamic adaptation in the interrogation process by providing in-depth real-time information and convenient verification tools. It helps interrogators grasp key points for in-depth questioning, fundamentally improving the efficiency and quality of case handling, thereby enhancing the system's intelligent interaction and adaptive capabilities, and optimizing user experience and case handling efficiency. Attached Figure Description

[0020] Figure 1 This is a flowchart of the script-based transcript generation method of the present invention.

[0021] Figure 2 This is a flowchart of the real-time audio and video acquisition and preprocessing process for recording according to the present invention.

[0022] Figure 3 This is a flowchart of the dual-track parallel feature extraction process of the present invention.

[0023] Figure 4 The flowchart illustrates the semantic transformation of the visual feature sequences in this invention.

[0024] Figure 5 This is a flowchart of the timing alignment and semantic fusion process of the present invention.

[0025] Figure 6 This is a flowchart illustrating the script-based structured definition and output generation process of this invention.

[0026] Figure 7 This is a flowchart of the script-based transcription operation based on the interactive time window function of the present invention.

[0027] Figure 8 This is a schematic diagram of the script-based transcript generation system of the present invention. Detailed Implementation

[0028] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0029] Example: like Figures 1-7 As shown, a script-based transcript generation method based on cross-modal semantic reasoning includes the following steps: Step S1: Synchronously collect and preprocess audio streams, video streams, and auxiliary text data in the interrogation scenario, and output audio stream A'(t), video frame sequence V'(x,y,t), and formatted text T(n); Step S2: Extract features from audio stream A'(t) and video frame sequence V'(x,y,t) in parallel using dual tracks, and output text sequence Text_Seq and visual feature sequence; Step S3: Perform semantic transformation on the visual feature sequence to generate a visual description sequence Vis_Seq in natural language; Step S4: Perform temporal alignment and semantic fusion between the text sequence Text_Seq and the visual description sequence Vis_Seq to generate a fused transcript sequence T_comprehensive; Step S5: Based on the predefined script structure Tscript, map the content in the fusion transcript sequence T_comprehensive to the role, declaration, and action fields to generate a script-style transcript; Step S6: Based on the interactive time window function, responding to the user's operation on the semantic units in the script-style transcript, play back the corresponding original audio and video segments.

[0030] Step S1 uses an integrated platform (such as an embedded system based on a Raspberry Pi or a dedicated server) to synchronously acquire audio and video. The synchronous acquisition and preprocessing methods include: Step S11: The system detects scene triggers and collects audio, video, and auxiliary text; The methods for collecting audio, video, and supplementary text are as follows: Step S111: Collect audio from the interrogation scene; The system employs a high-sensitivity microphone array or professional recording equipment to acquire speech signals from the scene. Multi-channel microphones are added to capture speaker audio, supporting sound source localization (distinguishing between the interrogator's and the subject's voices through array signal processing). The acquisition format is PCM or WAV, with a sampling rate of at least 44.1kHz and a bit depth of 16-24bit to ensure high fidelity. Step S112: Collect video footage from the interrogation scene; It uses a high-definition camera to record visual data, acquiring a spatiotemporal data stream V(x,y,t), where x and y are the frame pixel coordinates, and t is the timestamp (synchronized in milliseconds). The resolution is at least 1080p, the frame rate is 30-60fps, and it supports multi-angle coverage (such as front + side cameras) to capture non-verbal cues (such as facial expressions and gestures). Step S113: Collect auxiliary text; Introducing a manual text input device allows on-site personnel to input notes or keywords in real time. An identity recognition module can also be introduced to verify user identity ID_match and generate identity tags that are bound to the data stream to prevent data confusion. The auxiliary text is manually recorded text data, including notes, keywords, identity information, and other auxiliary data entered in real time.

[0031] Step S12: Synchronize the audio, video, and auxiliary text through time. The audio data is denoted as A(t), the video as V(x,y,t), and the auxiliary text as T(n), where n is the text segment index, and the collection duration is dynamically adjusted according to the scenario. Time synchronization is achieved through the NTP protocol or a hardware clock, ensuring that the timestamps of audio and video are aligned (error <50ms). Step S13: Separate the original audio and video stream (such as MP4 format) into independent tracks: audio stream A(t) and video frame sequence V(x,y,t); Audio and video separation uses FFmpeg or OpenCV libraries; Step S14: Audio stream A(t) preprocessing: Analyze the spectrum by short-time Fourier transform (STFT), filter out non-verbal audio segments, and generate clean audio A'(t); Step S15: Video V(x,y,t) preprocessing: Use optical flow or Kalman filtering motion compensation to preprocess the video, reduce jitter and blur, and generate an optimized video V'(x,y,t); Step S16: Preprocessing of auxiliary text T(n): Format the auxiliary text data (such as identity information and other auxiliary data) T(n) (such as removing redundancy and adding timestamps) and align it with audio and video; The auxiliary text T(n) can be aligned with the audio and video through keyword matching.

[0032] Step S17: Set the standard audio signal-to-noise ratio and video PSNR, compare the audio A'(t) and video V'(x,y,t) with the standard audio signal-to-noise ratio and video PSNR to verify data integrity. If the data is qualified, input it into step S2; otherwise, re-acquire or discard the segment. Step S2 is implemented using multi-threaded / parallel computing theory (such as PyTorch's DataParallel). Audio and video tracks are processed independently to reduce latency. The inputs are the preprocessed audio stream A'(t) and video frame sequence V'(x,y,t) from S1, and the outputs are Text_Seq and visual feature sequences (visual feature vectors). The feature extraction method for the independent processing of the audio and video tracks is as follows: Step S21: Perform speaker separation and speech recognition on the audio stream A'(t) to generate a text sequence Text_Seq with time sequence and speaker labels; The voiceprint separation and speech recognition method in step S21 includes Step S211: Diarization: Speaker separation is performed on audio A'(t), identifying multiple speakers (such as interrogators and subjects). Processing is done using tools including, but not limited to, Pyannote or Speaker (for transcript voice roles) and Diarization libraries (deep learning-based voice role separation libraries such as VBx or ECAPA-TDNN models). Input audio segments are divided into 5-10 second windows, and output audio data segments with speaker labels and time signatures. If synchronized video lip movement data is available, lip shape features can be fused to improve separation accuracy. Step S212: Audio Speech Recognition (ASR): Transcribe the audio data segments of each speaker after separation. Input the audio into an end-to-end ASR model (such as Whisper) to generate preliminary text. Enhance recognition in noisy scenes using audio-visual fusion ASR (such as LipNet or AVSR models). Add absolute timestamps: Label each text segment with t_start (start time) and t_end (end time). The timestamps are used to accurately locate the start and end times of the speech segments by calling VAD models (such as WebRTC VAD, Silero VAD, MediaPipe VAD), forcibly matching the discrete timestamps with the temporal information of the actual speech content; Step S213: Integrate audio stream features; The integrated result is the sequence Text_Seq = {(t_start, t_end, speaker (voice actor), content (content))}, where speaker (voice actor) is the label (such as "interrogator A" or "object X"), and content (content) is the transcribed text; Step S214: Post-process the audio stream speech recognition content using natural language processing (NLP) tools (such as spaCy) to remove filler words and correct errors.

[0033] Step S22: Perform pose estimation, facial analysis, and environmental background feature extraction on the video frame sequence V'(x,y,t) to generate a visual feature sequence with temporal sequence and confidence level; The method for video frame sampling and key behavior feature extraction in step S22 (video frame sequence (video track) is as follows: Step S221: Video frame sampling: Sampling is based on Nyquist's theorem. Using OpenCV's frame extraction function, V'(x,y,t) is sampled uniformly or adaptively. Based on motion detection, a frame sequence is generated at an adjustable rate of 5-10 frames per second (considering the recording scenario and the comprehensive consideration of motion feature capture and computational cost, generating a frame sequence every 5-10 seconds is the optimal time interval for engineering). Step S222: Pose estimation: Capture body and hand movements; Use open-source multi-person pose estimation models (such as the OpenPose model) to process the associated key points of the sampled frames, capture dynamic poses, extract human key points such as hand movements, head poses, and gaze offsets, and output 2D / 3D key point coordinates of human poses with timestamps. Step S223: Facial analysis: Identify expression units and micro-expressions; The sampled frames were processed using an open-source face analysis model (such as the OpenFace model) to extract feature vectors of facial action unit (AU) activation state, facial action unit AU intensity, basic expression, head pose estimation, and eye state detection. Step S224: Environmental Analysis: Identify key objects in the scene; Use an object detection model (such as YOLOv11 model) to detect and identify objects in the scene (such as items on a table or background elements), classify the detected and identified objects, and output the bounding boxes and label vectors of all detected and identified objects. Step S225: Integrate video stream features: Vectorize the extracted results into a key behavioral feature sequence, {(t_vis, pose_type&value&confidence, face_type&value&confidence, background_type&value&confidence)}; Use confidence (the confidence value of each feature extracted from pose estimation, facial unit analysis, and environmental background features is the standard output of its corresponding processing model) to filter (>0.8, based on the conventional output confidence threshold setting of specific models such as OpenPose and OpenFace) to remove noise and ensure that the feature sequence is synchronized in time (error <100ms, based on the conventional output setting of specific models such as OpenPose and OpenFace). In this example: t_vis is the frame time, pose_type="sitting posture"&value=normal&confidence(confidence)=0.95, face_type="nervous"&value=slight&confidence(confidence)=0.83, background_type="file"&value=desktop file&confidence(confidence)=0.91.

[0034] Step S3, the semantic transformation of the visual feature sequence, includes two sub-stages: decision filtering and semantic transformation. The input is the visual feature sequence from S2 (e.g., {(t_vis, pose_type&value&confidence(confidence), face_type&value&confidence(confidence), background_type&value&confidence(confidence))}), and the output is the visual description sequence Vis_Seq = {(t_vis, description(record description)(pose_type&value, face_type&value, background_type&value), confidence(confidence))}. Specifically, a hybrid framework of dedicated dictionary rules and VLM (language-visual model) is used, supporting real-time or post-processing, and includes the following steps: Step S31: Determine the filter: Step S311: Apply a predefined threshold to each visual feature to filter out invalid jitter or noise, and use a rule engine to determine the validity of the feature; For example, for head pose (Head_Yaw): if |yaw_angle| > 30° and duration > 1s, then retain it; otherwise, mark it as invalid and add a confidence check (confidence > 0.8). Step S312: Multi-feature aggregation: Use clustering algorithms (such as K-means) to group related features into event clusters (e.g., in this example, combining AU intensity and pose), calculate composite thresholds (e.g., expression intensity E_intensity > 0.5 and micro-expression change > threshold) to reduce redundancy; Step S313: Output intermediate sequence: Generate filtered video feature vector with timestamp t_vis; Step S32: Convert the video feature vector into a visual description sequence Vis_Seq based on the language-visual model; The filtered video feature vector with timestamp t_vis is input into the language-visual model LLaVA (VLM), which outputs a visual description sequence Vis_Seq with timestamp and confidence (inherited from S2): {(t_vis, description (transcription description), confidence (confidence))}, where confidence (confidence) is inherited or averaged.

[0035] Step S4, temporal alignment and fusion, is divided into three sub-stages: temporal mapping, insertion logic, and contradiction detection. The inputs are the visual description sequence Vis_Seq = {(t_vis, description (transcription description), confidence (confidence level))} from S3 and the text sequence Text_Seq = {(t_start, t_end, speaker (transcription voice role), content (content))} from S2. The output is the fused transcript sequence T_comprehensive. Specifically, an algorithm-driven framework (such as Python's time series library) is used to support real-time fusion. The temporal alignment and fusion method in step S4 includes... Step S41: Adaptive temporal mapping of time windows based on the results of Voice Activity Detection (VAD); Window Definition and Mapping: Define a sliding time window W (range 1-5 seconds, adaptive based on event density), traverse each text segment of the text sequence Text_Seq, map the Vis_Seq events to its timeline, for each t_vis, find overlapping or adjacent (t_start, t_end), calculate the intersection ratio IoT (Intersection overTime) = (min(t_end, t_vis_end) - max(t_start, t_vis_start)) / (t_end - t_start), set a threshold of 0.5, optimize based on experiments, when IoT > 0.5, the mapping is successful; Pause detection: Use the VAD model to identify pause gaps between Text_Seq. If there is no speech for more than 0.5 seconds, directly insert the Vis_Seq description at the end of the gap; Pre-processing: If t_vis is within the speech segment, calculate the semantic relevance S_rel = cosine_similarity(embed(description(transcription description)), embed(content(content))), and embed the vector using BERT or Sentence-BERT. If S_rel > 0.7, then prepare for embedding.

[0036] Step S42: Establish a semantic-level insertion mechanism. When the confidence of a description in the visual description sequence Vis_Seq is higher than the threshold X (e.g., 0.8), and its semantic similarity with the current text window (cosine similarity calculated by Sentence-BERT) is higher than the threshold Y (e.g., 0.9), insert the description in parenthetical notes after the most semantically relevant word segment in the text sequence. Word-level embedding: Perform NLP (Natural Language Processing) word segmentation on the content of Text_Seq (example: using jieba or spaCy), and precisely insert it after the corresponding word based on the relative position of t_vis (normalized_pos = (t_vis - t_start) / (t_end - t_start)) (example: if pos=0.5, embed it in the middle of the sentence), and convert it to dynamic. For example, embed "[avoiding eye contact]" as a side note into "I...[avoiding eye contact]...really don't know".

[0037] Insertion rules: For sentences that are not simple beginnings or ends, use a dependency parse tree (dependency tree is a core tool for syntactic analysis in natural language processing (NLP)) to determine the insertion point and ensure grammatical naturalness (example: after inserting the predicate). If confidence is less than 0.9, add an uncertainty marker such as "[possible: eye movement]".

[0038] Step S43: Establish a rule base. When negative semantic keywords such as 'deny' and 'no' appear in Text_Seq, and high-confidence descriptions such as 'nodding' and 'leaning forward' (usually associated with affirmation) appear in the corresponding Vis_Seq, a contradiction marker is triggered. At the same time, LLM can be used to perform deep semantic analysis on the context to assist in the judgment and detection of cross-modal contradictions. Logical judgment module: Threshold triggers activation of built-in judgment rule base + LLM check mutual exclusion only when S_rel > 0.6 and confidence > 0.8. Extract the semantics of Text_Seq (e.g., "negative" = "no") and match them with the descriptions of Vis_Seq ("nodding" = "affirmative"). If they are mutually exclusive ("negative" vs. "nodding" in the rule table scores -1), generate a highlight mark "[Warning: Body Semantic Conflict]" and output it to the transcript as a footnote; Step S44: Timing alignment and fusion output; After three steps—temporal mapping, semantic-level insertion, and contradiction detection—Vis_Seq = {(t_vis, description (transcription description), confidence (confidence level))} from step S3, Text_Seq = {(t_start, t_end, speaker (transcription voice role), content (content))} from step S2, and the formatted text (such as auxiliary data like identity information) T(n) from step S1 are merged to output the fused transcript sequence T_comprehensive.

[0039] Step S5 is divided into three sub-stages: structure definition, role transformation, and action insertion. The input is T_comprehensive (fusion sequence, including formatted text T(n), video stream features t_vis, and text sequence Text_Seq) from step S4, and the output is a Tscript structured text file or data object, specifically including... Step S51: Define the script structure (Tscript); Define the structure specification: Define Tscript as a hierarchical JSON or Markdown format to ensure that the narrative requirements are met. The script structure is Tscript = {scenes: [{role: string, statement: string, actions: array, timing: {t_start, t_end}}]}, where scenes are dynamic temporal segments, ensuring that each scene contains a character, statement, and action; Define a validation mechanism: Use schema validation (such as JSON Schema) to ensure structural integrity, and automatically fill in default values ​​if elements are missing (such as no roles).

[0040] Step S52: Convert narrative character tags; Identity to Label Mapping: Automatically convert ID_match (identity information inherited from S1 hand-formatted text (such as auxiliary data like identity information) T(n)) of T_comprehensive into canonical labels. Generate labels using a preset mapping table or LLM hints (e.g., "convert ID 'user_001' to an interrogation role, such as [Interrogator A]"). If ID_match is "officer", convert it to "[Interrogator A]"; if it is "suspect", convert it to "[Suspect X]". If there are multiple people in the scene, sort the tags based on the order of speaking (e.g., A / B / C) to ensure narrative coherence and form a multi-role tagging system based on the order of speaking; Step S53: Process and format the script-style transcript text based on the large language model; The system maps identities to tags and multi-role tags based on speaking order to roles in Tscript, auxiliary data of formatted text T(n) to scenes in Tscript, text sequences Text_Seq from T_comprehensive to statement: string in Tscript, and video stream features t_vis from T_comprehensive to actions: array in Tscript. Finally, it uses a Large Language Model (LLM) for polishing and formatting, outputting a script-style transcript text Tscript = {scenes: [{role: string, statement: string, actions: array: timing: {t_start, t_end}}]}.

[0041] The implementation of script-based note-taking operations based on interactive time window functions consists of function definition, click-triggered playback, and other note-taking operations. The inputs are the Tscript (script structure, including semantic units nj such as the note text) from step S5 and the video frame sequence V'(x,y,t). Video content is dynamically accessed through timestamp mapping and media APIs to achieve interactive access to the original video (such as video playback) based on the note-taking function call. Specific methods are as follows: Step S61: Define the interactive time window function W_callback(nj); Define the function signature and parameters: Define W_callback as the callback function, and nj as the semantic unit identifier in the transcript (corresponding to the scene_id of the Tscript in step S5). The function extracts the Text_Seq timing associated with nj (t_start, t_end, inherited from step S2 and directly contained in the Tscript). Delta t is preset to 5-10 seconds (configurable), and the extended range [t_start - Delta t, t_end + Delta t] is calculated.

[0042] Synchronous mapping: The function queries the Text_Seq time sequence and Tscript metadata corresponding to W_callback (nj), and uses a time index hash table to quickly locate the mapping nj to the video timeline; To ensure real-time performance: functions asynchronously call the media API, and buffer extended segments to reduce latency (<500ms); Step S62: Click to trigger video playback; Step S621: User Interaction Binding: In the note-taking interaction, bind an event listener (e.g., onClick) to the nj (e.g., clickable text segments in the note data). When clicked, call W_callback. The function sends an HTTP Range request to the server via the media API (e.g., HTML5 tags or the Video.js library) to retrieve t_start - Deltat, plays up to t_end + Deltat, and when playback ends, the function pauses and releases the buffer to avoid memory leaks. Errors such as a missing file fallback to prompting the user. Step S622: Contextual backtracking enhancement: Overlay subtitles (retrieved from Text_Seq content) and highlight (retrieved from T_comprehensive, such as contradiction markers) during playback, emphasizing synchronization (video frame rate matches timestamp).

[0043] Step S623: Error handling: If nj has no timing, the function falls back to the default full video playback.

[0044] Step S63: Other recording operations based on the interactive time window function; Step S631: Semantic Unit Highlighting and Navigation: Define the W_nav(nj) function. When nj is clicked, not only is the video played back, but related preceding and following units are also highlighted in the transcript. NLP similarity navigation is used to improve the contextual browsing of the transcript and avoid isolated viewing. Step S632: Contradiction Marker Query: Define the W_query(nj) function. When a contradiction is highlighted, the function retrieves the mutual exclusion details in S4 and displays a side panel showing the semantics of Text_Seq versus the description of Vis_Seq, thus supporting evidence auditing and enhancing objective verification.

[0045] Step S633: Time Window Adjustment and Export: Extend the W_adjust(nj, new_delta) function to allow users to dynamically modify Delta t (e.g., in a slider UI), recalculate the range, and export the segment (for MP4 and other video clips). This enables user-defined backtracking, suitable for long interrogation analysis.

[0046] This application constructs a complete technology chain from multimodal data acquisition to script-based structured transcript generation and interaction, with its core lying in cross-modal semantic-level reasoning and deep fusion. Specifically: (1) Temporal alignment and semantic fusion mechanism of cross-modal features This application uses a time window algorithm and semantic relevance calculation to precisely insert visual behavioral features (such as "eye evasion") into the dialogue text at the word segmentation level (such as "I... [eye evasion]... really don't know"), achieving seamless semantic integration rather than mechanical splicing.

[0047] (2) A video feature semanticization method based on rule-based and VLM / LLM hybrid approach The extracted visual feature vectors (such as AU intensity and key point coordinates) are subjected to action threshold filtering and multi-feature aggregation, and then combined with a visual language model to transform them into natural language descriptions that conform to the judicial context (such as transforming "Head_Yaw=35°" into "turn head to look to the right"), laying the foundation for subsequent semantic reasoning and script generation.

[0048] (3) Script-based structured output definition and dynamic generation logic A dedicated script structure (Tscript) is defined, the core of which is a sequence of triples {role, statement, actions}. This structure can not only clearly distinguish character dialogue, but also dynamically embed non-verbal behaviors as "actions", thereby constructing a transcript with narrative logic (cause, process, climax, and turning point), breaking through the rigid generation mode of traditional template "fill-in-the-blank".

[0049] (4) Interactive time window function that supports evidence backtracking By defining interactive functions such as W_callback(nj), each semantic unit in the transcript is precisely bound to the timestamp of the original audio and video stream. When a user clicks on any content in the transcript, the corresponding time period of audio and video playback is triggered, realizing closed-loop verification and deep auditing of "text-visual-auditory" and greatly improving the auditability and credibility of the transcript.

[0050] like Figure 8 As shown, this application also provides a script-based transcript generation system based on cross-modal semantic reasoning, including... The transcript implementation audio and video acquisition and preprocessing module is used to simultaneously acquire audio streams, video streams, and auxiliary text data in the interrogation scene, and also supports start, stop, and pause operations for transcript recording; The dual-track parallel feature extraction module is connected to the audio and video acquisition and preprocessing module for recording. It is used to perform voiceprint separation and speech recognition on the audio stream to generate a text sequence with time sequence and speaker labels. It is also used to perform pose estimation, facial analysis and environmental background feature extraction on the video stream, integrate the feature extraction results of the video stream and quantize them into a key behavioral feature sequence. The video feature semanticization module, connected to the dual-track parallel feature extraction module, applies a predefined threshold to the visual feature sequence to filter out invalid jitter or noise, uses a rule engine to determine the validity of the features, and uses a clustering algorithm (such as K-means) to group the relevant features into events to generate a filtered list of timestamped t_vis vectors. This list is then input into the VLM (Language-Vision Large Model LLaVA) and outputs a sequence with timestamps and confidence (inherited from S2) in natural language description. The temporal alignment and fusion module, connected to the video feature semanticization module and the dual-track parallel feature extraction module, is used to perform temporal alignment and fusion of Vis_Seq = {(t_vis, description (transcription description), confidence (confidence))} output by the video feature semanticization module and Text_Seq = {(t_start, t_end, speaker (transcription voice role), content (content))} output by the dual-track parallel feature extraction module, and output the fused transcript sequence T_comprehensive; The script-style structured definition and generation output module, connected to the time sequence alignment and fusion module, is used to process the fused transcript sequence through the LLM large language model and format it into a script-style transcript Tscript containing scenes, characters, content text, actions, or backgrounds. The script-based transcription operation module, based on an interactive time window function, along with the semantically defined W_callback callback function and the script-based transcription Tscript, is used to respond to user operations on the script-based transcription. It dynamically accesses video content through timestamp mapping and media API, enabling the playback of the corresponding original audio and video segments by calling the time window function based on the transcription.

[0051] The embodiments use different interrogation scenarios, employing existing intelligent transcript technology and the script-based transcript generation method and system based on cross-modal semantic reasoning of this application as intelligent judicial assistance technologies to test the accuracy of interrogations.

[0052] The interrogation scene is a fraud case interrogation scene: Scene description: The interrogator asked the suspect if he was involved in fraud, and the suspect answered "no," but the video shows him nodding and avoiding eye contact.

[0053] Input: S1 captures audio and video streams (suspect's voice + facial video) and manual notes ("suspect's abnormal expression").

[0054] Process: S2 Dual-track extraction (audio ASR generates Text_Seq "No", video OpenPose / AU extracts Head_Yaw=35°, AU intensity>0.5); S3 Semanticization "[Eye evasion] [Nod]"; S4 Temporal fusion embedding "I... [Eye evasion]... really [Nod] no", detect contradictions and generate "[Warning: Body semantic conflict]"; S5 Generate Tscript "[Suspect X]: I... [Eye evasion]... really [Nod] no [Warning: Body semantic conflict]"; S6 Click nj to trigger playback (Delta t=5s, front and back expansion to verify body gradation).

[0055] Output: Script-style transcript report, improving objectivity by 25%, with interactive backtracking to confirm clues of lying.

[0056] Results: The system achieved an accuracy rate of 95% and reduced manual review time by 30%.

[0057] The interrogation scene is a theft case interrogation scene: Scene description: The suspect described himself as "shopping in a supermarket at the time," and the video shows unusual hand movements (hiding items).

[0058] Input: S1 captures high-definition video (hand gestures) and audio (description speech).

[0059] Process: S2 extracts Text_Seq "shopping", video YOLO detects "[hands hidden]"; S3 semantically adds "[body leaning forward, hand gestures occlusion]"; S4 embeds "shopping in the supermarket at the time [body leaning forward, hand gestures occlusion]", without contradiction; S5 Tscript "[Suspect X]: shopping in the supermarket at the time (body leaning forward, hand gestures occlusion)"; S6 click to trigger video playback, overlay AR heatmap to highlight hands.

[0060] Output: Complete script transcript, incorporating non-verbal evidence, increasing the integrity of the evidence chain by 20%.

[0061] Results: Real-time interactive verification reduced the false positive rate by 15%.

[0062] The interrogation took place in a traffic accident reporting scenario. Scene description: The person who reported the incident stated "vehicle collision, no injury", and the video shows the head posture (head down to avoid the camera).

[0063] Input: S1 collects audio and video of the police report.

[0064] Process: S2 ASR "No injury", video OpenPose extracts "[head down]"; S3 "[avoiding eye contact]"; S4 embeds "vehicle collision, no [avoiding eye contact] injury", detects contradiction "[warning: may conceal injury]"; S5 Tscript "[reporter A]: vehicle collision, no (avoiding eye contact) injury [warning: may conceal injury]"; S6 replay extended segment to confirm limb semantics.

[0065] Output: Script-style transcripts support police decision-making, improving efficiency by 35%.

[0066] Results: Accurate cross-modal reasoning, suitable for extended scenarios (such as meeting minutes).

[0067] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A script-based transcript generation method based on cross-modal semantic reasoning, characterized in that: Includes the following steps: Step S1: Synchronously collect and preprocess audio streams, video streams, and auxiliary text data in the interrogation scenario, and output audio stream A'(t), video frame sequence V'(x,y,t), and formatted text T(n); Step S2: Extract features from audio stream A'(t) and video frame sequence V'(x,y,t) in parallel using dual tracks, and output text sequence Text_Seq and visual feature sequence; Step S3: Perform semantic transformation on the visual feature sequence to generate a visual description sequence Vis_Seq in natural language; Step S4: Perform temporal alignment and semantic fusion between the text sequence Text_Seq and the visual description sequence Vis_Seq to generate a fused transcript sequence T_comprehensive; Step S5: Based on the predefined script structure Tscript, map the content in the fusion transcript sequence T_comprehensive to the role, declaration, and action fields to generate a script-style transcript; Step S6: Based on the interactive time window function, responding to the user's operation on the semantic units in the script-style transcript, play back the corresponding original audio and video segments.

2. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 1, characterized in that: The synchronous acquisition and preprocessing method includes: Step S11: The system detects scene triggers and collects audio, video, and auxiliary text. Step S12: Synchronize the audio, video, and auxiliary text through time. The audio data is denoted as A(t), the video as V(x,y,t), and the auxiliary text as T(n), where n is the text segment index, and the collection duration is dynamically adjusted according to the scenario. Step S13: Separate the audio and video streams into independent tracks: audio stream A(t) and video frame sequence V(x,y,t); Step S14: Audio stream A(t) preprocessing: Analyze the spectrum through short-time Fourier transform, filter out non-verbal audio segments, and generate audio A'(t); Step S15: Video V(x,y,t) preprocessing: Perform video preprocessing using optical flow or Kalman filtering motion compensation to generate video V'(x,y,t); Step S16: Auxiliary text T(n) preprocessing: Format the auxiliary text data T(n) and align it with the audio and video; Step S17: Set the standard audio signal-to-noise ratio and video PSNR, compare the audio A'(t) and video V'(x,y,t) with the standard audio signal-to-noise ratio and video PSNR to verify data integrity. If the data is qualified, input it into step S2; otherwise, re-acquire or discard the segment.

3. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 1, characterized in that: The feature extraction method for the independent audio and video dual-track processing in step S2 is as follows: Step S21: Perform speaker separation and speech recognition on the audio stream A'(t) to generate a text sequence Text_Seq with time sequence and speaker labels; Step S22: Perform pose estimation, facial analysis, and environmental background feature extraction on the video frame sequence V'(x,y,t) to generate a visual feature sequence with temporal sequence and confidence level.

4. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 3, characterized in that: The voiceprint separation and speech recognition method in step S21 includes Step S211: Voiceprint separation: Speaker separation is performed on audio A'(t) to identify multiple speakers; input audio segments, output audio data segments with speaker labels and time; Step S212: Speech Recognition: Transcribe the audio data segments of each speaker after separation, input the audio into the end-to-end ASR model, generate preliminary text, use audio-visual fusion ASR to enhance recognition in noisy scenes, add absolute timestamps: label each text segment with t_start and t_end, the timestamps are located by calling the VAD model to locate the start and end times of the speech segment, and the discrete timestamps are forcibly matched with the temporal information of the actual speech content; Step S213: Integrate audio stream features; The integrated result is the sequence Text_Seq = {(t_start, t_end, speaker, content)}, where speaker is the label and content is the transcribed text; Step S214: Post-process the audio stream speech recognition content using natural language processing (NLP) tools to remove filler words and correct errors.

5. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 3, characterized in that: The video frame sampling and key behavior feature extraction method for the video frame sequence in step S22 is as follows: Step S221: Video frame sampling: Sampling is based on Nyquist's theorem. Using OpenCV's frame extraction function, V'(x,y,t) is uniformly or adaptively sampled, and a frame sequence is generated based on motion detection to set the frame. Step S222: Pose estimation: Capture body and hand movements; The open-source multi-person pose estimation model is used to process the associated key points of the sampled frames, capture dynamic poses, extract human key points, and output the 2D / 3D joint coordinates of human pose with timestamps. Step S223: Facial analysis: Identify expression units and micro-expressions; The sampled frames were processed using an open-source face analysis model to extract feature vectors for facial action unit activation state, facial action unit AU intensity, basic expression, head pose estimation, and eye state detection. Step S224: Environmental Analysis: Identify key objects in the scene; The object detection model is used to detect and identify objects in the scene, and the detected and identified objects are classified, outputting the bounding boxes and label vectors of all detected and identified objects; Step S225: Integrate video stream features: Vectorize the extracted results into a key behavioral feature sequence, {(t_vis, pose_type&value&confidence, face_type&value&confidence, background_type&value&confidence)}; Use confidence filtering to remove noise and synchronize the feature sequence in time.

6. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 1, characterized in that: Step S3 includes: Step S31: Determine the filter: Step S311: Apply a predefined threshold to each visual feature to filter out invalid jitter or noise, and use a rule engine to determine the validity of the feature; Step S312: Multi-feature aggregation: Cluster the relevant features using a clustering algorithm to group the events, calculate the composite threshold, and reduce redundancy; Step S313: Output intermediate sequence: Generate filtered video feature vector with timestamp t_vis; Step S32: Convert the video feature vector into a visual description sequence Vis_Seq based on the language-visual model.

7. The script-based transcript generation method based on cross-modal semantic reasoning according to claim 1, characterized in that: The timing alignment and fusion method in step S4 includes: Step S41: Adaptive temporal mapping of time windows based on speech activity detection results; Step S42: Establish a semantic-level insertion mechanism. When the confidence of a description in the visual description sequence Vis_Seq is higher than the threshold X, and its semantic similarity with the current text window is higher than the threshold Y, insert the description in parenthetical notes after the most semantically relevant word segment in the text sequence. Step S43: Establish a rule base. When a negative semantic keyword appears in Text_Seq and a high-confidence description appears in the corresponding Vis_Seq, a contradiction marker is triggered. At the same time, LLM is used to perform deep semantic analysis on the context to assist in the judgment and detection of cross-modal contradictions. Step S44: Timing alignment and fusion output.

8. A script-based transcript generation system based on cross-modal semantic reasoning, characterized in that: include The transcript implementation module includes audio and video acquisition and preprocessing, which simultaneously acquires audio streams, video streams, and auxiliary text data from the interrogation scene, and supports start, stop, and pause operations for transcript recording. The dual-track parallel feature extraction module is connected to the audio and video acquisition and preprocessing module for recording. It is used to perform voiceprint separation and speech recognition on the audio stream to generate a text sequence with time sequence and speaker labels. It is also used to perform pose estimation, facial analysis and environmental background feature extraction on the video stream, integrate the feature extraction results of the video stream and quantize them into a key behavioral feature sequence. The video feature semanticization module, connected to the dual-track parallel feature extraction module, is used to apply a predefined threshold to the visual feature sequence to filter out invalid jitter or noise, use a rule engine to judge the validity of the features, and use a clustering algorithm to group the relevant features into events to generate a filtered list of timestamped t_vis vectors, which are then input into the VLM and output a sequence with timestamps and confidence levels in natural language description. The temporal alignment and fusion module, connected to the video feature semanticization module and the dual-track parallel feature extraction module, is used to perform temporal alignment and fusion of Vis_Seq = {(t_vis, description, confidence)} output by the video feature semanticization module and Text_Seq = {(t_start, t_end, speaker, content} output by the dual-track parallel feature extraction module, and output the fused transcript sequence T_comprehensive. The script-style structured definition and generation output module, connected to the time sequence alignment and fusion module, is used to process the fused transcript sequence through the LLM large language model and format it into a script-style transcript Tscript containing scenes, characters, content text, actions, or backgrounds. The script-based transcription operation module, based on an interactive time window function, along with the semantically defined W_callback callback function and the script-based transcription Tscript, is used to respond to user operations on the script-based transcription. It dynamically accesses video content through timestamp mapping and media API, enabling the playback of the corresponding original audio and video segments by calling the time window function based on the transcription.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the script-based transcript generation system based on cross-modal semantic reasoning as described in any one of claims 1-7.

10. A computer program, characterized in that, When the computer program is executed by a processor, it is used to perform the script-based transcript generation system based on cross-modal semantic reasoning as described in any one of claims 1-7.

Images