Multi-mode intelligent semantic understanding and abstract generation system and method based on HDMI (High Definition Multimedia Interface) stream

Abstract

The invention provides a multi-mode intelligent semantic comprehension and abstract generation system and method based on an HDMI stream, and the system comprises an HDMI input module which is used for receiving a video signal outputted by external equipment; the image content streaming analysis module is configured to perform content segmentation, optical character recognition, layout structure extraction and graphic element detection on the video signals and output structured image data; the audio acquisition module is used for acquiring an audio signal and preprocessing the audio signal; the voice recognition module is used for transferring the preprocessed audio signal into a voice text; and the multi-modal fusion and semantic understanding module is configured to fuse the structured image data and the voice text, generate summary information and output the summary information to the result output module. According to the invention, real-time synchronous analysis of the multi-modal content and intelligent generation of the abstract are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia signal processing and artificial intelligence edge computing, in particular to a multi-modal intelligent semantic understanding and summary generation system and method based on HDMI stream. BACKGROUND

[0002] The existing multimedia content collection relies on application programs at the software level of the operating system. Users need to install various application software such as screen recording software, audio collection software, and document conversion tools on the front-end device. These application software often requires administrator permission and is difficult to deploy in enterprise environments or educational institutions. At the same time, compatibility problems of different operating system platforms can lead to high device adaptation costs and make it difficult to achieve cross-platform unified deployment.

[0003] In addition, traditional multimedia signal processing schemes usually scatter video recording, audio collection, text recognition, and semantic analysis functions in different software modules, and lack effective timing synchronization mechanisms between modules, resulting in low multi-modal information fusion accuracy. In particular, in real-time scenarios such as meetings, speeches, and teaching, the delay of content understanding seriously affects user experience. SUMMARY

[0004] To solve the defects of the prior art, the purpose of the present application is to provide a multi-modal intelligent semantic understanding and summary generation system and method based on HDMI stream, which realizes real-time collection, synchronous analysis, and intelligent generation of summaries of multi-modal content.

[0005] To achieve the above purpose, the present application provides a multi-modal intelligent semantic understanding and summary generation system based on HDMI stream, comprising: An HDMI input module for receiving a video signal output by an external device; An image content stream analysis module configured to perform content segmentation, optical character recognition, layout structure extraction, and graphic element detection on the video signal, and output structured image data; An audio collection module for collecting an audio signal and pre-processing the audio signal; A speech recognition module for transcribing the pre-processed audio signal into speech text; A multi-modal fusion and semantic understanding module configured to fuse the structured image data and the speech text, generate summary information, and output the summary information to a result output module.

[0006] Further, the structured image data includes optical character recognition text information, table structure information, layout information, and graphic element description.

[0007] Furthermore, the image content streaming analysis module adopts a data streaming convolutional neural network structure, including: a feature extraction layer, a layout structure perception layer, and a content classification layer; The feature extraction layer uses residual connections and attention mechanisms to extract multi-scale feature representations from the image, resulting in a multi-scale feature map. The layout structure perception layer automatically identifies different types of content regions from the multi-scale feature map by using a spatial relationship model between page elements established based on a graph neural network. The content classification layer is used to combine contextual information to accurately classify, locate boundaries, and identify content of different types of content regions, thereby obtaining structured image data.

[0008] Furthermore, the content classification layer includes: An optical character recognition unit is used to recognize the title area and paragraph area and generate the optical character recognition text information; The table parsing unit is used to parse the table area to obtain the table structure information; The layout structure recognition unit is used to identify the layout structure area and obtain the layout information. The graphic element detection unit is used to extract key elements from the graphic and generate graphic element descriptions.

[0009] Furthermore, the layout structure recognition unit uses a document tree parsing method to recognize the layout structure; the layout structure includes hierarchical headings, paragraph organization, table boundaries, and mixed text and image layout.

[0010] Furthermore, the audio acquisition module synchronously acquires audio signals using a circular or linear array of multiple omnidirectional microphones, and outputs the audio signals to the speech recognition module after preprocessing them using a beamforming algorithm.

[0011] Furthermore, the beamforming algorithm is either a minimum variance distortionless response algorithm or a generalized sidelobe canceller algorithm, which enables directional acquisition of the speaker's voice and suppression of background noise.

[0012] Furthermore, the speech recognition module uses an end-to-end deep neural network to recognize the preprocessed audio signal and performs real-time speech transcription through streaming decoding to generate speech text.

[0013] Furthermore, the multimodal fusion and semantic understanding module employs a cross-modal attention mechanism to semantically align the structured image data and the speech text, and uses a miniature large language model to understand the context and generate a summary.

[0014] Furthermore, the multimodal fusion and semantic understanding module includes: The cross-modal alignment layer establishes the semantic correspondence between optical character recognition text information, page layout information and speech text in the structured image data through the learned attention matrix, and automatically matches the temporal synchronization relationship between image content and speech content using a temporal alignment algorithm. The semantic fusion layer, a deep learning model based on attention mechanisms, maps feature representations from different modalities to a unified semantic space; it captures intra-modal and inter-modal dependencies through self-attention and cross-attention mechanisms; and it employs residual connections and layer normalization to ensure stable training and inference of deep networks. The summary generation layer generates summary information based on the semantic space and the dependency relationships, using a miniature large language model that has undergone parameter pruning and quantization optimization.

[0015] Furthermore, the result output module includes: An HDMI output interface is used to output the original or processed video signal to a display terminal. The USB interface is used to export data to external devices; The network transmission module is used to provide remote data transmission and access.

[0016] Furthermore, it also includes a control processing unit for coordinating and controlling the start / stop, data routing, and resource allocation of the HDMI input module, the image content streaming analysis module, the audio acquisition module, the multimodal fusion and semantic understanding module, and the result output module; the control processing unit embeds an AI acceleration unit to improve processing speed.

[0017] To achieve the above objectives, this application also provides a method for multimodal intelligent semantic understanding and summarization based on HDMI streams, employing the multimodal intelligent semantic understanding and summarization system based on HDMI streams as described above, including the following steps: Receive video signals from external devices via the HDMI input module; The processed original video signal is subjected to content segmentation, optical character recognition, layout structure extraction, and graphic element detection to obtain structured image data; The audio signals collected by the microphone array are preprocessed and transcribed to obtain speech text; By utilizing a cross-modal attention mechanism and a miniature large language model, the structured image data and the speech text are fused to generate summary information.

[0018] The multimodal intelligent semantic understanding and summarization system based on HDMI stream provided in this application acquires video and audio signals through an HDMI input module and an audio acquisition module, and then realizes real-time synchronous analysis of multimodal content through a multimodal fusion and semantic understanding module, thereby improving the efficiency and accuracy of multimodal content acquisition, understanding and result generation.

[0019] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of a multimodal intelligent semantic understanding and summarization system based on HDMI stream according to Embodiment 1 of this application; Figure 2 This is a schematic diagram of the processing flow of the image content streaming analysis module according to Embodiment 1 of this application; Figure 3 This is a schematic diagram of the processing flow of the audio acquisition module and the speech recognition module according to Embodiment 1 of this application; Figure 4 This is a schematic diagram of the processing flow of the multimodal fusion and semantic understanding module according to Embodiment 1 of this application; Figure 5 This is a schematic diagram of the hardware system architecture and module connection relationship according to Embodiment 1 of this application; Figure 6 This is a schematic diagram of the AI ​​inference acceleration and optimization strategy according to Embodiment 1 of this application; Figure 7 This is a schematic diagram illustrating the application scenario and system integration according to Embodiment 1 of this application; Figure 8 This is a schematic diagram illustrating the application of this application in an educational and teaching scenario according to Embodiment 2. Figure 9 This is a flowchart of a multimodal intelligent semantic understanding and summarization method based on HDMI stream according to Embodiment 3 of this application. Detailed Implementation

[0021] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0022] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0023] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments".

[0024] It should be noted that the terms "first" and "second" may be used in this application only to distinguish different devices, components or parts, and are not used to define the order of functions performed by these devices, components or parts or their interdependence.

[0025] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "More" should be understood as two or more.

[0026] The following terms may be used in the embodiments of this application: Streaming CNN: A convolutional neural network architecture optimized for video streams; Document structure awareness: an algorithm that can understand the layout and hierarchical structure of a document; Transformer: A deep learning model that incorporates an attention mechanism; Conformer: A speech recognition network architecture that combines CNN and Transformer; Transformer-Transducer: An architecture that unifies streaming and non-streaming speech recognition models into a single model; Miniature LLM: A lightweight, compressed, and optimized large language model; Parametric pruning: an optimization technique that reduces model size by removing redundant parameters; Model quantization: a technique to reduce the numerical precision of a model in order to improve inference efficiency; NPU: Neural Processing Unit, dedicated to accelerating AI inference; CTC Decoding: Connection-oriented temporal classification decoding, used for sequence labeling tasks.

[0027] Definitions for other terms will be provided in the following description.

[0028] Example 1 Figure 1 The following is a schematic diagram of the multimodal intelligent semantic understanding and summarization system based on HDMI stream according to Embodiment 1 of this application. Figure 1 The specific implementation methods of this application are described in detail.

[0029] The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to this application embodiment includes an HDMI input module 101, an image content streaming analysis module 102, an audio acquisition module 103, a speech recognition module 104, a multimodal fusion and semantic understanding module 105, a control processing unit 106, and a result output module 107. Each module will be described in detail below.

[0030] The HDMI input module 101 acquires and processes HDMI signals to receive video signals output from external devices. The HDMI input module 101 is used to receive high-definition video signals from external devices with any HDMI signal source, such as computers, mobile phones, tablets, and presentation equipment. It supports HDMI 1.4 to HDMI 2.1 standards and is compatible with multiple resolution formats such as 720p, 1080p, and 4K. The HDMI input module 101 achieves automatic signal format detection and matching through hardware-level EDID (Extended Display Identification Data) auto-negotiation and uses a dedicated video decoding chip for frame buffering and synchronization processing to ensure stable video stream reception, with a frame rate support of 30fps to 60fps.

[0031] The image content streaming analysis module 102 is used to analyze images in real time and output structured content. It is configured to perform content segmentation, optical character recognition, layout structure extraction and graphic element detection on the video signal received by the HDMI input module 101 in real time, and output structured image data.

[0032] Figure 2 The following is a schematic diagram of the processing flow of the image content streaming analysis module according to Embodiment 1 of this application, which will be referred to below. Figure 2 The image content streaming analysis module 102 of this application embodiment will be described in further detail.

[0033] In the embodiments of this application, the image content streaming analysis module 102 processes the received video frames in real time. First, image preprocessing is performed (each frame is analyzed within 100ms), including geometric correction (addressing projector trapezoidal distortion), brightness equalization (compensating for differences in display devices), and moiré removal (eliminating screen capture interference). Then, a streaming CNN (convolutional neural network) structure is used to process each frame, outputting structured image data, such as structured OCR (Optical Character Recognition) text content, layout information, and descriptions of graphic elements.

[0034] The described streaming convolutional neural network structure includes a feature extraction layer, a layout structure perception layer, and a content classification layer, enabling real-time detection of various content blocks such as PPT pages, table structures, graphic elements, and titles / paragraphs. The feature extraction layer employs residual connections and attention mechanisms (e.g., ResNet-50 backbone network + FPN multi-scale fusion) to extract multi-scale feature representations from the image, obtaining a multi-scale feature map. The layout structure perception layer automatically identifies different types of content regions, such as titles, paragraphs, tables, images, and formulas, from the multi-scale feature map using a spatial relationship model between page elements built based on graph neural networks (e.g., a region detection and classification model based on Mask R-CNN). The content classification layer, by combining contextual information, performs precise classification, boundary localization, and content recognition for different types of content regions, obtaining structured image data.

[0035] In the embodiments of this application, the content classification layer includes: an optical character recognition unit for recognizing the title area and paragraph area and generating optical character recognition text information; a table parsing unit for parsing the table area and obtaining table structure information; a page structure recognition unit for recognizing the page structure area and obtaining page layout information; and a graphic element detection unit for extracting key elements from the graphics and generating graphic element descriptions.

[0036] In the embodiments of this application, the image content streaming analysis module 102 further includes an input layer for receiving input video signals and supporting dynamic adjustment of resolution (1920×1080×3 in this embodiment).

[0037] In the embodiments of this application, the optical character recognition unit adopts CRNN+CTC decoding and supports recognition of multiple languages ​​including Chinese, English, and Japanese.

[0038] In the embodiments of this application, the layout structure recognition unit is based on document tree parsing technology and can recognize complex layout structures such as hierarchical headings, paragraph organization, table boundaries, and mixed text and images, with a structure recognition accuracy of ≥95%.

[0039] In the embodiments of this application, the image content streaming analysis module 102 optimizes the OCR recognition accuracy according to an adaptive threshold algorithm and dynamically adjusts the recognition parameters based on factors such as image quality, lighting conditions, and font size. For practical application scenarios such as projector displays and screen capture, it integrates preprocessing technologies such as image geometric correction, moiré removal, and brightness equalization, enabling it to support multilingual character recognition including Chinese, English, and Japanese, with a high character recognition accuracy.

[0040] In the embodiments of this application, the graphic element detection unit can identify graphic types such as flowcharts, pie charts, bar charts, and mind maps, and extract key information for semantic understanding.

[0041] The audio acquisition module 103 is used to acquire audio signals and preprocess them before outputting them to the speech recognition module 104. The speech recognition module 104 is used to transcribe the audio signals output by the audio acquisition module 103 into speech text in real time.

[0042] Figure 3 The following is a schematic diagram of the processing flow of the audio acquisition module and the speech recognition module according to Embodiment 1 of this application. Figure 3 The audio acquisition module 103 and the speech recognition module 104 of the embodiments of this application will be described in detail.

[0043] In the embodiments of this application, the audio acquisition module 103 employs a circular or linear array configuration consisting of 8 to 16 omnidirectional microphones, with a microphone spacing of 3-5 cm and a sampling rate supporting 16 kHz to 48 kHz. The audio acquisition module 103 is configured to: achieve directional acquisition of the speaker's voice and effective suppression of background noise based on an adaptive beamforming algorithm. For example, it may employ a minimum variance distortion-free response (MVDR) algorithm or a generalized sidelobe canceller (GSC) algorithm to dynamically adjust the directional pattern of the array according to the direction of the sound source, or combine a time-frequency domain multi-channel noise reduction algorithm with Wiener filtering and spectral subtraction to minimize environmental noise, echo, and reverberation while maintaining speech clarity.

[0044] To achieve speaker diarization (audio source), the audio acquisition module 103 is configured to automatically identify and separate the speech of multiple speakers based on a deep clustering algorithm, assigning an independent audio channel to each speaker. This supports simultaneous speaking scenarios for 2-8 people and improves speaker recognition accuracy.

[0045] In the embodiments of this application, the speech recognition module 104 employs an end-to-end deep neural network model to perform real-time speech transcription of the preprocessed audio signal, generating speech text. It uses the latest Conformer or Transformer-Transducer network structure, combining the local feature extraction capability of convolutional neural networks with the global sequence modeling capability of Transformers, achieving end-to-end speech recognition. Furthermore, the model has been pre-trained on large-scale multilingual speech data, supporting the recognition of major languages ​​such as Chinese, English, and Japanese, and possessing dialect adaptation and professional terminology recognition capabilities.

[0046] In the embodiments of this application, the speech recognition module 104 employs streaming decoding technology to achieve real-time speech transcription, with recognition latency controlled within 200ms. A jointly trained language model is used for post-processing, automatically adding punctuation, correcting recognition errors, and optimizing sentence expression. In multi-speaker scenarios, combined with speaker separation results, timestamps and identity identifiers are added to the speech content of different speakers. The speech recognition accuracy can reach 99% under clear speech conditions and maintain over 97% in noisy environments.

[0047] In some exemplary embodiments, the audio acquisition module 103 uses an 8-microphone circular array (radius 4cm) to synchronously acquire audio signals at a sampling rate of 16kHz and 16-bit quantization. Optimal weights are calculated in real-time using the MVDR beamforming algorithm, with the main lobe pointing towards the speaker and the null point aligned with the interference source. Wiener filtering is used for time-frequency domain noise reduction, suppressing environmental noise by 15-20dB while maintaining the naturalness of the speech. The Conformer speech recognition network parameters are: 12 encoder layers, 8 attention heads, 512 model dimensions, 31 convolutional kernels, and 10,000 hours of multilingual speech data for training.

[0048] The multimodal fusion and semantic understanding module 105 is configured to fuse the structured image data output by the image content streaming analysis module 102 and the speech text output by the speech recognition module 104 to generate summary information and output it.

[0049] Figure 4 This is a schematic diagram of the processing flow of the multimodal fusion and semantic understanding module according to Embodiment 1 of this application, as shown below. Figure 4 As shown, the multimodal fusion and semantic understanding module 105 of this application embodiment receives three types of input: OCR text, layout information, and speech text. It adopts a multi-level fusion structure to achieve deep integration of information such as image, text, and speech content, and uses a temporal alignment algorithm to perform temporal synchronization matching and semantic feature alignment to establish a connection between the speech content and the currently displayed screen. The semantic relationships between different modalities are learned through a multimodal Transformer network (6-layer encoder, 8 attention heads).

[0050] In the embodiments of this application, the multimodal fusion and semantic understanding module 105 includes a cross-modal alignment layer, a semantic fusion layer, and a summary generation layer.

[0051] The cross-modal alignment layer establishes semantic correspondences between OCR text, layout information, and speech text using a learned attention matrix. Considering the temporal and logical relevance between the presentation content and the spoken content, the cross-modal alignment layer employs a temporal alignment algorithm to automatically match the temporal synchronization between image content and speech content.

[0052] The semantic fusion layer, based on multimodal Transformer fusion, maps semantic representations from different modalities to a unified semantic space. Through self-attention and cross-attention mechanisms, it captures complex dependencies within and between modalities. Residual connections and layer normalization techniques are employed to ensure stable training and inference of deep networks.

[0053] The abstract generation layer integrates miniature large language models (such as TinyGPT, DistilBERT, etc.) that have undergone parameter pruning and quantization optimization, with the number of model parameters controlled within the range of 100M-500M.

[0054] In the embodiments of this application, the parameters of the Large Language Model (LLM) are as follows: Model size: 300M parameters (based on GPT-2 architecture compression); Quantization precision: INT8; Inference latency: ≤150ms; Abstract quality: ROUGE-L score ≥ 0.75.

[0055] The multimodal fusion and semantic understanding module 105 of this application embodiment intelligently generates summaries based on the fused multimodal semantic representation and utilizes a summarization generation algorithm (combining extractive and generative methods) with adaptive optimization capabilities. The specific steps of the summarization generation algorithm include: identifying important sentences, keywords, and core concepts in the document through a key information extraction algorithm; and generating fluent and coherent summary text using a sequence-to-sequence (Seq2Seq) generation model.

[0056] To meet the needs of different application scenarios, the multimodal fusion and semantic understanding module 105 supports the generation of various summary styles, such as meeting minutes style (highlighting decisions and action items), academic report style (emphasizing arguments and evidence), and teaching content style (focusing on knowledge points and logical structure). Through few-shot learning and domain adaptation techniques, it can quickly adapt to new application domains and content types.

[0057] In some exemplary embodiments, the multimodal fusion and semantic understanding module 105 also includes a summary quality assessment module, which monitors summary quality in real time and performs adaptive optimization based on automatic assessment metrics such as ROUGE and BLEU and manually annotated data. When a decline in summary quality is detected, the model parameters are automatically adjusted or a backup algorithm is switched to ensure the stability of the output quality.

[0058] The control processing unit 106 is used to coordinate and control the start and stop, data routing, and resource allocation of the above-mentioned modules. The control processing unit embeds an AI inference acceleration unit to accelerate AI model inference and supports dynamic model loading and online upgrades.

[0059] The control processing unit 106 in this embodiment is based on the RK3566 chip and the Android underlying system, integrating all the above modules at the hardware level, and supports 4K resolution and plug-and-play operation. Figure 5 As shown, the hardware integration and system optimization are based on the RK3566 main control chip platform, integrating an ARM Cortex-A55 quad-core CPU, a Mali-G52 2EE GPU, and a dedicated NPU (Neural Processing Unit). It features 4GB of LPDDR4 memory, 32GB of eMMC storage, and supports microSD card expansion.

[0060] In the embodiments of this application, the Android underlying system has been deeply customized and optimized, removing redundant services and applications and focusing on multimedia processing and AI inference tasks. For example... Figure 6 As shown, a real-time operating system scheduling strategy is employed to ensure low latency and high stability in audio and video processing. The integrated AI acceleration unit supports mainstream model formats such as TensorFlow Lite, ONNX, and OpenVINO, achieving efficient inference on limited hardware resources through techniques such as model quantization, operator fusion, and memory optimization. For example, knowledge distillation technology is used to transfer knowledge from large pre-trained models to lightweight models, significantly reducing computational resource requirements while maintaining semantic understanding capabilities. It also supports flexible control of summary length, generating content summaries of different granularities, such as brief summaries (50-100 words) and detailed summaries (200-500 words).

[0061] According to embodiments of this application, the multimodal intelligent semantic understanding and summarization generation system based on HDMI stream supports OTA (Over-The-Air) online upgrades, enabling remote model updates, system vulnerability fixes, and new feature additions. Based on a modular architecture, each functional module can be upgraded independently, avoiding the impact of a system-wide restart on user experience.

[0062] The output module 107 includes an HDMI output interface, which is used to transmit the original video signal or the processed video signal received by the HDMI input module 101 to the display terminal 108, thereby realizing a lossless signal relay function and ensuring that the user's normal display needs are not affected.

[0063] Signal relay: refers to the amplification and retransmission of signals through devices such as repeaters to extend the transmission distance of signals and ensure communication quality and stability.

[0064] The result output module 107 of this application embodiment supports multiple interfaces and protocols to meet the integration needs of different application scenarios. For example... Figure 5 As shown, the output module 107 also includes a USB 3.0 interface, supporting high-speed data transmission, which can export recognition results, summary information, audio and video files, etc., to external storage devices; a Gigabit Ethernet interface, supporting real-time data streaming and remote access, enabling seamless integration with enterprise intranets, conferencing systems, and knowledge management platforms; and RESTful API and WebSocket interfaces, supporting real-time data acquisition and control operations for third-party applications. The output data format supports multiple standard formats such as JSON, XML, PDF, and Word, facilitating subsequent processing and archiving. Integrated MQTT protocol support allows access to IoT platforms and smart office systems.

[0065] The result output module 107 of this application embodiment supports outputting recognized text, structured content and smart summaries through various methods such as USB interface, network interface, and local storage, and provides a standard API interface for connecting to third-party content management systems.

[0066] Furthermore, the HDMI stream-based multimodal intelligent semantic understanding and summarization system also includes a security module with an integrated hardware encryption chip to support security functions such as data transmission encryption, authentication, and access control. For example, user data and AI models are stored using AES-256 encryption to ensure information security and privacy protection.

[0067] According to embodiments of this application, the multimodal intelligent semantic understanding and summarization system based on HDMI streams supports real-time mode and batch processing mode. In real-time mode, it provides instant summary output, while in batch processing mode, it generates detailed structured documents.

[0068] Figure 7 This is a schematic diagram illustrating the application scenario and system integration according to Embodiment 1 of this application, as follows: Figure 7As shown, the multimodal intelligent semantic understanding and summarization system based on HDMI stream in this application embodiment can realize plug-and-play acquisition of HDMI signals, real-time synchronous analysis of multimodal content, high-precision structured understanding, and localized intelligent summarization. It is suitable for scenarios such as meeting recording, speech transcription, and teaching content acquisition and archiving. It supports automatic switching of multiple languages ​​and scene adaptive configuration to meet the urgent needs of modern office, education, and meeting scenarios for intelligent content processing.

[0069] The following are experimental data and performance comparisons of embodiments of this application, wherein the experimental environment settings are as follows: Test venue: Standard conference room (6m×4m×3m); Ambient noise: 35-45dB; Number of participants: 2-10; Test content: PPT presentation, technical discussion, academic report; Test duration: 200 hours in total.

[0070] The key performance indicators are compared as follows:

[0071] The detailed test results analysis of this application is as follows: 1) OCR recognition performance test: Test sample: 1000 slides of different types of PPTs; Scenarios included: projection display, screenshot, mobile phone photography; Recognition accuracy: 99.2% at the character level, 98.8% at the word level; Special features: Formula recognition 96%, table recognition 97%.

[0072] 2) Speech recognition performance test: Test audio: 100 hours of multi-scene recordings; Environmental conditions: Quiet / Moderate noise / High noise; Language distribution: Chinese 70%, English 25%, others 5%; Overall accuracy: 97.3% (quiet), 95.8% (medium noise), 93.2% (strong noise).

[0073] 3) Multimodal fusion effect test: Test scenario: 50 meetings of different types; Fusion accuracy: Content association accuracy rate of 96%; Timing synchronization: Audio-visual synchronization error <50ms; Semantic consistency: 94% cross-modal semantic matching.

[0074] 4) System integration compatibility testing: Test equipment: 100 different HDMI devices; Compatibility success rate: 99% (only 1 older device is incompatible); Automatic adaptation time: average 3.2 seconds; Signal quality: lossless pass-through, delay <10ms.

[0075] 5) Long-term stability test: Continued operation time: 30 days x 24 hours; System restart count: 0 times (excluding proactive maintenance); Memory leak: No significant increase; Performance degradation: <1%; Fault recovery: Automatic recovery in an average of 5 seconds.

[0076] Experimental verification shows that the technical indicators of this application in practical applications have all met or exceeded the design goals, providing an advanced, reliable and practical technical solution for the intelligent processing of multimedia content.

[0077] The multimodal intelligent semantic understanding and summarization system based on HDMI stream provided in this application has the following advantages: (1) Hardware-level plug-and-play capability: Through HDMI hardware direct connection, it does not rely on operating system software and achieves true plug-and-play. Deployment time is shortened to a few minutes, device compatibility is greatly improved, and it can support a variety of mainstream operating system platforms such as Windows, macOS, Linux, Android, and iOS.

[0078] (2) Multimodal real-time synchronous processing: An innovative streaming processing architecture is adopted to achieve millisecond-level synchronization of audio, video, and text, which significantly reduces the latency of multimodal information fusion and greatly improves real-time performance. In meeting scenario testing, the timeliness of content understanding significantly improves the user experience.

[0079] (3) High-precision structured content recognition: Based on document structure awareness OCR algorithm, it has high accuracy in recognizing complex content such as PPT, tables, and mixed text and images. Especially in practical application environments such as projection display and tilted shooting, the recognition robustness is significantly enhanced. At the same time, the page layout recognition accuracy is high, and it can accurately distinguish different areas such as titles, body text, tables, and images, providing a reliable foundation for subsequent semantic understanding.

[0080] (4) Multi-speaker far-field speech recognition: Combining microphone arrays and spatial audio processing technology improves speech recognition accuracy and supports complex scenarios where multiple people speak simultaneously. Compared with single-microphone solutions, both recognition accuracy and speaker separation accuracy are improved in noisy environments.

[0081] (5) Localized Intelligent Semantic Understanding: Employing edge deployment of a micro-large language model, fully localized semantic understanding and summary generation are achieved while avoiding data privacy risks. Summary generation latency is controlled through model compression and hardware acceleration. It has offline availability and is not limited by network environment.

[0082] (6) System integration and energy consumption optimization: By integrating multiple functional modules, the equipment cost and power consumption are reduced; at the same time, standardized API interfaces and multiple output interfaces are provided to reduce system integration costs.

[0083] (7) Universality of application scenarios: Through scenario adaptation, the processing parameters are automatically adjusted to meet the needs of different scenarios such as meetings, teaching, and speeches. Multilingual support and professional terminology recognition capabilities significantly enhance its applicability in international conferences and professional fields.

[0084] Example 2 This embodiment is based on Embodiment 1 and has undergone several optimizations and functional enhancements for complex application scenarios.

[0085] In terms of hardware upgrades: the microphone array has been upgraded to 16 high-precision MEMS microphones in a dual circular array configuration (8 in the inner ring and 8 in the outer ring). An enhanced AI inference acceleration unit is employed, integrating a dedicated Transformer accelerator, resulting in a 50% improvement in INT8 inference performance. Expanded storage is achieved with 64GB eMMC + 128GB SSD caching, supporting local processing of large volumes of data. An active cooling system has been added to ensure stability during prolonged high-load operation.

[0086] In terms of layout structure recognition, modeling the spatial and semantic relationships between page elements using graph neural networks can further improve the accuracy of structured recognition. This includes automatic numbering of multi-level headings, parsing of list hierarchy relationships, recognition of cross-page table connections, semantic association of mixed text and images, and the contextual relationship between formulas and text.

[0087] In multi-speaker scenarios, the 16-microphone array provides more accurate spatial positioning capabilities and supports 3D sound source localization. It supports scenarios with up to 8 people speaking simultaneously. It employs speaker embedding learning based on deep clustering, adaptive threshold speaker number estimation, real-time speaker identity tracking, and optimized automatic speaker role recognition (speaker, questioner, and participant).

[0088] Model optimization: The multilingual intelligent summarizing micro-LLM has been expanded to 500M parameters, increasing its multilingual summarization capabilities. It supports cross-language content understanding and translation, automatic recognition and explanation of technical terms, and adaptive expression based on cultural background.

[0089] An intelligent scene recognition module has been added. After the system starts, it automatically analyzes the characteristics of the input content and identifies the application scenario, so as to automatically adjust the processing strategy and summary style according to the scenario. For example, academic conferences emphasize arguments and evidence, highlighting innovative points; business meetings focus on decisions and action items; teaching and training emphasize knowledge points and logical structure; and technical demonstrations highlight technical details and operating procedures.

[0090] It integrates a real-time quality monitoring module to evaluate processing results from multiple dimensions. When quality anomalies are detected, it automatically adjusts algorithm parameters or activates backup strategies. This includes real-time monitoring of OCR confidence, speech recognition quality assessment, summary coherence checks, and multimodal consistency verification.

[0091] This embodiment supports user preference learning and personalized configuration. For example, it allows for customization of the terminology database, setting preferences for summary style, filtering of sensitive words in speech recognition, and customization of output format templates.

[0092] Taking educational application scenarios as an example, the system is deeply customized and optimized, taking into full account the special characteristics of the teaching content and the actual needs of educational users. Figure 8 As shown. Similarly, dedicated versions are developed for international conferences, medical consultations, legal meetings, and so on.

[0093] In the embodiments of this application, some common failure modes are analyzed and corresponding solutions are provided, as follows: Unstable HDMI signal: This manifests as intermittent signal loss and resolution fluctuations, caused by cable quality issues, loose connections, and EDID conflicts. Solutions include: adding signal buffering and retry mechanisms, implementing intelligent EDID negotiation algorithms, and providing real-time signal quality monitoring. Multimodal synchronization error: This manifests as a time mismatch between audio and video content, caused by processing latency differences and clock drift. Solutions include: ensuring hardware-level timestamp synchronization, adding adaptive latency compensation algorithms, and providing a manual calibration interface.

[0094] AI inference performance degradation manifests as decreased recognition accuracy and increased latency, caused by model overheating, memory fragmentation, and cache invalidation. Solutions include: implementing dynamic load balancing, adding intelligent heat dissipation control, and providing a model hot update mechanism.

[0095] Example 3 In the embodiments of this application, a multimodal intelligent semantic understanding and summarization method based on HDMI stream is also provided. The method uses the multimodal intelligent semantic understanding and summarization system based on HDMI stream as described above to realize intelligent multimedia content processing and summarization.

[0096] Figure 9The flowchart of the multimodal intelligent semantic understanding and summarization method based on HDMI stream according to Embodiment 3 of this application will be referenced below. Figure 9 This application provides a detailed description of the multimodal intelligent semantic understanding and summarization method based on HDMI streams.

[0097] First, in step S901: receive video signals from external devices via the HDMI input module.

[0098] In step S902: The processed original video signal is subjected to content segmentation, optical character recognition, layout structure extraction and graphic element detection to obtain structured image data.

[0099] In step S903: The audio signal acquired by the microphone array is preprocessed and transcribed to obtain the speech text.

[0100] In step S904: The structured image data and the speech text are fused using a cross-modal attention mechanism and a miniature large language model to generate summary information.

[0101] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimodal intelligent semantic understanding and summarization system based on HDMI stream, characterized in that, include: An HDMI input module is used to receive video signals output from external devices; The image content streaming analysis module is configured to perform content segmentation, optical character recognition, layout structure extraction, and graphic element detection on the video signal, and output structured image data. An audio acquisition module is used to acquire audio signals and preprocess the audio signals; The speech recognition module is used to transcribe the preprocessed audio signal into speech text; The multimodal fusion and semantic understanding module is configured to fuse the structured image data and the speech text, generate summary information, and output it to the result output module.

2. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The structured image data includes: optical character recognition text information, table structure information, page layout information, and descriptions of graphic elements.

3. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The image content streaming analysis module adopts a data streaming convolutional neural network structure, including: a feature extraction layer, a layout structure perception layer, and a content classification layer; The feature extraction layer uses residual connections and attention mechanisms to extract multi-scale feature representations from the image, resulting in a multi-scale feature map. The layout structure perception layer automatically identifies different types of content regions from the multi-scale feature map by using a spatial relationship model between page elements established based on a graph neural network. The content classification layer is used to combine contextual information to accurately classify, locate boundaries, and identify content of different types of content regions, thereby obtaining structured image data.

4. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 3, characterized in that, The content classification layer includes: An optical character recognition unit is used to recognize the title area and paragraph area and generate the optical character recognition text information; The table parsing unit is used to parse the table area to obtain the table structure information; The layout structure recognition unit is used to identify the layout structure area and obtain the layout information. The graphic element detection unit is used to extract key elements from the graphic and generate graphic element descriptions.

5. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 4, characterized in that, The layout structure recognition unit uses a document tree parsing method to recognize the layout structure; the layout structure includes hierarchical headings, paragraph organization, table boundaries, and mixed text and image layout.

6. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The audio acquisition module synchronously acquires audio signals using a circular or linear array of multiple omnidirectional microphones, and outputs the audio signals to the speech recognition module after preprocessing them using a beamforming algorithm.

7. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 6, characterized in that, The beamforming algorithm is either a minimum variance distortionless response algorithm or a generalized sidelobe canceller algorithm, which enables directional acquisition of the speaker's voice and suppression of background noise.

8. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The speech recognition module uses an end-to-end deep neural network to recognize the preprocessed audio signal and performs real-time speech transcription through streaming decoding to generate speech text.

9. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The multimodal fusion and semantic understanding module employs a cross-modal attention mechanism to semantically align the structured image data and the speech text, and uses a miniature large language model to understand the context and generate a summary.

10. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 9, characterized in that, The multimodal fusion and semantic understanding module includes: The cross-modal alignment layer establishes the semantic correspondence between optical character recognition text information, page layout information and speech text in the structured image data through the learned attention matrix, and automatically matches the temporal synchronization relationship between image content and speech content using a temporal alignment algorithm. The semantic fusion layer, a deep learning model based on attention mechanisms, maps feature representations from different modalities to a unified semantic space; it captures intra-modal and inter-modal dependencies through self-attention and cross-attention mechanisms; and it employs residual connections and layer normalization to ensure stable training and inference of deep networks. The summary generation layer generates summary information based on the semantic space and the dependency relationships, using a miniature large language model that has undergone parameter pruning and quantization optimization.

11. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, The result output module includes: An HDMI output interface is used to output the original or processed video signal to a display terminal. The USB interface is used to export data to external devices; The network transmission module is used to provide remote data transmission and access.

12. The multimodal intelligent semantic understanding and summarization system based on HDMI stream according to claim 1, characterized in that, It also includes a control processing unit for coordinating and controlling the start / stop, data routing, and resource allocation of the HDMI input module, the image content streaming analysis module, the audio acquisition module, the multimodal fusion and semantic understanding module, and the result output module; the control processing unit embeds an AI acceleration unit to improve processing speed.

13. A method for multimodal intelligent semantic understanding and summarization based on HDMI streams, employing the multimodal intelligent semantic understanding and summarization system based on HDMI streams as described in any one of claims 1-12, characterized in that, Includes the following steps: Receive video signals from external devices via the HDMI input module; The processed original video signal is subjected to content segmentation, optical character recognition, layout structure extraction, and graphic element detection to obtain structured image data; The audio signals collected by the microphone array are preprocessed and transcribed to obtain speech text; By utilizing a cross-modal attention mechanism and a miniature large language model, the structured image data and the speech text are fused to generate summary information.

Images