Method for displaying subtitles based on semantic recognition of a commercial display mainboard and commercial display mainboard

Abstract

The application relates to the technical field of display device manufacturing, and provides a method for displaying subtitles based on semantic recognition of a commercial display mainboard and the commercial display mainboard. In the method, an AI subtitle processing module performs local voice front-end processing, recognition and semantic analysis on an audio frame stream output by a player decoding pipeline, generates semantic text data with confidence and punctuation, a timestamp synchronization module generates a subtitle time slice and labels start and end times according to audio PTS and a system clock, a subtitle display constraint solving module combines physical form parameters, display configuration parameters and a current video frame, obtains subtitle display area parameters and unrenderable masks through saliency and OCR analysis, a subtitle display engine completes glyph shaping, bidirectional layout and GPU real-time rendering based on the above parameters, and realizes accurate alignment of subtitles and videos through micro-rate correction. The method realizes full-process localization processing, guarantees low delay, intelligent avoidance and high-quality display of the generated subtitles, and is suitable for diversified commercial display scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display device manufacturing, semantic recognition and the like, and in particular to a method for displaying subtitles based on semantic recognition of a commercial display mainboard and the commercial display mainboard. BACKGROUND

[0002] In the existing display device manufacturing technology for commercial display purposes, the subtitle generation and display system mostly relies on cloud-based voice recognition and processing services. When the player outputs an audio and video stream, the audio data needs to be uploaded to the cloud for recognition and analysis, and then the recognition result is returned to the local for subtitle rendering. In a weak network or offline scenario, cloud recognition cannot be used. Even in a good network environment, the process of uploading, recognizing and returning the audio data will bring significant delay, causing subtitle generation lag, and affecting the viewing experience of the audience. In addition, the subtitle system often directly aligns with the audio timestamp in a simple manner, without considering fast-forward, fast-play, drag-and-drop skip and other situations in the playback process, resulting in misalignment of subtitles in advance or delay. Moreover, the subtitles are usually fixedly displayed at the bottom of the screen, lacking intelligent perception and avoidance mechanisms for the content of the video picture. When there are faces, LOGOs, existing text information or key visual content in the picture, the subtitles are likely to overlap with them, affecting the picture integrity and visual effect. Moreover, the subtitle rendering method is usually mainly based on simple text superposition, lacks adaptive processing for different screen forms (landscape, portrait, and special-shaped screens), and is limited in display effects such as typesetting, stroke, and shadow, resulting in poor readability of the subtitles in complex commercial display scenarios.

[0003] In summary, the existing technology has problems such as strong dependence on network, large delay, poor time performance, insufficient adaptability, and single display effect, and it is difficult to meet the requirements of commercial display devices for high real-time performance, high accuracy and high readability of subtitle generation and display in diversified scenarios. SUMMARY

[0004] In view of the above-mentioned problems of the existing technology, the present application provides a method for displaying subtitles based on semantic recognition of a commercial display mainboard and the commercial display mainboard, which can integrate voice recognition, timestamp synchronization, intelligent solution of subtitle display area, and GPU efficient rendering modules locally on the commercial display mainboard, realize local processing of the whole process, ensure accurate alignment, intelligent avoidance and high-quality display of subtitles and videos, and meet the requirements of commercial display devices for high real-time performance, high accuracy and high readability of subtitle generation and display in diversified scenarios.

[0005] In a first aspect, the present application provides a method for displaying subtitles based on semantic recognition of a commercial display mainboard, wherein the commercial display mainboard integrates an AI subtitle processing module, a timestamp synchronization module, a subtitle display constraint solving module and a subtitle display engine, and the method comprises the following steps:

[0006] S1, the AI subtitle processing module reads audio data with presentation time stamp PTS from the audio frame stream output by the player decoding pipeline frame by frame, performs voice front-end processing, speech recognition and semantic analysis locally to obtain semantic text data with confidence and punctuation; the timestamp synchronization module converts the semantic text data into frame-aligned subtitle time slices and labels the start and end time and confidence threshold based on the PTS of the audio frame and the system clock;

[0007] S2, the subtitle display constraint solving module obtains a renderable heat map through saliency and OCR analysis based on the physical form parameters and display configuration parameters of the commercial display mainboard and in combination with the current video frame provided by the player decoding pipeline, and then solves the subtitle display area parameters and unrenderable mask according to the preset avoidance priority and safety margin, and provides the subtitle display area parameters to the subtitle display engine for rendering control;

[0008] S3, the subtitle display engine receives the subtitle time slices output by the timestamp synchronization module and the subtitle display area parameters output by the subtitle display constraint solving module, performs glyph shaping and bidirectional typesetting on the subtitle lines in its internal, and generates a typeset subtitle layer according to the subtitle display area parameters; the subtitle display engine performs real-time rendering of the subtitle layer on the GPU overlay plane, completes Alpha blending and stroke shadow overlay, and aligns the subtitle start and end time with the video presentation time according to the threshold range micro-rate correction strategy of the timestamp synchronization module, and finally outputs the timed subtitle overlay picture to the video compositor for composition and display.

[0009] In a second aspect, the application provides a commercial display mainboard for performing semantic recognition and subtitle display control on a playing file using the above-mentioned method for displaying subtitles based on semantic recognition of a commercial display mainboard.

[0010] Compared with the prior art, the application has the following beneficial effects:

[0011] The application provides a method and a commercial display mainboard for displaying subtitles based on semantic recognition. The method comprises the following steps: an AI subtitle processing module reads audio data with a presentation timestamp (PTS) from an audio frame stream output by a player decoding pipeline frame by frame, performs voice front-end processing, voice recognition and semantic analysis locally, and obtains semantic text data with confidence and punctuation; a timestamp synchronization module converts the semantic text data into a frame-aligned subtitle time slice and labels start and end times and a confidence threshold based on the PTS of the audio frame and a system clock; a subtitle display constraint solving module obtains a renderable heat map through saliency and OCR analysis based on physical form parameters and display configuration parameters of the commercial display mainboard, a current video frame provided by the player decoding pipeline and preset avoidance priorities and safety margins, solves subtitle display area parameters and an unrenderable mask, and provides the subtitle display area parameters to a subtitle display engine for rendering control; the subtitle display engine receives the subtitle time slice output by the timestamp synchronization module and the subtitle display area parameters output by the subtitle display constraint solving module, performs glyph shaping and bidirectional typesetting on subtitle lines in the subtitle display engine, generates a typeset subtitle layer according to the subtitle display area parameters, performs real-time rendering of the subtitle layer on a GPU overlay plane, completes Alpha blending and stroke shadow overlay, aligns the start and end times of the subtitles and the video presentation time according to the threshold range micro-rate correction strategy of the timestamp synchronization module, and finally outputs the timed subtitles to a video compositor for composition and display. The method can integrate voice recognition, timestamp synchronization, intelligent subtitle display area solving and GPU efficient rendering modules locally on the commercial display mainboard, realize local processing in the whole process, ensure accurate alignment, intelligent avoidance and high-quality display of subtitles and videos, and meet the requirements of high real-time performance, high accuracy and high readability of subtitle generation and display of commercial display devices in diversified scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0013] Figure 1 FIG. 1 is a flowchart of a method for displaying subtitles based on semantic recognition of a commercial display mainboard according to an embodiment of the application. DETAILED DESCRIPTION

[0014] In order to make the person skilled in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the present application.

[0015] Embodiment one

[0016] Referring to Figure 1 The present embodiment provides a method for displaying subtitles based on semantic recognition of a commercial display mainboard, the commercial display mainboard integrating an AI subtitle processing module, a timestamp synchronization module, a subtitle display constraint solving module and a subtitle display engine, the method comprising the following steps:

[0017] S1, the AI subtitle processing module reads audio data with a presentation timestamp (PTS) from the audio frame stream output by the player decoding pipeline frame by frame, performs speech front-end processing, speech recognition and semantic analysis locally, and obtains semantic text data with confidence and punctuation; the timestamp synchronization module converts the semantic text data into a frame-aligned subtitle time slice and labels the start and end time and the confidence threshold based on the PTS of the audio frame and the system clock; wherein PTS refers to Presentation Time Stamp, i.e. presentation timestamp.

[0018] S2, the subtitle display constraint solving module obtains a renderable heat map through saliency and OCR analysis based on the physical form parameters and display configuration parameters of the commercial display mainboard, and combines the current video frame provided by the player decoding pipeline, and solves the subtitle display area parameters and unrenderable mask according to the preset avoidance priority and safety margin, and provides the subtitle display area parameters to the subtitle display engine for rendering control;

[0019] S3, the subtitle display engine receives the subtitle time slice output by the timestamp synchronization module and the subtitle display area parameters output by the subtitle display constraint solving module, performs glyph shaping and bidirectional typesetting on the subtitle lines in its internal, and generates a typeset subtitle layer according to the subtitle display area parameters; the subtitle display engine performs real-time rendering of the subtitle layer on the GPU overlay plane, completes Alpha blending and stroke shadow overlay, and aligns the subtitle start and end time with the video presentation time according to the threshold range micro-rate correction strategy of the timestamp synchronization module, and finally outputs the timed subtitle overlay picture to the video compositor for composition and display. Wherein, Alpha blending is a layer overlay technique in computer graphics, which determines the composition of foreground image and background image through the Alpha channel (transparency information) of the pixel.

[0020] In this embodiment, by integrating the AI subtitle processing module, timestamp synchronization module, subtitle display constraint solving module and subtitle display engine on the commercial display mainboard, and completing audio analysis, speech recognition, semantic analysis, timestamp conversion, subtitle display area solving and GPU rendering output according to steps S1 to S3, the problems of dependence on external cloud computing, large processing delay, asynchronous subtitles and video, conflict between subtitles and important areas of the picture, and single display effect in the subtitle generation and display system can be solved. The scheme of this embodiment completes full-link processing on the local mainboard, directly converts the speech frame into semantic text and generates a subtitle time slice, and then combines the display constraint module to perform saliency and OCR analysis on the video frame, thereby generating subtitle display area parameters, ensuring that the subtitles do not block the main vision and key content, and enhancing the adaptability. At the same time, the subtitle display engine not only completes bidirectional typesetting and glyph shaping, but also supports GPU real-time rendering and Alpha blending, and can output clear and readable subtitle layers under low delay conditions. More importantly, the timestamp synchronization module introduces a threshold range micro-rate correction strategy, which can dynamically adjust the start and end time of the subtitles when fast forwarding, doubling or delaying occurs in video playback, realize high-precision alignment, achieve low-delay, strong robustness, adaptability to various screen forms and accurate time alignment of subtitle generation and display on the local commercial display mainboard, and improve the usability and professionalism of subtitles in actual scenarios.

[0021] Preferably, the physical form parameters of the commercial display mainboard include landscape, portrait or special-shaped screen types, the display configuration parameters of the commercial display mainboard include resolution, screen ratio and key content avoidance area, and the subtitle display area parameters include area coordinates, line width, line height and baseline, and jointly constrain the layout boundary of the subtitle text together with the non-renderable mask.

[0022] In this embodiment, the physical form parameters (landscape, portrait or special-shaped screen types) and display configuration parameters (resolution, screen ratio and key content avoidance area) of the commercial display mainboard are taken as input constraint conditions, and the area coordinates, line width, line height and baseline in the subtitle display area parameters are explicitly defined, and the non-renderable mask is combined to jointly constrain the layout boundary of the subtitle text, which can solve the problems of poor adaptability of the subtitle display method under different screen types, frequent overlap of subtitles with key content of the video or exceeding the display area. In this embodiment, the subtitle display engine can automatically adjust the typesetting area according to different screen forms, so that the subtitles are always located within the appropriate visible range; through accurate control of the line width, line height and baseline parameters, the neatness and readability of the subtitle typesetting are ensured; and the setting of the non-renderable mask and the avoidance area further ensures that the subtitles do not cover the LOGO, watermark or OSD information in the video.

[0023] Preferably, the commercial display mainboard further integrates an MCU control module; the MCU control module performs capability description analysis and function abstraction on peripherals connected to the commercial display mainboard, obtains an overlapping function set and a function device mapping relationship; in a subtitle rendering synchronization domain, when scene brightness, indicator lights or volume need to be linked, the MCU control module performs synchronous control on multiple peripherals based on a single control instruction and collects state readback, thereby realizing local semantic recognition and subtitle generation, key area avoidance and time synchronization rendering, and consistency of peripheral control in an offline or weak network scenario.

[0024] In this embodiment, an MCU control module is integrated in the commercial display mainboard, capability description analysis and function abstraction are performed on connected peripherals, an overlapping function set and a function device mapping relationship are generated, and in a subtitle rendering synchronization domain, synchronous control of multiple peripherals is realized through a single control instruction and state readback, which can solve the problem of split of subtitle display and peripheral linkage in a commercial display system, different synchronization of actions between different peripherals, and inability of subtitles to form an overall effect with peripherals in a weak network or offline state. The scheme of this embodiment completes capability analysis and abstraction of peripherals locally through the MCU module, defines volume, brightness, indicator lights, etc. as an overlapping function set, and forms a device mapping table, so that multiple peripherals can be executed atomically through a single instruction. In the subtitle rendering process, the MCU can ensure that peripheral actions and subtitle generation are completely synchronized, and even in a weak or no network scenario, local linkage of subtitles and peripherals can still be completed independently. Through the state readback mechanism, the execution result of the peripheral can also be confirmed to ensure reliable action, so that in commercial display applications, subtitles can not only be accurately generated and displayed, but also form time consistency and scenario linkage effects with brightness, volume and other peripherals, thereby enhancing the immersion and overall coordination of the subtitle system.

[0025] Preferably, the AI subtitle processing module comprises an acoustic front-end sub-module, an endpoint recognition sub-module, a language adaptation sub-module, a confidence evaluation sub-module, and a result integration and output sub-module; wherein the acoustic front-end sub-module performs resampling, endpoint detection, and dynamic gain control on the received audio frames to improve the signal-to-noise ratio of the voice and stabilize the energy distribution; the endpoint recognition sub-module locally loads a lightweight deep neural network on the commercial display mainboard, decodes the processed audio frames, supports CTC decoding and attention-based sequence-to-sequence reasoning in parallel, and obtains a preliminary transcription result; the language adaptation sub-module receives the preliminary transcription result, switches between different sub-word tables and language models based on language score to adapt to Chinese-English mixed or multi-lingual spoken language; the confidence evaluation sub-module performs numerical level confidence calculation and segment quality determination on the text output by the language adaptation sub-module, triggers delayed confirmation and backfilling for low-confidence segments, and adds confidence labels to each word or segment; the result integration and output sub-module receives the text with confidence labels, performs field standardization and data structure packaging, generates semantic text data, and submits the semantic text data to the timestamp synchronization module for generating a subtitle time slice.

[0026] In the present embodiment, the acoustic front-end sub-module, the endpoint recognition sub-module, the language adaptation sub-module, the confidence evaluation sub-module, and the result integration and output sub-module are arranged in the AI subtitle processing module, and a local lightweight deep neural network is used to complete voice decoding and confidence labeling, which can solve the problems of low signal-to-noise ratio of voice, high recognition delay, poor multi-language adaptability, and insufficient recognition result reliability in commercial display subtitle recognition. Specifically, the acoustic front-end sub-module effectively improves the audio quality through resampling, endpoint detection, and dynamic gain control, ensuring the stability of the input for subsequent recognition; the endpoint recognition sub-module runs a deep neural network locally on the mainboard, supports CTC decoding and attention-based sequence-to-sequence reasoning, balances real-time performance and accuracy, and avoids dependence on the cloud; the language adaptation sub-module switches between different sub-word tables and language models based on language score, ensuring recognition effect in Chinese-English mixed and multi-lingual spoken language environments; the confidence evaluation sub-module performs numerical level confidence calculation on the output text, and low-confidence segments are avoided from being directly displayed through a delayed confirmation and backfilling mechanism; the result integration and output sub-module structures the recognition result and submits it to the timestamp synchronization module to generate a subtitle time slice, thereby realizing robust, low-delay, multi-lingual adaptation, and confidence-labeled subtitle semantic text generation on the local mainboard, improving the practicality and user experience of the subtitle system in complex commercial display environments.

[0027] Preferably, the AI ​​subtitle processing module further includes a semantic analysis submodule, a reading rhythm control submodule, and a confidence mapping submodule. After completing the initial transcription, the semantic analysis submodule performs punctuation restoration, named entity recognition and terminology standardization, sensitive word normalization, and digital unit normalization to obtain structured text. The reading rhythm control submodule receives the structured text and divides it into line segments and paragraph segments according to the target reading speed, while generating soft line break markers. When the width of the line segment exceeds the line width parameter of the subtitle display engine, the reading rhythm control submodule prioritizes breaking the line at the soft line break; otherwise, it calls the equivalent phrase mapping table to shrink long paragraphs into short words. The confidence mapping submodule receives the aforementioned confidence labels and maps different confidence values ​​to corresponding display styles to distinguish between backfilled text and high-confidence text, and submits the integrated text to the timestamp synchronization module to generate corresponding subtitle time segments.

[0028] In this embodiment, a semantic analysis submodule, a reading rhythm control submodule, and a confidence mapping submodule are added to the AI ​​subtitle processing module. These submodules respectively perform punctuation restoration, named entity recognition, terminology standardization, sensitive word normalization, and numerical unit normalization. They also perform reading rhythm segmentation and soft line break control on the text, and map different confidence values ​​to different display styles. This addresses the shortcomings of subtitle recognition results in terms of structure, rhythm, and credibility. In this embodiment, the semantic analysis submodule structures and standardizes the text, ensuring concise and clear display. The reading rhythm control submodule combines the target reading speed with a soft line break strategy to make subtitle segmentation more natural, avoiding line breaks or compression. The confidence mapping submodule distinguishes between low-confidence and high-confidence text, for example, by using different styles or colors to highlight them, thereby improving the usability and credibility of the subtitles. Through this embodiment, the subtitle output is not only structured and standardized but also possesses a good reading rhythm and visual hierarchy. Users can intuitively distinguish between credible and unconfirmed content, thus obtaining a more natural and reliable subtitle experience.

[0029] Preferably, the subtitle display constraint solving module includes a saliency detection submodule and an OCR text block detection submodule. After receiving physical shape parameters, display configuration parameters, and the current video frame, the subtitle display constraint solving module calls the saliency detection submodule to perform saliency analysis on the video frame to generate a saliency heatmap, and calls the OCR text block detection submodule to perform text block recognition on the video frame to generate a text block set. The saliency heatmap, the text block set, and the preset key content avoidance area are combined to construct a multi-objective cost function. The subtitle display constraint solving module uses occlusion of the main visual area, coverage of existing OSD, and intrusion into the safety margin as penalty factors to solve for the subtitle display area parameters within the feasible region. If the solution fails, the subtitle display constraint solving module outputs the area parameters and transparency suggestion values ​​for the screen adaptive bottom band mode and provides them to the subtitle display engine for rendering control.

[0030] In this embodiment, the saliency detection submodule and the OCR text block detection submodule are designed in the subtitle display constraint solving module. The saliency analysis and text block recognition are performed on the video frame. The multi-objective cost function is constructed in combination with the preset key content avoidance area. The penalty factors such as shielding the main vision, covering the OSD, and invading the safety margin are used to solve the subtitle display area parameters in the feasible domain. If the solving fails, the bottom band mode is degraded and the transparency suggestion value is provided, which can solve the problem that the subtitle system conflicts with the main vision, existing text or logo under complex video pictures. The scheme of this embodiment generates a heat map through saliency detection to identify the area with high visual attention in the video frame. The existing text block is recognized through OCR to avoid the overlap of the subtitle. In combination with the safety margin constraint, the subtitle is ensured to be in a reasonable display range without affecting the main picture elements. The multi-objective cost function ensures comprehensive consideration of different factors, making the subtitle area solving more intelligent. When it cannot be solved in extreme cases, it can also automatically switch to the bottom band mode to ensure that the subtitle is still visible. In this embodiment, the subtitle display position can intelligently avoid important visual areas and existing text elements, and dynamically adapt to different picture content, so as to ensure that the subtitle is readable and does not interfere with the video content, and improve the fusion degree and professionalism of the subtitle and the video picture.

[0031] Preferably, after receiving the subtitle display area parameters, the subtitle display engine first limits the layout area of the subtitle text according to the mask of the special-shaped screen, and then recalculates the line width, line height and baseline of the subtitle text according to the nine-square safety area and minimum margin strategy to adapt to the switching of landscape, portrait and special-shaped screens. When multi-language subtitle display is required, the subtitle display engine performs bidirectional algorithm and complex script form replacement on multi-language subtitle text, and enables glyph caching and kerning adjustment during the font rendering process of the subtitle text to reduce rasterization time. The subtitle display engine performs Alpha blending and stroke shadow superposition on the GPU superposition plane for the subtitle layer after typesetting and font rendering, and outputs the time-synchronized subtitle superposition picture to the video compositor for composition and display.

[0032] In the embodiment, after receiving the subtitle display area parameters, the subtitle display engine limits the layout area according to the mask of the special-shaped screen, and recalculates the line width, line height and baseline in combination with the nine-square safe area and the minimum margin strategy, and executes the bidirectional algorithm and complex script replacement in the multi-language subtitle scene, and enables the glyph cache and the kerning fine-tuning in the font rendering, and finally completes the Alpha blending and the stroke shadow superposition on the GPU superposition plane, which can solve the problems of distortion in typesetting, high rendering delay and unclear font of subtitles in the special-shaped screen and multi-language script environment. The scheme of the embodiment ensures clear typesetting of subtitles in various screen forms through the mask of the special-shaped screen and the nine-square safe area strategy; ensures correct direction and accurate glyph in multi-language mixed arrangement through the bidirectional algorithm and the script replacement mechanism; significantly reduces the rendering delay through the glyph cache and the kerning fine-tuning mechanism, so that the GPU rendering is more efficient; and the Alpha blending and the stroke shadow improve the readability and the visual effect. In the embodiment, the subtitle engine can realize efficient, accurate and beautiful multi-language subtitle rendering in a complex environment, and improves the display quality and the cross-language adaptation capability.

[0033] Preferably, the timestamp synchronization module comprises a phase-locked loop modeling submodule and a jitter buffering submodule; after receiving the audio PTS and the video presentation time, the timestamp synchronization module calls the phase-locked loop modeling submodule to establish a phase-locked loop model to track the synchronization relationship between the audio PTS and the video presentation time, and calls the jitter buffering submodule to absorb the timestamp jitter generated in the transmission and buffering links of the audio frame stream output from the player decoding pipeline; when the difference between the subtitle time slice and the video presentation time exceeds a first threshold value, the timestamp synchronization module triggers instantaneous realignment, and when the difference exceeds a second threshold value, the timestamp synchronization module triggers a smooth resampling strategy and applies a threshold range micro-rate correction to subsequent time slices; when the speed-up playback and random dragging occur, the timestamp synchronization module rearranges and discards the subtitle time slices according to the player clock events, and provides the corrected subtitle time slices to the subtitle display engine for time rendering.

[0034] In the embodiment, the timestamp synchronization module includes a phase-locked loop modeling submodule and a jitter buffer submodule, which track and model audio PTS and video presentation time, absorb timestamp jitter in the decoding and transmission links, and trigger instantaneous realignment when the difference between subtitle time slices and video presentation time exceeds a first threshold and trigger a smooth resampling strategy when the difference exceeds a second threshold, apply micro-rate correction to subsequent time slices, and rearrange and discard subtitle time slices according to player clock events in the case of fast playback and dragging, thereby solving problems such as unsynchronized subtitles and video playback, time drift, and subtitle misalignment during fast playback and dragging.

[0035] Preferably, the commercial display mainboard is provided with a resource management controller for monitoring the load of the CPU and GPU and reporting the monitoring results to the subtitle display engine; the subtitle display engine encapsulates the subtitle time slices output by the timestamp synchronization module into a subtitle stream after receiving the subtitle time slices, and the subtitle stream contains a start time, an end time, a line slice sequence, and a style instruction; the subtitle display engine establishes a subtitle rendering queue in its internal and constructs the subtitle rendering queue into a ring buffer to be rendered in turn according to a time priority; when the resource management controller reports an increase in CPU or GPU pressure, the subtitle display engine reduces the stroke and shadow radius, reduces the maximum number of threads and refresh frequency, and degrades to monochrome and shadow-free rendering under a preset limit, but still maintains clear fonts and baseline alignment according to a degradation protocol.

[0036] In this embodiment, a resource management controller is arranged in the commercial display mainboard to monitor the load conditions of the CPU and GPU in real time and report the monitoring results to the subtitle display engine. After receiving the subtitle time slice output by the time stamp synchronization module, the subtitle display engine encapsulates it as a subtitle stream containing the start time, end time, line slice sequence and style instruction, and establishes an internal ring buffer subtitle rendering queue to ensure time priority. At the same time, when the resource pressure rises, the rendering system gradually reduces the stroke and shadow radius, reduces the number of issues and refresh frequency according to the degradation protocol, and in extreme cases, degenerates to single-color shadow-free rendering but maintains clear font and baseline alignment, which can solve the problem of subtitle stuttering, increased rendering delay or even display failure of the subtitle rendering system in resource-constrained or hardware power-constrained scenarios. In this embodiment, the resource management controller mechanism can actively monitor hardware load and transmit feedback information to the subtitle engine in real time, so that the subtitle rendering has dynamic scheduling capability. The rendering queue of the ring buffer ensures that subtitles can be dequeued in time priority and will not cause large-scale loss of subtitles due to transient overload. The design of the degradation protocol provides a gradual solution: when the resource is tight, the special effect complexity (stroke and shadow radius) is reduced first, then the number of parallel rendering is reduced, and finally the basic monochrome rendering is degraded, so that the subtitle can still be displayed stably and synchronized under different load pressures, ensuring the robustness and user experience of the system. Even in extreme scenarios, subtitles can still exist in a simplified form, avoiding the serious experience problem of subtitle disappearance, thereby improving the usability and reliability of the overall system.

[0037] Preferably, the commercial display mainboard enters the startup phase of the system running environment during the power-on initialization process, and the MCU control module traverses the peripheral bus connected to the commercial display mainboard in this startup phase to discover available peripherals, and reads the capability description of each peripheral, which includes function identification, value range, step and state reading method. The MCU control module aggregates the capability description and generates an overlapping function set and a function device mapping table based on the configuration strategy. When the subtitle display engine or the AI subtitle processing module issues a scenario-based linkage request, the MCU control module constructs a single control instruction and performs atomic execution and uniform time reference alignment on multiple peripherals, completes synchronous control and state readback, and reports the state readback result to the state manager component built-in the commercial display mainboard.

[0038] In this embodiment, in the startup stage of the commercial display mainboard, the MCU control module traverses the peripheral bus, discovers available peripherals, reads the capability description of the function identifier, value range, step and state mode, then aggregates the capability description and generates the overlapping function set and function device mapping table based on the configuration strategy; when the scene linkage request is issued by the subtitle display engine or the AI subtitle processing module, the MCU control module constructs a single control instruction and performs atomic execution and uniform time reference alignment on multiple peripherals, while completing state readback and reporting to the state manager component, which can solve the problems of lack of unified discovery mechanism in the startup stage, lack of atomicity in execution and lack of alignment reference in peripheral control. Through the scheme of this embodiment, the MCU discovers peripherals and establishes capability description at system startup, avoiding subsequent linkage due to unknown peripherals. At the same time, the atomic execution of the single instruction ensures the uniform time reference of cross-device operation, ensuring that the peripheral action is synchronized with the subtitle. The state readback mechanism provides a feedback link for the upper module, avoiding the problem of unknown execution failure of the peripheral.

[0039] Preferably, the MCU control module defines volume, mute, brightness, backlight, color temperature, fan speed and indicator light state as basic entries of the overlapping function set; when executing a single control instruction, the MCU control module first performs two-phase commit: in the pre-commit phase, the capabilities and states of each peripheral are checked and resources are reserved, and in the commit phase, the target values are simultaneously issued according to the synchronous trigger mark and the versioned state is aggregated; when any peripheral fails the check, the MCU control module broadcasts a rollback instruction and restores each peripheral to the pre-commit state, to ensure cross-device state consistency and instruction idempotency.

[0040] In this embodiment, the MCU control module defines volume, mute, brightness, backlight, color temperature, fan speed and indicator light state as basic entries of the overlapping function set, and adopts a two-phase commit mechanism when executing a single control instruction: in the pre-commit phase, the capabilities and states of the peripherals are checked and resources are reserved, and in the commit phase, the target values are simultaneously issued and the versioned state is aggregated; if any peripheral fails the check, a rollback instruction is broadcast to restore to the pre-commit state, which can solve the problem of inconsistent overall state and irreversible operation caused by partial device failure in the cross-peripheral control process. In this embodiment, the two-phase commit mechanism makes cross-device control transactional and idempotent, ensuring that either all executions are successful or all rollbacks are performed, without partial execution. At the same time, through versioned state management, the peripherals can maintain consistency in different batches of execution, avoiding state drift.

[0041] Preferably, the MCU control module is compatible with I2C, SPI, UART, RS485 and CAN at the physical link layer, and exposes a unified interface through a driver abstraction layer; when transmitting control messages between the MCU control module and each peripheral, the MCU control module appends CRC check to the control messages and sets a timeout retransmission to improve reliability; when the MCU control module detects link congestion or temporary unavailability of a peripheral, the MCU control module enters a backoff retry period, and switches to a soft bypass mode after the number of retries reaches a preset threshold, and performs a downgrade action on the reachable peripheral and records an unreachable device list; the MCU control module judges the link recovery by periodic heartbeat detection or state query, and after confirming the link recovery, the MCU control module fills in the control actions not performed during the abnormal period according to the recorded timestamp sequence, and accounts with the state information returned by the peripheral.

[0042] In this embodiment, the MCU control module is compatible with I2C, SPI, UART, RS485 and CAN at the physical link layer, and provides a unified interface through a driver abstraction layer; a CRC check and a timeout retransmission mechanism are appended when transmitting messages; when the link is congested or the peripheral is temporarily unavailable, the MCU control module enters a backoff retry period, and switches to a soft bypass mode after exceeding the threshold, and performs a downgrade operation on the reachable peripheral and records an unreachable list; after the link is recovered, the unexecuted actions are filled in based on the timestamp sequence and accounted with the state, which can solve the problem of control failure caused by unreliable link, temporary interruption or packet loss in peripheral communication. The multi-layer mechanism in this embodiment ensures reliability: CRC and timeout retransmission solve the reliability at the data level, backoff retry and soft bypass ensure uninterrupted control, and the filling and accounting mechanism ensures state synchronization after link recovery.

[0043] Preferably, the resource manager sets quotas and priorities for CPU, GPU and memory bandwidth on the commercial display mainboard, and puts the inference tasks of the AI subtitle processing module and the rendering tasks of the subtitle display engine into different priority queues; when the temperature or power consumption of the commercial display mainboard approaches the threshold, the resource manager calls the dynamic voltage and frequency adjustment function built-in the commercial display mainboard to reduce the voltage and frequency of CPU and GPU to reduce power consumption, and simultaneously issues a model adaptive downgrading instruction to the AI subtitle processing module, so that it switches to a quantization model during inference, and reduces the overlap rate of waveform windows in the speech front-end processing link to reduce the amount of calculation; at the same time, the resource manager issues a rendering simplification instruction to the subtitle display engine to reduce the complexity of special effects; while the downgrading and rendering simplification are being executed, the resource manager ensures that the timestamp synchronization module maintains a stable correction period, and guarantees the minimum number of lines and font clarity of the subtitle display engine to meet the requirements of time and readability.

[0044] In this embodiment, the resource manager sets quotas and priorities for CPU, GPU and memory bandwidth on the commercial display mainboard, and puts the AI subtitle processing task and the rendering task into different priority queues; when the temperature or power consumption approaches the threshold, the dynamic voltage and frequency adjustment (DVFS) function is called to reduce the frequency of CPU / GPU, and the model adaptive reduction instruction is issued to the AI subtitle processing module to switch to the quantization model, and the calculation complexity of the speech front-end processing is reduced; and the rendering simplification instruction is issued to the subtitle display engine to reduce the special effect complexity; in this process, the timestamp synchronization module still maintains a stable correction period and the lowest subtitle clarity, which can solve the problem of unstable, crash or low time accuracy of the subtitle system under high load or overheating of the commercial display system.

[0045] Preferably, the subtitle display engine internally includes a font and layout subsystem for providing font shaping and layout optimization when the subtitle display engine performs text rendering; the font and layout subsystem establishes a text shaping cache and glyph path reuse mechanism in the subtitle display engine, and hits the glyph cache according to the glyph hash and style parameters to reduce rasterization overhead; in a complex script scenario, the font and layout subsystem performs baseline alignment and ascender / descender optimization on the subtitle text, and inserts a minimum dwell time at punctuation according to the reading rhythm to improve readability; on a high DPI screen, the font and layout subsystem adaptively adjusts the stroke and shadow radius according to the pixel ratio, and limits the maximum number of transparent overlay layers, while ensuring that the baseline of the layout result and the subtitle display area parameters remain consistent.

[0046] In this embodiment, the font and layout subsystem is designed inside the subtitle display engine, a text shaping cache and glyph path reuse mechanism are established, and the glyph cache is hit based on glyph hash and style parameters to reduce rasterization overhead; baseline alignment and ascender / descender optimization are performed in a complex script scenario, and a minimum dwell time is inserted at punctuation to improve reading rhythm; on a high DPI screen, the stroke and shadow radius are adaptively adjusted according to the pixel ratio, the number of transparent overlay layers is limited, and consistency with the baseline of the subtitle display area is maintained, which can solve the problems of uneven layout, large font rendering overhead, and font blurring on high-resolution screens in complex language scenarios. The scheme of this embodiment reduces repeated calculations through the cache and reuse mechanism, improves performance, improves readability through baseline and rhythm control, and ensures that the font is still clear on screens of different resolutions through DPI adaptation, thereby realizing efficient, beautiful and consistent font rendering across devices, and users can obtain clear and comfortable reading experience on various display devices.

[0047] Preferably, the commercial display mainboard further comprises an observable and adaptive module, which is used to periodically collect end-to-end delay, time error, recognition real-time rate, rendering time consumption and peripheral device execution time consumption during the operation of the commercial display mainboard system, and form a sliding statistical result; when the sliding mean and variance of the time error exceed a threshold value, the observable and adaptive module generates an optimization instruction, the instruction time stamp synchronization module adjusts the buffer size of the jitter buffer sub-module and modifies the threshold range micro rate correction upper limit; when the recognition real-time rate decreases, the observable and adaptive module generates an optimization instruction, the instruction AI subtitle processing module reduces the semantic analysis complexity and merges short line pieces; when the peripheral device execution time consumption is abnormal, the observable and adaptive module generates an optimization instruction, the instruction MCU control module adjusts the control batch size and grouping synchronization domain; the execution results of all optimization instructions are fed back to the resource manager, so as to maintain the system stable and available under different loads and environments.

[0048] It should be noted that the observable and adaptive module is added in the commercial display mainboard, the end-to-end delay, time error, recognition real-time rate, rendering time consumption and peripheral device time consumption are periodically collected, and a sliding statistical result is formed; when the time error exceeds a threshold value, an optimization instruction is generated to adjust the buffer size and micro rate correction upper limit of the time stamp synchronization module; when the recognition real-time rate decreases, the instruction AI subtitle processing module reduces the semantic analysis complexity and merges short line pieces; when the peripheral device time consumption is abnormal, the instruction MCU module adjusts the batch size and grouping synchronization domain; all optimization results are fed back to the resource manager, which can solve the problems of large performance fluctuation, unstable subtitle time and recognition under dynamic load. Through the adaptive feedback closed loop mechanism of the embodiment, the system can real-time optimize the parameters of the key modules and maintain the overall performance stable.

[0049] Embodiment two

[0050] Referring to Figure 1 , the embodiment provides a commercial display mainboard, which integrates an AI subtitle processing module, a time stamp synchronization module, a subtitle display constraint solving module and a subtitle display engine. The method for displaying subtitles based on semantic recognition of the commercial display mainboard in the above embodiment is used for semantic recognition and display subtitle control of a playing file.

[0051] It should be noted that the above embodiments are only preferred specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. The protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for displaying subtitles based on semantic recognition of a commercial display motherboard, characterized in that, The commercial display motherboard integrates an AI subtitle processing module, a timestamp synchronization module, a subtitle display constraint solving module, and a subtitle display engine. The method includes the following steps: S1. The AI ​​subtitle processing module reads audio data with presentation timestamps (PTS) frame by frame from the audio frame stream output by the player's decoding pipeline, performs speech front-end processing, speech recognition, and semantic analysis locally, and obtains semantic text data with confidence level and punctuation. The timestamp synchronization module uses the PTS of the audio frame and the system clock as a basis to convert the semantic text data into subtitle time slices aligned with the frame and marks the start and end times and confidence threshold. S2. The subtitle display constraint solving module is based on the physical shape parameters and display configuration parameters of the commercial display motherboard, and combined with the current video frame provided by the player decoding pipeline. It obtains a renderable heat map through saliency and OCR analysis, and then solves the subtitle display area parameters and non-renderable mask according to the preset avoidance priority and safety margin. The subtitle display area parameters are then provided to the subtitle display engine for rendering control. S3. The subtitle display engine receives the subtitle time slice output by the timestamp synchronization module and the subtitle display area parameters output by the subtitle display constraint solving module. Within it, it performs glyph shaping and bidirectional layout on the subtitle lines and generates the layout subtitle layer according to the subtitle display area parameters. The subtitle display engine renders the subtitle layer in real time on the GPU overlay plane, completes alpha blending and stroke shadow overlay, and aligns the subtitle start and end times with the video presentation time according to the threshold range micro-rate correction strategy of the timestamp synchronization module. Finally, it outputs the time-aligned subtitle overlay image to the video compositor for composite display.

2. The method according to claim 1, characterized in that, The physical form parameters of the commercial display motherboard include horizontal screen, vertical screen or irregular screen type. The display configuration parameters of the commercial display motherboard include resolution, screen ratio and key content avoidance area. The subtitle display area parameters include area coordinates, line width, line height and baseline, and together with the non-renderable mask, constrain the layout boundary of the subtitle text.

3. The method according to claim 1, characterized in that, The AI ​​subtitle processing module includes an acoustic front-end submodule, an edge recognition submodule, a language adaptation submodule, a confidence assessment submodule, and a result integration and output submodule. The acoustic front-end submodule performs resampling, endpoint detection, and dynamic gain control on the received audio frames to improve the speech signal-to-noise ratio and stabilize energy distribution. The edge recognition submodule loads a lightweight deep neural network locally on the commercial display motherboard to decode the processed audio frames, supporting CTC decoding and attention-based sequence-to-sequence inference in parallel to obtain preliminary transcription results. The language adaptation submodule receives the preliminary... The transcription results are used to switch between different sub-vocabularies and language models based on language scores to adapt to mixed Chinese-English or multilingual spoken language. The confidence assessment submodule performs numerical confidence calculation and segment quality judgment on the text output by the language adaptation submodule, triggers delayed confirmation and backfilling for low-confidence segments, and adds confidence labels to each word or segment. The result integration and output submodule receives the text with confidence labels, performs field normalization and data structuring encapsulation, generates semantic text data, and submits the semantic text data to the timestamp synchronization module for generating subtitle time slices.

4. The method according to claim 3, characterized in that, The AI ​​subtitle processing module also includes a semantic analysis submodule, a reading rhythm control submodule, and a confidence mapping submodule. After completing the initial transcription, the semantic analysis submodule performs punctuation restoration, named entity recognition and terminology standardization, sensitive word normalization and digital unit normalization to obtain structured text. The reading rhythm control submodule receives the structured text and divides it into line segments and paragraph segments according to the target reading speed, while generating soft line break marks. When the width of the line segment exceeds the line width parameter of the subtitle display engine, the reading rhythm control submodule prioritizes breaking the line at the soft line break, otherwise it calls the equivalent phrase mapping table to shrink the long paragraph into short words. The confidence mapping submodule receives the aforementioned confidence labels and maps different confidence values ​​to corresponding display styles to distinguish backfilled text from highly reliable text. It then submits the integrated text to the timestamp synchronization module to generate corresponding subtitle time slices.

5. The method according to claim 1, characterized in that, The subtitle display constraint solving module includes a saliency detection submodule and an OCR text block detection submodule. After receiving physical shape parameters, display configuration parameters, and the current video frame, the subtitle display constraint solving module calls the saliency detection submodule to perform saliency analysis on the video frame to generate a saliency heatmap, and calls the OCR text block detection submodule to perform text block recognition on the video frame to generate a text block set. The saliency heatmap, the text block set, and the preset key content avoidance area are combined to construct a multi-objective cost function. The subtitle display constraint solving module uses occlusion of the main visual field, coverage of existing OSD, and intrusion into the safety margin as penalty factors to solve for the subtitle display area parameters within the feasible region. If the solution fails, the subtitle display constraint solving module outputs the area parameters and transparency suggestion values ​​for the screen adaptive bottom band mode and provides them to the subtitle display engine for rendering control.

6. The method according to claim 2, characterized in that, After receiving the subtitle display area parameters, the subtitle display engine first limits the layout area of ​​the subtitle text according to the irregular screen mask, and then recalculates the line width, line height and baseline of the subtitle text according to the nine-grid safe area and minimum margin strategy to adapt to the switching between landscape, portrait and irregular screens. When multilingual subtitles are required, the subtitle display engine performs bidirectional algorithms and complex script form replacement on the multilingual subtitle text, and enables glyph caching and letter spacing fine-tuning during the font rendering process of the subtitle text to reduce rasterization time. The subtitle display engine performs alpha blending and stroke shadow overlay on the GPU overlay plane for the subtitle layer after typesetting and font rendering, and outputs the timed subtitle overlay image to the video compositor for composite display.

7. The method according to claim 1, characterized in that, The timestamp synchronization module includes a phase-locked loop (PLL) modeling submodule and a jitter buffer submodule. Upon receiving the audio PTS and video presentation time, the timestamp synchronization module calls the PLL modeling submodule to establish a PLL model to track the synchronization relationship between the audio PTS and video presentation time, and calls the jitter buffer submodule to absorb the timestamp jitter generated during the transmission and buffering of the audio frame stream output from the player's decoding pipeline. When the difference between the subtitle time slice and the video presentation time exceeds a first threshold, the timestamp synchronization module triggers instantaneous realignment. When the difference exceeds a second threshold, the timestamp synchronization module triggers a smooth resampling strategy and applies threshold-range micro-rate correction to subsequent time slices. During accelerated playback and random dragging, the timestamp synchronization module rearranges and discards subtitle time slices based on the player's clock event, and provides the corrected subtitle time slices to the subtitle display engine for time-synchronized rendering.

8. The method according to claim 1, characterized in that, The commercial display motherboard is equipped with a resource controller to monitor the CPU and GPU load and report the monitoring results to the subtitle display engine. After receiving the subtitle time slice output by the timestamp synchronization module, the subtitle display engine encapsulates the subtitle time slice into a subtitle stream, which includes a start time, end time, line sequence, and style instructions. The subtitle display engine establishes a subtitle rendering queue internally and constructs the subtitle rendering queue as a circular buffer to render subtitles sequentially according to time priority. When the resource controller reports an increase in CPU or GPU pressure, the subtitle display engine reduces the stroke and shadow radius, the maximum concurrent output, and the refresh rate according to a degradation protocol. Under preset extreme conditions, it degrades to monochrome, shadowless rendering, but still maintains font clarity and baseline alignment.

9. A commercial display motherboard, characterized in that, The commercial display motherboard uses the method for displaying subtitles based on semantic recognition of the commercial display motherboard as described in any one of claims 1-8 to perform semantic recognition and subtitle display control on the playback file.

Images