A scene-aware intelligent subtitle generation method and system

Abstract

This invention relates to a scene-aware intelligent subtitle generation method and system. By dynamically extracting real-time visual features from the video and parsing the audio stream, it integrates multimodal information to establish an intelligent decision-making model. This enables deep collaboration between subtitle generation and presentation and the video content and semantic context. Based on scene transitions, character actions, changes in key objects and focal areas, it intelligently determines the timing and location of subtitle triggering. An avoidance algorithm ensures that subtitles do not obscure key visual elements. Integrated voice emotion analysis dynamically adjusts the font color, outline, and transparency of the subtitles, thereby enhancing emotional expressiveness and visual comfort while accurately conveying semantic information. This invention effectively solves the problems of fragmented audiovisual information, image obstruction, monotonous styles, and lack of emotional adaptation in traditional subtitle generation technologies, significantly improving the overall user experience in various video viewing scenarios.

Description

Technical Field

[0001] This application relates to the fields of video processing and multimedia technology, and in particular to a scene-aware intelligent subtitle generation method and system. Background Technology

[0002] Traditional video captioning technologies mostly rely solely on audio information for text recognition and timeline alignment, lacking depth perception and dynamic adaptation capabilities for visual scenes. Existing methods typically place captions statically in fixed areas, which not only easily obscures key information on the screen but also struggles to adapt to changing video content and complex scene compositions. Especially in highly dynamic scenes or multi-person dialogue environments, the way captions are presented often clashes with the visual focus, severely impacting the user's viewing experience.

[0003] Furthermore, conventional subtitle generation systems generally neglect collaborative decision-making among multimodal information, failing to intelligently adjust the triggering timing and display logic of subtitles based on visual events in the scene. Simultaneously, existing solutions are relatively weak in analyzing speech emotion and semantic structure, resulting in subtitles only performing basic translation functions, lacking emotional expression and contextual adaptation capabilities, and failing to meet users' dual demands for high-quality subtitles in terms of information accuracy and visual comfort.

[0004] On the other hand, most current subtitle rendering mechanisms lack the ability to adapt to the display environment. For example, problems such as insufficient contrast between subtitle color and background, obscuring key visual elements, and monotonous styles that fail to convey emotion are common. Especially on mobile or small-screen devices, existing technologies struggle to achieve intelligent avoidance and visual coordination between subtitles and dynamic images, and cannot dynamically optimize styles by combining voice emotions, thus limiting their application in diverse audiovisual scenarios. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] According to a first aspect of the present invention, the present invention claims protection for a scene-aware intelligent subtitle generation method, comprising:

[0007] S1, dynamically extract the real-time visual features of the video to be subtitled, including scene switching frames, changes in the amplitude of character movements, coordinates of key objects, and coordinates of the focal area of ​​the screen;

[0008] S2, parse the audio stream of the video to be subtitled, identify speech pauses and segment semantic units, and extract speech sentiment values ​​at the same time;

[0009] S3, establish a multimodal decision model, integrate the real-time visual features and semantic units to generate a dynamic subtitle triggering strategy. When the focus area of ​​the screen is detected to be stable and continuous for a preset duration and the change value of the action amplitude is less than a preset threshold, the subtitle display is activated, intelligent sentence segmentation is performed at the speech pause point, and the coordinates of key objects in the current screen are associated.

[0010] S4. Based on the speaker's face coordinates or the center coordinates of key objects in the current screen, dynamically calculate the subtitle display area and use an avoidance algorithm to make the subtitles avoid the focal area of ​​the screen. When the speaker is located on the side of the screen, activate the eye correction positioning mode and place the subtitles in the area of ​​the extension line of the person's line of sight.

[0011] S5, extract the background main color of the display area of ​​the subtitle in real time, and automatically adjust the subtitle font color and outline color through a color contrast optimization algorithm to ensure that the contrast meets the preset requirements;

[0012] S6 dynamically adjusts font effects based on semantic sentiment values, using gradient color rendering for positive emotions and increasing transparency compensation for negative emotions.

[0013] Furthermore, S1 also includes:

[0014] Compare the differences in HSV color histogram distribution frame by frame, and mark the switching point as a scene switching frame when the difference exceeds the dynamic threshold for 5 consecutive frames.

[0015] By analyzing pixel-level motion vectors in adjacent frames, the percentage value of global motion intensity in the image is calculated as the value of change in motion amplitude.

[0016] An improved YOLO model is used to output the center point coordinates and bounding box of people / objects in the image in real time, so as to obtain the coordinates of key objects.

[0017] Heatmaps are generated based on visual saliency algorithms, and areas with a heat value >85% are marked as focal areas for focal area determination;

[0018] Scene switching frame detection uses a dual threshold determination mechanism. When the difference in HSV histograms of 5 consecutive frames is greater than the threshold β, or the replacement rate of the main object in the YOLO object detection results is greater than 60%, it is determined to be a valid scene switch.

[0019] Furthermore, S2 also includes:

[0020] By combining short-time energy zero-crossing rate and silence segment detection, breakpoints are inserted at the boundaries of silence segments as speech pause points;

[0021] By using a bidirectional LSTM neural network to identify grammatical boundaries, the continuous speech stream is segmented into complete semantic groups for semantic unit segmentation.

[0022] Extract the fundamental frequency and formants of the voiceprint features and the text features, and output a continuous value from -1 negative to +1 positive as the sentiment tendency value;

[0023] Speech sentiment analysis employs a spectrogram-text cross-modal model to extract the joint feature vector of Mel frequency cepstral coefficients and semantic keywords, and outputs the sentiment tendency confidence score.

[0024] Furthermore, S3 also includes:

[0025] Set the subtitle trigger condition to activate the display window when the focus area remains stable for 500ms and the motion intensity is <15%;

[0026] It employs an intelligent sentence segmentation mechanism to force line breaks at speech pauses and binds the current semantic unit to the coordinates of key objects activated in the image.

[0027] The semantic unit segmentation adopts a multi-head attention mechanism. When an interrogative word or exclamation word is detected, a 0.3-second display delay is forcibly added after the word.

[0028] Furthermore, S4 also includes:

[0029] An avoidance algorithm is used to monitor the changes in the coordinates of the focus area in real time and dynamically adjust the position of the subtitle box to ensure that the minimum distance between the subtitle box and the focus area is greater than the length proportional to the screen width.

[0030] Perform eye-correction positioning. When the character is located in the left or right 1 / 3 area of ​​the screen, locate the subtitle anchor point along the line extending from the direction of the character's pupil gaze.

[0031] When performing the eye-correction positioning, a fan-shaped area with the pupil coordinates of the person as the origin and the line of sight ±15° is established, and the subtitle anchor point is dynamically snapped to the nearest coordinate point to the center of the screen within this fan-shaped area.

[0032] Furthermore, S5 also includes:

[0033] The subtitle area is sampled in a grid pattern, and after noise is eliminated by median filtering, the main hue is determined and the main color is extracted.

[0034] Automatically select font or stroke color with a brightness difference of >70% from the background color, and adjust the stroke width proportionally to the font size to ensure contrast.

[0035] When performing color optimization calculations, the average LAB color space value of a 5×5 pixel grid in the subtitle area is extracted;

[0036] The complementary color candidate set is calculated based on the brightness difference ΔL: when ΔL < 30, a high-contrast stroke is enabled, and the stroke width is adaptively set to 8%-12% of the font size.

[0037] Furthermore, S6 also includes:

[0038] Perform gradient color rendering, filling positive emotional statements with a gradient from warm yellow to orange, with the gradient direction advancing horizontally;

[0039] Transparency is compensated by applying a 40%-60% semi-transparency effect to negative emotional statements to reduce visual impact.

[0040] Furthermore, the method also includes:

[0041] Using an ad insertion engine, when a preset keyword appears in a semantic unit, a semi-transparent ad label is overlaid at the coordinates of the associated key object, and its transparency is adjusted in the opposite direction to the object's movement speed.

[0042] The insertion of the advertising logo must simultaneously meet the following conditions: the duration of the current scene is greater than 3 seconds, the overlap between the focal area of ​​the screen and the coordinates of the key object is greater than 70%, and the emotional tendency of the voice is neutral or positive.

[0043] Furthermore, the method also includes:

[0044] An anti-conflict mechanism is established. When multiple speakers are detected on screen at the same time, dialogue bubble-style subtitles are generated at the waist coordinates of the corresponding person based on the sound source localization result. Different subtitles for different speakers use differentiated border color encoding.

[0045] According to a second aspect of the present invention, the present invention claims protection for a scene-aware intelligent subtitle generation system, comprising:

[0046] One or more processors;

[0047] A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the scene-aware intelligent subtitle generation method.

[0048] This invention relates to a scene-aware intelligent subtitle generation method and system. By dynamically extracting real-time visual features from the video and parsing the audio stream, it integrates multimodal information to establish an intelligent decision-making model. This enables deep collaboration between subtitle generation and presentation and the video content and semantic context. Based on scene transitions, character actions, changes in key objects and focal areas, it intelligently determines the timing and location of subtitle triggering. An avoidance algorithm ensures that subtitles do not obscure key visual elements. Integrated voice emotion analysis dynamically adjusts the font color, outline, and transparency of the subtitles, thereby enhancing emotional expressiveness and visual comfort while accurately conveying semantic information. This invention effectively solves the problems of fragmented audiovisual information, image obstruction, monotonous styles, and lack of emotional adaptation in traditional subtitle generation technologies, significantly improving the overall user experience in various video viewing scenarios. Attached Figure Description

[0049] Figure 1 A flowchart illustrating the workflow of a scene-aware intelligent subtitle generation method claimed in this application.

[0050] Figure 2 A second workflow diagram of a scene-aware intelligent subtitle generation method claimed in an embodiment of this application;

[0051] Figure 3 A third flowchart illustrating a scene-aware intelligent subtitle generation method claimed in an embodiment of this application;

[0052] Figure 4 This is a structural block diagram of a scene-aware intelligent subtitle generation system claimed in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0054] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0055] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0056] According to a first embodiment of the present invention, the present invention claims protection for a scene-aware intelligent subtitle generation method, referring to... Figure 1 ,include:

[0057] S1, dynamically extract the real-time visual features of the video to be subtitled, including scene switching frames, changes in the amplitude of character movements, coordinates of key objects, and coordinates of the focal area of ​​the screen;

[0058] S2, parse the audio stream of the video to be subtitled, identify speech pauses and segment semantic units, and extract speech sentiment values ​​at the same time;

[0059] S3, establish a multimodal decision model, integrate the real-time visual features and semantic units to generate a dynamic subtitle triggering strategy. When the focus area of ​​the screen is detected to be stable and continuous for a preset duration and the change value of the action amplitude is less than a preset threshold, the subtitle display is activated, intelligent sentence segmentation is performed at the speech pause point, and the coordinates of key objects in the current screen are associated.

[0060] S4. Based on the speaker's face coordinates or the center coordinates of key objects in the current screen, dynamically calculate the subtitle display area and use an avoidance algorithm to make the subtitles avoid the focal area of ​​the screen. When the speaker is located on the side of the screen, activate the eye correction positioning mode and place the subtitles in the area of ​​the extension line of the person's line of sight.

[0061] S5, extract the background main color of the display area of ​​the subtitle in real time, and automatically adjust the subtitle font color and outline color through a color contrast optimization algorithm to ensure that the contrast meets the preset requirements;

[0062] S6 dynamically adjusts font effects based on semantic sentiment values, using gradient color rendering for positive emotions and increasing transparency compensation for negative emotions.

[0063] Furthermore, referring to Figure 2 S1 further includes:

[0064] Compare the differences in HSV color histogram distribution frame by frame, and mark the switching point as a scene switching frame when the difference exceeds the dynamic threshold for 5 consecutive frames.

[0065] By analyzing pixel-level motion vectors in adjacent frames, the percentage value of global motion intensity in the image is calculated as the value of change in motion amplitude.

[0066] An improved YOLO model is used to output the center point coordinates and bounding box of people / objects in the image in real time, so as to obtain the coordinates of key objects.

[0067] Heatmaps are generated based on visual saliency algorithms, and areas with a heat value >85% are marked as focal areas for focal area determination.

[0068] The scene switching frame detection adopts a dual threshold determination mechanism. When the difference in HSV histograms of 5 consecutive frames is greater than the threshold β, or the replacement rate of the main object in the YOLO object detection results is greater than 60%, it is determined to be a valid scene switching.

[0069] In this embodiment, the dual threshold determination mechanism includes:

[0070] HSV Histogram Difference: After converting the image to HSV space, a 64-dimensional histogram is generated, and a decision is triggered when the Bach distance is >0.7.

[0071] Object replacement rate detection: Calculate the overlap of the top 5 most significant objects between adjacent scenes. When the disappearance rate of the core object is >60%, a switch is triggered.

[0072] Furthermore, referring to Figure 3 S2 further includes:

[0073] By combining short-time energy zero-crossing rate and silence segment detection, breakpoints are inserted at the boundaries of silence segments as speech pause points;

[0074] By using a bidirectional LSTM neural network to identify grammatical boundaries, the continuous speech stream is segmented into complete semantic groups for semantic unit segmentation.

[0075] Extract the fundamental frequency and formants of the voiceprint features and the text features, and output a continuous value from -1 negative to +1 positive as the sentiment tendency value.

[0076] Among them, the speech sentiment analysis adopts a spectrogram-text cross-modal model to extract the joint feature vector of Mel frequency cepstral coefficients and semantic keywords, and outputs the sentiment tendency confidence score.

[0077] In this embodiment, the voice sentiment analysis model includes spectral features of a 128-dimensional feature vector extracted from the Mel spectrum, text features of a 300-dimensional word embedding extracted from sentiment keywords, and cross-modal fusion outputting a 0-1 confidence level through a two-stream neural network.

[0078] The specific implementation of the spectrogram-text cross-modal model includes:

[0079] Input features: Mel spectrum 128-dimensional feature vector, text sentiment keyword 300-dimensional word embedding vector;

[0080] Model architecture: A two-stream neural network is adopted. One stream processes spectral features, and the other processes text features. Feature fusion is performed through a cross-modal attention mechanism;

[0081] Output: Sentiment tendency confidence (a continuous value between 0 and 1);

[0082] Training method: End-to-end training is carried out using audio-text paired data with sentiment annotations, and the loss function is cross-entropy loss.

[0083] Furthermore, the S3 further includes:

[0084] Set the subtitle trigger condition to activate the display window when the focus area remains stable for 500 ms and the motion intensity < 15%;

[0085] Adopt an intelligent sentence-breaking mechanism, force a line break at the speech pause point, and bind the current semantic unit to the coordinates of the key objects activated in the picture;

[0086] The semantic unit segmentation adopts a multi-head attention mechanism. When a question word or an exclamation word is detected, a 0.3-second display delay is forced after the word.

[0087] Among them, in this embodiment, when semantic segmentation is enhanced, a multi-head attention mechanism is adopted to parallelly analyze three-way features of the phoneme sequence, the grammatical structure, and the sentiment keywords;

[0088] Based on the word delay rule, when question words such as "ma" and "ne" or exclamation words such as "a" and "ya" are detected, the subtitle display is extended by 300 ms.

[0089] The specific implementation of the multi-modal decision model includes:

[0090] Input: Visual features (scene switching frame, action amplitude, key object coordinates, focus area) and semantic units (text content, sentiment value);

[0091] Model architecture: A multi-modal fusion network based on an attention mechanism is adopted. Visual and text features respectively extract high-level features through CNN and Bi-LSTM, and are fused through cross-modal attention weights;

[0092] Output: Dynamic subtitle trigger strategy (whether to display, display position, display duration, etc.);

[0093] Training method: Supervised learning is carried out using manually annotated subtitle trigger strategy data, and the optimization objective is the strategy accuracy.

[0094] Furthermore, the S4 further includes:

[0095] An avoidance algorithm is used to monitor the changes in the coordinates of the focus area in real time and dynamically adjust the position of the subtitle box to ensure that the minimum distance between the subtitle box and the focus area is greater than the length proportional to the screen width.

[0096] Perform eye-correction positioning. When the character is located in the left or right 1 / 3 area of ​​the screen, locate the subtitle anchor point along the line extending from the direction of the character's pupil gaze.

[0097] When performing the eye-correction positioning, a fan-shaped area with the pupil coordinates of the person as the origin and the line of sight ±15° is established, and the subtitle anchor point is dynamically snapped to the nearest coordinate point to the center of the screen within this fan-shaped area.

[0098] In this embodiment, when eye correction is implemented, a 15-degree sector area with the pupil as the vertex is established, and the Euclidean distance between each point in this area and the center of the screen is calculated 30 times per second, and the upper left corner of the subtitle is anchored to the coordinate point with the smallest distance.

[0099] Furthermore, S5 also includes:

[0100] The subtitle area is sampled in a grid pattern, and after noise is eliminated by median filtering, the main hue is determined and the main color is extracted.

[0101] Automatically select font or stroke color with a brightness difference of >70% from the background color, and adjust the stroke width proportionally to the font size to ensure contrast.

[0102] When performing color optimization calculations, the average LAB color space value of a 5×5 pixel grid in the subtitle area is extracted;

[0103] The complementary color candidate set is calculated based on the brightness difference ΔL: when ΔL < 30, a high-contrast stroke is enabled, and the stroke width is adaptively set to 8%-12% of the font size.

[0104] In this embodiment, during color optimization, the predetermined subtitle area is divided into 25 square grids and converted to the LAB color space to calculate the average brightness of each grid. When the average brightness difference is less than 30, a 3-pixel basic stroke is started, with the stroke width equal to the current font size × 0.1, forcing the stroke color to have a hue difference of more than 90° from the background color.

[0105] Furthermore, S6 also includes:

[0106] Apply a gradient color rendering, filling the positive emotional statements with a gradient from warm yellow to orange, with the gradient direction advancing horizontally.

[0107] Transparency is compensated by applying a 40%-60% semi-transparency effect to negative emotional statements to reduce visual impact.

[0108] Furthermore, the method also includes:

[0109] Using an ad insertion engine, when a preset keyword appears in a semantic unit, a semi-transparent ad label is overlaid at the coordinates of the associated key object, and its transparency is adjusted in the opposite direction to the object's movement speed.

[0110] The insertion of the advertising logo must simultaneously meet the following conditions: the duration of the current scene is greater than 3 seconds, the overlap between the focal area of ​​the screen and the coordinates of the key object is greater than 70%, and the emotional tendency of the voice is neutral or positive.

[0111] In this embodiment, the ad insertion engine is triggered by keywords and matches semantic units in real time using a preset brand thesaurus; the center point of the ad logo is aligned with the center coordinates of the key object for coordinate association, and dynamic transparency is adopted, with the transparency increasing by 15% for every 10px / s increase in the object's movement speed.

[0112] The advertisement employs a triple constraint, including a time constraint: the current scene must last for more than 3000ms; a spatial constraint: the distance between the advertisement anchor point and the center of the focal area of ​​the screen must be less than 5% of the screen width; and a sentiment constraint: the sentiment value must be greater than 0 (positive) or between -0.3 and 0.3 (neutral).

[0113] Furthermore, the method also includes:

[0114] An anti-conflict mechanism is established. When multiple speakers are detected on screen at the same time, dialogue bubble-style subtitles are generated at the waist coordinates of the corresponding person based on the sound source localization result. Different subtitles for different speakers use differentiated border color encoding.

[0115] In this embodiment, for multi-person dialogue processing, the horizontal angle of the sound source is located by a microphone array, the sound source angle is mapped to the X-axis coordinate of the screen, a rounded rectangular subtitle box is generated at the corresponding person's belt position, and the border color is assigned according to the speaker ID: red / blue / green three-color encoding.

[0116] According to a second embodiment of the present invention, referring to Figure 4 This invention claims protection for a scene-aware intelligent subtitle generation system, comprising:

[0117] One or more processors;

[0118] A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the scene-aware intelligent subtitle generation method.

[0119] The embodiments of the present invention are implemented in the intelligent subtitle system for live sports broadcasts as follows:

[0120] Test environment:

[0121] The hardware features an NVIDIA A100×4 GPU cluster, a 96-core CPU, and an 8-channel microphone array.

[0122] The video sample is a 4K live broadcast of the UEFA Champions League final, 45 minutes long, including 12 slow-motion replays, 210 semantic units, and 5 commentators;

[0123] Comparison group: YouTube automatic captioning system.

[0124] First, scene transition detection is performed based on visual feature extraction;

[0125] Each frame is decomposed into HSV space, divided into 16×16 color blocks to generate a 64-dimensional histogram, and the Bach distance of 5 consecutive frames is calculated. When the cumulative difference is >0.75 (threshold β), the YOLO model is run synchronously to detect the main object, including players, goals, or soccer balls.

[0126] When the goal replay is switched, the HSV histogram difference is 0.82 > β, and the main object changes from "goalkeeper" to "celebrating player", with a replacement rate of 85%.

[0127] The system marks the switching point within 83ms of the corresponding video.

[0128] For motion amplitude analysis, calculate the displacement vector of SIFT feature points in adjacent frames and output the global motion intensity percentage value.

[0129] The amplitude of the action at the moment of shooting is ΔM=0.63, and the threshold α=0.15, which is judged as a high dynamic range image;

[0130] The focus area is tracked using the SiamRPN++ model. When the football occupies more than 15% of the screen, it is marked as the focus area. The focus coordinate error is determined to be less than 0.8% of the screen size.

[0131] The speech was analyzed using semantic unit segmentation. After microphone array noise reduction, clean human voice was extracted. Based on the energy threshold detection, a silence segment >350ms was used as the breakpoint. After detecting the interjection "Beautiful!", a forced delay of 0.3 seconds was added. The semantic unit accuracy was 98.1%, compared to 89.3% in the control group.

[0132] For sentiment analysis, a cross-modal model was used to extract 128-dimensional features from the Mel spectrum. The fundamental frequency jumped from 85Hz to 210Hz. The text keyword "unbelievable" triggered positive semantics, and the output sentiment value was +0.91, indicating strong positivity.

[0133] Caption triggering for multimodal decision-making includes:

[0134] In the scene of a yellow card decision, the close-up of the referee's gesture lasts for 1.2 seconds (>0.5 seconds), with a movement amplitude ΔM=0.09<α. The caption "Referee shows yellow card" is activated at the point where the commentator pauses.

[0135] The trigger rule for grading importance is to detect the proper noun "Messi" and the number "10", increase the font size by 12% (16px → 18px), add a gold pulse animation, and flash it with a 500ms cycle.

[0136] A spatial adaptive avoidance algorithm is employed, and specific implementations include:

[0137] During a penalty shootout, the focus area is the goalkeeper's face, with coordinates X:45%, Y:50%. The subtitles are automatically shifted to the bottom right corner, with coordinates X:85%, Y:80%, and the distance between the subtitles and the focus area is 22% of the screen width.

[0138] The eye gaze was corrected, and the pupil coordinates of the narrator were detected as X:28%, Y:50%. A fan-shaped area was established in the direction of the gaze with an azimuth angle of 120°±15°. The point closest to the center of the screen was located at X:35%, Y:65%, and the subtitle anchor point was snapped to this coordinate.

[0139] For multi-person dialogue processing, when three narrators are arguing, the sound source is located, where A (left 30°), B (middle 0°), and C (right -25°), and mapped to the screen coordinates: A (X:25%,Y:60%), B (X:50%,Y:60%), C (X:75%,Y:60%); and the waist bubble text is: A (red box), B (blue box), C (green box).

[0140] When optimizing colors, adaptive color matching is performed for the background color. In a specific implementation example, in a field grass scene, the subtitle pre-defined area is divided into a 5×5 grid. The background main color LAB value (L:48,a:-28,b:18) is detected, and the brightness difference ΔL=52 (>30) is selected. White font is selected, and the stroke width = font size 12px × 10% = 1.2px.

[0141] For emotional rendering, specific examples are as follows: in positive scenes, such as when a goal is scored, the gradient color is rendered as a horizontal gradient of #FFFF00 to #FFA500, while in negative scenes, such as when a foul is committed, the transparency is increased to 55% + gray tone.

[0142] A specific example of triggering verification for ad insertion engine is a sports drink ad. The semantic keyword "hydration" hits the word library, the scene lasts for 4.8 seconds (>3 seconds), the overlap between the player's water bottle and the object coordinates in the focus area is 92%, the sentiment value is +0.75 (positive), and a 40% transparent brand logo is superimposed on the water bottle.

[0143] Rules for dynamic transparency: When the kettle moves at a speed of 20px / s, the transparency is 40%; when the speed increases to 50px / s, the transparency is 70%.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0146] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for generating intelligent subtitles based on scene awareness, characterized in that, Comprise: S1, dynamically extract real-time visual features of the to-be-generated caption video, including scene switching frames, character motion amplitude change values, key object appearance coordinates and picture focus area coordinates; S2, analyze the audio stream of the to-be-generated caption video, identify speech pause points and segment semantic units, and extract speech emotion tendency values; S3, establish a multi-modal decision model, fuse the real-time visual features and semantic units to generate a dynamic caption triggering strategy, activate the caption display when the picture focus area is detected to be stable for a preset duration and the motion amplitude change value is less than a preset threshold, perform intelligent line breaking at the speech pause point, and associate the key object coordinates in the current picture; S4, dynamically calculate the caption display area according to the speaker's face coordinates or the key object center coordinates in the current picture, use an avoidance algorithm to make the caption avoid the picture focus area, and when the speaker is located at the side of the picture, start the eye correction positioning mode and place the caption in the character line of sight extension area; S5, real-time extract the background dominant color of the display area of the caption, automatically adjust the caption font color and outline color through color contrast optimization algorithm to ensure that the contrast meets the preset requirements; S6, dynamically adjust the font special effects according to the semantic emotion tendency values, use gradient color rendering for positive emotions, and increase the transparency compensation for negative emotions. 2.The scene-aware intelligent subtitle generation method of claim 1, wherein, The S1 further comprises: Frame-by-frame comparison of HSV color histogram distribution difference, when the difference of 5 consecutive frames exceeds the dynamic threshold, mark the switching point as a scene switching frame; Calculate the picture global motion intensity percentage value as the motion amplitude change value through adjacent frame pixel level motion vector analysis; Use the improved YOLO model to output the character / object center point coordinates and bounding box in the picture in real time to get the key object coordinates; Generate a heat map based on visual saliency algorithm, mark the area with heat value > 85% as focus area for focus area judgment; The scene switching frame detection uses a double threshold judgment mechanism, when the HSV histogram difference of 5 consecutive frames is > threshold β, or the subject object replacement rate in the YOLO object detection result is > 60%, it is determined as an effective scene switching. 3.The scene-aware intelligent subtitle generation method of claim 1, wherein, The S2 further comprises: Insert a breakpoint at the silent section boundary as a speech pause point by combining the short-time energy zero-crossing rate and silence section detection; Cut the continuous speech stream into complete semantic groups for semantic unit segmentation through bidirectional LSTM neural network to identify syntax boundaries; Extract the fundamental frequency and formant of voiceprint features and text features, output continuous values from -1 negative to +1 positive as emotion tendency values; The speech emotion analysis uses a spectrum-text cross-modal model to extract the joint feature vector of mel-frequency cepstrum coefficient and semantic keyword, and outputs the emotion tendency confidence. 4.The scene-aware intelligent subtitle generation method of claim 1, wherein, The S3 further comprises: Set the caption triggering condition to activate the display window when the focus area is stable for 500ms and the motion intensity is < 15%; Use an intelligent line breaking mechanism to force line breaks at speech pause points, and bind the current semantic unit with the activated key object coordinates in the picture; The semantic unit segmentation uses a multi-head attention mechanism, when a question tone word or an exclamation word is detected, a 0.3 second display delay is forced after the tone word.

5. The method for intelligent subtitle generation based on scene awareness according to claim 1, characterized in that, The S4 further comprises: Adopting an avoidance algorithm to monitor the focal point area coordinate changes in real time, dynamically adjusting the subtitle frame position to ensure that the minimum distance from the focal point area is greater than the first proportional length of the screen width; Performing eye correction positioning, when the character is located in the left and right 1 / 3 area of the screen, the subtitle anchor point is positioned along the line extended in the direction of the character's pupil line of sight; When performing the eye correction positioning, a fan-shaped area with the character's pupil coordinate as the origin and the line of sight direction ±15° is established, and the subtitle anchor point is dynamically adsorbed to the coordinate point closest to the center of the picture in the fan-shaped area.

6. The scene-aware intelligent subtitle generation method of claim 1, wherein, The S5 further comprises: Grid sampling is performed on the predetermined area of the subtitle, and the main color is determined after median filtering to eliminate noise points to extract the main color tone; The font or stroke color with a lightness difference greater than 70% from the background color is automatically selected, and the stroke width is adjusted in direct proportion to the font size to ensure contrast; The LAB color space average of the 5x5 pixel grid in the subtitle area is extracted when performing color optimization; According to the lightness difference ΔL, a complementary color candidate set is calculated: when ΔL<30, a high-contrast stroke is enabled, and the stroke width is adaptively set to 8%-12% of the font size.

7. The scene-aware intelligent subtitle generation method of claim 1, wherein, The S6 further comprises: Performing gradient color rendering, filling the positive emotional statements from warm yellow to orange, and the gradient direction is horizontally pushed forward; Compensating for transparency, applying a 40%-60% semi-transparent effect to negative emotional statements to reduce visual impact. 8.The scene-aware intelligent subtitle generation method of claim 1, wherein, The method further comprises: Using an advertisement implantation engine, when a preset keyword appears in a semantic unit, a semi-transparent advertisement logo is superimposed at the associated key object coordinate, and the transparency is inversely adjusted with the object movement speed; The implantation trigger of the advertisement logo needs to meet the following conditions simultaneously: the current scene duration is greater than 3 seconds, the overlap degree of the picture focal point area and the key object coordinate is greater than 70%, and the voice emotional tendency value is neutral or positive. 9.The scene-aware intelligent subtitle generation method of claim 1, wherein, The method further comprises: Establishing a conflict prevention mechanism, when multiple speakers are detected in the picture at the same time, generating a dialogue bubble-style subtitle at the corresponding character's waist coordinate according to the sound source positioning result, and using differentiated border color coding for subtitles of different speakers. 10.A scene-aware intelligent subtitle generation system, characterized in that, Comprise: One or more processors; A memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the scene-aware intelligent subtitle generation method according to any one of claims 1-9.

Images