Real-time 3D subtitle generation method and device based on depth perception
By combining depth perception technology and a graphics rendering engine, the position of the subtitles is automatically adjusted, which solves the problem of occlusion when the video scene changes, thus improving the accuracy of the subtitles and the user experience.
Patent Information
- Application Number
- CN202410986159.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing real-time 3D subtitle generation methods and devices lack dynamic adaptability when video scenes change, which makes subtitles easy to be obscured and affects the user's viewing experience.
It uses depth perception technology to capture depth images of video scenes, combines speech recognition and graphics rendering engines to automatically adjust the position of subtitles to avoid obstruction, and improves user interactivity through interactive displays and ranging sensors.
It achieves dynamic adaptability of subtitles, reduces the workload of manual editing, improves the accuracy and visual expressiveness of subtitles, and enhances the user experience.
Smart Images

Figure CN119011968A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of real-time 3D subtitle generation equipment, and in particular to a real-time 3D subtitle generation method and device based on depth perception. Background Art
[0002] A real-time 3D subtitle generation device is a technical device or system that can instantly generate and display three-dimensional effect subtitles during video playback or live broadcast. This device combines multiple technologies, including but not limited to computer graphics, video processing technology, and possible depth perception technology, to dynamically insert, position and render text information with three-dimensional visual effects in the video stream. However, when using the real-time 3D subtitle generation method and device, 3D subtitles are generally generated through traditional graphics processing technology and subtitle editing software, which lacks dynamic adaptability. When the video scene changes, the subtitles may be easily blocked, affecting the user's viewing experience. Summary of the invention
[0003] In order to overcome the problem that when using the real-time 3D subtitle generation method and device, 3D subtitles are generally generated through traditional graphics processing technology and subtitle editing software, which lacks dynamic adaptability. When the video scene changes, the subtitles may be easily blocked, affecting the user's viewing experience.
[0004] The technical solution of the present invention is: a real-time 3D subtitle generation method based on depth perception, comprising the following steps:
[0005] S11: using a depth camera to capture a depth image in a video scene, and preprocessing the depth image to improve the accuracy of subsequent processing;
[0006] S12: Real-time analysis of the audio stream in the video through speech recognition software or API;
[0007] S13: Fusing the subtitles generated by speech recognition with the depth image;
[0008] S14: Rendering the subtitles onto the video frame in 3D using a graphics rendering engine;
[0009] S15: Fusing the rendered 3D subtitles with the original video frames to ensure that the subtitles are seamlessly connected with the video content;
[0010] S16: Optimize resource allocation through terminal processor and intelligent algorithm to ensure real-time performance.
[0011] Preferably, when the audio stream in the video is identified and generated, the following steps are included:
[0012] S21: Real-time analysis of the audio stream in the video by speech recognition software to convert speech into text, including the joint use of acoustic models, language models and pronunciation dictionaries;
[0013] S22: Generate subtitle text based on the recognized text, including adjusting the display time, format and layout of the subtitles to ensure the readability and aesthetics of the subtitles.
[0014] Preferably, when positioning the 3D subtitles, the following steps are included:
[0015] S31: Fusing the subtitles generated by speech recognition with the depth image, and determining the initial position of the subtitles according to the distance information in the depth image;
[0016] S32: Through depth perception, detect whether the subtitle position overlaps or is blocked by the foreground object in the video. If blocked, the algorithm needs to automatically adjust the position of the subtitle to ensure that it is always clearly visible;
[0017] S33: As the video scene changes, the position of the subtitles follows the movement or repositioning to avoid occlusion.
[0018] Preferably, by automatically generating subtitles that match the video, the workload of manual editing is greatly reduced, and the accuracy and timeliness of the subtitles are improved. By generating 3D subtitles, not only the visual expressiveness of the subtitles is improved, but also the practicality and user experience of the subtitles can be improved through functions such as automatic avoidance of occlusion and dynamic adjustment of position.
[0019] Preferably, when performing 3D subtitle rendering, the following steps are included:
[0020] S41: using a graphics rendering engine to render the subtitles onto the video frames in 3D form, including applying lighting, shadows, and material effects to enhance the three-dimensional sense and visual impact of the subtitles;
[0021] S42: Add a perspective effect to the subtitles according to the distance information in the depth image, so that the subtitles appear to be more consistent with the visual laws of the 3D space.
[0022] Preferably, when fusing and outputting the video, the following steps are included:
[0023] S51: Fusing the rendered 3D subtitles with the original video frames to ensure seamless connection between the subtitles and the video content;
[0024] S52: Output the fused video frame to a display device in real time or transmit it to the network for live broadcast.
[0025] Preferably, when interacting with the user, the following aspects are included:
[0026] A11: Users can change the subtitle style, position and animation effects by themselves;
[0027] A12: The user's voice commands are collected and processed by the terminal processor.
[0028] A real-time 3D subtitle generating device based on depth perception comprises: a subtitle generating device body, a sensor component, a display component and a wearing component, wherein the surface of the subtitle generating device body is provided with the sensor component, the bottom surface of the subtitle generating device body is provided with the display component, and one side of the subtitle generating device body is provided with the wearing component.
[0029] Preferably, the distance between the user and surrounding obstacles is detected by the sensor component, 3D subtitles are generated by the display component, and the wearing component facilitates the user to wear the main body of the subtitle generating device.
[0030] Preferably, the sensing component includes an interactive display screen, a ranging sensor and a lighting lamp. The surface of the subtitle generating device body is provided with an interactive display screen, a ranging sensor is provided on one side of the subtitle generating device body, two groups of ranging sensors are provided, and a lighting lamp is provided in front of the ranging sensor. When in use, the interactive display screen facilitates the user to interact with the subtitle generating device body, the ranging sensor is used to detect the distance between the user and the obstacle, and the lighting lamp is used to illuminate the scene around the user.
[0031] Preferably, the display component includes a positioning frame, a liquid crystal display screen and a brightness sensor. The bottom surface of the subtitle generating device body is provided with a positioning frame, the interior of the positioning frame is provided with a liquid crystal display screen, and the inner top surface of the positioning frame is provided with a brightness sensor. When in use, the user's face is positioned by the positioning frame, and the 3D subtitles are generated and displayed by the liquid crystal display screen. The brightness sensor is used to prevent the liquid crystal display screen from being too bright and irritating the user's eyes.
[0032] Preferably, the wearing component includes a mounting bracket, an elastic band and a voice receiver. Mounting brackets are provided on both sides of the subtitle generating device body, an elastic band is provided at one end of the mounting bracket, and a voice receiver is provided on the inner side of the mounting bracket. When in use, the elastic band is installed through the mounting bracket, and the elastic band is used to facilitate the user to wear the subtitle generating device body, and the user's voice commands are received through the voice receiver.
[0033] Beneficial effects of the present invention:
[0034] 1. Compared with the traditional real-time 3D subtitle generation method and device, 3D subtitles are generally generated through traditional graphics processing technology and subtitle editing software, which lacks dynamic adaptability. When the video scene changes, the subtitles may be easily blocked, affecting the user's viewing experience. The real-time 3D subtitle generation method and device can detect whether the subtitle position overlaps or blocks the current object in the video through a built-in special 3D subtitle position determination method. When occlusion occurs, the subtitle position can be automatically adjusted to prevent occlusion;
[0035] 2. By automatically generating subtitles that match the video, the workload of manual editing is greatly reduced, and the accuracy and timeliness of subtitles are improved. By generating 3D subtitles, not only the visual expression of subtitles is improved, but also the practicality and user experience of subtitles can be improved through functions such as automatic avoidance of occlusion and dynamic adjustment of position;
[0036] 3. The interactive display screen allows users to interact with the main body of the subtitle generation device, the distance between the user and the obstacle is detected by the distance measuring sensor, the scene around the user is illuminated by the lighting lamp, the user's face is located by the positioning frame, 3D subtitles are generated and displayed on the LCD display screen, and the brightness sensor prevents the LCD display screen from being too bright and irritating the user's eyes. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 What is shown is a first stereoscopic structural schematic diagram of a real-time 3D subtitle generation device based on depth perception of the present invention;
[0038] Figure 2 Shown is a second stereoscopic structural schematic diagram of a real-time 3D subtitle generation device based on depth perception of the present invention;
[0039] Figure 3 Shown is a schematic diagram of the internal three-dimensional structure of a real-time 3D subtitle generation device based on depth perception of the present invention;
[0040] Explanation of the accompanying drawings: 1. Subtitle generating device body; 2. Sensing component; 3. Display component; 4. Wearing component; 201. Interactive display screen; 202. Distance measuring sensor; 203. Illuminating lamp; 301. Positioning frame; 302. LCD display screen; 303. Brightness sensor; 401. Mounting bracket; 402. Elastic band; 403. Voice receiver. DETAILED DESCRIPTION
[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0042] See also Figure 1-3The present invention provides an embodiment: a real-time 3D subtitle generation method based on depth perception, comprising the following steps:
[0043] S11: using a depth camera to capture a depth image in a video scene, and preprocessing the depth image to improve the accuracy of subsequent processing;
[0044] S12: Real-time analysis of the audio stream in the video through speech recognition software or API;
[0045] S13: Fusing the subtitles generated by speech recognition with the depth image;
[0046] S14: Rendering the subtitles onto the video frame in 3D using a graphics rendering engine;
[0047] S15: Fusing the rendered 3D subtitles with the original video frames to ensure seamless connection between the subtitles and the video content;
[0048] S16: Optimize resource allocation through terminal processor and intelligent algorithm to ensure real-time performance.
[0049] Preferably, when the audio stream in the video is identified and generated, the following steps are included:
[0050] S21: Real-time analysis of the audio stream in the video by speech recognition software to convert speech into text, including the joint use of acoustic models, language models and pronunciation dictionaries;
[0051] S22: Generate subtitle text based on the recognized text, including adjusting the display time, format and layout of the subtitles to ensure the readability and aesthetics of the subtitles.
[0052] Preferably, when positioning the 3D subtitles, the following steps are included:
[0053] S31: Fusing the subtitles generated by speech recognition with the depth image, and determining the initial position of the subtitles according to the distance information in the depth image;
[0054] S32: Through depth perception, detect whether the subtitle position overlaps or is blocked by the foreground object in the video. If blocked, the algorithm needs to automatically adjust the position of the subtitle to ensure that it is always clearly visible;
[0055] S33: As the video scene changes, the position of the subtitles follows the movement or repositioning to avoid occlusion.
[0056] Preferably, by automatically generating subtitles that match the video, the workload of manual editing is greatly reduced, and the accuracy and timeliness of the subtitles are improved. By generating 3D subtitles, not only the visual expressiveness of the subtitles is improved, but also the practicality and user experience of the subtitles can be improved through functions such as automatic avoidance of occlusion and dynamic adjustment of position.
[0057] Preferably, when performing 3D subtitle rendering, the following steps are included:
[0058] S41: using a graphics rendering engine to render the subtitles onto the video frames in 3D form, including applying lighting, shadows, and material effects to enhance the three-dimensional sense and visual impact of the subtitles;
[0059] S42: Add a perspective effect to the subtitles according to the distance information in the depth image, so that the subtitles appear to be more consistent with the visual laws of the 3D space.
[0060] Preferably, when fusing and outputting the video, the following steps are included:
[0061] S51: Fusing the rendered 3D subtitles with the original video frames to ensure seamless connection between the subtitles and the video content;
[0062] S52: Output the fused video frame to a display device in real time or transmit it to the network for live broadcast.
[0063] Preferably, when interacting with the user, the following aspects are included:
[0064] A11: Users can change the subtitle style, position and animation effects by themselves;
[0065] A12: The user's voice commands are collected and processed by the terminal processor.
[0066] A real-time 3D subtitle generating device based on depth perception comprises: a subtitle generating device body 1, a sensor component 2, a display component 3 and a wearing component 4, wherein the surface of the subtitle generating device body 1 is provided with the sensor component 2, the bottom surface of the subtitle generating device body 1 is provided with the display component 3, and one side of the subtitle generating device body 1 is provided with the wearing component 4.
[0067] Preferably, the distance between the user and surrounding obstacles is detected by the sensor component 2, 3D subtitles are generated by the display component 3, and the wearing component 4 facilitates the user to wear the subtitle generation device body 1.
[0068] Preferably, the sensing component 2 includes an interactive display screen 201, a distance measuring sensor 202 and a lighting lamp 203. The surface of the subtitle generating device body 1 is provided with an interactive display screen 201, and a distance measuring sensor 202 is provided on one side of the subtitle generating device body 1. The distance measuring sensor 202 is provided in two groups, and a lighting lamp 203 is provided in front of the distance measuring sensor 202. When in use, the interactive display screen 201 is used to facilitate the user to interact with the subtitle generating device body 1, the distance between the user and the obstacle is detected by the distance measuring sensor 202, and the lighting lamp 203 is used to illuminate the scene around the user.
[0069] Preferably, the display component 3 includes a positioning frame 301, a liquid crystal display screen 302 and a brightness sensor 303. The bottom surface of the subtitle generating device body 1 is provided with a positioning frame 301, the interior of the positioning frame 301 is provided with a liquid crystal display screen 302, and the inner top surface of the positioning frame 301 is provided with a brightness sensor 303. When in use, the user's face is positioned by the positioning frame 301, and the 3D subtitles are generated and displayed by the liquid crystal display screen 302. The brightness sensor 303 prevents the liquid crystal display screen 302 from being too bright and irritating the user's eyes.
[0070] Preferably, the wearing component 4 includes a mounting bracket 401, an elastic band 402 and a voice receiver 403. The mounting brackets 401 are arranged on both sides of the subtitle generating device body 1, the elastic band 402 is arranged at one end of the mounting bracket 401, and the voice receiver 403 is arranged on the inner side of the mounting bracket 401. When in use, the elastic band 402 is installed through the mounting bracket 401, and the elastic band 402 facilitates the user to wear the subtitle generating device body 1, and the voice receiver 403 receives the user's voice commands.
[0071] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.
Claims
1. A real-time 3D subtitle generation method based on depth perception; characterized by: The following steps are included: S11: using a depth camera to capture a depth image in a video scene, and preprocessing the depth image to improve the accuracy of subsequent processing; S12: Real-time analysis of the audio stream in the video through speech recognition software or API; S13: Fusing the subtitles generated by speech recognition with the depth image; S14: Rendering the subtitles onto the video frame in 3D using a graphics rendering engine; S15: Fusing the rendered 3D subtitles with the original video frames to ensure that the subtitles are seamlessly connected with the video content; S16: Optimize resource allocation through terminal processor and intelligent algorithm to ensure real-time performance.
2. The method for generating real-time 3D subtitles based on depth perception according to claim 1, characterized in that: When the audio stream in the video is recognized and generated, the following steps are included: S21: Real-time analysis of the audio stream in the video by speech recognition software to convert speech into text, including the joint use of acoustic models, language models and pronunciation dictionaries; S22: Generate subtitle text based on the recognized text, including adjusting the display time, format and layout of the subtitles to ensure the readability and aesthetics of the subtitles.
3. The method for generating real-time 3D subtitles based on depth perception according to claim 2, characterized in that: When positioning the 3D subtitles, the following steps are included: S31: Fusing the subtitles generated by speech recognition with the depth image, and determining the initial position of the subtitles according to the distance information in the depth image; S32: Through depth perception, detect whether the subtitle position overlaps or is blocked by the foreground object in the video. If blocked, the algorithm needs to automatically adjust the position of the subtitle to ensure that it is always clearly visible; S33: As the video scene changes, the position of the subtitles follows the movement or repositioning to avoid occlusion.
4. The method for generating real-time 3D subtitles based on depth perception according to claim 3, characterized in that: When rendering 3D subtitles, the following steps are included: S41: using a graphics rendering engine to render the subtitles onto the video frames in 3D form, including applying lighting, shadows, and material effects to enhance the three-dimensional sense and visual impact of the subtitles; S42: Add a perspective effect to the subtitles according to the distance information in the depth image, so that the subtitles appear to be more consistent with the visual laws of the 3D space.
5. The method for generating real-time 3D subtitles based on depth perception according to claim 4, characterized in that: When merging and outputting videos, the following steps are included: S51: Fusing the rendered 3D subtitles with the original video frames to ensure seamless connection between the subtitles and the video content; S52: Output the fused video frame to a display device in real time or transmit it to the network for live broadcast.
6. The method for generating real-time 3D subtitles based on depth perception according to claim 5, characterized in that: When interacting with users, this includes: A11: Users can change the subtitle style, position and animation effects by themselves; A12: The user's voice commands are collected and processed by the terminal processor.
7. A real-time 3D subtitle generation device based on depth perception, characterized in that: A real-time 3D subtitle generation device based on depth perception comprises: a subtitle generation device body (1), a sensor component (2), a display component (3) and a wearing component (4), wherein the surface of the subtitle generation device body (1) is provided with the sensor component (2), the bottom surface of the subtitle generation device body (1) is provided with the display component (3), and one side of the subtitle generation device body (1) is provided with the wearing component (4).
8. The real-time 3D subtitle generation device based on depth perception according to claim 7, characterized in that: The sensing component (2) comprises an interactive display screen (201), a distance measuring sensor (202) and an illumination lamp (203); the surface of the subtitle generating device body (1) is provided with the interactive display screen (201); one side of the subtitle generating device body (1) is provided with a distance measuring sensor (202); two groups of distance measuring sensors (202) are provided; and an illumination lamp (203) is provided on the projection of the distance measuring sensor (202).
9. The real-time 3D subtitle generation device based on depth perception according to claim 7, characterized in that: The display component (3) comprises a positioning frame (301), a liquid crystal display screen (302) and a brightness sensor (303); the bottom surface of the main body (1) of the subtitle generating device is provided with a positioning frame (301); the interior of the positioning frame (301) is provided with a liquid crystal display screen (302); and the inner top surface of the positioning frame (301) is provided with a brightness sensor (303).
10. The real-time 3D subtitle generation device based on depth perception according to claim 7, characterized in that: The wearing component (4) comprises a mounting bracket (401), an elastic band (402) and a voice receiver (403); the mounting brackets (401) are arranged on both sides of the main body (1) of the subtitle generating device; the elastic band (402) is arranged at one end of the mounting bracket (401); and the voice receiver (403) is arranged on the inner side of the mounting bracket (401).
