Method, device and readable storage medium for multimodal media simulating video media in dubbing

Abstract

The present invention proposes a method, device and readable storage medium for multimodal media to simulate video media in dubbing. In the field of dubbing, traditional video media have the problems of high cost and poor flexibility. In view of this, the present invention dynamically loads multimodal media such as audio, pictures, text, etc. when the program is running, and simulates video effects through specific steps. First, initialize the project, then configure resources, then request and parse data, then load UI controls and synchronize data with controls, add interactive operations and extended functions, and finally release resources. This technology effectively reduces costs, realizes resource sharing, improves operational efficiency and flexibility, has cross-platform compatibility, has important practical value in dubbing and other multimedia application scenarios, and can bring users a more efficient and convenient experience.

Description

Technical Field

[0001] The present invention relates to the field of multimedia processing and application technology, and in particular to a method, a device and a readable storage medium thereof for simulating video media using multimodal media in dubbing. Background Art

[0002] In today's digital information dissemination and learning process, various media play an important role in the field of dubbing learning. The text sentence dubbing solution has the advantages of less resource consumption and convenient interaction. Its electronic text data storage and transmission costs are low, it supports a variety of interactive operations and is easy to manage and update; the picture book solution can overcome language barriers and enhance the appeal of content with its intuitive information transmission and good memory effect. It also has advantages in dissemination, sharing and storage; the audio listening dubbing solution can efficiently transmit voice information, which is conducive to auditory memory, has a wide range of application scenarios, and can also combine cutting-edge technology to enhance the learning experience; the video viewing and video synthesis dubbing solution integrates multiple elements, is intuitive and vivid, has rich and diverse content, and has strong communication interactivity.

[0003] However, the video medium has significant defects. In terms of cost, whether it is storage hardware, transmission bandwidth, or equipment investment, technology learning and human time costs are extremely high. For example, the storage and transmission of high-definition videos require a lot of resources, the production and playback equipment are expensive, and learning related technologies is difficult and time-consuming. In terms of flexibility and resource management, the video content is fixed, the editing steps are complicated, resources cannot be shared, and modification synchronization is difficult, resulting in resource waste and bloated management. In terms of compatibility and reversibility, compatibility issues often occur when playing videos on different devices and platforms, and the quality and details of the video will be damaged after multiple edits and cannot be restored to their original state. In terms of retrieval function, as a single file, it is extremely inconvenient to batch tag, classify and store the rich multimodal elements within the video, and quickly find them, which seriously affects the effective use of learning resources and management efficiency.

[0004] Therefore, there is an urgent need for a new method, device and readable storage medium for multimodal media simulation of video media in dubbing to solve the problems existing in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and readable storage medium for simulating video media in multimodal media in dubbing. Although the existing video media in the current technology has advantages in the fields of dubbing learning, it has many defects such as high cost, poor flexibility, inability to share resources, low compatibility, irreversibility and inconvenient retrieval, which seriously affect its utilization efficiency and resource management.

[0006] The core technology of the present invention is mainly to dynamically load multimodal media such as audio, pictures, text, etc. when the program is running, and simulate the video media effect through a series of steps to achieve the advantages of cost reduction and efficiency improvement, flexible sharing of resources, improved user experience and cross-platform compatibility.

[0007] In a first aspect, the present invention provides a method for multimodal media simulating video media in dubbing, the method comprising the following steps:

[0008] S0, Initialization project:

[0009] Build a project and configure and start a development server; where the project is a software development tool or technology platform;

[0010] S1. Configure resources:

[0011] Acquire audio, image, and text multimodal file resources and map their relationships through automated, semi-automated, or manual methods;

[0012] S2, request data flow:

[0013] Adopt the front-end and back-end separation solution, build a file server on the back-end, and request and verify data on the front-end;

[0014] S3. Parsing data:

[0015] Preset algorithms are used to process different media, so that when resource structures or metadata of different media are changed, they can be updated and modified uniformly;

[0016] S4. Load UI controls:

[0017] Dynamically load media resource files in the order of audio, image, and text and bind corresponding controls, and set the appearance and functions of the controls to simulate video;

[0018] S5. Real-time synchronization of data and controls:

[0019] With the help of the timed trigger event or simulated event of the audio control, the synchronization of various data and the corresponding control is realized in the preset related methods; wherein the audio control is a component used to bind and play audio files; the various data include playing status, time, progress bar, full screen status, audio, subtitles, and background pictures;

[0020] S6. Add interactive operations:

[0021] Monitor audio control events and mouse and touch events on the bottom panel of the display interface to implement interactive functions, including loading, playing, pausing, and progress bar dragging;

[0022] S7. Release resources:

[0023] Stop loading resources when simulating a paused video state;

[0024] The resources are completely released before the simulated video-related pages or components are destroyed.

[0025] Furthermore, in step S2, the automated way of acquiring resources includes automatically acquiring audio, image, and text multimodal file resources using audio and video separation technology and automatically mapping their relationships;

[0026] The semi-automatic way of acquiring resources includes using internal tools or third-party software to assist in acquiring and organizing multimodal resources, and manually performing partial configuration and relationship mapping;

[0027] The manual method of acquiring resources includes manually collecting, producing and organizing audio, image, and text multimodal file resources by the design, teaching and research, and creative teams, and clarifying the correspondence and usage scenarios between them.

[0028] Furthermore, in step S3, the preset algorithms include a subtitle parsing and conversion algorithm, a highlight subtitle synchronization algorithm, and a multi-frame sequence image synchronization algorithm.

[0029] Further, in step S5, the preset related method includes:

[0030] Initialization synchronization method, which is triggered when the audio file is loaded and the corresponding control is initialized, is used to set the playback state to the initial stop state, determine the start and end time according to the audio playable range, and reset the progress bar position to zero and close the full screen display, thereby completing the setting of the initial state of the simulated video;

[0031] A dynamic refresh synchronization method is triggered at a preset frequency during audio playback. Within this dynamic refresh synchronization method, time synchronization is achieved by updating the current playback time in real time and adjusting the start and end times according to the audio length; progress bar synchronization relies on calculating and setting its position based on the playback progress, and can respond to external touch or mouse drag operations to change the playback position; audio synchronization is achieved by keeping consistent with the audio playback progress; subtitle synchronization is achieved by filtering and displaying the corresponding subtitles in the parsed subtitle data based on the current audio playback time; background image synchronization determines and displays the corresponding images in the parsed image set data based on the audio playback time.

[0032] Furthermore, in step S5, when the audio is played to a certain time point, the corresponding subtitle content is displayed synchronously and the position of the progress bar is updated; when the user drags the progress bar, the audio playback position and subtitle display also jump to the corresponding time point accordingly, and the switching of the background image or the animation effect matches the rhythm of the audio and subtitles.

[0033] Furthermore, in step S6, the progress bar dragging accurately calculates the correspondence between the user's dragging position and the audio playback time, achieving smooth jumping and playback; the volume adjustment is achieved by changing the volume property of the audio control, and the current volume level is fed back to the user in real time; the full-screen switching function is compatible with different browsers and devices, ensuring that the video content can be displayed and interacted normally in full-screen mode, and can be restored to the original interface state when exiting the full-screen mode.

[0034] Furthermore, the project is a Vue3 project, which is a progressive JavaScript framework for building user interfaces.

[0035] In a second aspect, the present invention provides a device for simulating video media with multimodal media in dubbing, comprising:

[0036] A project initialization module builds a project and configures and starts a development server; wherein the project is a software development tool or technology platform;

[0037] Resource configuration module, which acquires audio, image, and text multimodal file resources and maps their relationships through automated, semi-automated, or manual methods;

[0038] The data stream request module adopts a front-end and back-end separation solution. The back-end builds a file server, and the front-end requests data and verifies it.

[0039] The data analysis module uses preset algorithms to process different media, so that when the resource structure or metadata of different media changes, they can be uniformly updated and modified;

[0040] UI control loading module, dynamically loads media resource files in the order of audio, image, and text and binds corresponding controls, and sets the control appearance and functions to simulate video;

[0041] The data space synchronization module uses the timed trigger events or simulated events of the audio control to synchronize various data with the corresponding control in the preset related methods; the audio control is a component used to bind and play audio files; the various data include playback status, time, progress bar, full screen status, audio, subtitles, and background images;

[0042] Add a module for interactive operations to monitor audio control events and mouse and touch events on the bottom panel of the display interface to implement interactive functions, including loading, playing, pausing, and progress bar dragging;

[0043] The resource release module stops loading resources when the simulated video is paused, and completely releases resources before the simulated video-related page or component is destroyed.

[0044] In a third aspect, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned method of multimodal media simulating video media in dubbing.

[0045] In a fourth aspect, the present invention provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process, and the process includes a method for simulating video media according to the above-mentioned multimodal media in dubbing.

[0046] The main contributions and innovations of the present invention are as follows:

[0047] 1. Reduce costs: The resource usage involved in analog video playback is greatly reduced, the storage hardware and transmission traffic bandwidth costs are reduced accordingly, and the operation does not require high-performance hardware and professional software, which significantly reduces the economic burden on enterprises and individuals. For example, storage costs can be reduced by up to 99%.

[0048] 2. Resource sharing: Multimodal media resources exist independently, and multiple analog videos can share the same resource, which neither generates new files nor affects the original resources. It is easy to manage and maintain, and can effectively eliminate duplicate and invalid resources, further improving the effect of reducing costs and increasing efficiency.

[0049] 3. Improve efficiency: The media resources involved are simple in basis, and the operation does not require complicated and time-consuming processes, such as rendering and transcoding in video synthesis, which greatly improves the operational efficiency and enables users to modify and create content more conveniently.

[0050] 4. Flexible and free: You can freely combine media and perform fine operations on individual media, such as implementing subtitle comparison, real-time search and modification of subtitles, adaptive background and other diversified functions, which enhances the interactive experience and creative space and overcomes the drawbacks of fixed video content.

[0051] 5. Wide compatibility: Compatible with Android, iOS, Hongmeng, Web, WeChat, mini-programs and other platforms, it can be used smoothly on different devices and platforms, with a wide range of application scenarios, which can be expanded to music playback, listening to books, meditation and other fields.

[0052] 6. Strong expansion: It can expand metadata storage, operation information, etc., solve the inconvenience and irreversibility of video retrieval, and enhance visual and sound effects, such as adding stereo surround sound, etc., which improves the overall functionality and practicality.

[0053] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0055] Figure 1 is a flow chart of a method for multimodal media simulating video media in dubbing according to an embodiment of the present invention;

[0056] Figure 2 is an overall structural diagram according to an embodiment of the present invention;

[0057] Figure 3 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0059] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0060] In the field of dubbing, traditional video media has problems such as high cost and poor flexibility.

[0061] Based on this, the present invention solves the problems existing in the prior art based on a new architecture.

[0062] Embodiment 1

[0063] The present invention aims to propose a method for multimodal media to simulate video media in dubbing. Specifically, taking the Web front end as an example, the VUE front end framework is used to make this invention technology into a component, which is convenient for global use. Figure 1 and Figure 2 , the method comprising:

[0064] S0. Initialize the project, build the Vue3 project, configure and start the development server;

[0065] In this embodiment, the initialization items specifically include:

[0066] ① Enter the following commands "node-v" and "npm-v" in the terminal to ensure that Node.js and npm are installed. If not, go to the official Node.js website to download and install the LTS version.

[0067] ②Enter the following command "npm create vue@latest" in the terminal to quickly build a Vue3 project using Vite.

[0068] ③ Follow the prompts to configure the project during creation.

[0069] ④Enter the following commands "cd (project name)" and "npminstall" in the terminal to install project dependencies.

[0070] ⑤Enter the following command "npmrundev" in the terminal to start the development server.

[0071] S1. Configure resources, obtain audio, image, and text multimodal file resources through automated, semi-automated, or manual methods, and map their relationships;

[0072] In this embodiment, configuring resources specifically includes:

[0073] Similar to the common "video synthesis solution", multimodal file resources such as audio / image / text need to be prepared in advance. At present, the present invention adopts the following methods to obtain the mapping of the corresponding audio / image / text matching relationship:

[0074] ① Automation: Using relatively mature audio and video separation technology, the separation accuracy is within a controllable range. The present invention uses a high-performance server at the back end, adopts traditional methods based on signal processing, plus deep learning, and cooperates with artificial intelligence to separate the existing video files into frame image files and audio files. At the same time, it also uses text recognition technology to identify watermark subtitles and generate corresponding subtitle files. This method is set in the program parameters, and it will automatically separate audio files, picture files, and text subtitle files with the same UUID (Universally Unique Identifier) ​​name, and then automatically associate and map resources.

[0075] ② Semi-automation: The company has developed existing tools such as video production tools, subtitle production tools, UGC / PGC / AIGC creation platforms, etc.; or use third-party commercial software: artificial intelligence text image, artificial intelligence speech generation synthesis, artificial intelligence text video, etc., to automatically generate pictures, audio or video corresponding to the needs based on keywords, and then reversely separate the video into picture files, audio files and subtitle files. This method can also automate batch processing, and with a small amount of manual operation, the resource relationship can be mapped.

[0076] ③ Manual: In addition, the company has multiple design, teaching and research and content creation teams, each of which performs their duties, designs images, synthesizes or records audio resources, and researches products to produce content subtitles according to needs, and then cooperates with automation tools and artificial intelligence operations. The rest is manually generated, input and verified to ensure that all content is legal and compliant. This method mostly manually processes the mapping relationship of related resources.

[0077] Through the above methods, the matching relationships such as audio / image / text required for the simulation video can be mapped out to prepare resources.

[0078] Right now Figure 1 Submit multimodal media resources in: Then submit multimodal media resources including audio resources, image resources, subtitle resources, metadata resources, etc. These resources are the basic materials that constitute the simulation video.

[0079] S2, request data flow, adopt the front-end and back-end separation solution, use PHP to build a file server on the back end, and use JavaScript's Axios plug-in on the front end to request and verify data;

[0080] In this embodiment, in order to improve the flexibility and maintainability of the system and facilitate the management and update of resources, a front-end and back-end separation solution is adopted in the development. The back-end of the present invention uses PHP language to build a simple file server for remote storage and retrieval of file resources (basic operation, the technology of the present invention is not further described); the front-end uses JavaScript language Axios network plug-in to request and send data, allowing the back-end to query the database, or traverse the stored files, or extract keywords in the metadata, and perform UUID designation to match the corresponding audio, image, text and other resources to obtain data streams of various media. In this step, it is necessary to verify whether various resources meet the conditions.

[0081] For example, when in use, the UUID is specified by querying the database, traversing the stored files, or extracting keywords in the metadata, and then the corresponding audio A, image B, text C and other resources are obtained through the mapping relationship or resource path.

[0082] Right now Figure 1 Resource handling in:

[0083] Local resources: For local resources, select them locally on a computer or upload them online, and then verify the files to ensure the correctness and integrity of the resources.

[0084] Remote resources: For remote resources, they are obtained by requesting data streams and performing verification processing for request failures.

[0085] S3, parse the data and process it in specific formats according to different media, including subtitle segmentation, highlighted subtitle analysis, etc. After obtaining the resources, whether local or remote resources, data parsing is required so that these resources can be correctly loaded and used later.

[0086] In this embodiment, the data streams of various modal media have been acquired through step S2, but cannot be used directly at this time. The data streams must first be processed in specific formats according to different media.

[0087] In this embodiment, core algorithms such as subtitle parsing and conversion algorithm, highlight subtitle synchronization algorithm, multi-frame sequence image synchronization algorithm, etc. are common to all platforms and can be uniformly updated and modified even if the resource structure or metadata changes.

[0088] For example, the following is a parsing algorithm for converting text data streams into subtitles:

[0089] function parseTextToSubtitle(textData) {

[0090] / / First, check whether the input text data is empty. If it is empty, return an empty array to avoid subsequent processing errors

[0091] if (!textData) {

[0092] return [];

[0093] }

[0094] let subtitleArray = [];

[0095] / / Split the text data into lines, assuming that each line represents a subtitle information

[0096] let lines = textData.split('\n');

[0097] for (let i = 0; i <lines.length; i++) {

[0098] let line = lines[i];

[0099] / / Assuming the format of subtitle information is "start time - end time | subtitle content", each line is split here

[0100] let parts = line.split('|');

[0101] if (parts.length === 2) {

[0102] let timeParts = parts[0].split('-');

[0103] if (timeParts.length === 2) {

[0104] / / Convert the time part from a string to a digital representation for subsequent processing

[0105] let startTime = parseFloat(timeParts[0]);

[0106] let endTime = parseFloat(timeParts[1]);

[0107] let subtitleContent = parts[1];

[0108] / / Create a subtitle object, including the start time, end time and subtitle content

[0109] let subtitleItem = {

[0110] startTime: startTime,

[0111] endTime: endTime,

[0112] content: subtitleContent

[0113] };

[0114] subtitleArray.push(subtitleItem);

[0115] }

[0116] }

[0117] }

[0118] return subtitleArray;

[0119] }

[0120] The algorithm receives a piece of text data as input, first checks whether the input is empty, and directly returns an empty array if it is empty. Then the input text data is split by line, because it is assumed that each line contains the information of a subtitle. For each line, it is split according to the characters and divided into two parts, the first part is the time information, and the second part is the subtitle content. For the time information part, it is split according to the - character to get the start time and end time, and convert it into a floating point number. Finally, the start time, end time and subtitle content are combined into a subtitle object, stored in the subtitle array, and finally the array storing all subtitle objects is returned as the result.

[0121] The core of this algorithm is to process and parse the input text data, segment and convert it according to the agreed format (start time - end time subtitle content), and finally form an array of subtitle objects that are convenient for subsequent processing and use. Of course, the specific format and processing details can be adjusted and expanded according to actual needs, such as adding more checks on the time format and additional processing of the subtitle content.

[0122] For example, the following is the parsing algorithm for converting subtitles into text data stream:

[0123] function ToSrt(data) {

[0124] var res = '';

[0125] const end_of_line = '\r\n';

[0126] / / Traverse the input subtitle data array

[0127] for (var i = 0; i <data.length; i++) {

[0128] var s = data[i];

[0129] / / Add the subtitle number to the result string and wrap it

[0130] res += s.id + end_of_line;

[0131] / / Add the start time and end time of the subtitle to the result string in a specific format and wrap it

[0132] res += s.start + '-->' + s.end + end_of_line;

[0133] / / Add the subtitle content to the result string, replace the line break with a specific line break format, and add two line breaks

[0134] res += s.text.replace('\n', end_of_line) + end_of_line + end_of_line;

[0135] }

[0136] return res;

[0137] }

[0138] The function of this algorithm is to convert an array containing subtitle information into a text data stream in a specific format. First, initialize an empty string to store the result. Then traverse the input subtitle array. For each subtitle object, first add its number to the result string and wrap it, then add its start time and end time to the result string in a specific format (connected with -->) and wrap it, and finally replace the line break in the text content of the subtitle with a specific format and add it to the result string, and add two line breaks. After processing all subtitle objects, the constructed text data stream string is finally returned. This conversion process enables subtitle data to be stored, transmitted, or further processed in a standardized text format for use in other systems or applications.

[0139] Among them, the conversion of text data stream into subtitles and the conversion of subtitles into text data stream are both parts of the subtitle parsing and conversion algorithm. The highlight subtitle synchronization algorithm, the multi-frame sequence image synchronization algorithm, etc. are all existing algorithms, and both are expanded on the basis of the subtitle parsing and conversion algorithm.

[0140] The highlighted subtitle synchronization algorithm usually involves the following main steps: first extract the subtitle text content from the subtitle file using the subtitle parsing and conversion algorithm mentioned above, then parse the timestamp according to the subtitle text content and map it to the playback progress, and finally highlight the corresponding text during playback according to the display time of the subtitle. Among them, how to map the playback progress is consistent with the following "S5, real-time synchronization of data and controls" to monitor <audio>The @timeupdate="OnAudioUpdate" of the UI control is used to synchronize the current playback progress and the subtitle display time; how to highlight the corresponding text during playback is similar to the "this.RefreshSubtitle()" method below. It determines whether the subtitle needs to be highlighted based on the current time, and controls the highlighting effect through CSS styles. The following is the highlighted subtitle synchronization algorithm:

[0141] function RefreshHighlights(currentTime, subtitles) {

[0142] / / Get the subtitle entry for the current time.

[0143] subtitles.forEach(subtitle =>{

[0144] if (currentTime>= subtitle.startTime&¤tTime<= subtitle.endTime){

[0145] / / The current time is within the time range of this subtitle and is highlighted.

[0146] document.getElementById(subtitle.id).classList.add('highlight');

[0147] } else {

[0148] / / The current time is not within the time range of the subtitle, remove the highlight.

[0149] document.getElementById(subtitle.id).classList.remove('highlight');

[0150] }

[0151] });

[0152] }

[0153] The multi-frame sequence image synchronization algorithm usually has the following main steps: first, use the above subtitle parsing and conversion algorithm to extract the paths of all multi-frame sequence images, the start and end time of playback, and other information of the image from the metadata file containing all multi-frame sequence images. After the analysis is completed, an array containing all multi-frame sequence images is loaded, and each frame is an independent image file. Then use the "S4, Load UI Controls" below to create a corresponding UI controls to listen <audio>The @timeupdate="OnAudioUpdate" of the UI control is used to synchronize and control the current playback progress and image rendering display, thereby achieving a similar picture animation effect. The following is a multi-frame sequence image synchronization algorithm:

[0154] function RefreshImages(currentTime, images) {

[0155] / / Get the image entry for the current time.

[0156] this.image = images.find(image=>return currentTime>=image.startTime&¤tTime<= image.endTime;

[0157] if(!!this.image){

[0158] / / Render the image by assigning a value to the specified image.

[0159] let element = document.getElementById('MyImage');

[0160] if(!!element){

[0161] / / Fault tolerance compatibility.

[0162] element.src = image.url;

[0163] }

[0164] }

[0165] }

[0166] S4. Load UI controls, dynamically load media resource files in the order of audio, image, and text and bind corresponding controls, set the control appearance and function to simulate video; according to the parsed data, load and initialize various multimodal combined visual controls, such as audio controls, image controls, subtitle controls, etc. These controls will display the corresponding content in the user interface.

[0167] In this embodiment, since the technology of the present invention focuses on the effect of simulating video media playback, the video can be considered to be composed of content elements such as audio, image / image set, subtitles, etc.

[0168] In this embodiment, by dynamically loading media resource files such as audio, image / image set, and text, when the program is running, the audio control corresponding to the audio media resource file is called to bind the audio content element, the image / image set media resource file corresponds to the image / imageList control to bind the image content element, the text media resource corresponds to the text control to bind the subtitle content element, etc., and then freely combine them to simulate the playback effect of the video media.

[0169] Because it involves interactive experience, the optimal solution for loading order is to load in the order of resource size. Generally, audio file size > image / image set file size > text size, etc., so the loading order of media resource files is audio, image, text, etc.

[0170] The present invention uses <audio>The control is bound to the corresponding audio file to load the audio content element. The control is bound to the corresponding image file to load the image content element. Bind the corresponding subtitle file to the control to load the subtitle content element as an example:

[0171] ①Create and set <audio>The labeled audio control enables the project to load audio content elements and have sound and sound effects during runtime:

[0172] <audio class="ui-audio" / / Set a class name "ui-audio" for the audio control, which facilitates styling it through the CSS style sheet to make its display on the page meet the design requirements.

[0173] src="url" / / Specify the source address of the audio file. Here, "url" should be the path or link to the actual audio file. By this attribute, the browser is told where to obtain the audio data.

[0174] preload="auto" / / Set the audio preloading method to "auto", indicating that the browser will automatically load the audio file in the background so that it can respond faster when the user needs to play it, reducing the playback delay.

[0175] @loadedmetadata="OnAudioInit" / / Bind an event handler function named "OnAudioInit" to the "loadedmetadata" event. When the metadata of the audio (such as duration, number of channels, etc.) is loaded, this event will be triggered, and then the "OnAudioInit" function will be executed. Usually, some operations related to audio playback initialization can be performed in this function.

[0176] @timeupdate="OnAudioUpdate" / / Bind the "OnAudioUpdate" event handler function to the "timeupdate" event. The "timeupdate" event is triggered at regular intervals during audio playback. By executing the "OnAudioUpdate" function, operations such as updating the audio playback progress and related interface elements (such as progress bars, subtitles, etc.) can be achieved to keep the audio playback synchronized with other elements.

[0177] @ended="OnAudioEnd" / / Bind the "OnAudioEnd" event handler function to the "ended" event. When the audio playback is completed, the "ended" event will be triggered, and then the "OnAudioEnd" function will be executed. Some cleanup or prompting operations after playback can be performed in this function.

[0178] ref="refAudio"> / / Set a reference name "refAudio" for the audio control. Through this reference, you can easily access and operate this audio control in JavaScript code, such as obtaining the current playback time of the audio, controlling playback pause, etc.

[0179] < / audio>

[0180] By using <audio>The label creates an audio control and sets a series of properties and event listeners for it. First, a style class name is specified for it through the class attribute for subsequent style adjustments. The src attribute specifies the source path of the audio file. The preload attribute is set to auto to enable the browser to preload the audio automatically. At the same time, the loadedmetadata, timeupdate, and ended events are respectively bound to the corresponding OnAudioInit, OnAudioUpdate, and OnAudioEnd functions, which will perform specific operations when the corresponding events are triggered, such as initializing audio playback, updating the playback progress, and handling the end of playback, etc. In addition, the ref attribute sets a reference name, which facilitates operating on the audio control in JavaScript code, thereby enabling the project to load the audio content element, play sounds and sound effects at runtime, and implement various functions and interactions related to audio playback.

[0181] ② Create and set the label image control, enabling the project to load the image content element and having a poster image and a background image during runtime:

[0182] <img

[0183] class="img-poster" / / Add a class named "img-poster" to this label to enable styling it through CSS. It may be used to display the poster image, and styles such as its size, position, and border can be set to conform to the layout and design requirements of the project.

[0184] src="background" / / Specify the source address of the image as "background". Here, it may be a variable or an expression that will ultimately be resolved to the actual image resource address for loading the image required for the poster.

[0185] v-if="!!background" / / Use the v-if directive of Vue.js. Only when the "background" variable exists and is true, this label and its corresponding image element will be rendered and displayed, avoiding displaying an empty element when there is no resource, and optimizing performance and page layout.

[0186] / >

[0187] <imgList / / This may seem to be a custom component or element. Judging from the naming, it may be used to manage and display a list of background images.

[0188] class="img-backgrounds" / / Add the "img-backgrounds" class to the element and set its style through CSS. It can be used to set the style of the background image list, such as arrangement, spacing, etc.

[0189] src="backgroundList" / / Specifies the source address of the image as "backgroundList", which may be a list variable or expression that stores multiple background image resources, and will eventually be resolved into an actual background image resource list.

[0190] v-if="backgroundList.length>0" / / Using Vue.js's v-if directive, this element and the background image elements it contains will be rendered and displayed only when the length of the "backgroundList" array is greater than 0, ensuring that this part is displayed only when there is an available background image list, avoiding rendering empty elements, improving performance and page neatness.

[0191] / >

[0192] This code uses the HTML Tags and related directives of Vue.js are designed to load image content elements for the project. The tag uses the class attribute to add the img-poster class for subsequent style settings. The src attribute specifies the source address of the image, which is background. The v-if directive is used to render the image only when the background variable is true, so that it can be used to display the poster image. For the imgList element (which may be a custom component), the img-backgrounds class is added for style adjustment. The source address specified by its src attribute is backgroundList. The v-if directive is used to render only when the length of the backgroundList array is greater than 0, which is used to display the background image list. In this way, the poster images and background images in the project can be effectively managed and displayed, while ensuring that they are only rendered when there are available image resources, avoiding unnecessary element rendering and performance overhead.

[0193] ③Create and set< / audio> The label text control enables the project to load subtitle content elements, and subtitles and multi-line subtitles are displayed at runtime:

[0194] / / Define a div container that contains subtitle content and use the "subtitles" class to facilitate overall style setting and layout management of the subtitle elements.

[0195] <p

[0196] class="txt-first" / / Add the "txt-first" class to the paragraph element so that it can be styled, possibly for displaying the first line of subtitles.

[0197] v-if="!!subtitleFirstText&&isPlaying == true" / / Using the v-if directive of Vue.js, the paragraph element will be rendered and displayed only when the "subtitleFirstText" variable exists and is true, and the "isPlaying" variable is true, which is used to control the display of the first line of subtitles in the playing state.

[0198] ref="refSubtitleFirst"> / / Set a reference for the paragraph element to facilitate accessing and operating the element through the reference in the JavaScript code of the Vue component.

[0199] {{ subtitleFirstText.first}} / / Use double curly braces for data binding and display the data in "subtitleFirstText.first" in the paragraph element. This may be the specific text content of the first line of subtitles.

[0200]

[0201] <p

[0202] class="txt-temp" / / Add the "txt-temp" class to the paragraph element to facilitate styling, which may be used to display temporary or multi-line subtitles.

[0203] style="`font-size:${item}rem`" / / Use inline styles to set the font size based on the value of the "item" variable, allowing dynamic font size adjustment.

[0204] v-for="(item, index) in firstsizes" / / Use the v-for directive of Vue.js to loop according to the elements in the "firstsizes" array and create a paragraph element for each element in the array, where "item" represents the array element and "index" represents the index of the element.

[0205] key="index" / / Add a unique key to the v-for loop to improve rendering performance and ensure the uniqueness of elements and the accuracy of updates.

[0206] ref="`refSubtitleTemp${index}`"> / / Set a reference for each paragraph element created in the loop. The reference name is generated according to the index, which makes it convenient to access and operate the corresponding element through the reference in the JavaScript code of the Vue component.

[0207] {{ subtitleFirstTemp}} / / Use double curly braces for data binding and display the data in "subtitleFirstTemp" in the paragraph element. This may be the specific text content of multiple lines of subtitles.

[0208]

[0209]

[0210] This code mainly uses Vue.js to create and set text controls for displaying subtitle content. First, a div element is used as a container, and its class attribute is subtitles, which is used to wrap all subtitle elements for unified management and style settings. In this div element, there is a p element, which uses the class attribute to add the txt-first class, and uses the v-if instruction to decide whether to render according to the values ​​of the subtitleFirstText variable and the isPlaying variable. It will only be displayed when subtitleFirstText has content and isPlaying is true. The displayed content is bound to the data of subtitleFirstText.first through double curly braces, which is used to display the first line of subtitles. In addition, there are a series of p elements, which use the class attribute to add the txt-temp class, use the v-for instruction to loop and create according to the elements in the firstsizes array, dynamically adjust the font size through inline styles, use the key attribute to add a unique key to each element, and use references to facilitate subsequent operations, and use double curly braces to bind the data of subtitleFirstTemp to display multiple lines of subtitles. These elements together constitute the subtitle display part of the project, which can display subtitles and multi-line subtitles at runtime, and can be dynamically displayed and styled according to the values ​​and conditions of variables, providing subtitle display functions and flexible style control.

[0211] After this step, the scattered multimodal media resource files are linked to the multimodal media controls and then freely "assembled" and bound, so that this project has content elements such as sound, picture, subtitles, etc. similar to real video at runtime, thus achieving the consistency of content elements of simulated video media.

[0212] Next, the present invention On the image control, continue to overlay the appearance of the video when it is playing:

[0213] ①Add a layer that covers the entire image control Label Translucent Mask and <button>The label button simulates click-to-play, delays displaying / hiding the control panel at the bottom, etc.

[0214] ②Add a semi-transparent gray color to the bottom Label, analog video control panel;

[0215] ③Add on the control panel The play / pause state button of the label, The start time of the tag, <slider> Progress bar of composite label,< / slider> The expiration date of the tag and < / button> <button> Full screen button display of the label, etc.

[0216] After this step, the project will display an interface similar to that of real video playback when running, thus achieving consistency in the appearance of the simulated video media.

[0217] S5. Synchronize data and controls in real time. With the help of the TimeUpdate event of the Audio control or simulate the event, synchronize the play status, time, progress bar, full-screen status, audio, subtitles, background images and other data with the control in the relevant methods. After the control is loaded and initialized, the control data needs to be synchronized to ensure the consistency of audio, images, subtitles and other elements in time and content to achieve the effect of simulating video.

[0218] In this embodiment, the above steps only associate each media resource file with the media control for static display, and they are still independent of each other. At this time, it is necessary to use the Audio control of each platform, which will provide a TimeUpdate timed trigger event (if there is no such event, it can be simulated by timer Timer or requestAnimationFrame and other methods, usually triggered once every 15 to 250 milliseconds) to synchronize data with controls and make the entire project "dynamic".

[0219] In this embodiment, the duration of loading the audio file is used as the duration of the simulated video. <audio>The control's loadedmetadata initialization event and timeupdate interface rendering refresh event, and declare the corresponding <audio>The event methods of the control are @loadedmetadata="OnAudioInit" and @timeupdate="OnAudioUpdate".

[0220] Among them, "OnAudioInit" is the audio file loaded and the corresponding <audio>When the control is initialized, the initialization method will be automatically triggered to execute, which can initialize the operation of the project when it is running. The code is as follows (for the sake of document simplicity, the following code will not be commented and explained in detail, and the same function can be achieved through different code forms):

[0221] / / Initialize events.

[0222] functionOnAudioInit(){

[0223] / / TODO: Initialization.

[0224] this.OnInit();

[0225] }

[0226] / / Render refresh event.

[0227] functionOnAudioUpdate(){

[0228] / / TODO: Refresh interface rendering.

[0229] this.RefreshTimes();

[0230] this.RefreshProgress();

[0231] this.RefreshAudio();

[0232] this.RefreshSubtitle();

[0233] this.RefreshBackgrounds();

[0234] }

[0235] ① Initialization of "video":

[0236] In the "OnAudioInit" initialization event, set the playback state to stop state, start / end / current playback time, progress bar position, close full screen state, etc. to initialize the entire simulation state.

[0237] functionOnInit(){

[0238] / / Initialize the playback state.

[0239] this.isPlaying=false;

[0240] / / Initialize full screen state.

[0241] this.isFullScreen=false;

[0242] / / Set the start / end time.

[0243] if(this.audioPlayer.seekable.length>0){

[0244] this.startTime=Times.FormatDuration(

[0245] this.audioPlayer.seekable.start(0) );

[0247] this.endTime=Times.FormatDuration(this.audioPlayer.seekable.end(0));

[0248] }else{

[0249] this.startTime=0;

[0250] this.endTime=this.audioPlayer.duration;

[0251] }

[0252] }

[0253] ② Synchronization of "video" playback status:

[0254] When you click the label translucent mask that covers the entire image control or the play button in the lower left corner of the bottom control panel, the audio file will be triggered to start playing, the "video" state isPlaying is set to true, and the play button state is changed to the playing icon. At this time, the simulated video is in the playing state.

[0255] / / Start playing button event.

[0256] functionOnPlayClick(){

[0257] this.player.play();

[0258] this.isPlaying=true;

[0259] this.isShowControls=true;

[0260] this._RefreshControls();

[0261] }

[0262] When the play button is clicked again, the audio file will be paused, the "video" status isPlaying will be set to false, and the playing icon will be changed to the pause icon. At this time, the simulated video is in a paused state.

[0263] / / Pause playback button event.

[0264] functionOnPlayClick(){

[0265] this.player.pause();

[0266] this.isPlaying=false;

[0267] this.isShowControls=false;

[0268] this._RefreshControls();

[0269] }

[0270] The control panel at the bottom can switch the play / pause status.

[0271] / / Switch the play / pause state button.

[0272] functionOnStatusClick(){

[0273] if (this.player.paused) {

[0274] this.player.play();

[0275] this.isPlaying=true;

[0276] }else{

[0277] this.player.pause();

[0278] this.isPlaying=false;

[0279] this.isShowControls=false;

[0280] }

[0281] this._RefreshControls();

[0282] }

[0283] ③ Synchronization of "video" start time, end time, and current play time:

[0284] In the "OnAudioUpdate" interface rendering refresh event, the start time, end time, and current play time are dynamically updated. The start time is fixed to 0, the end time is the duration of the current audio, and the currently displayed play time needs to be dynamically calculated based on the play progress.

[0285] functionRefreshTimes(){

[0286] this.startTime=0;

[0287] this.currentTime=Times.FormatDuration(

[0288] (this.progress / 100)*this.player.duration );

[0290] this.endTime=Times.FormatDuration(this.player.duration);

[0291] }

[0292] ④ Synchronization of "Video" progress bar:

[0293] This progress bar increases dynamically as time goes by, and sets the corresponding playback progress. You can control the progress bar by touching or dragging the mouse to play at a fixed point. At this time, you need to obtain the touch / mouse click position, and then calculate the current drag progress:

[0294] functionRefreshProgress(posX){

[0295] let pb = this.$refs.refProgress;

[0296] letrect=pb.getBoundingClientRect();

[0297] letpos=posX-rect.left;

[0298] this.progress=Math.max(0,Math.min((pos / rect.width)*100,100));

[0299] }

[0300] ⑤ Synchronize the status of "video" in full screen or not:

[0301] / / Set full screen by clicking the "Full Screen" button.

[0302] functionEnterFullScreen(){

[0303] letelem=document.documentElement;

[0304] if (elem.requestFullscreen) {

[0305] elem.requestFullscreen();

[0306] }elseif(elem.mozRequestFullScreen){

[0307] / / Firefox

[0308] elem.mozRequestFullScreen();

[0309] }elseif(elem.webkitRequestFullscreen){

[0310] / / Chrome,SafariandOpera

[0311] elem.webkitRequestFullscreen();

[0312] }elseif(elem.msRequestFullscreen){

[0313] / / IE / Edge

[0314] elem.msRequestFullscreen();

[0315] }

[0316] }

[0317] / / Cancel full screen by clicking the "Full Screen" button.

[0318] functionExitFullScreen(){

[0319] if(document.exitFullscreen){

[0320] document.exitFullscreen();

[0321] } else if (document.mozCancelFullScreen) {

[0322] / / Firefox

[0323] document.mozCancelFullScreen();

[0324] } else if (document.webkitExitFullscreen) {

[0325] / / Chrome, Safari and Opera

[0326] document.webkitExitFullscreen();

[0327] } else if (document.msExitFullscreen) {

[0328] / / IE / Edge

[0329] document.msExitFullscreen();

[0330] }

[0331] }

[0332] / / Detect whether it is full screen currently.

[0333] function IsFullScreen() {

[0334] if (document.fullscreenElement) {

[0335] return true;

[0336] } else if (document.mozFullScreenElement) {

[0337] / / Firefox

[0338] return true;

[0339] } else if (document.webkitFullscreenElement) {

[0340] / / Chrome, Safari and Opera

[0341] returntrue;

[0342] }elseif(document.msFullscreenElement){

[0343] / / IE / Edge

[0344] returntrue;

[0345] }else{

[0346] returnfalse;

[0347] }

[0348] }

[0349] ⑥Audio / sound effects synchronization:

[0350] In this embodiment, the duration of loading the audio file is used as the duration of the simulated video, so the audio / sound effect is automatically synchronized when monitoring @timeupdate="OnAudioUpdate".

[0351] If you want to customize the playback time of the simulated video, you need to create a timer or requestAnimationFrame method to refresh the interface rendering during initialization. Then get the audio <audio>Control element to synchronize the playback time in real time.

[0352] functionRefreshAudio(time){

[0353] letaudio=this.$refs.refAudio;

[0354] if(!!audio){

[0355] this.player.currentTime=time;

[0356] }

[0357] }

[0358] ⑦ Synchronization of subtitles / multi-line subtitles:

[0359] Similarly, in the "OnAudioUpdate" interface rendering refresh event, dynamically update the parsed subtitle data in "Step S3, Parsing Data":

[0360] functionRefreshSubtitle(){

[0361] this.$nextTick(()=>{

[0362] this.subtitleFirstText=this.subtitleList.find((item)=>{

[0363] return(

[0364] this.player.currentTime*1000>=item.start&&

[0365] this.player.currentTime*1000<=item.end );

[0367] });

[0368] this.subtitleFirstTemp=!!this.subtitleFirstText

[0369] ?this.subtitleFirstText.first

[0370] '';

[0371] letsubtitle=this.$refs.refSubtitleFirst;

[0372] lettemps=this.firstsizes.map((item,index)=>{

[0373] return{

[0374] size:item,

[0375] element:this.$refs[`refSubtitleTemp${index}`][0],

[0376] };

[0377] });

[0378] if(

[0379] !subtitle

[0380] temps.length!=this.firstsizes.length

[0381] !this.subtitleFirstTemp

[0382] ){

[0383] return;

[0384] }

[0385] subtitle.style.fontSize='';

[0386] this.$nextTick(()=>{

[0387] function_CalcSize(temps,index=0){

[0388] lettemp=temps[index];

[0389] if(index>=temps.length!temp!temp.element){

[0390] return'';

[0391] }

[0392] letheight=temp.element.offsetHeight;

[0393] if(height>temp.size*100*(index+1)+10){

[0394] return_CalcSize(temps,index+1);

[0395] }else{

[0396] return `${temp.size}rem`;

[0397] }

[0398] }

[0399] subtitle.style.fontSize=_CalcSize(temps);

[0400] });

[0401] });

[0402] }

[0403] ⑧ Synchronization of background images / picture sets:

[0404] Similarly, in the "OnAudioUpdate" interface rendering refresh event, dynamically update the parsed picture set data in "Step S3, Parsing Data":

[0405] functionRefreshBackgrounds(){

[0406] this.$nextTick(()=>{

[0407] this.background=this.backgroundList.find((item)=>{

[0408] return(

[0409] this.player.currentTime*1000>=item.start&&

[0410] this.player.currentTime*1000<=item.end );

[0412] });

[0413] });

[0414] }

[0415] After this step, the entire "video" starts to "move". During development, the variables, methods, general libraries, etc. are synchronized with the data information of the real video, thereby achieving consistency in the data structure of the simulated video medium.

[0416] Among them, "OnAudioUpdate" is a rendering refresh method that is continuously triggered and executed every 15 to 250 milliseconds when the audio is playing. This method can be used to continuously refresh the interface display.

[0417] S6. Add interactive operations and listen <audio>The control events and the mouse and touch events of the bottom panel realize operations such as loading, playing, pausing, and interactive functions such as dragging the progress bar; realize interactive operations between users and simulated videos, such as playing, pausing, dragging the progress bar, adjusting the volume, etc., to provide a good user experience.

[0418] In this embodiment, considering <audio>The controls will provide complete status events, such as @loadeddata=OnAudioStart, @loadedmetadata=OnAudioInit, @ended=OnAudioEnd, etc. We can monitor various events in real time and complete various display operations such as loading, buffering, playing, pausing, stopping, etc.

[0419] Add mouse and touch events to the bottom panel, such as @mousedown=OnProgressDragStart, @mousemove=OnProgressDragging, @mouseup=OnProgressDragEnd, @mouseleave=OnProgressDragEnd, @touchstart=OnProgressDragStart, @touchmove=OnProgressDragging, @touchend=OnProgressDragEnd, to monitor mouse and touch-dragging progress bar operations to achieve cross-end compatibility for interface rendering and refresh.

[0420] Among them, "mouse events" are mainly for scenarios where mouse operations are used, such as interacting with the user interface of the simulated video by clicking and dragging the mouse on a computer or other device. Specifically, they include events such as @mousedown (mouse press), @mousemove (mouse move), and @mouseup (mouse release). By listening to these events, operations such as clicking the play button and dragging the progress bar can be implemented. For example, when the user presses the progress bar area with the mouse (@mousedown event triggers) and drags the mouse (@mousemove event continuously triggers) to change the position of the progress bar, the system can capture these operations and adjust the audio playback position and the display progress of the video screen accordingly, thereby enabling the user to control the playback progress of the simulated video; when the user releases the mouse (@mouseup event triggers), the final operation confirmation and interface update are completed.

[0421] "Touch events" are mainly for devices with touch functions, such as mobile phones and tablets. Common touch events include @touchstart (touch start), @touchmove (touch move), @touchend (touch end), etc. These events enable users to interact with simulated videos with natural touch gestures on touch devices. For example, a user touches and drags the progress bar on the phone screen (@touchstart and @touchmove event responses) to fast forward or rewind the video, and completes the operation and updates the video status when the touch ends (@touchend event).

[0422] The "mouse and touch events" are mentioned in the present invention to emphasize that the present invention can provide cross-device (mouse operation device and touch operation device) interactive operation functions to ensure smooth and convenient user interaction experience on different types of terminals. This is one of the important technical features of the present invention in terms of interactivity and plays a key role in improving product usability and user satisfaction.

[0423] After this step, the entire "video" has the same event logic as the real video, thus achieving the consistency of the interactive experience of the simulated video medium.

[0424] S7. Expand more functions (preferred steps), add function buttons such as listening training, Chinese-English comparison, and speech speed setting; expand more functions according to needs, such as adding learning tools, special effects settings, etc., to enhance the practicality and fun of simulation videos.

[0425] In this embodiment, as a preferred embodiment, a "listening training" function button is added to realize closing subtitles;

[0426] Added a "Chinese-English comparison" function button to switch between Chinese and English subtitles, and can adaptively display multiple lines of subtitles;

[0427] Added "Speech Speed ​​Setting" function button to achieve fast learning;

[0428] Added a "Blur Context" function button to blur some subtitles or images to improve the sense of language;

[0429] Added "background animation" function button to achieve audio-visual integration;

[0430] Added "Rating and Comments" function button to realize learning ability perception...

[0431] S8. Release resources, stop resource loading and release media resources in OnPause and BeforeDestroy events.

[0432] In this embodiment, when the interaction is idle, the OnPause event stops the continued loading of resources; when leaving the page or destroying the page, the timer and event response are stopped in the BeforeDestroy event, and various media resources are released. This achieves the goal of ready-to-use and ready-to-release, without any cache remaining, saving storage space for users.

[0433] Among them, OnPause event: This event is triggered when the simulated video is in the pause state. In this event, the system will stop loading media resources such as audio, images, and text to avoid unnecessary resource consumption during the pause period. For example, when pausing playback, it will no longer request new subtitle data from the server or load new background image resources, thereby optimizing system performance and saving network bandwidth and device resources.

[0434] BeforeDestroy event: This event is triggered when a page or component is about to be destroyed, for example, when the user closes the page containing the simulated video or switches to other applications, causing the current page to lose focus and be reclaimed by the system. In the BeforeDestroy event, the system will perform a series of operations to completely release various media resources previously occupied, including stopping all running timers (such as timers used for audio playback progress updates or animation effects), canceling all event responses (such as no longer responding to audio control play, pause and other events), and releasing the audio stream stored in the memory, clearing the image cache, and releasing the memory space related to subtitles, etc., to ensure that the system will not have any factors that may cause resource waste or performance degradation after the page is closed or the component is destroyed, effectively improving the system's resource utilization and overall performance.

[0435] These two events are mentioned in the present invention in order to clarify the key technical features and operations of the present invention in resource management, and to emphasize that it can effectively utilize system events to achieve efficient utilization and reasonable release of resources, which is of great significance for improving system performance, reducing resource usage and providing a good user experience.

[0436] < / audio> < / audio> < / audio> < / audio> < / audio> < / audio> Figure 2 What is shown in the figure are the various components from the bottom layer to the upper layer and their interrelationships, which ultimately achieve the effect of simulating video.

[0437] Bottom layer: The bottom layer is APP / browser, which is the platform environment for the entire simulation video system to run, indicating that the system can be used in various applications or browsers.

[0438] Middle layer:

[0439] Audio file + Audio control: The audio file is combined with the Audio control to realize the sound playback. This is the implementation method of the sound part in the analog video.

[0440] Picture file + Image control: The picture file is displayed through the Image control to form a poster effect, providing static visual elements for the simulation video, such as the cover or background of the video.

[0441] Sequence frame picture file + images control: Sequence frame picture file works with images control to realize single-frame background, sequence frame animation background or motion effect, simulating dynamic picture effects in video.

[0442] Subtitle file + Text control: The subtitle file is combined with the Text control to display subtitle content, including Chinese and English subtitles, highlighted subtitles, etc., to enhance the information transmission and learning functions of the video.

[0443] Configuration file + Metadata storage: The configuration file is associated with the Metadata storage and is used to record metadata. The metadata can contain relevant information about the video, such as title, duration, author, etc., to facilitate the management and retrieval of simulation videos.

[0444] Upper layer: The top layer is the UI (user interface), which integrates the above elements to form a complete user interaction interface, allowing users to interact with the simulated video just like watching a real video.

[0445] Embodiment 2

[0446] Based on the same concept, the present invention also proposes a device for simulating video media with multimodal media in dubbing, comprising:

[0447] A project initialization module builds a project and configures and starts a development server; wherein the project is a software development tool or technology platform;

[0448] Resource configuration module, which acquires audio, image, and text multimodal file resources and maps their relationships through automated, semi-automated, or manual methods;

[0449] The data stream request module adopts a front-end and back-end separation solution. The back-end builds a file server, and the front-end requests data and verifies it.

[0450] The data analysis module uses preset algorithms to process different media, so that when the resource structure or metadata of different media changes, they can be uniformly updated and modified;

[0451] UI control loading module, dynamically loads media resource files in the order of audio, image, and text and binds corresponding controls, and sets the control appearance and functions to simulate video;

[0452] The data space synchronization module uses the timed trigger events or simulated events of the audio control to synchronize various data with the corresponding control in the preset related methods; the audio control is a component used to bind and play audio files; the various data include playback status, time, progress bar, full screen status, audio, subtitles, and background images;

[0453] Add a module for interactive operations to monitor audio control events and mouse and touch events on the bottom panel of the display interface to implement interactive functions, including loading, playing, pausing, and progress bar dragging;

[0454] The resource release module stops loading resources when the simulated video is paused, and completely releases resources before the simulated video-related page or component is destroyed.

[0455] Embodiment 3

[0456] This embodiment also provides an electronic device, referring to Figure 3 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0457] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0458] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0459] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0460] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement the method of simulating video media with multimodal media in any dubbing in the above embodiments.

[0461] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0462] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0463] Input / output devices 408 are used to input or output information.

[0464] Embodiment 4

[0465] This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process. The process includes a method for simulating video media in dubbing of multimodal media according to the first embodiment.

[0466] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0467] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0468] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 1 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0469] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0470] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.< / button> < / audio> < / audio> < / audio>

Claims

1. A method for simulating video media with multimodal media in dubbing, characterized in that: The following steps are involved: S0, Initialization project: Build a project and configure and start a development server; where the project is a software development tool or technology platform; S1. Configure resources: Acquire audio, image, and text multimodal file resources and map their relationships through automated, semi-automated, or manual methods; The automated way of acquiring resources includes automatically acquiring audio, image, and text multimodal file resources using audio and video separation technology and automatically mapping their relationships; The semi-automatic way of acquiring resources includes using internal tools or third-party software to assist in acquiring and organizing multimodal resources, and manually performing partial configuration and relationship mapping; The manual resource acquisition method includes manually collecting, producing and organizing audio, image and text multimodal file resources by the design, teaching and research and creation teams, and clarifying the corresponding relationship and usage scenarios between them; S2, request data flow: Adopt the front-end and back-end separation solution, build a file server on the back-end, and request and verify data on the front-end; S3. Parsing data: Preset algorithms are used according to different media to make unified updates and modifications when resource structures or metadata of different media are changed; the preset algorithms include subtitle parsing and conversion algorithms, highlight subtitle synchronization algorithms, and multi-frame sequence image synchronization algorithms; S4. Load UI controls: Dynamically load media resource files in the order of audio, image, and text and bind corresponding controls, and set the appearance and functions of the controls to simulate video; S5. Real-time synchronization of data and controls: With the help of the timed trigger event or simulated event of the audio control, the synchronization of various data and the corresponding control is realized in the preset related method; wherein the audio control is a component used to bind and play audio files; the various data include playing status, time, progress bar, full screen status, audio, subtitles, background pictures; the preset related method includes: Initialization synchronization method, which is triggered when the audio file is loaded and the corresponding control is initialized, is used to set the playback state to the initial stop state, determine the start and end time according to the audio playable range, and reset the progress bar position to zero and close the full screen display, thereby completing the setting of the initial state of the simulated video; A dynamic refresh synchronization method is triggered at a preset frequency during audio playback. In this dynamic refresh synchronization method, time synchronization is achieved by updating the current playback time in real time and adjusting the start and end times according to the audio duration; progress bar synchronization relies on calculating and setting its position according to the playback progress, and can respond to external touch or mouse drag operations to change the playback position; audio synchronization is achieved by keeping consistent with the audio playback progress; subtitle synchronization is achieved by filtering and displaying the corresponding subtitles in the parsed subtitle data according to the current audio playback time; background image synchronization determines and displays the corresponding image in the parsed image set data according to the audio playback time; S6. Add interactive operations: Monitor audio control events and mouse and touch events on the bottom panel of the display interface to implement interactive functions, including loading, playing, pausing, and progress bar dragging; S7. Release resources: Stop loading resources when simulating a paused video state; The resources are completely released before the simulated video-related pages or components are destroyed.

2. The method of simulating video media with multimodal media in dubbing as claimed in claim 1, characterized in that: In step S5, when the audio is played to a certain time point, the corresponding subtitle content is displayed synchronously and the position of the progress bar is updated; when the user drags the progress bar, the audio playback position and subtitle display also jump to the corresponding time point accordingly, and the switching of the background image or the animation effect matches the rhythm of the audio and subtitles.

3. The method of simulating video media with multimodal media in dubbing as claimed in claim 1, characterized in that: In step S6, the progress bar dragging accurately calculates the correspondence between the user's dragging position and the audio playback time to achieve smooth jump and playback; volume adjustment is achieved by changing the volume properties of the audio control, and the current volume level is fed back to the user in real time; the full-screen switching function is compatible with different browsers and devices, ensuring that the video content can be displayed and interacted normally in full-screen mode, and can be restored to the original interface state when exiting the full-screen mode.

4. The method for simulating video media by multimodal media in dubbing as claimed in any one of claims 1 to 3, characterized in that: The project is a Vue3 project, which is a progressive JavaScript framework for building user interfaces.

5. A device for simulating video media with multimodal media in dubbing, characterized in that: include: A project initialization module builds a project and configures and starts a development server; wherein the project is a software development tool or technology platform; A resource configuration module, which acquires audio, image, and text multimodal file resources and maps their relationships in an automated, semi-automated, or manual manner; the automated method of acquiring resources includes automatically acquiring audio, image, and text multimodal file resources using audio and video separation technology and automatically mapping their relationships; the semi-automated method of acquiring resources includes acquiring and organizing multimodal resources with the assistance of internal tools or third-party software, and manually performing partial configuration and relationship mapping; the manual method of acquiring resources includes manually collecting, producing, and organizing audio, image, and text multimodal file resources by the design, teaching and research, and creative teams, and clarifying the corresponding relationships and usage scenarios between them; The data stream request module adopts a front-end and back-end separation solution. The back-end builds a file server, and the front-end requests data and verifies it. The data analysis module uses preset algorithms to process different media so that when the resource structure or metadata of different media changes, they can be uniformly updated and modified; the preset algorithms include subtitle analysis and conversion algorithms, highlight subtitle synchronization algorithms, and multi-frame sequence image synchronization algorithms; UI control loading module, dynamically loads media resource files in the order of audio, image, and text and binds corresponding controls, and sets the control appearance and functions to simulate video; The data space synchronization module uses the timed trigger event or simulation event of the audio control to realize the synchronization of multiple data and the corresponding control in the preset related method; wherein the audio control is a component used to bind and play audio files; the multiple data include playback status, time, progress bar, full screen status, audio, subtitles, background picture; the preset related method includes: Initialization synchronization method, which is triggered when the audio file is loaded and the corresponding control is initialized, is used to set the playback state to the initial stop state, determine the start and end time according to the audio playable range, and reset the progress bar position to zero and close the full screen display, thereby completing the setting of the initial state of the simulated video; A dynamic refresh synchronization method is triggered at a preset frequency during audio playback. In this dynamic refresh synchronization method, time synchronization is achieved by updating the current playback time in real time and adjusting the start and end times according to the audio duration; progress bar synchronization relies on calculating and setting its position according to the playback progress, and can respond to external touch or mouse drag operations to change the playback position; audio synchronization is achieved by keeping consistent with the audio playback progress; subtitle synchronization is achieved by filtering and displaying the corresponding subtitles in the parsed subtitle data according to the current audio playback time; background image synchronization determines and displays the corresponding image in the parsed image set data according to the audio playback time; Add a module for interactive operations to monitor audio control events and mouse and touch events on the bottom panel of the display interface to implement interactive functions, including loading, playing, pausing, and progress bar dragging; The resource release module stops loading resources when the simulated video is paused, and completely releases resources before the simulated video-related page or component is destroyed.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for simulating video media in dubbing by multimodal media as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes the method for simulating video media in dubbing by multimodal media according to any one of claims 1 to 4.

Images