Display device and streaming media file synthesis method

Through the coordinated work of the user interface, player, detector and processor of the display device, the generation and encapsulation of streaming media files is solved, and the problem of synchronous recording and synthesis of multimedia files is realized while playing and recording is improved. User experience.

CN114666516BActive Publication Date: 2024-12-03HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210144774.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-17
Publication Date
2024-12-03
Estimated Expiration
2042-02-17

AI Technical Summary

Technical Problem

The prior art cannot realize synchronous recording and synthesis of multimedia files. Users need to record the multimedia files first and then synthesis, and cannot perform synthesis while recording and playing.

Method used

Receive triggering operations through the user interface of the display device, the player plays the first streaming media file and calls back the data, the detector records the second streaming media file, the processor generates audio and video elementary stream data, and encapsulates it into new Streaming media files.

Benefits of technology

Real-time synthesis is realized while playing and recording streaming media files, solving the problem of synchronous recording and synthesis, and improving user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114666516B_ABST
    Figure CN114666516B_ABST
Patent Text Reader

Abstract

The present application provides a display device and a method for synthesizing streaming media files, and relates to the field of data processing technology. The display device includes: a user interface, configured to receive a trigger operation input by a user; a player, configured to play a first streaming media file and call back the data corresponding to the first streaming media file; a detector, configured to record a second streaming media file and obtain the data corresponding to the second streaming media file; a processor, configured to generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file; an encapsulator, configured to encapsulate the audio elementary stream data and the video elementary stream data to synthesize a third streaming media file. The present application can synthesize a new streaming media file according to the played streaming media file and the recorded streaming media file while playing the streaming media file and recording the streaming media file.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a display device and a streaming media file synthesis method. Background Art

[0002] With the development of audio and video technology, especially the rise of short video platforms in recent years, more and more users will participate in video production. For example, users often have the need to synthesize videos and publish them on short video platforms.

[0003] In the related art, when a user wants to record a streaming media file and synthesize the recorded streaming media file with an existing streaming media file into a new video file, the user needs to first record and obtain the streaming media file, and then use the recorded streaming media file as a material to synthesize with the existing streaming media file to generate a new streaming media file, and it is impossible to synchronize the recording and synthesis of multimedia files. For example: when a user wants to record a video and select a specified audio as background music, the user needs to record the video first, and then select the specified audio to synthesize the new video. It is impossible to select the background music before recording the video, play the background music, and record the video and synthesize the new video. For another example: when a user wants to record an audio and use the recorded audio as the background music of a specified video, the user needs to record the audio first, and then select the specified video to synthesize the new video. It is impossible to select the video before recording the audio, play the video, and record the audio and synthesize the new video. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a display device and a streaming media file synthesis method, which can synthesize new streaming media files based on the played streaming media files and the recorded streaming media files while playing streaming media files and recording streaming media files.

[0005] In a first aspect, the present application provides a display device, including:

[0006] The user interface is configured to: receive a trigger operation input by a user, wherein the trigger operation is used to trigger the playing of a first streaming media file, the recording of a second streaming media file, and the synthesis of a third streaming media file based on the first streaming media file and the second streaming media file;

[0007] The player is configured to: play the first streaming media file, and call back data corresponding to the first streaming media file during the process of playing the first streaming media file;

[0008] The detector is configured to: record the second streaming media file and obtain data corresponding to the second streaming media file;

[0009] The processor is configured to: generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file;

[0010] The encapsulator is configured to encapsulate the audio elementary stream data and the video elementary stream data to synthesize the third streaming media file.

[0011] In some embodiments, the player is configured to: obtain the first streaming media file, decapsulate the first streaming media file to obtain the elementary stream data corresponding to the first streaming media file, cache the elementary stream data corresponding to the first streaming media file in a cache queue, sequentially decode the data in the cache queue to obtain the playback data corresponding to the first streaming media file, play the first streaming media file according to the playback data corresponding to the first streaming media file, and call back the playback data corresponding to the first streaming media file;

[0012] The processor is configured to: encode the playback data corresponding to the first streaming media file to generate the audio elementary stream data or the video elementary stream data.

[0013] In some embodiments, the player is configured to: obtain the first streaming media file, decapsulate the first streaming media file to obtain the basic stream data corresponding to the first streaming media file, cache the basic stream data corresponding to the first streaming media file in a cache queue, sequentially read and decode the data in the cache queue through a decoding module to obtain the playback data corresponding to the first streaming media file, play the first streaming media file according to the playback data corresponding to the first streaming media file, and call back the basic stream data corresponding to the first streaming media file read by the decoding module;

[0014] The processor is configured to: determine the elementary stream data corresponding to the first streaming media file as the audio elementary stream data or the video elementary stream data.

[0015] In some embodiments, the packager is configured to:

[0016] Periodically obtain the player timestamp based on the preset duration;

[0017] Determine whether a difference between a timestamp of elementary stream data corresponding to the first streaming media file called back and a timestamp of the player is less than a preset value;

[0018] If so, encapsulate the called-back video elementary stream data.

[0019] In some embodiments, the user interface is further configured to: receive a start position and an end position of a target segment input by a user; the target segment is a segment in the first streaming media file used to synthesize the third streaming media file;

[0020] The player is further configured to determine a target position, wherein the target position is a position of a key frame that is before the start position and closest to the start position;

[0021] The detector is configured to: start recording the second streaming media file when the playback position of the first streaming media file exceeds the target position; and stop recording the second streaming media file when the playback position of the first streaming media file exceeds the end position.

[0022] In some embodiments, the player is further configured to:

[0023] When the playback position of the first streaming media file exceeds the end position, the playback of the first streaming media file is stopped.

[0024] In some embodiments, the first streaming media file is a video file; the second streaming media file is an audio file;

[0025] The detector is configured to start recording the pulse code modulation (PCM) data corresponding to the audio file when the rendering of the first video frame of the first streaming media file is completed.

[0026] In some embodiments, the first streaming media file includes: a video file and a first audio file, and the second streaming media file is a second audio file;

[0027] The player is configured to: during the process of playing the first streaming media file, call back the elementary stream data corresponding to the video file and the PCM data corresponding to the first audio file;

[0028] The processor is configured to generate an audio data stream according to PCM data corresponding to the first audio file and data corresponding to the second streaming media file, and determine the basic stream data corresponding to the video file as the video basic stream data.

[0029] In some embodiments, the player is further configured to: cache the PCM data corresponding to the first audio file into a first data queue;

[0030] The detector is further configured to: cache the PCM data corresponding to the second streaming media file into a second data queue;

[0031] The processor is configured to: mix the PCM data in the first data queue and the second data queue based on a preset rule to generate mixed audio data, and encode the mixed audio data to generate the audio elementary stream data.

[0032] In a second aspect, the present application provides a streaming media file synthesis method, comprising:

[0033] Receive a trigger operation input by a user, wherein the trigger operation is used to trigger playing of a first streaming media file, recording of a second streaming media file, and synthesis of a third streaming media file based on the first streaming media file and the second streaming media file;

[0034] Playing the first streaming media file, and calling back data corresponding to the first streaming media file during the process of playing the first streaming media file;

[0035] Recording the second streaming media file to obtain data corresponding to the second streaming media file;

[0036] Generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file;

[0037] The audio elementary stream data and the video elementary stream data are encapsulated to synthesize the third streaming media file.

[0038] In some embodiments, playing the first streaming media file and calling back data corresponding to the first streaming media file during the playing of the first streaming media file includes:

[0039] Acquire the first streaming media file, decapsulate the first streaming media file to obtain basic stream data corresponding to the first streaming media file, cache the basic stream data corresponding to the first streaming media file in a cache queue, sequentially decode the data in the cache queue to obtain playback data corresponding to the first streaming media file, play the first streaming media file according to the playback data corresponding to the first streaming media file, and call back the playback data corresponding to the first streaming media file;

[0040] The step of generating audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file comprises:

[0041] The playback data corresponding to the first streaming media file is encoded to generate the audio elementary stream data or the video elementary stream data.

[0042] In some embodiments, playing the first streaming media file and calling back data corresponding to the first streaming media file during the playing of the first streaming media file includes:

[0043] Obtaining the first streaming media file, decapsulating the first streaming media file to obtain basic stream data corresponding to the first streaming media file, caching the basic stream data corresponding to the first streaming media file into a cache queue, sequentially reading and decoding the data in the cache queue through a decoding module to obtain playback data corresponding to the first streaming media file, playing the first streaming media file according to the playback data corresponding to the first streaming media file, and calling back the basic stream data corresponding to the first streaming media file read by the decoding module;

[0044] The step of generating audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file comprises:

[0045] The elementary stream data corresponding to the first streaming media file is determined to be the audio elementary stream data or the video elementary stream data.

[0046] In some embodiments, encapsulating the audio elementary stream data and the video elementary stream data comprises:

[0047] Periodically obtain the player timestamp based on the preset duration;

[0048] Determine whether a difference between a timestamp of elementary stream data corresponding to the first streaming media file called back and a timestamp of the player is less than a preset value;

[0049] If so, encapsulate the called-back video elementary stream data.

[0050] In some embodiments, the method further comprises:

[0051] Receive the starting position and ending position of a target segment input by a user; the target segment is a segment in the first streaming media file used to synthesize the third streaming media file;

[0052] Determine a target position, where the target position is a position of a key frame that is before the starting position and closest to the starting position;

[0053] When the playback position of the first streaming media file exceeds the target position, starting to record the second streaming media file;

[0054] When the playback position of the first streaming media file exceeds the end position, the recording of the second streaming media file is stopped.

[0055] In some embodiments, the method further comprises:

[0056] When the playback position of the first streaming media file exceeds the end position, the playback of the first streaming media file is stopped.

[0057] In some embodiments, the first streaming media file is a video file; the second streaming media file is an audio file;

[0058] The recording of the second streaming media file and obtaining data corresponding to the second streaming media file includes:

[0059] When the rendering of the first video frame of the video file is completed, the pulse code modulation PCM data corresponding to the audio file starts to be recorded.

[0060] In some embodiments, the first streaming media file includes: a video file and a first audio file, and the second streaming media file is a second audio file;

[0061] The calling back the data corresponding to the first streaming media file during the process of playing the first streaming media file includes:

[0062] During the playing of the first streaming media file, the elementary stream data corresponding to the video file and the PCM data corresponding to the first audio file are called back;

[0063] The step of generating audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file comprises:

[0064] An audio data stream is generated according to the PCM data corresponding to the first audio file and the data corresponding to the second streaming media file, and the elementary stream data corresponding to the video file is determined as the video elementary stream data.

[0065] In some embodiments, generating the audio data stream according to the PCM data corresponding to the first audio file and the data corresponding to the second streaming media file includes:

[0066] Cache the PCM data corresponding to the first audio file into the first data queue

[0067] Cache the PCM data corresponding to the second audio file into a second data queue;

[0068] Mixing the PCM data in the first data queue and the second data queue to generate mixed audio data based on a preset rule;

[0069] The mixed audio data is encoded to generate the audio elementary stream data.

[0070] In a third aspect, the present application provides a computer-readable storage medium, including: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the streaming media file synthesis method shown in the second aspect is implemented.

[0071] In a fourth aspect, the present application provides a computer program product, comprising: when the computer program product runs on a computer, the computer implements the streaming media file synthesis method shown in the second aspect.

[0072] When the display device provided in the embodiment of the present application receives a triggering operation for triggering the playback of a first streaming media file, recording a second streaming media file, and synthesizing a third streaming media file based on the first streaming media file and the second streaming media file through a user interface, the display device plays the first streaming media file through a player, and calls back the data corresponding to the first streaming media file during the playback of the first streaming media file, records the second streaming media file through a detector, obtains the data corresponding to the second streaming media file, generates audio basic stream data and video basic stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file through a processor, and encapsulates the audio basic stream data and the video basic stream data through an encapsulator to synthesize the third streaming media file. Since the display device provided by the embodiment of the present application can generate audio basic stream data and video basic stream data according to the corresponding data of the first streaming media file and the corresponding data of the second streaming media file in the process of playing the first streaming media file and recording the second streaming media file, and encapsulate the audio basic stream data and the video basic stream data to synthesize the third streaming media file, the display device provided by the embodiment of the present application can synthesize the played streaming media files and the recorded streaming media files in real time when playing the streaming media files and recording the streaming media files, so as to realize the synthesis of new streaming media files according to the played streaming media files and the recorded streaming media files while playing the streaming media files and recording the streaming media files. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0075] Figure 1A schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application;

[0076] Figure 2 A hardware configuration block diagram of a control device 100 according to one or more embodiments of the present application;

[0077] Figure 3 A hardware configuration block diagram of a display device 200 according to one or more embodiments of the present application;

[0078] Figure 4 This is a diagram of the architecture of a player in one or more embodiments of the present application;

[0079] Figure 5 A schematic diagram of software configuration of the display device 200 in one or more embodiments of the present application;

[0080] Figure 6 One of the flow charts of the steps of the multimedia file synthesis method provided in this application;

[0081] Figure 7 The second step flow chart of the multimedia file synthesis method provided by this application;

[0082] Figure 8 The third step flow chart of the multimedia file synthesis method provided by this application;

[0083] Fig. 9 This is a flowchart of the steps of the multimedia file synthesis method provided by the present application;

[0084] Fig.10 The fifth step flow chart of the multimedia file synthesis method provided by this application;

[0085] Fig.11 This is the sixth step flow chart of the multimedia file synthesis method provided by the present application;

[0086] Fig.12 The seventh step flow chart of the multimedia file synthesis method provided by the present application;

[0087] Fig.13 This is a flowchart of the steps of the multimedia file synthesis method provided by the present application;

[0088] Fig.14 The ninth step flow chart of the multimedia file synthesis method provided by the present application;

[0089] Fig.15 The present invention provides an architecture diagram of a multimedia file synthesis method. DETAILED DESCRIPTION

[0090] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0091] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0092] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0093] The terms "including" and "having" and any variations thereof in this application are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to all the components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0094] In the related art, it is impossible to record and synthesize multimedia files synchronously, but it is necessary to record the multimedia files first, and then synthesize the recorded multimedia files with the existing multimedia files. To solve the above problems, the embodiment of the present application provides a display device and a multimedia file synthesis method. When the display device receives a trigger operation for triggering the playback of a first streaming media file, recording a second streaming media file, and synthesizing a third streaming media file based on the first streaming media file and the second streaming media file through a user interface, the display device plays the first streaming media file through a player, and calls back the data corresponding to the first streaming media file during the playback of the first streaming media file, records the second streaming media file through a detector, obtains the data corresponding to the second streaming media file, generates audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file through a processor, and encapsulates the audio elementary stream data and the video elementary stream data through an encapsulator to synthesize the third streaming media file. Since the display device provided by the embodiment of the present application can generate audio basic stream data and video basic stream data according to the corresponding data of the first streaming media file and the corresponding data of the second streaming media file in the process of playing the first streaming media file and recording the second streaming media file, and encapsulate the audio basic stream data and the video basic stream data to synthesize the third streaming media file, the display device provided by the embodiment of the present application can synthesize the played streaming media files and the recorded streaming media files in real time when playing the streaming media files and recording the streaming media files, so as to realize the synthesis of new streaming media files according to the played streaming media files and the recorded streaming media files while playing the streaming media files and recording the streaming media files.

[0095] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, a mobile phone, etc. Figure 2 This is a specific implementation of the display device of the present application.

[0096] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application, such as Figure 1As shown, the user can operate the display device 200 through the control device 100 and / or the mobile terminal 300. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user instructions through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices can also be used to control the display device 200.

[0097] In some embodiments, the mobile terminal 300 can install software applications for the display device 200, and realize connection communication through the network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 also communicates data with the server 400 through various communication methods. The display device 200 can be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide an intelligent network TV function that provides a computer support function.

[0098] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play the role of an interactive intermediary between the user and the display device 200. The communication interface 130 is used to communicate with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button or an alternative module.

[0099] Figure 3 FIG. 2 shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment. Figure 4The display device 200 shown includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0100] In some embodiments, the above-mentioned display device is a terminal device with a display function, such as a television, a mobile phone, a computer, a learning machine, etc.

[0101] In some embodiments, the controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

[0102] An output interface (display 260 and / or audio output interface 270), configured to output user interaction information;

[0103] The communicator 220 is used to communicate with the server 400 or other devices.

[0104] In some embodiments, the user interface 280 can receive a triggering operation for triggering the playing of a first streaming media file, recording of a second streaming media file, and synthesis of a third streaming media file based on the first streaming media file and the second streaming media file. The player (when the first multimedia file is a video file, the player is a display 260; when the first multimedia file is an audio file, the player is an audio output interface 270; when the first multimedia file includes both a video file and an audio file, the player includes a display 260 and an audio output interface 270) can play the first streaming media file and call back the data corresponding to the first streaming media file during the playing of the first streaming media file; the detector 230 (when the second multimedia file is a video file, specifically an image collector; when the second multimedia file is an audio file, a sound collector) can record the second streaming media file and obtain the data corresponding to the second streaming media file (when the second multimedia file is a video file, the data corresponding to the second streaming media file is image data; when the second multimedia file is an audio file, the data corresponding to the second streaming media file is pulse code modulation data); the controller 250 including a video processor and an audio processor can generate audio basic stream data and video basic stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file, and encapsulate the audio basic stream data and the video basic stream data to synthesize the third streaming media file.

[0105] Reference Figure 4 As shown, in some embodiments, the implementation process of the player (display 260 or audio output interface 270) playing the first streaming media file includes: the resource loading module 41 obtains the first streaming media file from the local storage space or the server, and inputs the first streaming media file into the decapsulation module 42; the decapsulation module 42 decapsulates the first streaming media file, obtains the basic stream data corresponding to the first streaming media file, and caches the basic stream data corresponding to the first streaming media file into the cache queue 43, the decoding module 44 reads the data in the cache queue 43 in sequence, and decodes the read data to obtain the playback data corresponding to the first streaming media file (when the first multimedia file is a video file, the playback data is image data; when the first multimedia file is an audio file, the playback data is pulse code modulation data), and finally the output module 45 plays the first streaming media file according to the playback data corresponding to the first streaming media file.

[0106] Based on the implementation process of the above-mentioned player playing the first streaming media file, there are two implementation methods for the player to call back the data corresponding to the first streaming media file during the process of playing the first streaming media file: the first method: in the process of playing the first streaming media file, the playback data corresponding to each frame played is called back from the decoding module, and then the called-back playback data is encoded to generate an audio basic stream or a video basic stream corresponding to the first streaming media file. The second method: in the process of playing the first streaming media file, the basic stream data corresponding to the first streaming media file is called back. In the above second implementation method, the basic stream data corresponding to the first streaming media file is directly called back instead of the playback data. Therefore, the above second implementation method can avoid the process of encoding the playback data, which consumes computing resources, and avoids changes in the image quality or sound quality in the first multimedia file. In addition, as mentioned above Figure 4 As shown, the basic stream data corresponding to the first streaming media file is generated by the decapsulation module 42 and stored in the cache queue 43, and the decoding module 44 reads the basic stream data corresponding to the first streaming media file from the cache queue 43. Therefore, in the above-mentioned second implementation method, the basic stream data corresponding to the first streaming media file can be called back from the output of the decapsulation module 42 or the cache queue 43 or the input of the decoding module 44. However, the timestamps of the basic stream data output by the decapsulation module 42 and the basic stream data of the cache queue 43 often deviate greatly from the timestamp of the player, while the timestamps of the basic stream data corresponding to the first streaming media file read by the decoding module 44 often deviate less from the timestamp of the player. Therefore, the basic stream data corresponding to the first streaming media file read by the decoding module 44 can be called back to avoid excessive deviation of the timestamps of the encapsulated audio basic stream data and video basic stream data.

[0107] In some embodiments, the controller 250 can also start a sub-thread, through which the player timestamp is periodically obtained with a preset duration (for example, 100 ms), and it is determined whether the difference between the timestamp of the basic stream data corresponding to the first streaming media file called back and the player timestamp is less than a preset value (for example, 2 seconds). If so, the basic stream data corresponding to the first streaming media file called back is sent to the encapsulation module to encapsulate the audio basic stream data and the video basic stream data, thereby completely avoiding excessive deviation in the timestamps of the encapsulated audio basic stream data and the video basic stream data.

[0108] In some embodiments, the present application can also support the user to specify the start and end positions of the first multimedia file for real-time synthesis with the recording file. The implementation method includes: the user interface 280 receives the start position and end position of the target segment for synthesizing the third streaming media file in the first streaming media file input by the user; the player determines the position of the key frame located before the start position and closest to the start position, and the detector starts recording the second streaming media file when the playback position of the first streaming media file exceeds the target position; when the timestamp of the elementary stream data corresponding to the first streaming media file called back exceeds the end position, the recording of the second streaming media file is stopped. Among them, the player can send a start recording instruction to the detector when the playback position of the first streaming media file exceeds the target position, so that the detector starts recording the second streaming media file; when the timestamp of the elementary stream data corresponding to the first streaming media file called back exceeds the end position, the detector sends an end recording instruction to the detector, so that the detector stops recording the second streaming media file. In addition, the player can stop playing the first streaming media file when the timestamp of the elementary stream data corresponding to the first streaming media file called back exceeds the end position.

[0109] Since image data needs to be rendered when playing a video file, it often takes a long time from receiving the video playback instruction to completing the rendering of the first video of the video file. When the first multimedia file is a video file and the second multimedia file is an audio file, if the recording of the second streaming media file starts immediately after receiving the trigger operation input by the user, it is likely to cause the timestamp deviation of the encapsulated audio elementary stream data and the video elementary stream data to be too large. In response to the above problem, in some embodiments, the detector can start recording the pulse code modulation (PCM) data corresponding to the audio file when the player completes the rendering of the first video frame of the video file.

[0110] In some scenarios, the first multimedia file includes both a video file and an audio file, and the second multimedia file is an audio file. For example, in the case of dubbing a video, the first multimedia file includes both a video file and background music, and the second multimedia file is a recorded human voice. In such scenarios, for the video data, the video basic stream data can be extracted in the manner provided by any of the above embodiments, and for the audio data, the background sound and the human voice need to be synthesized, and the audio basic stream data corresponding to the synthesized audio data is obtained. In some embodiments, the player can call back the basic stream data corresponding to the video file and the PCM data corresponding to the first audio file during the playback of the first streaming media file, and the controller 250 can generate an audio data stream according to the PCM data corresponding to the first audio file and the data corresponding to the second streaming media file, and determine the basic stream data corresponding to the video file as the video basic stream data. In some embodiments, the implementation method of the controller 250 generating an audio data stream based on the PCM data corresponding to the first audio file and the data corresponding to the second streaming media file may include: the player caches the PCM data corresponding to the first audio file to a first data queue, and the detector caches the PCM data corresponding to the second audio file to a second data queue; based on preset rules, the PCM data in the first data queue and the second data queue are mixed to generate mixed audio data, and the mixed audio data is encoded to generate the audio basic stream data.

[0111] Figure 5 for Figure 3 The software configuration diagram of the display device 200 is shown as follows. Figure 5As shown, the system is divided into four layers, from top to bottom, they are the application layer (Applications layer for short), the Java interface layer, the native implementation layer, the application framework layer (Application Framework layer for short), and the hardware implementation layer. The application layer is responsible for the process control between multiple modules, the Java interface layer is mainly responsible for the management of the business, and each business is mainly implemented based on the native layer. Usually, the player business includes a loading module, a decapsulation module, a video decoding module, an audio decoding module, a video rendering module, an audio rendering module, etc. The encapsulation and encoding business includes an audio encoding module, a video encoding module, an encapsulation module, etc. The recording business includes a video recording module, an audio recording module, an audio separation module, etc. The loading module is mainly responsible for loading the audio files and / or video files selected by the user in advance from the server or local storage space. The decapsulation module is mainly responsible for decapsulating the audio files and / or video files loaded by the loading module, and obtaining the video elementary stream data of the video file and / or the audio elementary stream data of the audio file. The video decoding module is mainly responsible for decoding the video elementary stream data decapsulated by the decapsulation module into image data. The video rendering module is mainly responsible for rendering the image data decoded by the video decoding module to display the audio file. The audio decoding module is mainly responsible for decoding the audio elementary stream data decapsulated by the decapsulation module into PCM data. The audio rendering module is mainly responsible for rendering the PCM data decoded by the audio decoding module to play the audio file. The video recording module is mainly responsible for collecting the image data corresponding to the video file. The audio recording module is mainly responsible for collecting the PCM data corresponding to the audio file. The audio separation module is mainly responsible for separating the PCM data of the audio file from the multimedia file selected by the user to play in advance when the multimedia file selected by the user to play in advance includes both video files and audio files. The video encoding module is mainly responsible for performing video encoding on the image data called back by the player service or the image data recorded by the video recording module to generate video elementary stream data. The audio encoding module is mainly responsible for performing audio encoding on one or more of the CPM data called back by the player service, the PCM data recorded by the audio recording module, and the PCM data separated by the audio separation module to generate audio elementary stream data. The encapsulation module is mainly responsible for encapsulating the video elementary stream data output by the video encoding module and the audio elementary stream data output by the audio encoding module to generate a synthesized multimedia file.For example: in the scenario of real-time video playback and audio recording, when the user triggers the playback / recording behavior, in the player service, the loading module loads the video file selected by the user in advance, the decapsulation module decodes the video file loaded by the loading module, obtains and outputs the video basic stream data of the video file, the video decoding module decodes the video basic stream data output by the decapsulation module to obtain and output image data, and the video rendering module renders the image data output by the video decoding module, thereby realizing the playback of the video file. In the recording service, the audio recording module collects the PCM data corresponding to the audio file. In the encapsulation and encoding service, when the player service calls back the image data, the video encoding module performs video encoding on the called back image data to obtain video elementary stream data, the audio encoding module performs audio encoding on the recorded PCM data to obtain audio elementary stream data, and the encapsulation module encapsulates the video elementary stream data obtained by video encoding and the audio elementary stream data obtained by audio encoding, and obtains a new multimedia file synthesized by the played video file and the recorded audio file; when the player service calls back the video elementary stream data, the audio encoding module performs audio encoding on the recorded PCM data to obtain audio elementary stream data, and the encapsulation module encapsulates the video elementary stream data obtained by audio encoding and the audio elementary stream data obtained by audio encoding, and obtains a new multimedia file synthesized by the played video file and the recorded audio file. The framework layer is the encapsulated functions implemented by the platform, especially the functions related to the hardware implementation, for example: the multimedia interface related to playback implemented by the framework layer encapsulates the relevant implementation of hard decoding.

[0112] In order to explain the present solution in more detail, the following will be described in an exemplary manner with reference to the accompanying drawings. It can be understood that the steps involved in the accompanying drawings may include more steps or fewer steps in actual implementation, and the order of these steps may also be different, so as to implement the streaming media file synthesis method provided in the embodiment of the present application. Figure 6 As shown, Figure 6 A flowchart of a method for synthesizing a streaming media file according to one or more embodiments of the present application is provided. The method for synthesizing a streaming media file:

[0113] S11. Receive a trigger operation input by a user.

[0114] The trigger operation is used to trigger the playing of a first streaming media file, the recording of a second streaming media file, and the synthesis of a third streaming media file based on the first streaming media file and the second streaming media file.

[0115] Exemplarily, the triggering operation may specifically be an operation of a preset virtual control by a user.

[0116] In some embodiments, the first streaming media file is a video file, and the second streaming media file is an audio file. For example, in a karaoke scene, a video file of a song is played (first streaming media file), a user's singing voice is recorded (second streaming media file), and a music video based on the song and the user's singing voice is synthesized (third streaming media file).

[0117] In some embodiments, the first streaming media file is an audio file, and the second streaming media file is a video file. For example, in a short video creation scenario, the selected background music is played (first streaming media file), the video content of the short video is recorded (second streaming media file), and the short video is synthesized based on the background music and the video content (third streaming media file).

[0118] In some embodiments, the first streaming media file includes both a video and audio file and an audio file, and the second streaming media file is an audio file. For example, in a scene of dubbing a video with background music, the video and background music are played (first streaming media file), the dubbing content is recorded (second streaming media file), and a short video is synthesized based on the video, background music and dubbing content (third streaming media file).

[0119] S12: Play the first streaming media file, and call back data corresponding to the first streaming media file during the process of playing the first streaming media file.

[0120] In some embodiments, playing the first streaming media file in the above step S12 includes the following steps a to d:

[0121] Step a: decapsulate the first streaming media file to obtain elementary stream data corresponding to the first streaming media file.

[0122] Step b: caching the elementary stream data corresponding to the first streaming media file into a cache queue.

[0123] Step c: decoding the data in the cache queue in sequence to obtain playback data corresponding to the first streaming media file.

[0124] Step d: playing the first streaming media file according to the playing data corresponding to the first streaming media file.

[0125] In some embodiments, in the above step S12, calling back the data corresponding to the first streaming media file during the process of playing the first streaming media file includes:

[0126] Call back the playback data corresponding to the first streaming media file.

[0127] That is, after the playback data corresponding to the first streaming media file is obtained by decoding in the above step c, the playback data corresponding to the first streaming media file is called back.

[0128] In some embodiments, in the above step S12, calling back the data corresponding to the first streaming media file during the process of playing the first streaming media file includes:

[0129] Call back the basic stream data corresponding to the first streaming media file read by the decoding module.

[0130] That is, when the playback data corresponding to the first streaming media file is read and decoded in the above step c, the basic stream data corresponding to the first streaming media file is called back.

[0131] S13: Record the second streaming media file to obtain data corresponding to the second streaming media file.

[0132] In some embodiments, the second streaming media file is a video file, and the data corresponding to the second streaming media file may be image data, for example: data in a YUV format or data in an RGB format.

[0133] In some embodiments, the second streaming media file is an audio file, and the data corresponding to the second streaming media file is PCM data.

[0134] S14. Generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file.

[0135] In some embodiments, the first streaming media file is a video file, and the second streaming media file is an audio file, and generating audio elementary stream data and video elementary stream data according to data corresponding to the first streaming media file and data corresponding to the second streaming media file includes:

[0136] Acquire video elementary stream data according to the data corresponding to the first streaming media file;

[0137] The audio elementary stream data is acquired according to the data corresponding to the second streaming media file.

[0138] In some embodiments, the first streaming media file is an audio file, and the second streaming media file is a video file, and generating audio elementary stream data and video elementary stream data according to data corresponding to the first streaming media file and data corresponding to the second streaming media file includes:

[0139] Acquire audio elementary stream data according to the data corresponding to the first streaming media file;

[0140] The video elementary stream data is acquired according to the data corresponding to the second streaming media file.

[0141] S15. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the third streaming media file.

[0142] In some embodiments, when the playback thread completes playback preparation, the encapsulation module can be initialized and track information of the audio elementary stream data and the video elementary stream data can be created to implement encapsulation of the audio elementary stream data and the video elementary stream data through the encapsulation module.

[0143] When the display device provided in the embodiment of the present application receives a triggering operation for triggering the playback of a first streaming media file, recording a second streaming media file, and synthesizing a third streaming media file based on the first streaming media file and the second streaming media file through a user interface, the display device plays the first streaming media file through a player, and calls back the data corresponding to the first streaming media file during the playback of the first streaming media file, records the second streaming media file through a detector, obtains the data corresponding to the second streaming media file, generates audio basic stream data and video basic stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file through a processor, and encapsulates the audio basic stream data and the video basic stream data through an encapsulator to synthesize the third streaming media file. Since the display device provided by the embodiment of the present application can generate audio basic stream data and video basic stream data according to the corresponding data of the first streaming media file and the corresponding data of the second streaming media file in the process of playing the first streaming media file and recording the second streaming media file, and encapsulate the audio basic stream data and the video basic stream data to synthesize the third streaming media file, the display device provided by the embodiment of the present application can synthesize the played streaming media files and the recorded streaming media files in real time when playing the streaming media files and recording the streaming media files, so as to realize the synthesis of new streaming media files according to the played streaming media files and the recorded streaming media files while playing the streaming media files and recording the streaming media files.

[0144] Reference Figure 7 As shown, when the first multimedia file is a video file, the second multimedia file is an audio file, and when the playback data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0145] S71: Receive a trigger operation input by a user.

[0146] The trigger operation is used to trigger the playing of a video file, the recording of an audio file, and the synthesis of a streaming media file based on the video file and the audio file.

[0147] S72: Play the video file, and call back the image data corresponding to the video file during the process of playing the video file.

[0148] S73. Encode the image data corresponding to the video file to obtain video elementary stream data corresponding to the video file.

[0149] In some embodiments, encoding the image data corresponding to the video file to obtain the video elementary stream data corresponding to the video file includes:

[0150] The image data corresponding to the video file is encoded in an H264 hard-coding manner to obtain the video elementary stream data corresponding to the video file.

[0151] S74: Record the audio file and obtain PCM data corresponding to the audio file.

[0152] In an embodiment of the present application, the playback of the video file and the recording of the audio file are started at the same time. In some embodiments, the implementation method of controlling the simultaneous start of the playback of the video file and the recording of the audio file is: when the playback preparation of the video file is completed, an indication message is sent to the recording thread, so that the playback of the video file and the recording of the audio file start at the same time.

[0153] S75. Perform audio encoding on the PCM data to generate audio elementary stream data.

[0154] S76. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the streaming media file.

[0155] When encapsulating, the timestamp information of the basic stream needs to be provided. For the audio basic stream, the timestamp of the audio frame can be written from the initial moment (0) based on the sampling rate and the number of channels. For example: when the audio sampling rate is 44.1khz, the number of channels is 2, and the encoding method is Advanced Audio Coding (ACC), the recorded PCM data is 16-bit little endian, then the size of each frame of PCM data is 1024x2x2=4096 bytes, and the timestamp length of each frame is 1024 / 44.1=23.22ms. For the video, since the player timestamp does not necessarily start from the initial moment (0), for example, the start timestamp of a live video source is generally not the initial moment, the timestamp of the video basic stream data needs to be subtracted from the timestamp of the first frame, that is, the timestamp relative to the first frame of video data is used for the encapsulation module.

[0156] Reference Figure 8As shown, when the first multimedia file is an audio file, the second multimedia file is a video file, and when the playback data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0157] S81. Receive a trigger operation input by a user.

[0158] The trigger operation is used to trigger the playing of an audio file, the recording of a video file, and the synthesis of a streaming media file based on the video file and the audio file.

[0159] S82: Play the audio file, and call back PCM data corresponding to the audio file during the playing of the audio file.

[0160] S83. Encode the PCM data corresponding to the audio file to obtain audio elementary stream data corresponding to the audio file.

[0161] S84: Record the video file to obtain image data corresponding to the video file.

[0162] Similarly, in an embodiment of the present application, the playback of the audio file and the recording of the video file are started at the same time. In some embodiments, the implementation method of controlling the simultaneous start of the playback of the audio file and the recording of the video file is: when the playback preparation of the audio file is completed, an indication message is sent to the video recording thread, so that the playback of the audio file and the recording of the video file start at the same time.

[0163] S85. Perform video encoding on the image data to generate video elementary stream data.

[0164] S86. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the streaming media file.

[0165] Similarly, the timestamp information of the basic stream needs to be provided during encapsulation. For the video basic stream, the timestamp of the video frame can be written from the initial time (0) based on the frame rate and video track information. For the audio, since the player timestamp does not necessarily start from the initial time (0), the timestamp of the audio basic stream data needs to be subtracted from the timestamp of the first frame, that is, the timestamp relative to the first frame of audio data is used for the encapsulation module.

[0166] Reference Fig. 9 As shown, when the first multimedia file is a video file, the second multimedia file is an audio file, and when the basic stream data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0167] S91: Receive a trigger operation input by a user.

[0168] The trigger operation is used to trigger the playing of a video file, the recording of an audio file, and the synthesis of a streaming media file based on the video file and the audio file.

[0169] S92: Play the video file, and during the process of playing the video file, call back the video elementary stream data read by the decoding module.

[0170] Reference Figure 4 As shown, in the process of the player playing the video file, the video basic stream data with the same timestamp is generated earlier than the image data. If the video basic stream data in the encapsulation module is called back and sent to the encapsulation module for encapsulation, the audio and video offsets with the same timestamp will be large, which will trigger a deadlock when playing on some players, resulting in abnormalities such as failure to play or frequent playback freezes. The above embodiment calls back the video basic stream data read by the decoding module, thereby reducing the audio and video offsets with the same timestamp, thereby avoiding abnormalities such as failure to play or frequent playback freezes when the player is playing.

[0171] S93: Record the audio file and obtain PCM data corresponding to the audio file.

[0172] S94: Perform audio encoding on the PCM data to generate audio elementary stream data.

[0173] S95. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the streaming media file.

[0174] Compared with calling back the image data corresponding to the video file during the process of playing the video file, encoding operations need to be performed while playing the video file. In particular, high-frame rate and high-bitrate video encoding consumes a lot of system computing and memory resources, which is very likely to cause video playback freezes, resulting in abnormalities such as the audio and video being out of sync between the played video and the recorded audio. Fig. 9 The embodiment shown in the figure selects to call back the video elementary stream data instead of the image data corresponding to the video when calling back the data corresponding to the video file. Fig. 9 The illustrated embodiment can avoid encoding of video data and change of video quality.

[0175] Reference Fig.10 As shown, when the first multimedia file is an audio file, the second multimedia file is a video file, and the basic stream data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0176] S101: Receive a trigger operation input by a user.

[0177] The trigger operation is used to trigger the playing of an audio file, the recording of a video file, and the synthesis of a streaming media file based on the video file and the audio file.

[0178] S102: Play the audio file, and call back the audio elementary stream data read by the decoding module during the playing of the video file.

[0179] As mentioned by peers, the audio basic stream data read by the callback decoding module can reduce the audio and video offsets of the same timestamp, thereby avoiding abnormalities such as deadlock caused by the player during playback, such as inability to play or frequent playback freezes.

[0180] S103: Record the video file to obtain image data corresponding to the video file.

[0181] S104: Perform video encoding on the image data to generate video elementary stream data.

[0182] S105: Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the streaming media file.

[0183] Compared with calling back the PCM data corresponding to the video file during the playback of the audio file, the system computing and memory resources required to perform encoding operations while playing the audio file are also required, which is very likely to cause video playback freezes, resulting in abnormalities such as the audio and video being out of sync between the played video and the recorded audio. Fig.10 The embodiment shown in the figure selects to call back the audio elementary stream data instead of the PCM data corresponding to the audio when calling back the data corresponding to the audio file. Fig.10 The illustrated embodiment can avoid encoding of audio data and avoid changes in audio quality.

[0184] Reference Fig.11 As shown, when the first multimedia file is a video file, the second multimedia file is an audio file, and when the basic stream data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0185] S111: Receive a trigger operation input by a user.

[0186] The trigger operation is used to trigger the playing of a video file, the recording of an audio file, and the synthesis of a streaming media file based on the video file and the audio file.

[0187] S112: Play the video file, and during the process of playing the video file, call back the video elementary stream data read by the decoding module.

[0188] S113: Record the audio file to obtain PCM data corresponding to the audio file.

[0189] S114: Perform audio encoding on the PCM data to generate audio elementary stream data.

[0190] S115. Periodically obtain the player timestamp based on a preset time period.

[0191] In some embodiments, the timestamp of the video frame currently played by the player may be obtained as the player timestamp.

[0192] In some embodiments, the above step S115 may include: periodically obtaining the player timestamp with a preset duration of 100 ms.

[0193] S116: Determine whether the difference between the timestamp of the called-back video elementary stream data and the timestamp of the player is less than a preset value.

[0194] Exemplarily, the preset value may be 2s, 3s, etc.

[0195] In the above step S116, if it is determined that the difference between the timestamp of the callback video elementary stream data and the timestamp of the player is greater than the preset value, continue to execute the above step S115; if it is determined that the difference between the timestamp of the callback video elementary stream data and the timestamp of the player is less than the preset value, execute the following step S117.

[0196] S117: Send the video elementary stream data to the encapsulator.

[0197] S118. Encapsulate the audio elementary stream data and the video elementary stream data.

[0198] The above embodiment obtains the player timestamp at preset time intervals. If the timestamp of the called-back video basic stream data and the player timestamp are within a certain threshold range, the video basic stream data is sent to the encapsulation module for encapsulation. Therefore, the above embodiment completely avoids and solves the problem of excessive deviation between the audio and video timestamp and the video timestamp.

[0199] Reference Fig.12 As shown, when the first multimedia file is an audio file, the second multimedia file is a video file, and the basic stream data corresponding to the first streaming media file is called back, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0200] S121: Receive a trigger operation input by a user.

[0201] The trigger operation is used to trigger the playing of a video file, the recording of an audio file, and the synthesis of a streaming media file based on the video file and the audio file.

[0202] S122: Play the audio file, and during the playing of the audio file, call back the audio elementary stream data read by the decoding module.

[0203] S123: Record the video file to obtain image data corresponding to the video file.

[0204] S124: Perform video encoding on the image data to generate video elementary stream data.

[0205] S125. Periodically obtain the player timestamp based on a preset duration.

[0206] In some embodiments, the timestamp of the audio frame currently played by the player may be obtained as the player timestamp.

[0207] S126: Determine whether the difference between the timestamp of the called-back audio elementary stream data and the timestamp of the player is less than a preset value.

[0208] Exemplarily, the preset value may be 2s, 3s, etc.

[0209] In the above step S116, if it is determined that the difference between the timestamp of the called-back audio elementary stream data and the timestamp of the player is greater than the preset value, continue to execute the above step S125; if it is determined that the difference between the timestamp of the called-back audio elementary stream data and the timestamp of the player is less than the preset value, execute the following step S127.

[0210] S127, sending the audio elementary stream data to the encapsulator.

[0211] S128. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the multimedia file.

[0212] The above embodiment obtains the player timestamp at preset time intervals. If the timestamp of the called-back audio basic stream data and the player timestamp are within a certain threshold range, the called-back audio basic stream data is sent to the encapsulation module for encapsulation. Therefore, the above embodiment completely avoids and solves the problem of excessive deviation between the audio and video timestamps and the video timestamps.

[0213] In some embodiments, the media file synthesis method provided in the embodiments of the present application also supports the user to select the starting position and the ending position of the first multimedia file to be synthesized with the second multimedia file in real time. Fig.13 As shown, the implementation method includes the following steps:

[0214] S1301: Receive the start position and end position of the target segment.

[0215] The target segment is a segment in the first streaming media file used to synthesize the third streaming media file.

[0216] S1302: Determine the target location.

[0217] The target position is the position of a key frame that is before the starting position and closest to the starting position.

[0218] S1303: Receive a trigger operation input by the user.

[0219] The trigger operation is used to trigger the playing of a first streaming media file, the recording of a second streaming media file, and the synthesis of a third streaming media file based on the first streaming media file and the second streaming media file.

[0220] S1304: Play the first streaming media file.

[0221] S1305: Determine whether the playback position of the first streaming media file exceeds the target position.

[0222] In the above step S1305, if the playback position of the first streaming media file does not exceed the target position, the process returns to step S1304; if the playback position of the first streaming media file exceeds the target position, the process executes the following step S1306.

[0223] S1306: Record the second streaming media file, obtain data corresponding to the second streaming media file, and start to call back the data corresponding to the first streaming media file.

[0224] S1307: Generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file.

[0225] S1308. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the third multimedia file.

[0226] S1309: Determine whether the playback position of the first streaming media file exceeds the end position.

[0227] In the above step S1309, if the playback position of the first streaming media file does not exceed the end position, the process returns to step S1306; if the playback position of the first streaming media file exceeds the target position, the process executes the following step S1310.

[0228] S1310: Stop recording the second streaming media file, and stop calling back data corresponding to the first streaming media file.

[0229] In some embodiments, the multimedia file synthesis method provided in the embodiments of the present application further includes:

[0230] When the playback position of the first streaming media file exceeds the end position, the playback of the first streaming media file is stopped.

[0231] Reference Fig.14 As shown, when the first streaming media file includes: a video file and a first audio file, the second streaming media file is a second audio file, and the second multimedia file is an audio file, the multimedia file synthesis method provided in the embodiment of the present application includes:

[0232] S141: Receive a trigger operation input by a user.

[0233] The trigger operation is used to trigger the playing of a first streaming media file, the recording of a second streaming media file, and the synthesis of a third streaming media file based on the first streaming media file and the second streaming media file.

[0234] S142: Play the first streaming media file, and call back the elementary stream data corresponding to the video file and the PCM data corresponding to the first audio file during the process of playing the first streaming media file.

[0235] S143: Record the second streaming media file to obtain PCM data corresponding to the second streaming media file.

[0236] S144. Generate an audio data stream according to the PCM data corresponding to the first audio file and the data corresponding to the second streaming media file.

[0237] S145. Encapsulate the audio elementary stream data and the video elementary stream data to synthesize the third streaming media file.

[0238] In some embodiments, generating an audio data stream according to PCM data corresponding to the first audio file and data corresponding to the second streaming media file includes:

[0239] The PCM data corresponding to the first audio file is cached in a first data queue; the PCM data corresponding to the second audio file is cached in a second data queue; based on a preset rule, the PCM data in the first data queue and the second data queue are mixed to generate mixed audio data; and the mixed audio data is encoded to generate the audio basic stream data.

[0240] Reference Fig.15As shown, in some embodiments, the implementation of the above embodiment includes: when a trigger operation input by a user is received, the first multimedia file is obtained through the resource loading module, and the first multimedia is sent to the decapsulation module for decapsulation. The decapsulation result of the first multimedia file includes one audio elementary stream data and one video elementary stream data, and the audio elementary stream data and the video elementary stream data will be cached in the audio elementary stream data queue and the video elementary stream data queue respectively. The video elementary stream data is obtained by calling back the video elementary stream data read by the decoding module. The video elementary stream data is obtained by calling back the PCM data corresponding to the first audio file and storing it in the first data queue, obtaining the PCM data corresponding to the second audio file through the sound collector, and storing it in the second data queue, mixing the PCM data in the first data queue and the second data queue based on a preset rule to generate mixed audio data, and encoding the mixed audio data to obtain the audio elementary stream data. Finally, the audio elementary stream data and the video elementary stream data are encapsulated to synthesize the third streaming media file.

[0241] Since the size of audio data of each frame is fixed, for example, the size of one frame of audio data in AAC format, 2 channels, 16 bits is fixed to 4096 bytes, so in some embodiments, the data in the first data queue and the second data queue can be synthesized by data volume.

[0242] In some embodiments, the multimedia file synthesis method provided in the embodiments of the present application can also adjust the delay of the second audio file as needed. For example, only the data in the first data queue is taken for the first five frames, and the audio data in the first data queue and the second data queue are synthesized starting from the sixth frame. For 44.1k sampled audio, the microphone sound can be delayed by 5x1024 / 44.1ms.

[0243] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned streaming media file synthesis method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0244] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0245] The present application provides a computer program product, including: when the computer program product is run on a computer, the computer is enabled to implement the above-mentioned streaming media file synthesis method.

[0246] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A display device, characterized in that: include: The user interface is configured to: receive a trigger operation input by a user, wherein the trigger operation is used to trigger the playing of a first streaming media file, the recording of a second streaming media file, and the synthesis of a third streaming media file based on the first streaming media file and the second streaming media file; The player is configured to: play the first streaming media file, and call back data corresponding to the first streaming media file during the process of playing the first streaming media file; The detector is configured to: record the second streaming media file and obtain data corresponding to the second streaming media file; The processor is configured to: generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file; The encapsulator is configured to: encapsulate the audio elementary stream data and the video elementary stream data to synthesize the third streaming media file; The player is configured to: obtain the first streaming media file, decapsulate the first streaming media file to obtain basic stream data corresponding to the first streaming media file, cache the basic stream data corresponding to the first streaming media file in a cache queue, sequentially read and decode the data in the cache queue through a decoding module to obtain playback data corresponding to the first streaming media file, play the first streaming media file according to the playback data corresponding to the first streaming media file, and call back the basic stream data corresponding to the first streaming media file read by the decoding module; The processor is configured to: determine the elementary stream data corresponding to the first streaming media file as the audio elementary stream data or the video elementary stream data.

2. The display device according to claim 1, characterized in that The wrapper is also configured to: Periodically obtain the player timestamp based on the preset duration; Determine whether a difference between a timestamp of elementary stream data corresponding to the first streaming media file called back and a timestamp of the player is less than a preset value; If so, encapsulate the called-back video elementary stream data.

3. The display device according to claim 1, characterized in that The user interface is further configured to: receive a start position and an end position of a target segment input by a user; the target segment is a segment in the first streaming media file used to synthesize the third streaming media file; The player is further configured to: determine a target position, the target position being a position of a key frame that is before the start position and closest to the start position; The detector is configured to: start recording the second streaming media file when the playback position of the first streaming media file exceeds the target position; When the playback position of the first streaming media file exceeds the end position, the recording of the second streaming media file is stopped.

4. The display device according to claim 3, characterized in that The player is further configured to: When the playback position of the first streaming media file exceeds the end position, the playback of the first streaming media file is stopped.

5. The display device according to claim 1, characterized in that The first streaming media file is a video file; the second streaming media file is an audio file; The detector is configured to start recording the pulse code modulation PCM data corresponding to the audio file when the rendering of the first video frame of the video file is completed.

6. The display device according to claim 1, characterized in that The first streaming media file includes: a video file and a first audio file, and the second streaming media file is a second audio file; The player is configured to: during the process of playing the first streaming media file, call back the elementary stream data corresponding to the video file and the PCM data corresponding to the first audio file; The processor is configured to generate an audio data stream according to PCM data corresponding to the first audio file and data corresponding to the second streaming media file, and determine the basic stream data corresponding to the video file as the video basic stream data.

7. The display device according to claim 6, characterized in that The player is further configured to: cache the PCM data corresponding to the first audio file into a first data queue; The detector is further configured to: cache the PCM data corresponding to the second audio file into a second data queue; The processor is configured to: mix the PCM data in the first data queue and the second data queue based on a preset rule to generate mixed audio data, and encode the mixed audio data to generate the audio elementary stream data.

8. A method for synthesizing streaming media files, characterized in that: include: Receive a trigger operation input by a user, wherein the trigger operation is used to trigger playing of a first streaming media file, recording of a second streaming media file, and synthesis of a third streaming media file based on the first streaming media file and the second streaming media file; Playing the first streaming media file, and calling back data corresponding to the first streaming media file during the playing of the first streaming media file; Recording the second streaming media file to obtain data corresponding to the second streaming media file; Generate audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file; Encapsulating the audio elementary stream data and the video elementary stream data to synthesize the third streaming media file; The playing of the first streaming media file and calling back the data corresponding to the first streaming media file during the playing of the first streaming media file include: obtaining the first streaming media file, decapsulating the first streaming media file to obtain the basic stream data corresponding to the first streaming media file, caching the basic stream data corresponding to the first streaming media file into a cache queue, sequentially reading and decoding the data in the cache queue by a decoding module to obtain the playback data corresponding to the first streaming media file, playing the first streaming media file according to the playback data corresponding to the first streaming media file, and calling back the basic stream data corresponding to the first streaming media file read by the decoding module; The generating of audio elementary stream data and video elementary stream data according to the data corresponding to the first streaming media file and the data corresponding to the second streaming media file includes: determining the elementary stream data corresponding to the first streaming media file as the audio elementary stream data or the video elementary stream data.