Live broadcast content processing method and device, equipment and storage medium
By implementing block-based memory management and simultaneous recording and uploading during live streaming, the problem of excessive disk I/O and memory usage during live streaming recording was solved, enabling fast uploading and high-concurrency recording tasks while reducing costs.
Patent Information
- Application Number
- CN202310508355.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-06
- Publication Date
- 2024-11-08
AI Technical Summary
Existing live streaming recording methods consume a lot of disk I/O and memory during long-duration or high-bitrate live streams, resulting in a low number of concurrent recording tasks and lengthy upload times.
During the live recording process, file writing and uploading operations are continuously performed. The memory is managed in blocks, and the recording sub-files are written to blocks in memory. When the number of blocks written reaches a preset value, the files are uploaded to the storage unit and the memory is released. The recording files are merged when the recording ends.
It enables simultaneous recording and uploading, reducing uploading time after recording ends, increasing the number of concurrent recording tasks on a single machine, and saving on live recording costs.
Smart Images

Figure CN118921501A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of cloud technology, and in particular to a live content processing method, device, equipment and storage medium. Background Art
[0002] Live broadcast is an emerging online social method that can share multimedia content such as audio and video from the live broadcast end with different online users in real time through cloud technology. With the rapid development of the live broadcast industry, cloud live broadcast recording has become an indispensable core function of cloud live broadcast platforms. The cloud live broadcast recording service needs to pull and push stream data from customers in real time for transpackaging, generate specific recording files at the end of the live broadcast, and then upload the generated recording files to cloud storage.
[0003] An existing live broadcast recording method specifically writes live broadcast stream data continuously into a local file system or host memory, generates a complete recording file when the live broadcast ends or the written content reaches a preset length, and then uploads the recording file to cloud storage.
[0004] However, if the live broadcast lasts for a long time or the live streaming bitrate is high, the generated recording file will be larger, which will occupy more disk IO and memory during the recording process. Therefore, the number of concurrent recording tasks for a single machine is small, and it will also lead to a relatively lengthy upload time. Summary of the invention
[0005] The embodiments of the present application provide a live content processing method, device, equipment and storage medium, which can reduce the time consumed in uploading live content after recording is completed and increase the number of concurrent recording tasks for a single machine.
[0006] In a first aspect, a live content processing method is provided, comprising:
[0007] Obtaining a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information;
[0008] Pulling a stream from a stream server according to the target stream address, recording the pulled stream media according to the recording parameter configuration information, and continuously performing file writing and uploading operations during the recording process until the recording ends, wherein the file writing and uploading operations are: writing a recording sub-file into a block of a memory, wherein the memory is pre-divided into a plurality of blocks, and when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, wherein M is a preset positive integer;
[0009] When the recording is finished, an indication message is sent to the storage unit, where the indication message is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a recording file.
[0010] In a second aspect, a live content processing method is provided, comprising:
[0011] In response to the live content access operation, displaying a live content selection page, the live content selection page including a live recording file released by a live platform;
[0012] Wherein, the live broadcast recording file is a file that the server pulls the stream from the stream server according to the target stream address after obtaining the recording task instruction, and the recording task instruction includes the target stream address and the recording parameter configuration information, and the pulled streaming media is recorded according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer, and when the recording is finished, the server sends an indication message to the storage unit, and the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files;
[0013] In response to an access trigger operation on a target live broadcast recording file, sending a request for obtaining the target live broadcast recording file to a server;
[0014] Receiving the target live broadcast recording file sent by the server;
[0015] According to the target live broadcast recording file, a corresponding page display is performed.
[0016] In a third aspect, a live content processing method is provided, comprising:
[0017] Obtaining a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information, wherein the recording parameter configuration information includes an MP4 file format;
[0018] Pull the stream from the stream server according to the target stream address, initialize the demultiplexing context and the multiplexing context used in the stream pulling process, and set the recording parameters according to the MP4 file format, wherein the recording parameters include the stream pulling source address, the recording in the MP4 file format and the output media type, and the custom parameters, wherein the custom parameters are used to instruct the recording unit to write the recording sub-file into the memory;
[0019] According to the recording parameters, the pulled streaming media is recorded. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into a block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer;
[0020] When the recording is finished, an indication message is sent to the storage unit, where the indication message is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live broadcast recording file.
[0021] In a fourth aspect, a live content processing device is provided, comprising:
[0022] An acquisition module, used for acquiring a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information;
[0023] A recording module, used for pulling a stream from a streaming server according to the target stream address, recording the pulled streaming media according to the recording parameter configuration information, and continuously performing file writing and uploading operations during the recording process until the recording is finished, wherein the file writing and uploading operations are as follows: writing a recording sub-file into a block of a memory, wherein the memory is pre-divided into a plurality of blocks, and when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, wherein M is a preset positive integer;
[0024] The sending module is used to send indication information to the storage unit when the recording is finished, and the indication information is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a recording file.
[0025] In a fifth aspect, a live content processing device is provided, including:
[0026] A display module, configured to display a live content selection page in response to a live content access operation, wherein the live content selection page includes a live recording file released by a live platform;
[0027] Wherein, the live broadcast recording file is a file that the server pulls the stream from the stream server according to the target stream address after obtaining the recording task instruction, and the recording task instruction includes the target stream address and the recording parameter configuration information, and the pulled streaming media is recorded according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer, and when the recording is finished, the server sends an indication message to the storage unit, and the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files;
[0028] A sending module, configured to send a request for obtaining the target live broadcast recording file to a server in response to an access trigger operation on the target live broadcast recording file;
[0029] A receiving module, used for receiving the target live broadcast recording file sent by the server;
[0030] The display module is also used to: display a corresponding page according to the target live broadcast recording file.
[0031] In a sixth aspect, a live content processing device is provided, including:
[0032] An acquisition module, used for acquiring a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information, wherein the recording parameter configuration information includes an MP4 file format;
[0033] A recording module, used to pull the stream from the stream server according to the target stream address, initialize the demultiplexing context and the multiplexing context used in the stream pulling process, and set the recording parameters according to the MP4 file format, wherein the recording parameters include the stream pulling source address, the recording in the MP4 file format and the output media type, and the custom parameters, wherein the custom parameters are used to instruct the recording unit to write the recording sub-file into the memory;
[0034] According to the recording parameters, the pulled streaming media is recorded. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into a block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer;
[0035] The sending module is used to send indication information to the storage unit when the recording is finished, and the indication information is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live recording file.
[0036] In the seventh aspect, a live content processing device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method in the first aspect or its various embodiments or the second aspect or its various embodiments or the method in the third aspect or its various embodiments.
[0037] In an eighth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute a method as in the first aspect or its various embodiments, or the second aspect or its various embodiments, or the third aspect or its various embodiments.
[0038] In the ninth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute the method of the first aspect or its various embodiments, the second aspect or its various embodiments, or the third aspect or its various embodiments.
[0039] In summary, in an embodiment of the present application, after obtaining a recording task instruction, the recording task instruction includes a target stream address and recording parameter configuration information, and the stream is pulled from the stream server according to the target stream address. According to the recording parameter configuration information, the pulled streaming media is recorded. During the live recording process, file writing and uploading operations are continuously performed, the memory is managed in blocks, and the recording sub-files are written to the blocks of the memory. When the number of blocks written with the recording sub-files is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released. At the end of the recording, an indication information is sent to the storage unit, instructing the storage unit to merge the multiple uploaded recording sub-files to obtain a live recording file. Therefore, as the live recording process continues, uploading can be achieved while recording, reducing the uploading time after the recording is completed, and achieving file output in seconds. Moreover, during the recording process, only the memory of the size of M blocks is occupied and no disk io resources are occupied at all, thereby increasing the number of concurrent recording tasks for a single machine and saving the cost of live recording. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 A schematic diagram of an application scenario of a live content processing method provided in an embodiment of the present application;
[0042] Figure 2 A flowchart of a live content processing method provided in an embodiment of the present application;
[0043] Figure 3 This is a schematic diagram of the structure of a box in MP4;
[0044] Figure 4 A schematic diagram of an application architecture of a live content processing method provided in an embodiment of the present application;
[0045] Figure 5 A flowchart of a live content processing method provided in an embodiment of the present application;
[0046] Figure 6 A flowchart of a live content processing method provided in an embodiment of the present application;
[0047] Figure 7 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application;
[0048] Figure 8 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application;
[0049] Fig. 9 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application;
[0050] Fig.10 It is a schematic block diagram of a live content processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0053] Before introducing the technical solution of the embodiment of the present application, the following is an introduction to the relevant knowledge of the embodiment of the present application:
[0054] 1. Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0055] 2. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.
[0056] At present, the storage method of the storage system is: create a logical volume, and when creating the logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object not only contains data but also contains additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object. The process of allocating physical storage space for the logical volume by the storage system is specifically: according to the capacity estimation of the objects stored in the logical volume (the estimation often has a large margin relative to the capacity of the actual objects to be stored) and the group of independent redundant disk array (RAID, Redundant Array of Independent Disk), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, so that the physical storage space is allocated to the logical volume.
[0057] 3. Live broadcasting enables viewers to watch the live scene at a remote location through the Internet. That is, the party initiating the live broadcast shares the multimedia content such as audio and video of the scene with the audience in real time. As an emerging social method, live broadcasting has more and more users on the Internet.
[0058] 4. Video information stream push refers to the process of continuously pushing live content, that is, live information stream composed of multimedia content, to the server so that the server can distribute the information stream to the terminals of the audience watching the live broadcast, thus realizing the live broadcast process.
[0059] 5. Streaming terminal: a terminal that generates multimedia information streams, also known as the terminal used by anchors for live broadcasts, i.e., the live broadcast terminal.
[0060] 6. Streaming terminal: The terminal that receives multimedia information streams, that is, the terminal used by the audience when watching live broadcasts, that is, the playback terminal. In a live broadcast, the number of playback devices can be 0 to N.
[0061] 7. The streaming server is the "bridge" between the live broadcast end and the playback end. In a live broadcast, the streaming device sends the generated multimedia information stream to the streaming server, and the streaming server distributes the multimedia information stream to the playback end.
[0062] 8. MP4, or MPEG-4 Part 14, is a standard digital multimedia container format. The extension of MPEG-4 Part 14 is .mp4. It is mainly used to store digital audio and digital video, but can also store subtitles and still images. It can accommodate video streams that support bitstreams (such as Advanced Video Coding) and is streaming media.
[0063] 9. FFmpeg: FFmpeg is an open source free software that can perform recording, conversion, and streaming functions in multiple audio and video formats. It mainly includes libavcodec, which is an audio and video decoder library used in multiple projects, and libavformat, an audio and video format conversion library.
[0064] 10. Muxing: abbreviation of multiplexing, which refers to the process of interweaving multiple types of media data into a streaming media or file.
[0065] 11. Demuxing: abbreviation of demultiplexing, which is the opposite of muxing. It refers to the process of separating media data of different types from a streaming media or file.
[0066] 12. Live recording is a service that converts the original live stream into audio and video (without modifying the audio, video data, corresponding timestamps, etc.) and stores the resulting file on the on-demand platform. Live recording can store the recorded file on the on-demand platform for on-demand playback.
[0067] In the live broadcast recording method of the prior art, a large amount of disk IO and memory are occupied during the recording process, so the number of concurrent recording tasks of a single machine is small, and it also leads to a relatively lengthy upload time.
[0068] In order to solve this technical problem, in an embodiment of the present application, during the live recording process, file writing and uploading operations are continuously performed, the memory is managed in blocks, and the recording sub-files are written into the blocks of the memory. When the number of blocks into which the recording sub-files have been written is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released. At the end of the recording, an indication message is sent to the storage unit, instructing the storage unit to merge the multiple uploaded recording sub-files to obtain a live recording file. Thus, as the live recording process continues, uploading can be achieved while recording, reducing the uploading time after the recording is completed, and achieving file output in seconds. Moreover, during the recording process, only the memory of the size of M blocks is occupied and no disk io resources are occupied at all, thereby increasing the number of concurrent recording tasks for a single machine and saving the cost of live recording.
[0069] It should be understood that the technical solution provided in the embodiments of the present application can be applied to the following scenarios, but is not limited to:
[0070] Exemplarily, the technical solution of the embodiment of the present application can be applied to a cloud live broadcast scenario to record live streaming media. Specifically, the original live stream can be encapsulated through audio and video conversion (without modifying audio, video data and other information) and stored in a file on a video-on-demand platform. The live broadcast recording can store the recorded file on the video-on-demand platform to facilitate on-demand playback or other processing of the recorded file.
[0071] Figure 1 A schematic diagram of an application scenario of a live content processing method provided in an embodiment of the present application, such as Figure 1 As shown, the application scenario mainly involves a stream pulling terminal 101, a stream pushing terminal 102, a stream server 103 and a recording server 104. The stream pulling terminal 101, the stream pushing terminal 102, the stream server 103 and the recording server 104 can communicate with each other through a wired or wireless network.
[0072] The stream pulling terminal 101 and / or the stream pushing terminal 102 include but are not limited to smart phones, tablets, laptops, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Optionally, the stream pulling terminal 101 and the stream pushing terminal 102 may be the same terminal or different terminals.
[0073] The streaming server 103 or the recording server 104 can be a server that provides various services. It can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This embodiment does not limit this.
[0074] In a live broadcast, the streaming terminal 102 sends the generated multimedia information stream to the streaming server 103 , and the streaming server 103 distributes the multimedia information stream to the streaming terminal 101 .
[0075] It should be noted that Figure 1 The number of the stream pulling terminal 101, the stream pushing terminal 102, and the stream server 103 is only for illustration, and any number of the stream pulling terminal 101, the stream pushing terminal 102, and the stream server 103 may be provided according to actual needs.
[0076] In one embodiment of the present application, the live content processing method may be executed by the recording server 104 .
[0077] Exemplarily, in one embodiment, before the push stream terminal 102 pushes the stream, that is, before the live broadcast starts, the recording task configuration information can be set through the live broadcast platform. The recording task configuration information includes multiple recording tasks and recording parameter configuration information of each recording task. Each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded. At the beginning of the live broadcast, the push stream terminal 102 pushes the stream to the streaming server 103. When the streaming server 103 receives the multimedia information stream pushed by the push stream terminal 102, it sends a push stream notification message to the recording server 104. The push stream notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media. After receiving the push stream notification message, the recording server 104 determines that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information, and then triggers the recording task instruction. The recording task instruction includes the target stream address and the recording parameter configuration information. Next, the recording server 104 pulls the stream from the streaming server according to the target stream address, and records the pulled streaming media according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished. The file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-files have been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, M is a preset positive integer, and at the end of the recording, sending an indication message to the storage unit, the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files to obtain a live recording file.
[0078] Exemplarily, in another embodiment, before streaming, that is, before the live broadcast begins, the streaming terminal 102 can set recording task configuration information through the live broadcast platform. The recording task configuration information includes multiple recording tasks and recording parameter configuration information for each recording task. Each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded. Optionally, the recording parameter configuration information may include recording start and end time, recording file format, file length, and file storage location information. Optionally, the distinction between multiple recording tasks configured for the same streaming media is mainly different in start and end time and recording parameter configuration information, and the input information of multiple recording tasks - the domain name of the streaming media and the identifier of the streaming media are the same. At the beginning of the live broadcast, the streaming terminal 102 pushes the stream to the streaming server 103. When the streaming server 103 receives the multimedia information stream pushed by the streaming terminal 102, it sends a streaming notification message to the live broadcast control center server. The streaming notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media. After receiving the streaming notification message, the live broadcast control center server determines that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information, and then sends a recording task instruction to the recording server 104. The recording task instruction includes the target stream address and recording parameter configuration information. Next, the recording server 104 pulls the stream from the streaming server according to the target stream address, and records the pulled streaming media according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished. The file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-files have been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, M is a preset positive integer, and at the end of the recording, sending an indication message to the storage unit, the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files to obtain a live recording file.
[0079] Among them, the storage unit in the embodiment of the present application can be a cloud storage server for storing various live broadcast data, interactive data, live broadcast recording files and other information generated during the live broadcast process, and specifically can be a cloud object storage server. The present application does not impose any restrictions on the data content stored in the storage unit and its storage method.
[0080] In some embodiments, for some important data obtained by the live broadcast platform, such as the live broadcast recording files in the embodiments of the present application, cloud storage can be used to achieve storage through a distributed cloud storage system. In this case, physical storage space can be allocated in advance for each logical volume created, such as a storage space composed of disks of a storage device or several storage devices. The terminal can store data on a certain logical volume, thereby storing the data on the file system, and then writing the physical storage space of the logical volume, and recording the data storage location, so that when the terminal logs in to the live broadcast platform to access data, the file system can access the data according to the storage location. The specific implementation process is not described in detail one by one in this application.
[0081] After the live broadcast recording file is stored in the storage unit, both the stream pulling terminal 101 and the stream pushing terminal 102 can access (eg, play on demand) the live broadcast recording file through the on-demand platform, or perform other processing on the live broadcast recording file.
[0082] The technical solution of the embodiment of the present application will be described in detail below:
[0083] Figure 2 A flowchart of a live content processing method provided in an embodiment of the present application, the method can be, for example, Figure 1 The recording server 104 shown performs, but is not limited to, Figure 2 As shown, the method may include the following steps:
[0084] S101. Obtain a recording task instruction, where the recording task instruction includes a target stream address and recording parameter configuration information.
[0085] Specifically, the recording task instruction includes a target stream address and recording parameter configuration information, wherein the target stream address is the stream address of the streaming media to be recorded, and the recording parameter configuration information may include a recording format, which is the format of the recording file obtained by recording, and the format of the recording file may be, for example, MP4 format, FLV or HLS, etc. Optionally, the recording parameter configuration information may also include storage location information, storage format and storage duration, etc. The storage location information is used to store the recording file in a storage unit according to the storage location information after the recording is completed.
[0086] In an practicable manner, the recording task instruction is triggered based on a push notification message and recording task configuration information, the push notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media, the recording task configuration information includes multiple recording tasks and recording parameter configuration information of each recording task, and each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded. In this manner, the push notification message can be sent by the streaming server after receiving the streaming media sent by the push terminal, and the streaming server can obtain the domain name, target stream address and the identifier of the streaming media according to the received streaming media. The recording task configuration information can be set by the user through the live broadcast platform, and each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded, and can also include the recording duration, and recording according to the specified time period, and can also include the recording mode, such as audio and video recording or pure audio recording or pure video recording, etc. For example, the user creates a recording task through the live broadcast platform, and the recording task is a recording task from 8 am to 10 am on May 10, XX, the recording format is FLV, the recording mode is video recording, the segment interval is 30 minutes, and the storage duration is permanent storage.
[0087] As an implementable method, obtaining the recording task instruction in S101 may specifically be:
[0088] S1011. Receive a streaming notification message sent by a streaming server.
[0089] S1012: Determine that at least one of a domain name of a streaming media and an identifier of a streaming media corresponding to a target streaming address exists in the recording task configuration information, and trigger a recording task instruction.
[0090] Specifically, after receiving the streaming media sent by the streaming terminal, the streaming server sends a streaming notification message to the recording server. The streaming notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media. If the recording server determines based on the streaming notification message that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information, it triggers the recording task instruction.
[0091] As an implementable method, obtaining the recording task instruction in S101 may specifically be:
[0092] Receive a recording task instruction sent by the live broadcast control server. The recording task instruction is sent by the live broadcast control server after receiving the push notification message sent by the streaming server and determining that at least one of the domain name of the streaming media and the identifier of the streaming media corresponding to the target streaming address exists in the recording task configuration information.
[0093] In this embodiment, after receiving the streaming media sent by the streaming terminal, the streaming server sends a streaming notification message to the live broadcast control server. The streaming notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media. If the live broadcast control server determines based on the streaming notification message that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information, it sends a recording task instruction to the recording server.
[0094] S102, pull the stream from the stream server according to the target stream address, and record the pulled streaming media according to the recording parameter configuration information. During the recording process, continue to perform file writing and uploading operations until the recording is completed. The file writing and uploading operations are: write the recording sub-file to the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks written with the recording sub-file is greater than M, upload the recording sub-files in the M blocks to the storage unit, and release the memory occupied by the M blocks, where M is a preset positive integer.
[0095] S103. When the recording is finished, an indication message is sent to the storage unit, where the indication message is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live broadcast recording file.
[0096] Specifically, the recording server pulls the stream from the streaming server according to the target stream address, which may be to send a stream pull request carrying the target stream address to the streaming server. After receiving the stream pull request, the streaming server sends the streaming media whose stream address is the target stream address to the recording server, and the recording server records the pulled streaming media according to the recording parameter configuration information. Specifically, during the recording process, the file writing and uploading operations are continuously performed until the recording is finished. The file writing and uploading operations are: writing the recording sub-file to the block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, M is a preset positive integer, for example, M is equal to 2, or M can also be other values, the smaller M is, the smaller the memory occupied during the recording process, and the more concurrent single-machine recording tasks.
[0097] In the embodiment of the present application, since the recording sub-file is written into the memory block, the memory occupied by the block is released after the recording sub-file in the block is uploaded. Therefore, as the live recording process continues, it is possible to upload while recording, which reduces the uploading time after the recording is completed, and the file is available in seconds. Moreover, during the recording process, only the memory of M blocks is occupied and no disk io resources are occupied. Therefore, the number of concurrent recording tasks of a single machine is increased, saving the cost of live recording.
[0098] The following is a specific implementation of recording the pulled streaming media according to the recording parameter configuration information in S102.
[0099] Optionally, the recording parameter configuration information includes a recording format. In S102, the pulled streaming media is recorded according to the recording parameter configuration information, which may be:
[0100] S1021. Initialize the demultiplexing context and multiplexing context used in the streaming process, and set recording parameters according to the recording format. The recording parameters include the streaming source address, recording file format, output media type, and custom parameters. The custom parameters are used to instruct the recording unit to write the recording sub-file into the memory.
[0101] S1022: Record the pulled streaming media according to the recording parameters.
[0102] Specifically, recording the pulled streaming media according to the recording parameters may be performed by recording the pulled streaming media according to the recording parameters through a recording unit.
[0103] Specifically, a recording task corresponds to a demultiplexing context, and the pulled streaming media corresponds to a demultiplexing context. A demultiplexing context receives audio and video frames one by one. In this embodiment, the demultiplexing context and muxing context used in the stream pulling process are first initialized, and the recording parameters are set according to the recording format, and then the pulled streaming media is recorded according to the recording parameters. The recording parameters include the stream pulling source address, the recording file format, the output media type and the custom parameters. The stream pulling source address is the target stream address, the recording file format is the recording format in the recording parameter configuration information, and the output media type can include audio and video, pure audio and pure video. The custom parameters are used to instruct the recording unit to write the recording sub-file to the memory. The recording unit can be, for example, ffmpeg. The custom parameters are used to instruct ffmpeg to write the recording sub-file to the memory. Specifically, it can be injected into ffmpeg by custom parameters to instruct ffmpeg to redirect the internal write file io interface to the memory io interface, so that the recording sub-file is written to the memory during recording. Among them, ffmpeg is an open source free software that can perform recording, conversion, and streaming functions in multiple audio and video formats. It mainly includes a decoder library for audio and video in multiple projects, as well as an audio and video format conversion library.
[0104] Optionally, in an implementable manner, in S1022, the pulled streaming media is recorded according to the recording parameters, which may be:
[0105] S11. For the pulled current demultiplexing context, read the audio frame and the video frame from the current demultiplexing context.
[0106] S12. According to the output media type, the audio frame and the video frame are transpacked, and the transpacked audio frame and video frame are interleaved and aligned.
[0107] S13. Obtain a recording sub-file according to the recording file format and the interleaved and aligned audio frames and video frames, and write the recording sub-file into the block of the memory.
[0108] In the above S11-S13 process, the supported recording file format may be MP4 format or FLV format, and a specific recording process when the recording format is MP4 format is shown below. First, the structure and composition of the MP4 file are described.
[0109] MP4 files are composed of multiple boxes, each of which contains specific information. Figure 3 This is a schematic diagram of the structure of a box in MP4, such as Figure 3 As shown, a box consists of a header and a body. The header specifies the size and type of the box. The body box is generally a container box, and boxes can be nested.
[0110] All data in an MP4 file is contained in a box. An MP4 file consists of several boxes. Each box has a type and length. A box can be understood as a data object block. A box can contain another box. An MP4 file will first have one and only one box of type ftyp, which is a mark of the MP4 format and contains some information about the file. After that, there will be one and only one box of type moov, which is a container box. The subbox contains metadata information of the media. The media data of an MP4 file is contained in a box of type "mdat" (media data box). This type of box is also a container box. There can be multiple or no boxes. The structure of the media data is described by metadata.
[0111] MP4 files are composed of several boxes, which are organized in a tree structure under the root node. Generally, MP4 files mainly contain three nodes: ftyp, mdat, and moov. Among them, ftyp is the file type, which describes the version of the specification to be followed, and mdat is the specific media data, including audio and video frames. Moov mainly includes media metadata information, as well as media data index information.
[0112] Optionally, the recording format is an MP4 file format, and the output recording file format is an MP4 file format. In an implementable manner, in S1022, the pulled streaming media is recorded according to the recording parameters, which may be:
[0113] S21. After successful initialization, MP4 file header information is generated by multiplexing context.
[0114] Specifically, when the initialization is successful, the MP4 file header information can be automatically generated through the multiplexing context.
[0115] S22: For the pulled current demultiplexing context, if it is determined according to the current demultiplexing context that the streaming is not finished, perform file writing and uploading operations according to the MP4 file format.
[0116] Optionally, in S22, the file writing and uploading operations are performed according to the MP4 file format, which may be specifically:
[0117] S221. Read audio frames and video frames from the current demultiplexing context.
[0118] S222. According to the output media type, transpackaging the audio frames and the video frames through the multiplexing context, and interweaving and aligning the transpackaging audio frames and the video frames.
[0119] S223, write the interleaved and aligned audio frames and video frames into the media data area of the MP4 file, and update the frame index and other data in the media data index information of the MP4 file at the same time, write the MP4 file into the blocks of the memory, and the MP4 file content in one block constitutes a recording sub-file.
[0120] S224. When the number of blocks written into the recording sub-file is greater than M, upload the recording sub-files in the M blocks to the storage unit, release the memory occupied by the M blocks, and update the block count of the blocks written into the recording sub-file, wherein the recording sub-file carries the identifier of the block where the recording sub-file is located.
[0121] In one embodiment, the first block contains MP4 file header information, and some fields in the MP4 file header information need to be updated at the end of the multiplexing process. The first block is complete in terms of data only at the end of recording. Therefore, optionally, the recording sub-file of the first block is not uploaded, and the recording sub-file of the first block is uploaded at the end of the push stream. Alternatively, the recording sub-file of the first block can be uploaded, and the recording sub-file of the first block can be uploaded again at the end of the push stream.
[0122] S23. If the streaming is determined to be finished according to the current demultiplexing context, the interleaving alignment process is ended, the media data in the multiplexing context cache is cleared, and the media data is written into the blocks in the memory, the media data stored in the MP4 file header information is updated, and the recording sub-file of the first block and the recording sub-file of the last block in the recording process are uploaded to the storage unit.
[0123] Optionally, in the present embodiment, when uploading the recording sub-file of each block, the recording sub-file carries the identification of the block, such as the block number, or the recording sub-file carries the file number and is numbered in sequential order, so that when the storage unit merges multiple uploaded recording sub-files, it can be merged according to the block number to obtain a live recording file.
[0124] Optionally, in an implementable manner, uploading the recorded sub-files in the M blocks to the storage unit in S102 may specifically be: calling an application program interface of the storage unit to upload the recorded sub-files in the M blocks to the storage unit.
[0125] The live content processing method provided by the embodiment of the present application obtains a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information, pulls the stream from the stream server according to the target stream address, and records the pulled streaming media according to the recording parameter configuration information. During the live recording process, the file writing and uploading operations are continuously performed, the memory is managed in blocks, and the recording sub-files are written into the blocks of the memory. When the number of blocks written with the recording sub-files is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released. At the end of the recording, an indication information is sent to the storage unit, indicating the storage unit to merge the uploaded multiple recording sub-files to obtain a live recording file. Therefore, as the live recording process continues, uploading can be achieved while recording, reducing the uploading time after the recording is completed, and achieving file output in seconds. Moreover, during the recording process, only the memory of the size of M blocks is occupied and no disk io resources are occupied at all, thereby increasing the number of concurrent recording tasks for a single machine and saving the cost of live recording.
[0126] The technical solution of the live content processing method provided in the embodiment of the present application is described in detail below in combination with a specific embodiment.
[0127] Figure 4 A schematic diagram of an application architecture of a live content processing method provided in an embodiment of the present application, such as Figure 4 As shown, the architecture includes a streaming terminal 110 , a streaming server 120 , a live broadcast control server 130 , a recording server 140 , a storage device 150 and a file access terminal 160 .
[0128] The streaming terminal 110 and the file access terminal 160 include but are not limited to smart phones, tablets, laptops, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Optionally, the streaming terminal 110 and the file access terminal 160 may be the same terminal or different terminals.
[0129] The streaming server 120, the live broadcast control server 130 and the recording server 140 may be servers that provide various services. They may be independent physical servers, or they may be server clusters or distributed systems composed of multiple physical servers. They may also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Specifically, they may be cloud virtual host servers, which is not limited in this embodiment.
[0130] Optionally, the live broadcast control server 130 and the recording server 140 may be one server.
[0131] In one embodiment of the present application, the live content processing method may be executed by the recording server 104 .
[0132] Exemplarily, in one embodiment, before streaming, that is, before the start of live broadcast, the streaming terminal 110 can set recording task configuration information through the live broadcast platform. The recording task configuration information includes multiple recording tasks and recording parameter configuration information of each recording task. Each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded. At the start of the live broadcast, the streaming terminal 110 pushes the stream to the streaming server 120. When the streaming server 120 receives the multimedia information stream pushed by the streaming terminal 110, it sends a streaming notification message to the live broadcast control center server 130. The streaming notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media. After receiving the streaming notification message, the live broadcast control center server 130 determines that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information, and then sends a recording task instruction to the recording server 104. The recording server 104 executes the following Figure 5 The live content processing method shown is to record the live broadcast and upload the live broadcast recording file to the storage device 150. The storage device 150 can be a cloud storage. The file access terminal 160 or the streaming terminal 110 can access the recorded live broadcast recording file after the live broadcast ends.
[0133] Figure 5 A flowchart of a live content processing method provided in an embodiment of the present application, the method can be, for example, Figure 1 or Figure 4 The recording server 104 shown in the figure executes, but is not limited to this. In this embodiment, the recording format is MP4 format as an example for explanation. Figure 5 As shown, the method may include the following steps:
[0134] S201. Obtain a recording task instruction, where the recording task instruction includes a target stream address and recording parameter configuration information, where the recording parameter configuration information includes a recording format.
[0135] For details about how to obtain recording task instructions, see Figure 2 The description in the illustrated embodiment will not be repeated here.
[0136] In this embodiment, the MP4 recording format is taken as an example for description.
[0137] S202. Initialize the demultiplexing context and multiplexing context used in the streaming process, and set recording parameters according to the recording format. The recording parameters include the streaming source address, recording file format, output media type, and custom parameters. The custom parameters are used to instruct the recording unit to write the recording sub-file into the memory.
[0138] In this embodiment, the recording file format is MP4 format.
[0139] Specifically, one recording task corresponds to one demultiplexing context, and the pulled streaming media corresponds to one demultiplexing context. One demultiplexing context receives audio and video frames one by one. In this embodiment, the demultiplexing context and muxing context used in the stream pulling process are first initialized, and the recording parameters are set according to the recording format, and then the pulled streaming media is recorded according to the recording parameters. The recording parameters include the stream pulling source address, the recording file format, the output media type and the custom parameters. The stream pulling source address is the target stream address, the recording file format is the recording format in the recording parameter configuration information, and the output media type can include audio and video, pure audio and pure video. The custom parameters are used to instruct the recording unit to write the recording sub-file to the memory. The recording unit can be, for example, ffmpeg. The custom parameters are used to instruct ffmpeg to write the recording sub-file to the memory. Specifically, it can be injected into ffmpeg by custom parameters to instruct ffmpeg to redirect the internal write file io interface to the memory io interface, so that the recording sub-file is written to the memory during recording. Among them, ffmpeg is an open source free software that can perform recording, conversion, and streaming functions in multiple audio and video formats. It mainly includes a decoder library for audio and video in multiple projects, as well as an audio and video format conversion library.
[0140] S203: After successful initialization, generate MP4 file header information by multiplexing context.
[0141] S204: Determine whether the streaming is finished for the pulled current demultiplexing context.
[0142] If it is determined that the streaming has not ended, S205 is executed; if it is determined that the streaming has ended, S210 is executed.
[0143] S205: Read audio frames and video frames from the current demultiplexing context.
[0144] S206. According to the output media type, the audio frames and video frames are transpacked through the multiplexing context, and the transpacked audio frames and video frames are interleaved and aligned, and the interleaved and aligned audio frames and video frames are written into the media data area (mdat) of the MP4 file, and the MP4 file is written into the blocks of the memory. The MP4 file content in a block constitutes a recording sub-file.
[0145] Specifically, ffmpeg can be used to write audio frames and video frames into the media data area (mdat) of the MP4 file according to the MP4 container standard, and update the frame index in moov (the index is mainly used for drag-and-play usage scenarios). The custom parameter instructs ffmpeg to write the recorded subfile into the memory, so the audio frames and video frames will be written directly into the memory.
[0146] S207: Determine whether the number of blocks written into the recording sub-file is greater than M.
[0147] Optionally, M may be 2. When the number of blocks written into the recording sub-file is greater than 2, it indicates that there is at least one complete block.
[0148] If yes, execute S208, if no, execute S209.
[0149] S208, uploading the recording sub-files in the M blocks to the storage unit, releasing the memory occupied by the M blocks, and updating the block count of the blocks written into the recording sub-files, wherein the recording sub-files carry the identifiers of the blocks where the recording sub-files are located.
[0150] Specifically, the first block contains the MP4 file header information. Some fields in the MP4 file header information need to be updated at the end of the multiplexing process. The first block is complete in terms of data only at the end of recording. Therefore, optionally, the recording sub-file of the first block is not uploaded, and the recording sub-file of the first block is uploaded at the end of the push stream. Alternatively, the recording sub-file of the first block can be uploaded, and the recording sub-file of the first block can be re-uploaded at the end of the push stream. In this embodiment, the recording sub-file of the first block is not uploaded as an example.
[0151] S209: Update the frame index in the media data index information of the MP4 file.
[0152] Specifically, the frame index in moov is updated, and the index is used in the drag-and-play usage scenario.
[0153] S210: Determine that the streaming is finished, end the interleaving alignment process, clear the media data in the multiplexing context cache, and write the media data into the blocks in the memory.
[0154] S211. Update the media data stored in the MP4 file header information.
[0155] Specifically, the media data stored in the MP4 file header information may be updated by ffmpeg.
[0156] S212: Upload the recording sub-file of the first block and the recording sub-file of the last block in the recording process to a storage unit.
[0157] S213: Close the demultiplexing context and the multiplexing context, and end the recording.
[0158] In this embodiment, during the live broadcast recording process, file writing and uploading operations are continuously performed, the memory is managed in blocks, and the recording sub-files are written into the blocks of the memory. When the number of blocks that have been written with the recording sub-files is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released. At the end of the recording, an instruction message is sent to the storage unit to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live broadcast recording file. Thus, as the live broadcast recording process continues, uploading can be achieved while recording, reducing the time consumed by uploading after the recording is completed, and the file is output in seconds. Moreover, during the recording process, only the memory of the size of M blocks is occupied and no disk io resources are occupied at all, thereby increasing the number of concurrent recording tasks of a single machine and saving the cost of live broadcast recording. The problem of long upload delay caused by the excessive size of MP4 files is solved, and MP4 files can be uploaded in seconds.
[0159] Figure 6 A flowchart of a live content processing method provided in an embodiment of the present application. The method of this embodiment can be executed by a terminal device, such as Figure 6 As shown, the method of this embodiment may include:
[0160] S301. In response to a live content access operation, a live content selection page is displayed, where the live content selection page includes a live recording file released by a live platform.
[0161] Among them, the live recording file is that after the server obtains the recording task instruction, the recording task instruction includes the target stream address and recording parameter configuration information, pulls the stream from the stream server according to the target stream address, and records the pulled streaming media according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is completed. The file writing and uploading operations are: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks written with the recording sub-file is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, M is a preset positive integer, and the server sends an indication message to the storage unit at the end of the recording, and the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files.
[0162] In this embodiment, the server may be a recording server, which pulls the stream from the streaming server according to the target stream address, and then records the pulled streaming media according to the recording parameter configuration information.
[0163] S302: In response to an access triggering operation on a target live broadcast recording file, a request for obtaining the target live broadcast recording file is sent to a server.
[0164] S303: Receive the target live broadcast recording file sent by the server.
[0165] S304: Display a corresponding page according to the target live broadcast recording file.
[0166] The live content processing method provided in the present embodiment continuously performs file writing and uploading operations during the recording process of the live recording file, manages the memory in blocks, writes the recording sub-files into the blocks of the memory, and when the number of blocks into which the recording sub-files have been written is greater than M, uploads the recording sub-files in the M blocks to the storage unit, and releases the memory occupied by the M blocks, and sends an indication message to the storage unit at the end of the recording, instructing the storage unit to merge the multiple uploaded recording sub-files to obtain the live recording file. Thus, as the live recording process continues, uploading can be achieved while recording, reducing the time consumed by uploading after the recording is completed, and achieving file output in seconds. Moreover, during the recording process, only the memory of the size of M blocks is occupied and no disk io resources are occupied at all, thereby increasing the number of concurrent recording tasks of a single machine and saving the cost of live recording.
[0167] Figure 7 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the live content processing device may include: an acquisition module 11, a recording module 12 and a sending module 13.
[0168] The acquisition module 11 is used to acquire a recording task instruction, which includes a target stream address and recording parameter configuration information;
[0169] The recording module 12 is used to pull the stream from the stream server according to the target stream address, and record the pulled stream according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is completed. The file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer;
[0170] The sending module 13 is used to send indication information to the storage unit when the recording is finished, and the indication information is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a recording file.
[0171] In one embodiment, the recording task instruction is triggered based on a streaming notification message and recording task configuration information. The streaming notification message includes a domain name of the streaming media, a target stream address, and an identifier of the streaming media. The recording task configuration information includes multiple recording tasks and recording parameter configuration information of each recording task. Each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded.
[0172] In one embodiment, the acquisition module 11 is used to:
[0173] Receive the push notification message sent by the streaming server;
[0174] It is determined that at least one of a domain name of a streaming media corresponding to a target streaming address and an identifier of the streaming media exists in the recording task configuration information, and a recording task instruction is triggered.
[0175] In one embodiment, the acquisition module 11 is used to:
[0176] Receive a recording task instruction sent by the live broadcast control server. The recording task instruction is sent by the live broadcast control server after receiving the push notification message sent by the streaming server and determining that at least one of the domain name of the streaming media and the identifier of the streaming media corresponding to the target streaming address exists in the recording task configuration information.
[0177] In one embodiment, the recording parameter configuration information includes a recording format, and the recording module 12 is used to:
[0178] Initialize the demultiplexing context and multiplexing context used in the streaming process, and set the recording parameters according to the recording format. The recording parameters include the streaming source address, recording file format, output media type, and custom parameters. The custom parameters are used to instruct the recording unit to write the recording sub-file into the memory;
[0179] Record the pulled streaming media according to the recording parameters.
[0180] In one embodiment, the recording module 12 is used to:
[0181] For the pulled current demultiplexing context, read the audio frame and the video frame from the current demultiplexing context;
[0182] According to the output media type, transpackaging the audio frame and the video frame through a multiplexing context, and interleaving and aligning the transpackaging audio frame and video frame;
[0183] A recording sub-file is obtained according to the recording file format and the interleaved and aligned audio frames and video frames, and the recording sub-file is written into the block of the memory.
[0184] In one embodiment, the recording format is an MP4 file format, the output recording file format is an MP4 file format, and the recording module 12 is used to:
[0185] After successful initialization, MP4 file header information is generated through multiplexing context;
[0186] For the pulled current demultiplexing context, if it is determined according to the current demultiplexing context that the streaming has not ended, the file is written and uploaded according to the MP4 file format;
[0187] If the streaming is determined to be finished according to the current demultiplexing context, the interleaving alignment process is ended, the media data in the multiplexing context cache is cleared, and the media data is written to the blocks in the memory, the media data stored in the MP4 file header information is updated, and the recording sub-file of the first block and the recording sub-file of the last block in the recording process are uploaded to the storage unit.
[0188] In one embodiment, the recording module 12 is specifically used for:
[0189] Read audio and video frames from the current demultiplexing context;
[0190] According to the output media type, the audio frame and the video frame are transpacked by a multiplexing context, and the transpacked audio frame and the video frame are interleaved and aligned;
[0191] The interleaved and aligned audio frames and video frames are written into the media data area of the MP4 file, and the frame index in the media data index information of the MP4 file is updated at the same time, and the MP4 file is written into the blocks of the memory, and the MP4 file content in one block constitutes a recording sub-file;
[0192] When the number of blocks written into the recording sub-file is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released, and the block count of the blocks written into the recording sub-file is updated at the same time, wherein the recording sub-file carries the identifier of the block where the recording sub-file is located.
[0193] In one embodiment, the recording module 12 is used to:
[0194] The application program interface of the storage unit is called to upload the recorded sub-files in the M blocks to the storage unit.
[0195] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 7 The live content processing device shown can execute Figure 2 or Figure 5 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the live content processing device are respectively to achieve Figure 4 For the sake of brevity, the corresponding processes in the corresponding method embodiments are not repeated here.
[0196] Figure 8 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the live content processing device may include: a display module 21, a sending module 22 and a receiving module 23.
[0197] The display module 21 is used to display a live content selection page in response to a live content access operation, and the live content selection page includes a live recording file released by a live platform;
[0198] Among them, the live broadcast recording file is that after the server obtains the recording task instruction, the recording task instruction includes the target stream address and the recording parameter configuration information, the stream is pulled from the stream server according to the target stream address, and the pulled streaming media is recorded according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is completed. The file writing and uploading operations are: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks written with the recording sub-file is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, M is a preset positive integer, and the server sends an indication information to the storage unit at the end of the recording, and the indication information is used to instruct the storage unit to merge the multiple uploaded recording sub-files;
[0199] The sending module 22 is used for sending a target live broadcast recording file acquisition request to the server in response to an access trigger operation on the target live broadcast recording file;
[0200] The receiving module 23 is used to receive the target live broadcast recording file sent by the server;
[0201] The display module 21 is also used to display a corresponding page according to the target live broadcast recording file.
[0202] Fig. 9 A schematic diagram of the structure of a live content processing device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the live content processing device may include: an acquisition module 31, a recording module 32 and a sending module 33.
[0203] The acquisition module 31 is used to acquire a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information, and the recording parameter configuration information includes an MP4 file format;
[0204] The recording module 32 is used to pull the stream from the stream server according to the target stream address, initialize the demultiplexing context and the multiplexing context used in the stream pulling process, and set the recording parameters according to the MP4 file format, the recording parameters including the stream pulling source address, the recording in the MP4 file format and the output media type and the custom parameters, the custom parameters are used to instruct the recording unit to write the recording sub-file into the memory;
[0205] According to the recording parameters, the pulled streaming media is recorded. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into a block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer;
[0206] The sending module 33 is used to send indication information to the storage unit when the recording is finished, and the indication information is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live recording file.
[0207] In one embodiment, the recording module 32 is used to:
[0208] After successful initialization, MP4 file header information is generated through the multiplexing context;
[0209] For the pulled current demultiplexing context, if it is determined according to the current demultiplexing context that the streaming is not finished, the file writing and uploading operations are performed according to the MP4 file format;
[0210] If the streaming is determined to be finished according to the current demultiplexing context, the interleaving alignment process is ended, the media data index information is written into the blocks in the memory, the media data stored in the MP4 file header information is updated, and the recording sub-file of the first block and the recording sub-file of the last block in the recording process are uploaded to the storage unit.
[0211] In one embodiment, the recording module 32 is specifically used for:
[0212] Reading audio frames and video frames from the current demultiplexing context, and transpackaging the audio frames and the video frames;
[0213] Interleaving and aligning the transpacked audio frames and video frames through a multiplexing context according to the output media type;
[0214] Writing the interleaved and aligned audio frames and video frames into the media data area of the MP4 file, updating the frame index in the media data index information of the MP4 file, and writing the MP4 file into the blocks of the memory, wherein the MP4 files in one block constitute a recording sub-file;
[0215] When the number of blocks written into the recording sub-file is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released, and the block count of the blocks written into the recording sub-file is updated at the same time, wherein the recording sub-file carries the identifier of the block where the recording sub-file is located.
[0216] In one embodiment, the recording module 32 is specifically used for:
[0217] The application program interface of the storage unit is called to upload the recorded sub-files in the M blocks to the storage unit.
[0218] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.
[0219] The live content processing device of the embodiment of the present application is described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware encoding processor to execute, or a combination of hardware and software modules in the encoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.
[0220] Fig. 9 1 is a schematic block diagram of a live content processing device provided in an embodiment of the present application. The live content processing device may be a server or a terminal device in the above method embodiment.
[0221] like Fig. 9 As shown, the live content processing device may include:
[0222] The memory 210 and the processor 220, the memory 210 is used to store the computer program and transmit the program code to the processor 220. In other words, the processor 220 can call and run the computer program from the memory 210 to implement the method in the embodiment of the present application.
[0223] For example, the processor 220 may be configured to execute the above method embodiments according to instructions in the computer program.
[0224] In some embodiments of the present application, the processor 220 may include but is not limited to:
[0225] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.
[0226] In some embodiments of the present application, the memory 210 includes but is not limited to:
[0227] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).
[0228] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 210 and executed by the processor 220 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the live content processing device.
[0229] like Fig. 9 As shown, the live content processing device may also include:
[0230] The transceiver 230 may be connected to the processor 220 or the memory 210 .
[0231] The processor 220 may control the transceiver 230 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 230 may include a transmitter and a receiver. The transceiver 230 may further include an antenna, and the number of antennas may be one or more.
[0232] It should be understood that the various components in the live content processing device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.
[0233] The embodiment of the present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the embodiment of the present application also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.
[0234] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.
[0235] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.
[0236] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0237] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0238] The above contents are only specific implementation methods of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.
Claims
1. A method for processing live content, characterized in that: include: Obtaining a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information; Pulling a stream from a stream server according to the target stream address, recording the pulled stream media according to the recording parameter configuration information, and continuously performing file writing and uploading operations during the recording process until the recording ends, wherein the file writing and uploading operations are: writing a recording sub-file into a block of a memory, wherein the memory is pre-divided into a plurality of blocks, and when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, wherein M is a preset positive integer; When the recording is finished, an indication message is sent to the storage unit, where the indication message is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live broadcast recording file.
2. The method according to claim 1, characterized in that The recording task instruction is triggered based on a streaming notification message and recording task configuration information, wherein the streaming notification message includes the domain name of the streaming media, the target stream address and the identifier of the streaming media, and the recording task configuration information includes multiple recording tasks and recording parameter configuration information of each recording task, wherein each recording task includes the domain name of the streaming media to be recorded and the identifier of the streaming media to be recorded.
3. The method according to claim 2, characterized in that The step of obtaining the recording task instruction includes: Receiving the streaming notification message sent by the streaming server; It is determined that at least one of a domain name of the streaming media corresponding to the target streaming address and an identifier of the streaming media exists in the recording task configuration information, and the recording task instruction is triggered.
4. The method according to claim 2, characterized in that: The step of obtaining the recording task instruction includes: Receive the recording task instruction sent by the live broadcast control server, and the recording task instruction is sent by the live broadcast control server after receiving the push notification message sent by the streaming server and determining that at least one of the domain name of the streaming media corresponding to the target stream address and the identifier of the streaming media exists in the recording task configuration information.
5. The method according to claim 1, characterized in that The recording parameter configuration information includes a recording format, and recording the pulled streaming media according to the recording parameter configuration information includes: Initialize the demultiplexing context and multiplexing context used in the streaming process, and set recording parameters according to the recording format, wherein the recording parameters include the streaming source address, the recording file format and the output media type, and the custom parameters are used to instruct the recording unit to write the recording sub-file into the memory; The pulled streaming media is recorded according to the recording parameters.
6. The method according to claim 5, characterized in that The step of recording the pulled streaming media according to the recording parameters includes: For the pulled current demultiplexing context, read the audio frame and the video frame from the current demultiplexing context; According to the output media type, transpackaging the audio frame and the video frame through a multiplexing context, and interleaving and aligning the transpackaging audio frame and video frame; A recording sub-file is obtained according to the recording file format and the interleaved and aligned audio frames and video frames, and the recording sub-file is written into the block of the memory.
7. The method according to claim 5, characterized in that The recording format is an MP4 file format, the output recording file format is an MP4 file format, and the pulled streaming media is recorded according to the recording parameters, including: After successful initialization, MP4 file header information is generated through the multiplexing context; For the pulled current demultiplexing context, if it is determined according to the current demultiplexing context that the streaming is not finished, the file writing and uploading operations are performed according to the MP4 file format; If the streaming is determined to be finished according to the current demultiplexing context, the interleaving alignment process is ended, the media data in the multiplexing context cache is cleared, and the media data is written into the blocks in the memory, the media data stored in the MP4 file header information is updated, and the recording sub-file of the first block and the recording sub-file of the last block in the recording process are uploaded to the storage unit.
8. The method according to claim 7, characterized in that The file writing and uploading operations are performed according to the MP4 file format, including: Reading audio frames and video frames from the current demultiplexing context; According to the output media type, transpackaging the audio frame and the video frame through a multiplexing context, and interleaving and aligning the transpackaging audio frame and video frame; Writing the interleaved and aligned audio frames and video frames into the media data area of the MP4 file, updating the frame index in the media data index information of the MP4 file, and writing the MP4 file into the blocks of the memory, wherein the MP4 file contents in one block constitute a recording sub-file; When the number of blocks written into the recording sub-file is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released, and the block count of the blocks written into the recording sub-file is updated at the same time, wherein the recording sub-file carries the identifier of the block where the recording sub-file is located.
9. The method according to claim 1, characterized in that: The uploading of the recorded sub-files in the M blocks to the storage unit includes: The application program interface of the storage unit is called to upload the recorded sub-files in the M blocks to the storage unit.
10. A method for processing live content, characterized in that: include: In response to the live content access operation, displaying a live content selection page, the live content selection page including a live recording file released by a live platform; Wherein, the live broadcast recording file is a file that the server pulls the stream from the stream server according to the target stream address after obtaining the recording task instruction, and the recording task instruction includes the target stream address and the recording parameter configuration information, and the pulled streaming media is recorded according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer, and when the recording is finished, the server sends an indication message to the storage unit, and the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files; In response to an access trigger operation on a target live broadcast recording file, sending a request for obtaining the target live broadcast recording file to a server; Receiving the target live broadcast recording file sent by the server; According to the target live broadcast recording file, a corresponding page display is performed.
11. A method for processing live content, characterized in that: include: Obtaining a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information, wherein the recording parameter configuration information includes an MP4 file format; Pull the stream from the stream server according to the target stream address, initialize the demultiplexing context and the multiplexing context used in the stream pulling process, and set the recording parameters according to the MP4 file format, wherein the recording parameters include the stream pulling source address, the recording in the MP4 file format and the output media type, and the custom parameters, wherein the custom parameters are used to instruct the recording unit to write the recording sub-file into the memory; According to the recording parameters, the pulled streaming media is recorded. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into a block of the memory, the memory is pre-divided into a plurality of blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer; When the recording is finished, an indication message is sent to the storage unit, where the indication message is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a live broadcast recording file.
12. The method according to claim 11, characterized in that The step of recording the pulled streaming media according to the recording parameters includes: After successful initialization, MP4 file header information is generated through the multiplexing context; For the pulled current demultiplexing context, if it is determined according to the current demultiplexing context that the streaming is not finished, the file writing and uploading operations are performed according to the MP4 file format; If the streaming is determined to be finished according to the current demultiplexing context, the interleaving alignment process is ended, the media data index information is written into the blocks in the memory, the media data stored in the MP4 file header information is updated, and the recording sub-file of the first block and the recording sub-file of the last block in the recording process are uploaded to the storage unit.
13. The method according to claim 12, characterized in that The file writing and uploading operations are performed according to the MP4 file format, including: Reading audio frames and video frames from the current demultiplexing context, and transpackaging the audio frames and the video frames; Interleaving and aligning the transpacked audio frames and video frames through a multiplexing context according to the output media type; Writing the interleaved and aligned audio frames and video frames into the media data area of the MP4 file, updating the frame index in the media data index information of the MP4 file, and writing the MP4 file into the blocks of the memory, wherein the MP4 files in one block constitute a recording sub-file; When the number of blocks written into the recording sub-file is greater than M, the recording sub-files in the M blocks are uploaded to the storage unit, and the memory occupied by the M blocks is released, and the block count of the blocks written into the recording sub-file is updated at the same time, wherein the recording sub-file carries the identifier of the block where the recording sub-file is located.
14. The method according to claim 11, characterized in that The uploading of the recorded sub-files in the M blocks to the storage unit includes: The application program interface of the storage unit is called to upload the recorded sub-files in the M blocks to the storage unit.
15. A live content processing device, characterized in that: include: An acquisition module, used for acquiring a recording task instruction, wherein the recording task instruction includes a target stream address and recording parameter configuration information; A recording module, used for pulling a stream from a streaming server according to the target stream address, recording the pulled streaming media according to the recording parameter configuration information, and continuously performing file writing and uploading operations during the recording process until the recording is finished, wherein the file writing and uploading operations are as follows: writing a recording sub-file into a block of a memory, wherein the memory is pre-divided into a plurality of blocks, and when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to a storage unit, and releasing the memory occupied by the M blocks, wherein M is a preset positive integer; The sending module is used to send indication information to the storage unit when the recording is finished, and the indication information is used to instruct the storage unit to merge the uploaded multiple recording sub-files to obtain a recording file.
16. A live content processing device, characterized in that: include: A display module, configured to display a live content selection page in response to a live content access operation, wherein the live content selection page includes a live recording file released by a live platform; Wherein, the live broadcast recording file is a file that the server pulls the stream from the stream server according to the target stream address after obtaining the recording task instruction, and the recording task instruction includes the target stream address and the recording parameter configuration information, and the pulled streaming media is recorded according to the recording parameter configuration information. During the recording process, the file writing and uploading operations are continuously performed until the recording is finished, and the file writing and uploading operations are as follows: writing the recording sub-file into the block of the memory, the memory is pre-divided into multiple blocks, when the number of blocks into which the recording sub-file has been written is greater than M, uploading the recording sub-files in the M blocks to the storage unit, and releasing the memory occupied by the M blocks, where M is a preset positive integer, and when the recording is finished, the server sends an indication message to the storage unit, and the indication message is used to instruct the storage unit to merge the multiple uploaded recording sub-files; A sending module, configured to send a request for obtaining the target live broadcast recording file to a server in response to an access trigger operation on the target live broadcast recording file; A receiving module, used for receiving the target live broadcast recording file sent by the server; The display module is also used to: display a corresponding page according to the target live broadcast recording file.
17. A live content processing device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 9 or 10 or 11 to 14.
18. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 9 or 10 or 11 to 14.
19. A computer program product comprising instructions, characterized in that When the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9 or 10 or 11 to 14.
