Streaming media data processing method, device and equipment, medium and product

Abstract

The embodiment of the invention provides a streaming media data processing method, device and equipment, a medium and a product, and is applied to a streaming media client, and the method comprises the steps: obtaining source data collected by a target collection assembly at a streaming media client side; converting the source data to generate streaming media data; sending the streaming media data to a streaming media service program at a streaming media client side; streaming the streaming media data from the streaming media service program, and playing the streaming media data obtained by streaming at the streaming media client locally; and pushing the streaming media data in the streaming media service program to a server. According to the embodiment of the invention, stream multiplexing can be realized on a stream media client side (namely a stream pushing terminal), the purpose that stream pulling of the stream media data does not need to occupy the downlink bandwidth of the stream media client side is achieved, the flow consumption of stream media data transmission is reduced, and the transmission efficiency of the stream media data is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of streaming media technology, and in particular to a streaming media data processing method, a streaming media data processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Streaming media data refers to a media format that is streamed and can be played over the internet. Existing synchronous streaming media data playback solutions, due to the exclusivity of acquisition components (such as cameras and microphones) (i.e., they do not support stream reuse), require that streaming terminals (such as those used by anchors) push streaming data to a server while also requiring that streaming terminals (and pull terminals, such as those used by viewers) remotely pull the streaming data from the server for local playback. This results in inefficient streaming data transmission. Summary of the Invention

[0003] The embodiments of the present application provide a streaming media data processing method, device and equipment, medium, and product, which can realize stream multiplexing on the streaming media client side (i.e., the streaming terminal), thereby achieving the purpose of pulling streaming media data without occupying the downlink bandwidth on the streaming media client side, reducing the traffic consumption of streaming media data transmission and improving the transmission efficiency of streaming media data.

[0004] In one aspect, an embodiment of the present application provides a method for processing streaming media data, which is applied to a streaming media client, and includes:

[0005] Obtain source data collected by the target collection component on the streaming media client side;

[0006] Convert and process source data to generate streaming media data;

[0007] Sending streaming media data to the streaming media service program on the streaming media client side;

[0008] Pulling streaming media data from the streaming media service program and playing the pulled streaming media data locally on the streaming media client; and

[0009] Push the streaming data in the streaming service program to the server.

[0010] In one aspect, an embodiment of the present application provides a streaming media data processing device, which is applied to a streaming media client, and includes:

[0011] An acquisition unit, configured to acquire source data collected by a target acquisition component on a streaming media client side;

[0012] A processing unit, configured to convert the source data and generate streaming media data;

[0013] The processing unit is further configured to send the streaming media data to a streaming media service program on the streaming media client side;

[0014] The processing unit is further used to pull streaming media data from the streaming media service program and play the pulled streaming media data locally on the streaming media client; and push the streaming media data in the streaming media service program to the server.

[0015] In one implementation, a streaming media client includes a detection service program and a first streaming service program; the detection service program is used to detect the service status of the streaming media service program according to a polling cycle; and the acquisition unit is used to acquire source data collected by a target acquisition component on the streaming media client side, specifically to perform the following steps:

[0016] When the polling cycle time point is reached, the detection service program is called to detect the service status of the streaming media service program;

[0017] If the service status of the streaming media service program is in the startup state, the first streaming service program is called to obtain the source data collected by the target collection component.

[0018] In one implementation, the number of acquisition components on the streaming media client side is M, where M is a positive integer; and the processing unit is further configured to perform the following steps:

[0019] When a streaming media acquisition request is generated in the streaming media client, determining the type of streaming media requested to be acquired by the streaming media acquisition request;

[0020] Based on the determined streaming media type, determining a target acquisition component from the M acquisition components;

[0021] Among them, the streaming media type includes audio type, video type or audio and video type; when the determined streaming media type is audio type, the target acquisition component is the audio acquisition component; when the determined streaming media type is video type, the target acquisition component is the video acquisition component; when the determined streaming media type is audio and video type, the target acquisition component is the audio and video acquisition component.

[0022] In one implementation, the target acquisition component includes a first acquisition component and a second acquisition component, the streaming media data includes first streaming media data corresponding to the first acquisition component and second streaming media data corresponding to the second acquisition component; the streaming media client includes an interface player, and the interface player includes at least a first playback area and a second playback area; and the processing unit is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0023] Obtaining a first size of the first playback area, adapting a first video frame in the first streaming media data obtained by pulling the stream according to the first size, and playing the adapted first video frame in the first playback area;

[0024] A second size of the second playback area is obtained, a second video frame in the second streaming media data obtained by pulling the stream is adapted according to the second size, and the adapted second video frame is played in the second playback area.

[0025] In one implementation, the processing unit is configured to perform the following steps when playing the first video frame after the adaptation process in the first playback area:

[0026] Obtaining the number of first decoded video frames that have not yet been played, where the first decoded video frames are obtained by decoding first streaming media data obtained by pulling the stream;

[0027] If the number of the first decoded video frames is greater than or equal to the first preset number, playing the adapted first video frames in the first playback area at the first playback speed;

[0028] If the number of the first decoded video frames is less than or equal to the second preset number, playing the adapted first video frames in the first playback area at the second playback speed;

[0029] The first playback speed is greater than a preset reference playback speed, and the second playback speed is less than the preset reference playback speed.

[0030] In one implementation, the interface player further includes a third playback area; and the processing unit is further configured to perform the following steps:

[0031] If a third acquisition component is detected to be newly added in the target acquisition component, the third streaming media data obtained by pulling the stream is played in the third playback area, where the third streaming media data is obtained by converting the source data collected by the third acquisition component;

[0032] If it is detected that any acquisition component in the target acquisition components is removed, the playback screen in the corresponding playback area of ​​the interface player is closed.

[0033] In one implementation, the processing unit is further configured to perform the following steps:

[0034] Outputting prompt information about the first acquisition component in the first playback area, and outputting prompt information about the second acquisition component in the second playback area;

[0035] Among them, the positional relationship between the first playback area and the second playback area includes at least one of the following: the first playback area and the second playback area are independent of each other and have no overlapping area, there is an overlapping area between the first playback area and the second playback area, and the first playback area and the second playback area are arranged and displayed in the interface player.

[0036] In one implementation, a streaming client includes an interface player; the interface player includes a floating window; and a processing unit is configured to perform the following steps when playing streaming data pulled locally on the streaming client:

[0037] The third size of the floating window is obtained, the video frames in the streaming media data are adapted according to the third size, and the adapted video frames are played in the floating window.

[0038] In one implementation, a streaming media client includes a first streaming service program and an interface player, and streaming media data is sent to the streaming media service program via the first streaming service program. A processing unit is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0039] Get the data reading configuration parameters in the interface player;

[0040] In the interface player, the configuration parameters are read according to the data and the streaming media data obtained by pulling the stream is played;

[0041] Among them, the data reading configuration parameters include parameters related to the target duration. The target duration is the duration from when the interface player obtains the streaming media data obtained by pulling the stream to when the first data frame in the streaming media data obtained by pulling the stream is played; the data reading configuration parameters include at least one of a byte parameter and a duration parameter; the byte parameter is used to indicate the maximum number of bytes read in a single time, and the duration parameter is used to indicate the maximum duration of a single read.

[0042] In one implementation, the streaming media client includes an interface player, the streaming media data includes audio stream data, and the audio stream data is played in the interface player; the interface player includes an audio smoothing processing program; and the processing unit is further configured to perform the following steps:

[0043] If the playback speed of the audio stream data by the interface player is greater than a preset reference playback speed, the audio smoothing processing program is called to perform smoothing processing on the audio stream data in the interface player.

[0044] In one implementation, the streaming media data is stored in a server; the streaming media data stored in the server supports cloud playback; and the processing unit is further configured to perform the following steps:

[0045] When the detection service program detects that the target acquisition component has been removed, the streaming data in the streaming service program is stopped from being pushed to the server, and a streaming data push stop instruction is sent to the server. The push stop instruction is used to instruct the server to stop the cloud playback of the streaming data.

[0046] In one implementation, the streaming media client includes a first streaming service program; the processing unit is configured to convert the source data and generate streaming media data by executing the following steps:

[0047] Obtaining encoding optimization parameters set in the first streaming service program;

[0048] Invoking a first streaming service program, encoding the source data according to the encoding optimization parameters, and encapsulating the encoded source data to obtain streaming media data in a first encoding format;

[0049] The coding optimization parameters include at least one of the following: a coding speed optimization parameter and a coding delay optimization parameter. The coding speed optimization parameter is used to enable a function of increasing the coding speed, and the coding delay optimization parameter is used to enable a function of reducing the coding delay.

[0050] In one implementation, the streaming media client further includes a second streaming service program; and the processing unit is configured to execute the following steps:

[0051] Invoking a second streaming service program to convert streaming media data in a first encoding format in the streaming media service program into streaming media data in a second encoding format supported by the server;

[0052] The second streaming service program is called to send the streaming media data in the second encoding format to the server.

[0053] In one implementation, the streaming client is a program that is secondary developed based on a software development kit; the streaming service program is encapsulated in the software development kit;

[0054] The software development kit further includes at least one of the following programs: an interface player, a detection service program, a first streaming service program, and a second streaming service program.

[0055] Accordingly, an embodiment of the present application provides a computer device, comprising:

[0056] a processor suitable for implementing a computer program;

[0057] A computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned streaming media data processing method.

[0058] Accordingly, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned streaming media data processing method.

[0059] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-described streaming media data processing method.

[0060] The embodiment of the present application is applied to a streaming media client. When it is specifically implemented, the source data collected by the target acquisition component on the streaming media client side can be obtained; the source data can be transformed and processed to generate streaming media data; the streaming media data can be sent to the streaming media service program on the streaming media client side; the streaming media data can be pulled from the streaming media service program, and the pulled streaming media data can be played locally on the streaming media client; and the streaming media data in the streaming media service program can be pushed to the server. It can be seen that the embodiment of the present application constructs a set of local streaming media service programs to receive streaming media data, so that the streaming media data can be pulled and played directly from the local, and the streaming media data can be pushed to the remote server. While realizing stream multiplexing on the streaming media client side, the pulling of streaming media data does not need to occupy the downlink bandwidth on the streaming media client side, reducing the traffic consumption of streaming media data transmission and improving the transmission efficiency of streaming media data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 This is a system architecture diagram of a streaming media data processing system provided by an embodiment of the present application;

[0063] Figure 2 This is a flow chart of a method for processing streaming media data provided by an embodiment of the present application;

[0064] Figure 3 This is a flow chart of another streaming media data processing method provided in an embodiment of the present application;

[0065] Figure 4aThis is a schematic diagram of the positions between playback areas provided in an embodiment of the present application;

[0066] Figure 4b This is another schematic diagram of the positions between the playback areas provided in an embodiment of the present application;

[0067] Figure 4c This is another schematic diagram of the positions between the playback areas provided in an embodiment of the present application;

[0068] Figure 5 This is a schematic diagram of an interface provided by an embodiment of the present application;

[0069] Figure 6 Schematic diagram of a streaming media data processing method provided in an embodiment of the present application;

[0070] Figure 7 This is another interface diagram provided in an embodiment of the present application;

[0071] Figure 8 This is another interface diagram provided in an embodiment of the present application;

[0072] Figure 9 This is a schematic diagram of adjusting a GPL version of FFmpeg to an LGPL version of FFmpeg provided in an embodiment of the present application;

[0073] Figure 10 This is another interface diagram provided in an embodiment of the present application;

[0074] Figure 11 This is a structural diagram of a streaming media data processing device provided in an embodiment of the present application;

[0075] Figure 12 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] Streaming media technology (or streaming transmission technology) refers to a technology that compresses a series of media data (such as audio data, video data, audio and video data, etc.) and transmits them in segments over the Internet in a pipeline-like manner, achieving real-time transmission of audio and video for viewing on the Internet. Streaming media data refers to continuous time-based media obtained by encapsulating a series of media data on the Internet based on streaming media technology, including but not limited to audio stream data (a data stream used for real-time transmission of audio data), video stream data (a data stream used for real-time transmission of video data), or audio and video stream data (a data stream used for real-time transmission of audio and video data). In other words, streaming media data can actually be segmented into multiple streaming segments (or streaming data packets, consisting of one or more data frames), each of which has corresponding address information. For example, the address information can be a URL (Uniform Resource Locator) address. In this way, a complete set of streaming media data can be formed based on the address information corresponding to multiple streaming segments.

[0078] A streaming terminal is a terminal device that encapsulates a series of locally collected media data and sends the encapsulated streaming media data to a server. A streaming media client can be deployed in a streaming terminal. A streaming media client can be a client that has the ability to play streaming media data. A client refers to a computer program that completes one or more specific tasks. The so-called streaming can refer to the process of sending streaming media data in the form of transmitting streaming media segments. Correspondingly, a streaming terminal is a terminal device that pulls the streaming media data uploaded by the streaming terminal from the server and plays it. The so-called streaming can refer to the process of receiving streaming media data in the form of transmitting streaming media segments.

[0079] The embodiment of the present application builds a set of local streaming media service programs on the streaming media client side; wherein, the streaming media service program can be a service program that implements push-pull streaming (including push streaming and pull streaming) functions based on the real-time data transmission protocol, that is, the streaming media service program can refer to a program that can implement live broadcast service, and live broadcast service refers to an activity that can provide real-time audio-visual content services (such as live broadcast, video conferencing, voice calls, etc.); illustratively, the embodiment of the present application can modify the service code of livego (a streaming media service program based on the RTMP protocol) so that the modified livego (i.e., the streaming media service program used in the embodiment of the present application) supports direct push mode (i.e., pulling streaming media data from the local); RTMP (Real-Time Messaging Protocol, a real-time messaging protocol) is a relatively common live push-pull streaming protocol at present. The main features of the RTMP protocol are: support for multiplexing, support for packet transmission, and belonging to the application layer protocol. In this way, the source data (such as audio data, video data, or audio and video data, etc.) collected by the target acquisition component on the streaming media client side can be converted into streaming media data, and the streaming media data can be sent to the streaming media service program on the streaming media client side. Furthermore, the streaming media client can, on the one hand, pull streaming media data from the streaming media service program and play the streaming media data pulled locally on the streaming media client; on the other hand, it can push the streaming media data in the streaming media service program to the server. Thus, the embodiment of the present application constructs a set of local streaming media service programs to receive streaming media data, so that the streaming media client side can support the direct push mode (i.e., directly pulling and playing streaming media data locally), and only needs to occupy the uplink bandwidth of the streaming media client side to push the streaming media data to the remote server, solving the problem of the existing synchronous playback solution occupying both uplink and downlink bandwidth, reducing the traffic consumption of streaming media data transmission and improving the transmission efficiency of streaming media data.

[0080] The embodiments of the present application can be applied to synchronous playback scenarios such as live broadcast scenarios, audio and video call scenarios, audio and video conferencing scenarios, and real-time violation detection scenarios. Among them, live broadcast scenarios include but are not limited to: game live broadcast scenarios, event live broadcast scenarios, shopping live broadcast scenarios, and entertainment live broadcast scenarios. For example, in a live broadcast scenario, the streaming media client can be the live broadcast client used by the anchor, so that the live broadcast client used by the anchor can play the live streaming media data locally on the one hand, and push the live streaming media data to the live broadcast server on the other hand. At this time, the live broadcast client used by the audience can pull the stream from the live broadcast server and play the live streaming media data. In an audio and video call scenario, the streaming media client can be the instant messaging client used by the first object participating in the audio and video call (any one of the multiple objects participating in the audio and video call). At this time, the instant messaging client used by the first object will play the call streaming media data collected by itself locally on the one hand, and push the call streaming media data to the instant messaging server on the other hand, so that the instant messaging client used by the second object participating in the audio and video call (other objects other than the first object among the multiple objects participating in the audio and video call) can pull the stream from the instant messaging server and play the call streaming media data. The audio and video conferencing scenario is similar to the audio and video call scenario. In the real-time violation detection scenario, the streaming media client can be a violation detection client (such as a client that plays video stream data collected at the game site to detect whether the players participating in the game are cheating). On the one hand, the violation detection client can play the detection streaming media data locally, and on the other hand, it can push the detection streaming media data to a remote cloud server so that the detection streaming media data can be viewed in the cloud, thereby realizing remote violation detection.

[0081] The following describes the system architecture applicable to the streaming media data processing method provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 , Figure 1 This is a structural diagram of a streaming media data processing system provided by an embodiment of the present application. The streaming media data processing system may include: at least one terminal device (such as Figure 1 The terminal device 101 and the terminal device 102 in the embodiment of the present invention are provided as follows: a server 103 and a content distribution network 104. The terminal device 101 and the terminal device 102 can communicate with each other through the server 103, and the terminal device 101 and the terminal device 102 can communicate with the content distribution network 104 respectively.

[0082] Both terminal device 101 and terminal device 102 may run a streaming client, which includes but is not limited to a browser, an instant messaging client, a live broadcast client, a conference client, and the like. Illustratively, terminal device 101 may run streaming client 1, and terminal device 102 may run streaming client 2, and streaming client 1 and streaming client 2 may exchange information through server 103. Terminal device 101 and terminal device 102 may include but are not limited to smartphones (such as smartphones running the Android system or smartphones running the Internetworking Operating System (IOS)), tablet computers, portable personal computers, mobile internet devices (MIDs), vehicle-mounted devices, head-mounted devices, smart homes, and smart robots.

[0083] Server 103 may correspond to a streaming client and is a server that provides technical support for the services provided by the streaming client. It is understood that when terminal device 101 pushes streaming media data to server 103, terminal device 101 is a streaming terminal. At this time, if terminal device 102 pulls the streaming media data uploaded by terminal device 101 from server 103, terminal device 102 is a streaming terminal. Server 103 may have the following functions: ① Receive streaming media data sent by the streaming terminal; ② Generate file information for the streaming data, which may include address information for each streaming segment, which can be used to indicate the location of the corresponding streaming segment within the streaming data; ③ Send the streaming data to the streaming terminal based on the streaming data file information. Server 103 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. It should be understood that there may be one or more servers 103.

[0084] Wherein, the push stream terminal may include an acquisition component (a component for collecting audio data, video data or audio and video data, such as a camera, a microphone, etc.), a streaming media service program, and a streaming media client (which may be integrated with an interface player). The acquisition component is responsible for collecting source data (i.e., a series of media data), and the streaming media client is responsible for converting the source data collected by the acquisition component into streaming media data, and pushing the streaming media data to the streaming media service program. The streaming media data in the streaming media service program can be distributed to the interface player in the streaming media client and the server at the far end. During specific implementation, the streaming media client can notify the interface player to attempt to pull the streaming media data in the streaming media service program, and play the streaming media data obtained by pulling the stream, and simultaneously distribute the streaming media data in the streaming media service program to the server. In one embodiment, the streaming media service program can be encapsulated in the streaming media client. At this moment, the streaming media client is a client that can realize the multiplexing of streams on the streaming media client side (i.e., the push stream terminal). That is to say, the streaming media client can push the streaming media data obtained by encapsulation to the server side on the one hand, and can directly pull the stream locally and play the streaming media data on the other hand.

[0085] In one embodiment, the stream pulling terminal may send a download request for streaming media data to the server 103, and the server 103 may send the streaming media data to the stream pulling terminal in response to the download request. The content distribution network 104 may include one or more node servers, and each node server may store streaming media data. In another embodiment, the stream pulling terminal may send a download request to the server 103, and the server 103 forwards the download request to the target node server in the content distribution network 104. The target node server in the content distribution network 104 may obtain the corresponding streaming media data for the stream pulling terminal according to the download request and return the streaming media data to the stream pulling terminal, wherein the target node server may be the node server in the content distribution network 104 that is shortest distance away from the stream pulling terminal.

[0086] The streaming media data processing method proposed in this application involves a large number of data storage services, so a large amount of computer operating costs are required. Then, the server 103 in the embodiment of this application can be a cloud server that provides basic cloud computing services (such as cloud storage). Among them, the so-called cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool that is used on demand and is flexible and convenient. Among them, cloud technology can include cloud storage technology. The so-called cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide data storage and business access functions to the outside world.

[0087] The streaming media data processing methods provided in the embodiments of the present application may also involve the field of artificial intelligence (AI). For example, AI technology can be used to detect violations in video stream data within streaming media data; or AI technology can be used to add subtitle information to video stream data included in streaming media data. Artificial intelligence technology refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject covering a wide range of fields, encompassing both hardware- and software-level technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technologies, operating / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology (CV), speech processing technology, natural language processing technology, and machine learning (ML) / deep learning (DL).

[0088] The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information must be subject to the knowledge or consent of the individual subject (or the presence of a legal basis for the acquisition of information), and subsequent data use and processing must be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as when obtaining source data collected by the acquisition component, the user's permission or consent must be obtained, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant region.

[0089] See Figure 2 , Figure 2 1 is a flow chart of a method for processing streaming media data provided by an embodiment of the present application. The method can be executed by a streaming media client, specifically a streaming media client running on a streaming terminal. The streaming media data processing method includes the following steps S201-S204:

[0090] S201: Acquire source data collected by a target collection component on a streaming media client side.

[0091] The target acquisition component can be a video acquisition component, an audio acquisition component or an audio and video acquisition component; the video acquisition component is used to acquire video data (including video frames), such as a camera or a screen capture program; the audio acquisition component is used to acquire audio data (including audio frames), such as a microphone; the audio and video acquisition component is used to acquire audio and video data (including audio data and video data), and can be composed of an audio acquisition component and a video acquisition component. Among them, the target acquisition component can be integrated into the streaming terminal, or it can be connected to the streaming terminal by wired or wireless means. In an embodiment of the present application, the streaming media client needs to obtain the source data acquired by the target acquisition component, and the source data can include one or both of the audio data and video data acquired in real time.

[0092] In one embodiment, the streaming client can respond to a streaming acquisition request (for requesting the acquisition of streaming data) generated in the streaming client to obtain source data collected by the target acquisition component on the streaming client side. For example, in a live broadcast scenario, the streaming client is a live broadcast client, and the live broadcast client can be integrated with a live broadcast function. When the anchor can initiate a live broadcast request through the live broadcast function in the live broadcast client, the live broadcast request is a streaming acquisition request. For another example, in an audio and video call scenario, the streaming client is an instant messaging client. When the instant messaging client initiates a voice call request, the voice call request is a streaming acquisition request. For another example, in a real-time violation detection scenario, the streaming client can be a violation detection client. When the violation detection client is started, a streaming acquisition request can be generated to obtain source data collected by the target acquisition component on the streaming client side.

[0093] The number of acquisition components on the streaming media client side is M, where M is a positive integer. In one embodiment, when a streaming media acquisition request is generated in the streaming media client, the type of streaming media requested to be collected by the streaming media acquisition request can be determined; based on the determined streaming media type, the target acquisition component is determined from the M acquisition components; and then the source data collected by the target acquisition component on the streaming media client side is obtained. Among them, the streaming media type includes audio type, video type or audio and video type; when the determined streaming media type is audio type, the target acquisition component is an audio acquisition component; when the determined streaming media type is video type, the target acquisition component is a video acquisition component; when the determined streaming media type is audio and video type, the target acquisition component is an audio and video acquisition component. For example, the live broadcast client can provide a live broadcast function that turns off the video screen. When the anchor turns on the live broadcast function, the streaming media acquisition request initiated by the live broadcast client can be used to request the collection of audio data, that is, only the microphone can be determined as the target acquisition component. For example, when a video call request (a streaming media acquisition request for requesting the acquisition of audio and video data) is initiated through the video call control in the instant messaging client, the microphone and camera can be identified as target acquisition components to obtain the audio data collected by the microphone and the video data collected by the camera.

[0094] S202: Convert the source data to generate streaming media data.

[0095] In an embodiment of the present application, the source data collected by the target acquisition component can be encoded to obtain encoded source data, and then the encoded source data can be encapsulated into streaming media data that complies with a real-time data transmission protocol (the streaming media service program must support the real-time data transmission protocol). For example, the encapsulation format can include TS (Transport Stream), FLV (Flash Video, streaming media), etc. The purpose of the encoding process is to collect and compress the source data collected by the target acquisition component to reduce the amount of data transmitted; if the source data includes video data and audio data, the video data (the video format used is usually YUV (color components in SECAM and PAL color spaces, color) format or RGB (Red Green Blue, three primary colors) format) can be encoded into flv1 format (a compressed encoding format), and the audio data (the audio format used is usually PCM (Pulse Code Modulation) format) can be encoded into ACC (Advanced Audio Coding) format.

[0096] S203: Send the streaming media data to the streaming media service program on the streaming media client side.

[0097] Since the source data is acquired in real time, in one embodiment, the streaming media client can encode the data frames (including one or both of audio frames and video frames) collected by the target acquisition component, and then store the encoded data frames (referred to as encoded data frames for short) in a memory pool or a preset queue. In this way, one or more encoded data frames can be obtained from the memory pool or the preset queue, and the one or more encoded data frames can be encapsulated into streaming media segments that comply with the real-time data transmission protocol and then sent to the streaming media service program. In other words, the transmission of streaming media data is actually achieved by transmitting multiple streaming media segments like a pipeline.

[0098] S204: Pull streaming media data from the streaming media service program and play the pulled streaming media data locally on the streaming media client; and push the streaming media data in the streaming media service program to the server.

[0099] A streaming client can pull streaming media data from a streaming service program and play the pulled streaming media data locally on the streaming client. In a specific implementation, the streaming media client can decapsulate the pulled streaming media data to obtain encoded source data, then decode the encoded source data to obtain the source data, and then play the source data locally. For example, the streaming media client is integrated with an interface player, and the streaming media client can play the source data in the interface player. It should be noted that when playing the pulled streaming media data, the streaming media client decodes the encoded data frames in the decapsulated streaming media data, and simultaneously renders the decoded data frames in the interface player, i.e., the decoding and rendering processes are performed simultaneously. In addition, the streaming media client can establish a session channel with the server and push the streaming media data from the streaming service program to the server in the form of streaming media segments through the session channel. After the streaming media data is pushed to the server, it can be stored in the server, and the streaming media data stored in the server can be played on the server side. In this way, local and background synchronous playback of streaming media data can be achieved through pure software technology.

[0100] The embodiment of the present application is applied to a streaming media client. When it is specifically implemented, the source data collected by the target acquisition component on the streaming media client side can be obtained; the source data can be transformed and processed to generate streaming media data; the streaming media data can be sent to the streaming media service program on the streaming media client side; the streaming media data can be pulled from the streaming media service program, and the pulled streaming media data can be played locally on the streaming media client; and the streaming media data in the streaming media service program can be pushed to the server. It can be seen that the embodiment of the present application constructs a set of local streaming media service programs to receive streaming media data, so that the streaming media data can be pulled and played directly from the local, and the streaming media data can be pushed to the remote server. While realizing stream multiplexing on the streaming media client side, the pulling of streaming media data does not need to occupy the downlink bandwidth on the streaming media client side, reducing the traffic consumption of streaming media data transmission and improving the transmission efficiency of streaming media data.

[0101] See Figure 3 , Figure 3 1 is a flow chart of another streaming media data processing method provided by an embodiment of the present application. The method can be executed by a streaming media client, specifically a streaming media client running in a streaming terminal. The streaming media data processing method includes the following steps S301-S304:

[0102] S301: Call a first streaming media service program to obtain source data collected by a target collection component on a streaming media client side, and convert the source data into streaming media data.

[0103] S302: Call the first streaming media service program to send streaming media data to the streaming media service program on the streaming media client side.

[0104] The streaming media client may include a first streaming service program, the first streaming service program may be used for encoding the source data collected by the target acquisition component and encapsulating it into streaming media data, and streaming media data is pushed to the streaming media service program. Optionally, the first streaming service program may be FFmpeg (Fast Forward Mpeg), FFmpeg is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams, FFmpeg is issued under the Lesser General Public License (LGPL) agreement or the General Public License (GPL) agreement, and anyone can use it freely, but must strictly abide by the LGPL or GPL agreement. In one embodiment, when a streaming media acquisition request is generated in the streaming media client, the first streaming service program can be called to initiate an acquisition component viewing instruction, and the acquisition component viewing instruction can trigger the streaming terminal running the streaming client to return the acquisition component to which the streaming terminal is connected, so that M acquisition components on the streaming media client side can be determined, and then the streaming media type requested to be collected by the streaming media acquisition request can be called to determine the target acquisition component (which can be one or more) from the M acquisition components. For example, the first streaming service program initiates a dshow instruction (a collection component viewing instruction used to capture collection components such as cameras, collection cards, and microphones) to obtain M collection components on the streaming media client side. When the determined streaming media type is a video type, the video collection component among the M collection components is matched according to "DirectShow video devices".

[0105] Optionally, when the M acquisition components include multiple video acquisition components, the multiple video acquisition components can acquire video data from different perspectives.

[0106] The streaming media client also includes a detection service program, which can be used for detecting the service state (including startup state and unstarted state) of the streaming media service program according to the polling cycle; the polling cycle refers to the interval duration at which the service state of the streaming media service program is detected, which can be set as needed, such as 20 seconds. In one embodiment, when the time point of the polling cycle is reached, if the polling cycle is 30 seconds, the last detection time point is 1:10:00, so when arriving at 1:10:30, it is believed that the time point of the polling cycle is reached, the detection service program can be called to detect the service state of the streaming media service program, if the service state of the streaming media service program is in the startup state, then the first push stream service program is called to obtain the source data collected by the target acquisition component, and the source data is converted into streaming media data, and then the streaming media data is sent to the streaming media service program based on the real-time data transmission protocol. Alternatively, after the streaming media client is started, the streaming media service program can be started.

[0107] In one implementation, calling the first streaming service program to convert source data into streaming media data includes: calling the first streaming service program to obtain encoding optimization parameters set in the first streaming service program; calling the first streaming service program to encode the source data according to the encoding optimization parameters, and encapsulating the encoded source data to obtain streaming media data in a first encoding format (which must comply with a real-time data transmission protocol); wherein the first encoding format refers to the encoding format used to encode the source data, such as the flv1 format. Optionally, the encoding optimization parameters include at least one of the following: an encoding speed optimization parameter and an encoding delay optimization parameter. The encoding speed optimization parameter is used to enable a function to increase the encoding speed, and the encoding delay optimization parameter is used to enable a function to reduce the encoding delay. Exemplarily, when the first streaming service program is FFmpeg, the preset parameters in FFmpeg mainly adjust the balance between encoding speed and quality, and are arranged from fast to slow in terms of encoding speed: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, verslow. -preset ultrafast can be added in FFmpeg to obtain the encoding speed optimization parameters in the embodiment of the present application. The tune parameters in FFmpeg are parameters that match the video type and visual optimization, including: film (movie type), animation (animation type), grain (used when retaining fine grain), stillimage (used when encoding static images), psnr (parameter for improving peak signal-to-noise ratio), ssim (parameter for improving structural measurement indicators), fastdecode (parameter for achieving fast decoding), zerolatency (zero delay), and -tune can be added in FFmpeg. Zerolatency obtains the encoding delay optimization parameters in the embodiment of the present application. It can be seen that the encoding speed of the first push stream can be improved and the encoding delay of the first push stream can be reduced by the encoding optimization parameters.

[0108] S303: Control the interface player to pull streaming media data from the streaming media service program, and play the pulled streaming media data on the interface player.

[0109] An interface player can be integrated into the streaming media client. In one implementation, streaming media data is pulled from the streaming media service program, and the streaming media data pulled is played locally on the streaming media client, including: when the detection service program in the streaming media client detects that the first push service program in the streaming media client triggers push (i.e., sends streaming media data to the streaming media service program), the streaming media client will call the detection service program to send a pull notification to the interface player. The interface player can be controlled to pull streaming media data from the streaming media service program based on the pull notification, and play the streaming media data pulled on the interface player. Specifically, the interface player can decapsulate the streaming media data to obtain the encoded source data, and then decode the encoded source data to obtain the source data, and then play the source data in the interface player. It should be noted that the streaming media data will be sent to the interface player in the form of streaming media segments, so the streaming media segments are actually decapsulated. Encapsulation involves placing one or more encoded data frames into a streaming media segment according to a specific format. A streaming media segment typically consists of two parts: a header and a body. The body includes one or more encoded data frames, while the header primarily contains information describing the body, such as the encoding format, data type, and the identifier of the streaming media segment. Decapsulation involves removing the encapsulation process to retrieve the encoded data frames within the streaming media segment. Encoding the source data and decoding the encoded source data can be understood as inverse processes: encoding is followed by decoding in a corresponding manner, thereby recreating the original source data as closely as possible.

[0110] In one embodiment, playing the pulled streaming media data locally on a streaming media client includes: obtaining data read configuration parameters in an interface player; and playing the pulled streaming media data in the interface player according to the data read configuration parameters. The data read configuration parameters include parameters related to a target duration, which is the duration from when the interface player obtains the pulled streaming media data to when the first data frame in the pulled streaming media data is played. The data read configuration parameters include at least one of a byte parameter and a duration parameter; the byte parameter indicates the maximum number of bytes to be read at a time, and the duration parameter indicates the maximum duration of a single read. Optionally, when the data reading configuration parameters include a byte parameter and a duration parameter, if the number of bytes read by the interface player is equal to the maximum number of bytes indicated by the byte parameter, and the time taken has not reached the maximum duration indicated by the duration parameter, then the amount of data read at a single time is determined by the byte parameter. Correspondingly, when the time taken by the interface player to read the data reaches the maximum duration indicated by the duration parameter, and the number of bytes read has not reached the maximum number of bytes indicated by the byte parameter, then the amount of data read at a single time is determined by the duration parameter; for example, if the byte parameter is 1000 bytes and the duration parameter is 10 seconds, then either 1000 bytes of data or 10 seconds of data will be read. Specifically, during the decapsulation process, the interface player must first parse the header information in the streaming media segment before it can obtain the encoded data frame in the streaming media segment. If the amount of data read at a single time is much larger than the amount of data in the header information in the streaming media segment, it will not be conducive to improving the speed of parsing the header information, thereby delaying the time to obtain the first encoded data frame, and thereby delaying the time to render the first data frame, which is not conducive to improving the user experience. Therefore, the embodiment of the present application can set the byte parameter and / or duration parameter based on the data volume of the header information in the streaming media segment, so that the data volume read in a single time is adapted to the data volume of the header information in the streaming media segment (for example, the data volume read in a single time is slightly larger than the data volume of the header information in the streaming media segment). Optionally, the byte parameter and / or duration parameter can be set based on the experience value of the expert. In one implementation, the interface player can be an FFME player (FFMediaElement.window, a multimedia player based on FFmpeg), the byte parameter can be set by the parameter value of probesize, and the duration parameter can be set by the parameter value of maxanalyzeduration.

[0111] In one embodiment, the target acquisition component includes a first acquisition component and a second acquisition component, the streaming media data includes first streaming media data (including video stream data) corresponding to the first acquisition component and second streaming media data (including video stream data) corresponding to the second acquisition component, and the interface player includes at least a first playback area and a second playback area. The streaming media data obtained by pulling the stream is played locally on the streaming media client, including: obtaining a first size (length × width) of the first playback area, adapting the first video frame in the first streaming media data obtained by pulling the stream according to the first size, and playing the adapted first video frame in the first playback area, that is, the length and width of the adapted first video frame are the same as the length and width of the first playback area. Obtaining a second size (length × width) of the second playback area, adapting the second video frame in the second streaming media data obtained by pulling the stream according to the second size, and playing the adapted second video frame in the second playback area, that is, the length and width of the adapted second video frame are the same as the length and width of the second playback area. That is to say, the interface player in the embodiment of the present application may have multiple built-in playback areas, and different playback areas can render video frames in different video stream data. Since the size of the playback area may be much smaller than the original size of the video frame, the video frame to be rendered will be adapted according to the size of the playback area during the rendering phase of the video frame. In this way, it can be rendered at the most appropriate visual size, reducing resource overhead. For example, the size of the video frame rendering parameter videoFilter can be optimized: scale = -1:{h}, h is the height of the corresponding playback area, and scale = -1:{h} is used to adjust the height of the video frame to the height of the corresponding playback area.

[0112] Optionally, prompt information about the first acquisition component (which may include the name of the first acquisition component, the format, type, size, etc. of the source data acquired by the first acquisition component) may be output in the first playback area, and prompt information about the second acquisition component (which may include the name of the second acquisition component, the format, type, size, etc. of the source data acquired by the second acquisition component) may be output in the second playback area. The positional relationship between the first playback area and the second playback area includes at least one of the following: the first playback area and the second playback area are independent of each other and have no overlapping areas (such as Figure 4a The two play areas are independent of each other and have no overlapping area. There is an overlapping area between the first play area and the second play area (such as Figure 4b The first and second play areas are arranged (eg, side by side or in columns) and displayed in the interface player (eg, Figure 4c Shows two play areas arranged side by side in the interface player).

[0113] In one implementation, the first video frame after adaptation is played in the first playback area, including: the interface player can periodically obtain the number of the first decoded video frames that have not been played, the first decoded video frame is obtained by decoding the first streaming media data obtained by pulling the stream, that is, the first video frame is the first decoded video frame that has been decoded and is to be rendered. If the number of the first decoded video frames is greater than or equal to the first preset number (which can be set as needed), it means that the number of cached first decoded video frames is too large, and the first video frame after adaptation is played in the first playback area at the first playback speed; if the number of the first decoded video frames is less than or equal to the second preset number (which can be set as needed), it means that the number of cached first decoded video frames is small, and the first video frame after adaptation is played in the first playback area at the second playback speed; wherein the first playback speed is greater than the preset reference playback speed, and the second playback speed is less than the preset reference playback speed. For example, the preset reference playback speed is 1. When the number of first decoded video frames is less than or equal to the second preset number (20), each cached first decoded video frame is played at a playback speed of 0.8. When the number of first decoded video frames is greater than or equal to the first preset number (100), each cached first decoded video frame is played at a playback speed of 1.2. Similarly, playing the adapted second video frame in the second playback area includes: obtaining the number of first decoded video frames that have not been played, obtaining the number of second decoded video frames that have not been played, the second decoded video frame being obtained by decoding the second streaming media data obtained by pulling the stream, that is, the second video frame is a decoded second decoded video frame to be rendered. If the number of second decoded video frames is greater than or equal to the first preset number, playing the adapted second video frame in the second playback area at the first playback speed; if the number of second decoded video frames is less than or equal to the second preset number, playing the adapted second video frame in the second playback area at the second playback speed; wherein the first playback speed is greater than the preset reference playback speed, and the second playback speed is less than the preset reference playback speed. This is because decoded video frames are usually stored in a buffer (such as a queue). Therefore, when there is too much data in the buffer, accelerated playback can speed up the consumption of buffer data and thus reduce latency; when there is less data remaining in the buffer, low-speed playback can be used to slow down the consumption of buffer data and thus prevent lag.

[0114] It should be noted that when collecting video data and audio data simultaneously, the aligned playback of video frames and audio frames is involved. At this time, if the video stream data is accelerated, the audio stream data must also be accelerated. In one embodiment, the interface player includes an audio smoothing processing program. When the interface player plays the audio stream data at a speed greater than a preset reference playback speed, the audio smoothing processing program is called to smooth the audio stream data in the interface player to maintain audio smoothness and enhance the user's auditory experience. Optionally, the audio smoothing processing program can be a functional module in SoundTouch (a sound processing library). SoundTouch mainly includes three functional modules: speed change, pitch change, and speed change and pitch change. SoundTouch can operate on streaming media data in real time.

[0115] In one embodiment, the interface player also includes a third playback area; when a new third acquisition component is detected in the target acquisition component through the detection service program, for example, a new acquisition component is connected to the streaming media client side, and the acquisition component is used to collect the above-determined streaming media type, then the newly connected acquisition component is determined as the third acquisition component, and the third streaming media data obtained by pulling the stream can be played in the third playback area. The third streaming media data is obtained by calling the first push service program to convert and process the source data collected by the third acquisition component. Optionally, if it is detected that any acquisition component in the target acquisition component is removed, such as when the camera in the target acquisition component is removed, the playback screen in the corresponding playback area of ​​the interface player is closed, and at the same time, the streaming media data corresponding to any acquisition component will be stopped from being pushed to the streaming media service program. It can be seen that the embodiment of the present application supports hot-swappable linkage and synchronization of the number of local playback screens.

[0116] In one embodiment, the interface player includes a floating window; the streaming media data obtained by pulling the stream is played locally on the streaming media client, including: obtaining the third size of the floating window, adapting the video frames in the streaming media data according to the third size, and playing the adapted video frames in the floating window. The floating window can be displayed on the surface of other clients and is a movable window. The embodiment of the present application supports the use of a floating window to display video stream data, and during the rendering process, the video frames to be rendered in the streaming media data are rendered according to the size of the playback area, thereby reducing resource overhead.

[0117] In one embodiment, a streaming media client includes an interface player; the interface player includes a playback area and a floating window; playing the streaming media data obtained by pulling the stream locally on the streaming media client includes: when playing the streaming media data obtained by pulling the stream in the playback area, closing the playback screen in the floating window; when playing the streaming media data obtained by pulling the stream in the floating window, closing the playback screen in the playback area. For example, Figure 5(This is an interface diagram provided by an embodiment of the present application) shows that the player can provide a control for opening a floating window. When the control is operated, a floating window will pop up (it can be one or more, and one floating window is used to play a streaming media data), and the streaming media data obtained by pulling the stream will be played in the floating window. At this time, the playback area can be filled with a video placeholder. It can be seen that when the playback area and the floating window are opened at the same time, there will be a pair of identical video screens. Therefore, when the floating window is opened, the embodiment of the present application will close the playback screen in the playback area with the same address, or close the playback screen in the floating window with the same address when the playback area is opened, so as to reduce the resource overhead of local playback, so that more cameras can be expanded to carry more screens in the later stage.

[0118] S304: Call the second streaming service program to stream the streaming media data in the streaming media service program to the server.

[0119] The streaming media client also includes a second streaming service program, which can be used to stream streaming media data in the streaming media service program to the server. In one embodiment, streaming media data in the streaming media service program to the server includes: calling the second streaming service program to convert the streaming media data in the first encoding format in the streaming media service program into streaming media data in the second encoding format supported by the server; calling the second streaming service program to send the streaming media data in the second encoding format to the server. Optionally, the second encoding format can be o264rt format, and data in other encoding formats can be converted into data in o264rt format by an o264rt encoder (an improved H.264 encoder with the advantages of low consumption, high speed, high efficiency, high quality, and weak network protection); for example, the encoded data frame in the streaming media data in the first encoding format adopts the flv1 format, and the o264rt encoder can convert the encoded data frame in flv1 format into the encoded data frame in o264rt format, and the encoded data frame in o264rt format is encapsulated to obtain streaming media data in the second encoding format. For example, the o264rt encoder may be added to FFmpeg: -complexity 0 -c:vlibo264rt -o264rt_params MinQp=1:MaxQp=503 to obtain a second streaming service program.

[0120] In one embodiment, the streaming media data is stored in the server; the streaming media data stored in the server supports cloud playback (i.e., directly viewing and playing the streaming media data stored in the server online); the stream pulling terminal can achieve fast playback of the streaming media data by directly playing the streaming media data in the cloud. If the detection service program detects that the target acquisition component has been removed, the streaming media data in the streaming media service program is stopped from being pushed to the server, and a streaming stop instruction for the streaming media data is sent to the server, and the streaming stop instruction is used to instruct the server to stop the cloud playback of the streaming media data. For example, when camera 1 is removed, the detection service program will notify the server to stop playing the streaming media data corresponding to camera 1. It can be seen that the embodiment of the present application supports synchronization of the number of hot-swappable linkage remote playback screens.

[0121] In an embodiment of the present application, when the detection service program detects that the streaming media service program is started, the first push streaming service program can be called to transmit the source data collected by the target acquisition component to the streaming media service program in the form of streaming media segments. With the help of the push-pull streaming function of the streaming media service program, the streaming media data is transmitted to the player on the one hand, and the streaming media data is transmitted to the remote server on the other hand, which can achieve synchronous viewing of local and background, save network resources, and only occupy the uplink bandwidth; and support hot-plug linkage to synchronize the number of local playback screens and remote playback screens when the streaming media client is running; through a series of low-latency optimization technologies (such as setting encoding optimization parameters, adding data reading configuration parameters, and adjusting the playback speed based on the number of decoded data frames), the interface player can achieve low latency and low energy consumption.

[0122] Optionally, the streaming media client can be a program that is secondary developed based on a software development kit; the streaming media service program is encapsulated in the software development kit; wherein the software development kit also includes at least one of the following programs: an interface player, a detection service program, a first streaming service program, and a second streaming service program. In other words, existing live broadcast clients, instant messaging clients, and other clients that can achieve synchronous playback can achieve the functions achieved by the streaming media client in the embodiment of the present application by integrating the software development kit provided in the embodiment of the present application. The following explanation is made by taking the streaming media client as an example of a violation detection client. Figure 6(is a scenario diagram of the streaming media data processing method provided by an embodiment of the present application), after the violation detection client is started, the livego streaming media service program and the first detection service program will be enabled. The first detection service program will poll to obtain the service status of the livego streaming media service program. When the service status of the livego streaming media service program is the started state, the first detection service program will detect and match the video acquisition component, notify the streaming media client to call the first push streaming service program to encapsulate the video data collected by the video acquisition component into streaming media data, and push the streaming media data to the livego streaming media service program. The video encoding format used for pushing is flv1 format. After the first detection service program detects that the first push streaming service program triggers the push streaming, it will notify the interface player in real time to try to pull the streaming media data in the livego streaming media service program for playback. It should be noted that after receiving the notification, the interface player will continue to try to pull the streaming playback until the playback is successful, and the interface player can support two playback scenarios: playback area and floating window. In addition, when the first detection service program detects that a video acquisition component has been removed, the first push streaming server will interrupt the push streaming and notify the interface player to remove the useless playback screen in time. The violation detection client may be integrated with an environmental detection control (i.e., a control for enabling remote violation detection), for example, Figure 7 The control "Start Game" in (another interface diagram provided by an embodiment of the present application) is an environment detection control. When the environment detection control is triggered, the second detection service program can be started to forward the existing active video stream data in the livego streaming service program (i.e., the streaming data being pushed to the livego streaming service program) to the remote server through the second push streaming service program. The forwarding process can perform real-time video transcoding, that is, converting the original video encoding flv1 format into o264rt format. The streaming data pushed to the server will be stored in the server, such as Figure 8 (which is another interface schematic diagram provided in an embodiment of the present application), the server's backend (a management interface) can pull streaming media data stored in the server for playback.

[0123] The first streaming service program can be the LGPL version of FFmpeg, and the interface player can be the FFME player. The LGPL version of FFmpeg and the FFME player rely on the same FFmpeg basic library. Figure 9(This is a schematic diagram of adjusting a GPL version of FFmpeg to an LGPL version of FFmpeg provided in an embodiment of the present application), if the GPL version of FFmpeg is downloaded from the official version, the GPL version of FFmpeg basic library needs to be adjusted to the LGPL version of FFmpeg basic library, specifically, the GPL version of the files: Avcode.dll, Avdevice.dll, Avfilter.dll, Avutil.dll, Swresample.dll, Swscale.dll are replaced with the LGPL version. The embodiment of the present application integrates soundtouch.dll (audio smoothing processing program) in the FFmpeg basic library. In addition, as Figure 10 As shown in the figure (which is another interface diagram provided by an embodiment of the present application), considering that streaming media data in the O264RT format has the advantages of low consumption, high speed, high efficiency, high quality, and weak network protection, converting streaming media data in the FLV1 format to streaming media data in the O264RT format is conducive to maintaining low-latency playback in the background. The solution provided by the embodiment of the present application allows the streaming terminal to play the streaming media data while pushing the stream to the server, and ultimately the time difference between the playback screen presented by the streaming terminal and the playback screen presented by the server background is controlled to 1 second.

[0124] See Figure 11 , Figure 11 This is a structural diagram of a streaming media data processing device provided in an embodiment of the present application. The streaming media data processing device can be a computer program (including program code) running in the above-mentioned streaming terminal, for example, the streaming media data processing device is a streaming media client; the streaming media data processing device can be used to execute the corresponding steps of the method provided in the embodiment of the present application. Figure 11 As shown, the streaming media data processing device 1100 may include at least one of the following units: an acquisition unit 1101 and a processing unit 1102.

[0125] The acquisition unit 1101 is used to acquire source data collected by the target acquisition component on the streaming media client side;

[0126] The processing unit 1102 is used to convert the source data to generate streaming media data;

[0127] The processing unit 1102 is further configured to send the streaming media data to the streaming media service program on the streaming media client side;

[0128] The processing unit 1102 is further configured to pull streaming media data from the streaming media service program and play the pulled streaming media data locally on the streaming media client; and push the streaming media data in the streaming media service program to the server.

[0129] In one implementation, the streaming media client includes a detection service program and a first streaming service program; the detection service program is used to detect the service status of the streaming media service program according to a polling cycle; and the acquisition unit 1101 is used to acquire source data collected by the target acquisition component on the streaming media client side, specifically to perform the following steps:

[0130] When the polling cycle time point is reached, the detection service program is called to detect the service status of the streaming media service program;

[0131] If the service status of the streaming media service program is in the startup state, the first streaming service program is called to obtain the source data collected by the target collection component.

[0132] In one implementation, the number of acquisition components on the streaming media client side is M, where M is a positive integer; the processing unit 1102 is further configured to perform the following steps:

[0133] When a streaming media acquisition request is generated in the streaming media client, determining the type of streaming media requested to be acquired by the streaming media acquisition request;

[0134] Based on the determined streaming media type, determining a target acquisition component from the M acquisition components;

[0135] Among them, the streaming media type includes audio type, video type or audio and video type; when the determined streaming media type is audio type, the target acquisition component is the audio acquisition component; when the determined streaming media type is video type, the target acquisition component is the video acquisition component; when the determined streaming media type is audio and video type, the target acquisition component is the audio and video acquisition component.

[0136] In one implementation, the target acquisition component includes a first acquisition component and a second acquisition component, the streaming media data includes first streaming media data corresponding to the first acquisition component and second streaming media data corresponding to the second acquisition component; the streaming media client includes an interface player, and the interface player includes at least a first playback area and a second playback area; the processing unit 1102 is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0137] Obtaining a first size of the first playback area, adapting a first video frame in the first streaming media data obtained by pulling the stream according to the first size, and playing the adapted first video frame in the first playback area;

[0138] A second size of the second playback area is obtained, a second video frame in the second streaming media data obtained by pulling the stream is adapted according to the second size, and the adapted second video frame is played in the second playback area.

[0139] In one implementation, the processing unit 1102 is configured to perform the following steps when playing the first video frame after the adaptation process in the first playback area:

[0140] Obtaining the number of first decoded video frames that have not yet been played, where the first decoded video frames are obtained by decoding first streaming media data obtained by pulling the stream;

[0141] If the number of the first decoded video frames is greater than or equal to the first preset number, playing the adapted first video frames in the first playback area at the first playback speed;

[0142] If the number of the first decoded video frames is less than or equal to the second preset number, playing the adapted first video frames in the first playback area at the second playback speed;

[0143] The first playback speed is greater than a preset reference playback speed, and the second playback speed is less than the preset reference playback speed.

[0144] In one implementation, the interface player further includes a third playback area; and the processing unit 1102 is further configured to perform the following steps:

[0145] If a third acquisition component is detected to be newly added in the target acquisition component, the third streaming media data obtained by pulling the stream is played in the third playback area, where the third streaming media data is obtained by converting the source data collected by the third acquisition component;

[0146] If it is detected that any acquisition component in the target acquisition components is removed, the playback screen in the corresponding playback area of ​​the interface player is closed.

[0147] In one implementation, the processing unit 1102 is further configured to perform the following steps:

[0148] Outputting prompt information about the first acquisition component in the first playback area, and outputting prompt information about the second acquisition component in the second playback area;

[0149] Among them, the positional relationship between the first playback area and the second playback area includes at least one of the following: the first playback area and the second playback area are independent of each other and have no overlapping area, there is an overlapping area between the first playback area and the second playback area, and the first playback area and the second playback area are arranged and displayed in the interface player.

[0150] In one implementation, the streaming client includes an interface player; the interface player includes a floating window; and the processing unit 1102 is configured to perform the following steps when playing the streaming data pulled locally on the streaming client:

[0151] The third size of the floating window is obtained, the video frames in the streaming media data are adapted according to the third size, and the adapted video frames are played in the floating window.

[0152] In one implementation, a streaming media client includes a first streaming service program and an interface player, and streaming media data is sent to the streaming media service program via the first streaming service program. The processing unit 1102 is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0153] Get the data reading configuration parameters in the interface player;

[0154] In the interface player, the configuration parameters are read according to the data and the streaming media data obtained by pulling the stream is played;

[0155] Among them, the data reading configuration parameters include parameters related to the target duration. The target duration is the duration from when the interface player obtains the streaming media data obtained by pulling the stream to when the first data frame in the streaming media data obtained by pulling the stream is played; the data reading configuration parameters include at least one of a byte parameter and a duration parameter; the byte parameter is used to indicate the maximum number of bytes read in a single time, and the duration parameter is used to indicate the maximum duration of a single read.

[0156] In one implementation, the streaming media client includes an interface player, the streaming media data includes audio stream data, and the audio stream data is played in the interface player; the interface player includes an audio smoothing processing program; and the processing unit 1102 is further configured to perform the following steps:

[0157] If the playback speed of the audio stream data by the interface player is greater than a preset reference playback speed, the audio smoothing processing program is called to perform smoothing processing on the audio stream data in the interface player.

[0158] In one implementation, the streaming media data is stored in a server; the streaming media data stored in the server supports cloud playback; and the processing unit 1102 is further configured to perform the following steps:

[0159] When the detection service program detects that the target acquisition component has been removed, the streaming data in the streaming service program is stopped from being pushed to the server, and a streaming data push stop instruction is sent to the server. The push stop instruction is used to instruct the server to stop the cloud playback of the streaming data.

[0160] In one implementation, the streaming media client includes a first streaming service program; the processing unit 1102 is used to convert the source data and generate streaming media data, specifically to perform the following steps:

[0161] Obtaining encoding optimization parameters set in the first streaming service program;

[0162] Invoking a first streaming service program, encoding the source data according to the encoding optimization parameters, and encapsulating the encoded source data to obtain streaming media data in a first encoding format;

[0163] The coding optimization parameters include at least one of the following: a coding speed optimization parameter and a coding delay optimization parameter. The coding speed optimization parameter is used to enable a function of increasing the coding speed, and the coding delay optimization parameter is used to enable a function of reducing the coding delay.

[0164] In one implementation, the streaming media client further includes a second streaming service program; and the processing unit 1102 is configured to execute the following steps:

[0165] Invoking a second streaming service program to convert streaming media data in a first encoding format in the streaming media service program into streaming media data in a second encoding format supported by the server;

[0166] The second streaming service program is called to send the streaming media data in the second encoding format to the server.

[0167] In one implementation, the streaming client is a program that is secondary developed based on a software development kit; the streaming service program is encapsulated in the software development kit;

[0168] The software development kit further includes at least one of the following programs: an interface player, a detection service program, a first streaming service program, and a second streaming service program.

[0169] It is understood that the functions of the various functional units of the streaming media data processing device described in the embodiments of the present application can be specifically implemented according to the methods in the above-mentioned method embodiments. The specific implementation process can be referred to the relevant description of the above-mentioned method embodiments and will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0170] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 12, the computer device 1200 includes a processor 1201, a communication interface 1202 and a computer-readable storage medium 1203. The processor 1201, the communication interface 1202 and the computer-readable storage medium 1203 can be connected via a bus or other means. The communication interface 1202 is used to receive and send data. The computer-readable storage medium 1203 can be stored in the memory of the computer device 1200. The computer-readable storage medium 1203 is used to store computer programs. The computer programs include program instructions. The processor 1201 is used to execute the program instructions stored in the computer-readable storage medium 1203. The processor 1201 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device 1200. It is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0171] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in the computer device 1200 for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the computer device 1200 and, of course, the extended storage medium supported by the computer device 1200. The computer-readable storage medium provides a storage space that stores the processing system of the computer device 1200. In addition, one or more instructions suitable for being loaded and executed by the processor 1201 are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0172] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1201 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned streaming media data processing method embodiment; in a specific implementation, the one or more instructions in the computer-readable storage medium are loaded by the processor 1201 and execute the following steps:

[0173] Obtain source data collected by the target collection component on the streaming media client side;

[0174] Convert and process source data to generate streaming media data;

[0175] Sending streaming media data to the streaming media service program on the streaming media client side;

[0176] Pull streaming media data from the streaming media service program and play the pulled streaming media data locally on the streaming media client; and push the streaming media data in the streaming media service program to the server.

[0177] In one implementation, the streaming media client includes a detection service program and a first streaming service program; the detection service program is used to detect the service status of the streaming media service program according to a polling cycle; when the processor 1201 is used to obtain source data collected by the target collection component on the streaming media client side, it is specifically used to perform the following steps:

[0178] When the polling cycle time point is reached, the detection service program is called to detect the service status of the streaming media service program;

[0179] If the service status of the streaming media service program is in the startup state, the first streaming service program is called to obtain the source data collected by the target collection component.

[0180] In one implementation, the number of acquisition components on the streaming media client side is M, where M is a positive integer; the processor 1201 is further configured to perform the following steps:

[0181] When a streaming media acquisition request is generated in the streaming media client, determining the type of streaming media requested to be acquired by the streaming media acquisition request;

[0182] Based on the determined streaming media type, determining a target acquisition component from the M acquisition components;

[0183] Among them, the streaming media type includes audio type, video type or audio and video type; when the determined streaming media type is audio type, the target acquisition component is the audio acquisition component; when the determined streaming media type is video type, the target acquisition component is the video acquisition component; when the determined streaming media type is audio and video type, the target acquisition component is the audio and video acquisition component.

[0184] In one implementation, the target acquisition component includes a first acquisition component and a second acquisition component, the streaming media data includes first streaming media data corresponding to the first acquisition component and second streaming media data corresponding to the second acquisition component; the streaming media client includes an interface player, and the interface player includes at least a first playback area and a second playback area; the processor 1201 is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0185] Obtaining a first size of the first playback area, adapting a first video frame in the first streaming media data obtained by pulling the stream according to the first size, and playing the adapted first video frame in the first playback area;

[0186] A second size of the second playback area is obtained, a second video frame in the second streaming media data obtained by pulling the stream is adapted according to the second size, and the adapted second video frame is played in the second playback area.

[0187] In one implementation, the processor 1201 is configured to perform the following steps when playing the first video frame after the adaptation process in the first playback area:

[0188] Obtaining the number of first decoded video frames that have not yet been played, where the first decoded video frames are obtained by decoding first streaming media data obtained by pulling the stream;

[0189] If the number of the first decoded video frames is greater than or equal to the first preset number, playing the adapted first video frames in the first playback area at the first playback speed;

[0190] If the number of the first decoded video frames is less than or equal to the second preset number, playing the adapted first video frames in the first playback area at the second playback speed;

[0191] The first playback speed is greater than a preset reference playback speed, and the second playback speed is less than the preset reference playback speed.

[0192] In one implementation, the interface player further includes a third playback area; and the processor 1201 is further configured to perform the following steps:

[0193] If a third acquisition component is detected to be newly added in the target acquisition component, the third streaming media data obtained by pulling the stream is played in the third playback area, where the third streaming media data is obtained by converting the source data collected by the third acquisition component;

[0194] If it is detected that any acquisition component in the target acquisition components is removed, the playback screen in the corresponding playback area of ​​the interface player is closed.

[0195] In one implementation, the processor 1201 is further configured to perform the following steps:

[0196] Outputting prompt information about the first acquisition component in the first playback area, and outputting prompt information about the second acquisition component in the second playback area;

[0197] Among them, the positional relationship between the first playback area and the second playback area includes at least one of the following: the first playback area and the second playback area are independent of each other and have no overlapping area, there is an overlapping area between the first playback area and the second playback area, and the first playback area and the second playback area are arranged and displayed in the interface player.

[0198] In one implementation, the streaming client includes an interface player; the interface player includes a floating window; and the processor 1201 is configured to perform the following steps when playing the streaming data pulled locally on the streaming client:

[0199] The third size of the floating window is obtained, the video frames in the streaming media data are adapted according to the third size, and the adapted video frames are played in the floating window.

[0200] In one implementation, a streaming media client includes a first streaming service program and an interface player, and streaming media data is sent to the streaming media service program via the first streaming service program. The processor 1201 is configured to perform the following steps when playing the streaming media data obtained by pulling the stream locally on the streaming media client:

[0201] Get the data reading configuration parameters in the interface player;

[0202] In the interface player, the configuration parameters are read according to the data and the streaming media data obtained by pulling the stream is played;

[0203] Among them, the data reading configuration parameters include parameters related to the target duration. The target duration is the duration from when the interface player obtains the streaming media data obtained by pulling the stream to when the first data frame in the streaming media data obtained by pulling the stream is played; the data reading configuration parameters include at least one of a byte parameter and a duration parameter; the byte parameter is used to indicate the maximum number of bytes read in a single time, and the duration parameter is used to indicate the maximum duration of a single read.

[0204] In one implementation, the streaming media client includes an interface player, the streaming media data includes audio stream data, and the audio stream data is played in the interface player; the interface player includes an audio smoothing processing program; and the processor 1201 is further configured to perform the following steps:

[0205] If the playback speed of the audio stream data by the interface player is greater than a preset reference playback speed, the audio smoothing processing program is called to perform smoothing processing on the audio stream data in the interface player.

[0206] In one implementation, the streaming media data is stored in a server; the streaming media data stored in the server supports cloud playback; and the processor 1201 is further configured to perform the following steps:

[0207] When the detection service program detects that the target acquisition component is removed, a streaming data push stop instruction is sent to the server, and the streaming data push stop instruction is used to instruct the server to stop the cloud playback of the streaming data.

[0208] In one implementation, the streaming media client includes a first streaming service program; the processor 1201 is configured to convert the source data and generate streaming media data by executing the following steps:

[0209] Obtaining encoding optimization parameters set in the first streaming service program;

[0210] Invoking a first streaming service program, encoding the source data according to the encoding optimization parameters, and encapsulating the encoded source data to obtain streaming media data in a first encoding format;

[0211] The coding optimization parameters include at least one of the following: a coding speed optimization parameter and a coding delay optimization parameter. The coding speed optimization parameter is used to enable a function of increasing the coding speed, and the coding delay optimization parameter is used to enable a function of reducing the coding delay.

[0212] In one implementation, the streaming media client further includes a second streaming service program; when the processor 1201 is configured to:

[0213] Invoking a second streaming service program to convert streaming media data in a first encoding format in the streaming media service program into streaming media data in a second encoding format supported by the server;

[0214] The second streaming service program is called to send the streaming media data in the second encoding format to the server.

[0215] In one implementation, the streaming client is a program that is secondary developed based on a software development kit; the streaming service program is encapsulated in the software development kit;

[0216] The software development kit further includes at least one of the following programs: an interface player, a detection service program, a first streaming service program, and a second streaming service program.

[0217] Based on the same inventive concept, the principles and beneficial effects of the computer device 1200 provided in the embodiment of the present application for solving the problem are similar to the principles and beneficial effects of the streaming media data processing method in the method embodiment of the present application for solving the problem. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0218] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0219] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described streaming media data processing method.

[0220] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0222] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technical object of a person skilled in the art that can be easily conceived of within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A streaming media data processing method, characterized in that: The method is applied to a streaming media client, and the method includes: Acquire source data collected by the target collection component on the streaming media client side; Converting the source data to generate streaming media data; Sending the streaming media data to the streaming media service program on the streaming media client side; Pulling the streaming media data from the streaming media service program, and playing the pulled streaming media data locally on the streaming media client; and Push the streaming media data in the streaming media service program to the server.

2. The method according to claim 1, wherein The streaming media client includes a detection service program and a first streaming service program; the detection service program is used to detect the service status of the streaming media service program according to a polling cycle; The acquiring of source data collected by the target collection component on the streaming media client side includes: When the time point of the polling cycle is reached, calling the detection service program to detect the service status of the streaming media service program; If the service status of the streaming media service program is in the startup state, the first streaming service program is called to obtain the source data collected by the target collection component.

3. The method according to claim 1, wherein The number of acquisition components on the streaming media client side is M, where M is a positive integer; the method further includes: When a streaming media acquisition request is generated in the streaming media client, determining the type of streaming media requested to be acquired by the streaming media acquisition request; Based on the determined streaming media type, determining the target acquisition component from the M acquisition components; Among them, the streaming media type includes audio type, video type or audio and video type; when the determined streaming media type is audio type, the target acquisition component is an audio acquisition component; when the determined streaming media type is video type, the target acquisition component is a video acquisition component; when the determined streaming media type is audio and video type, the target acquisition component is an audio and video acquisition component.

4. The method according to claim 1, wherein The target acquisition component includes a first acquisition component and a second acquisition component, and the streaming media data includes first streaming media data corresponding to the first acquisition component and second streaming media data corresponding to the second acquisition component; The streaming media client includes an interface player, and the interface player includes at least a first play area and a second play area; Playing the streaming media data obtained by pulling the stream locally on the streaming media client includes: Obtaining a first size of the first playback area, adapting a first video frame in the first streaming media data obtained by pulling the stream according to the first size, and playing the adapted first video frame in the first playback area; A second size of the second playback area is obtained, a second video frame in the second streaming media data obtained by pulling the stream is adapted according to the second size, and the adapted second video frame is played in the second playback area.

5. The method according to claim 4, wherein Playing the adapted first video frame in the first playback area includes: Obtaining a number of first decoded video frames that have not yet been played, where the first decoded video frames are obtained by decoding the first streaming media data obtained by pulling the stream; If the number of the first decoded video frames is greater than or equal to a first preset number, playing the adapted first video frames in the first playback area at a first playback speed; If the number of the first decoded video frames is less than or equal to the second preset number, playing the adapted first video frames in the first playback area at a second playback speed; The first playback speed is greater than a preset reference playback speed, and the second playback speed is less than the preset reference playback speed.

6. The method according to claim 4, wherein The interface player further includes a third play area; and the method further includes: If a third acquisition component is detected to be newly added to the target acquisition component, third streaming media data obtained by pulling the stream is played in the third playback area, where the third streaming media data is obtained by converting the source data collected by the third acquisition component; If it is detected that any one of the target acquisition components is removed, the play screen in the corresponding play area of ​​the interface player is closed.

7. The method according to claim 4, wherein The method further comprises: Outputting prompt information about the first acquisition component in the first playback area, and outputting prompt information about the second acquisition component in the second playback area; Among them, the positional relationship between the first playback area and the second playback area includes at least one of the following: the first playback area and the second playback area are independent of each other and have no overlapping area, there is an overlapping area between the first playback area and the second playback area, and the first playback area and the second playback area are arranged and displayed in the interface player.

8. The method according to claim 1, wherein The streaming media client includes an interface player; The interface player includes a floating window; Playing the streaming media data obtained by pulling the stream locally on the streaming media client includes: A third size of the floating window is acquired, video frames in the streaming media data are adapted according to the third size, and the adapted video frames are played in the floating window.

9. The method according to claim 1, wherein The streaming media client includes an interface player; the interface player includes a play area and a floating window; Playing the streaming media data obtained by pulling the stream locally on the streaming media client includes: When the streaming media data obtained by pulling the stream is played in the playback area, closing the playback screen in the floating window; When the streaming media data obtained by pulling the stream is played in the floating window, the playing screen in the playing area is closed.

10. The method according to claim 1, wherein The streaming media client comprises a first streaming service program and an interface player, and the streaming media data is sent to the streaming media service program via the first streaming service program; Playing the streaming media data obtained by pulling the stream locally on the streaming media client includes: Obtaining data reading configuration parameters in the interface player; Reading configuration parameters according to the data in the interface player, and playing the streaming media data obtained by pulling the stream; Among them, the data reading configuration parameters include parameters related to the target duration, and the target duration is the duration from when the interface player obtains the streaming media data obtained by pulling the stream to when the first data frame in the streaming media data obtained by pulling the stream is played; the data reading configuration parameters include at least one of a byte parameter and a duration parameter; the byte parameter is used to indicate the maximum number of bytes read in a single time, and the duration parameter is used to indicate the maximum duration of a single read.

11. The method according to claim 1, wherein The streaming media client includes an interface player, the streaming media data includes audio stream data, and the audio stream data is played in the interface player; The interface player includes an audio smoothing processing program; The method further comprises: If the playback speed of the audio stream data by the interface player is greater than a preset reference playback speed, the audio smoothing processing program is called to perform smoothing processing on the audio stream data in the interface player.

12. The method according to claim 1, wherein The streaming media data is stored in the server, and the streaming media data stored in the server supports cloud playback; the method further includes: When the detection service program detects that the target acquisition component has been removed, the streaming data in the streaming service program is stopped from being pushed to the server, and a push stop instruction for the streaming data is sent to the server, wherein the push stop instruction is used to instruct the server to stop the cloud playback of the streaming data.

13. The method according to claim 1, wherein The streaming media client includes a first streaming service program; the converting and processing the source data to generate streaming media data includes: Obtaining encoding optimization parameters set in the first streaming service program; calling the first streaming service program, encoding the source data according to the encoding optimization parameters, and encapsulating the encoded source data to obtain streaming media data in a first encoding format; The coding optimization parameters include at least one of the following: a coding speed optimization parameter and a coding delay optimization parameter. The coding speed optimization parameter is used to enable a function of increasing the coding speed, and the coding delay optimization parameter is used to enable a function of reducing the coding delay.

14. The method according to claim 13, wherein The streaming media client further includes a second streaming service program; and the step of streaming the streaming media data in the streaming media service program to the server includes: Invoking the second streaming service program to convert the streaming media data in the first encoding format in the streaming media service program into streaming media data in a second encoding format supported by the server; The second streaming service program is called to send the streaming media data in the second encoding format to the server.

15. The method according to any one of claims 1 to 14, wherein: The streaming media client is a program that is secondary developed based on a software development kit; the streaming media service program is encapsulated in the software development kit; The software development kit further includes at least one of the following programs: an interface player, a detection service program, a first streaming service program, and a second streaming service program.

16. A streaming media data processing device, characterized in that: The device is applied to a streaming media client, and the device includes: An acquisition unit, configured to acquire source data collected by a target acquisition component on the streaming media client side; A processing unit, configured to convert the source data to generate streaming media data; The processing unit is further configured to send the streaming media data to the streaming media service program on the streaming media client side; The processing unit is further configured to pull the streaming media data from the streaming media service program and play the pulled streaming media data locally on the streaming media client; and push the streaming media data in the streaming media service program to a server.

17. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the streaming media data processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the streaming media data processing method according to any one of claims 1 to 15.

19. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the streaming media data processing method according to any one of claims 1 to 15 is implemented.

Images