Streaming media data processing method and device, client and computer program product

Abstract

The invention discloses a streaming media data processing method, a streaming media data processing device, electronic equipment and a computer program product. Taking a cloud game as an example, different complexities of game pictures or load fluctuation of a server side can directly cause fluctuation of collection intervals between frames of the collected game pictures, so that the delay and lagging phenomena occur in the process of rendering the game pictures by a client side. Therefore, from the essence of the delay lagging phenomenon, whether the current frame is the abnormal frame is determined through the acquisition interval, namely, whether the acquisition interval of the current frame has abnormal fluctuation is determined, and under the condition that the current frame is determined to be the abnormal frame, the current frame is submitted for decoding through the determined optimized submitting time, so that the decoding efficiency is improved. Therefore, the adverse effect of the abnormal fluctuation of the collection interval on the streaming media content presented by the client is reduced, and the fluency of the streaming media content and the user experience are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of streaming media, and in particular relates to a streaming media data processing method, a streaming media data processing device, a client, and a computer program product. Background Art

[0002] The popularity of streaming media has brought great convenience to people's leisure and entertainment. Online video on demand, live video and cloud games are all streaming applications. Although streaming media has improved the efficiency of playing and presenting streaming content, during the transmission process, due to the complexity of streaming content or the load fluctuation of the server, the client may experience delays and freezes, which in turn affects the smoothness of playback and user experience.

[0003] Therefore, a streaming media technology that can ensure smooth playback and user experience needs to be proposed. Summary of the invention

[0004] The present application provides a streaming media data processing method, a streaming media data processing device, a client, and a computer program product, which can improve the fluency of the streaming media content played by the client, thereby enhancing the user experience.

[0005] In a first aspect, the present application provides a streaming media data processing method, which is applied to a client, and the data processing method includes:

[0006] Determine whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media image; if the acquisition interval is greater than the interval threshold, the current frame is an abnormal frame;

[0007] If the current frame is an abnormal frame, the optimized submission time of the current frame is determined according to the collection interval and the preset buffer time;

[0008] Submit the current frame for decoding according to the optimized submission time.

[0009] In a second aspect, the present application provides a streaming media data processing device, comprising:

[0010] A first determination module is used to determine whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture; when the acquisition interval is greater than the interval threshold, the current frame is an abnormal frame;

[0011] The second determination module is used to determine the optimized submission time of the current frame according to the collection interval and the preset buffer time if the current frame is an abnormal frame;

[0012] The submission module is used to submit the current frame for decoding according to the optimized submission time.

[0013] In a third aspect, the present application provides a client, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0015] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0016] Compared with the prior art, the present application has the following beneficial effects: the complexity of the streaming media picture content and the fluctuation of the server load will cause the collection interval of different frames of the streaming media picture to fluctuate, which will in turn cause the server to present the streaming media picture with delay and freeze. Based on this, the present application determines whether the current frame is an abnormal frame through the collection interval, that is, determines whether the collection interval of the current frame has abnormal fluctuations. When the current frame is determined to be an abnormal frame, the current frame is submitted for decoding through the determined optimized submission time, thereby reducing the adverse effects of the abnormal fluctuation of the collection interval on the presentation of streaming media content by the client, thereby improving the fluency of the streaming media content and the user experience.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the collection and decoding process of streaming media images provided in an embodiment of the present application;

[0020] Figure 2 It is a flow chart of a method for processing streaming media data provided by an embodiment of the present application;

[0021] Figure 3 It is another schematic diagram of the collection and decoding process of streaming media images provided by an embodiment of the present application;

[0022] Figure 4 It is a flowchart of another streaming media data processing method provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the effect of the acquisition interval-decoding interval provided in an embodiment of the present application;

[0024] Figure 6 is a structural diagram of a streaming media data processing device provided in an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of the client provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] In order to more intuitively understand the defects of current streaming media technology, we will take cloud gaming as an example. Cloud gaming places the running and rendering process of the game on the server. Even clients with lower performance can achieve cross-platform operation through cloud gaming technology, allowing users to play high-quality games anytime and anywhere. The server is responsible for real-time acquisition and transmission of game images to the client to ensure the presentation of high-definition images. However, due to the complexity of the game screen and changes in the server load, it is easy to cause screen delays and freezes, which in turn affects the smoothness of the game and the user experience.

[0028] In order to solve this problem, this application takes the delay and freeze phenomenon as the starting point. The root cause is: during the transmission of streaming media content, if the client immediately submits for decoding and playback after receiving the streaming media picture, then as long as there is a fluctuation in the network, the playback of the streaming media content will be affected. In order to reduce the impact of network fluctuations, refer to Figure 1 ,The client adopts a buffer algorithm, that after receiving the streaming media picture, it determines the normal ,submission time based on the receiving time and buffer time, thereby covering the frame size fluctuation and ,network fluctuation during the submission and decoding period.

[0029] Although the buffer algorithm can cover the fluctuations in frame size and network during the period of submission for decoding, it cannot cover the fluctuations in the collection interval. Figure 1The acquisition interval between the third frame and the second frame fluctuates. Due to the complexity of the streaming media content and the fluctuation of the server load, the acquisition interval of the third frame fluctuates. If the third frame is still submitted for decoding according to the normal submission time, the normal submission time is estimated too late due to the fluctuation of the sampling interval. The streaming media picture has been received, but it has not yet reached the normal submission time, so it will not be submitted, which leads to a large decoding interval between the third frame and the second frame. That is, the client cannot regularly render the received media picture, causing the streaming media content seen by the user to be delayed and stuck due to the fluctuation of the acquisition interval, resulting in poor fluency of streaming media content that greatly reduces the user experience.

[0030] Therefore, starting from the real cause of the delay and freeze phenomenon, this application proposes a data processing method for streaming media, which submits the current frame for decoding by optimizing the submission time different from the normal submission time, so as to reduce the adverse effects of abnormal fluctuations in the acquisition interval on the presentation of streaming media content by the client, thereby improving the fluency of streaming media content and the user experience. The control method proposed in this application will be described below through specific embodiments.

[0031] The streaming media data processing method provided in the embodiment of the present application can be applied to any client that plays streaming media content transmitted based on streaming media technology. Specifically, the client can be a mobile phone, a tablet computer, a car-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and other electronic devices. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0032] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with the client as the execution subject.

[0033] Figure 2 A schematic flow chart of a streaming media data processing method provided by the present application is shown, and the streaming media data processing method includes:

[0034] Step 210: The client receives the current frame of the streaming media picture.

[0035] Before the emergence of streaming media technology, users had to download the complete media file to their local computer before they could watch it. This method not only took up a lot of storage space, but also required waiting for the download to be completed before starting to play, affecting the user's instant viewing experience. The emergence of streaming media technology has completely changed this model. Its core principle is to divide the streaming media content into small data packets and transmit them to the client in real time through the network for playback, without the need to download the complete file in advance.

[0036] The advantages of streaming media technology are its immediacy and efficiency. First, users can start watching while data is loading, which reduces waiting time and enables "downloading and playing at the same time". Second, streaming media only transmits the data that users need to watch at the moment, saving storage space on the client.

[0037] Based on this, the client can receive the data required by the user for current viewing, that is, the current frame of the streaming media picture.

[0038] Step 220: The client determines whether the current frame is an abnormal frame according to the collection interval between the previous frame and the current frame.

[0039] The acquisition interval between the previous frame and the current frame of the streaming media picture can well reflect whether the sampling interval of the current frame is affected by the complexity of the streaming media picture and / or the server load fluctuation. In order to accurately determine whether the acquisition interval fluctuation is abnormal, the interval threshold can be set based on the normal acquisition interval, or based on the sum of the normal acquisition interval and the allowed fluctuation value. The interval threshold can be considered to be the critical value of the acquisition interval fluctuation. Once the interval threshold is exceeded, the fluctuation of the acquisition interval is an abnormal situation. Accordingly, if the acquisition interval of the current frame is greater than the preset interval threshold, the current frame can be considered to be an abnormal frame.

[0040] Exemplarily, in order to improve the adaptability of the present application to different network conditions and more accurately determine abnormal frames, the interval threshold can be dynamically determined according to network information. Specifically, it can be dynamically determined according to the network speed.

[0041] Step 230: If the current frame is an abnormal frame, the client determines the optimized submission time of the current frame according to the collection interval and the preset buffer time.

[0042] Step 240: The client submits the current frame for decoding according to the optimized submission time.

[0043] For abnormal frames, in order to avoid the fluctuation of the acquisition interval disrupting the regular decoding interval, thereby causing delays and freezes in the streaming media content presented by the client, the client can determine an optimized submission time that is different from the normal submission time. Among them, the optimized submission time is earlier than the normal submission time, which helps to submit the current frame in time, minimize the adverse effects of the fluctuation of the acquisition interval, and improve the smoothness of the streaming media content and the user experience. The optimized submission time can be determined based on the acquisition interval and the preset buffer time, and the buffer time can be determined based on the Kalman estimation algorithm and the fixed system jitter.

[0044] In this embodiment, for the current frame with abnormal acquisition interval fluctuations, the client can determine the optimized submission time based on the acquisition interval buffer time. By optimizing the submission time to submit the current frame for decoding, the adverse effects of the abnormal fluctuations of the acquisition interval on the client's presentation of streaming media content can be reduced, thereby improving the fluency of the streaming media content and the user experience.

[0045] In some embodiments, in order to accurately calculate the optimized submission time, step 210 specifically includes:

[0046] Step A1: The client determines the normal submission time of the current frame according to the reception time and buffering time of the current frame.

[0047] Step A2: The client determines the optimized submission time based on the normal submission time and the collection interval.

[0048] The optimized submission time is earlier than the normal submission time, and the difference between the optimized submission time and the normal submission time is determined according to the collection interval. Based on this, the client can first calculate the normal submission time, and then determine the optimized submission time by the normal submission time and the collection interval.

[0049] The normal submission time is determined based on the receiving time and the buffering time. Specifically, it can be the sum of the receiving time and the buffering time. The receiving time can be real or expected. The expected receiving time is preferred. The normal submission time calculated based on the expected value can cover the frame size fluctuation and network fluctuation from the time when the server collects the data to the time when the client submits the data for decoding as much as possible.

[0050] Exemplarily, the expected receiving time may be calculated based on a Kalman estimation algorithm, that is, the actual acquisition time and receiving time of each frame are used to iterate the model, thereby estimating the expected receiving time.

[0051] From the above content, it can be seen that the difference between the optimized submission time and the normal submission time is determined according to the collection interval. Therefore, the client can determine the optimized submission time according to the normal submission time and the collection interval.

[0052] In some embodiments, the abnormal acquisition interval causes a lag in the current frame acquisition. In order to reduce the adverse effects of the lag time on the subsequent streaming media content presentation, the aforementioned step A2 specifically includes:

[0053] Step A21: The client determines the time difference of the abnormal fluctuation based on the collection interval and the preset normal collection interval.

[0054] Step A22: The client determines the optimized submission time based on the time difference and the normal submission time.

[0055] The client can calculate how long the current frame acquisition is delayed by the acquisition interval and the normal acquisition interval, that is, the time difference. In order to avoid the delay of the normal submission time due to the time difference, the client can optimize the late normal submission time by the time difference and obtain an optimized submission time earlier than the normal submission time, thereby reducing the impact of the time difference on the decoding of the current frame and ensuring the smoothness of the streaming media content presented by the client.

[0056] In some embodiments, the aforementioned step A22 specifically includes:

[0057] Step A221: The client determines whether the time difference is less than an optimization time threshold, where the optimization time threshold is set based on the buffer time.

[0058] In the process of optimizing the normal submission time based on the time difference, if the time difference is large, unlimited optimization cannot be achieved. For example, if the calculated optimized submission time is earlier than the reception time of the current frame, the current frame cannot be submitted based on the optimized submission time.

[0059] Based on this, in order to deal with this situation, an optimization time threshold may be set, and it may be determined whether the time difference is less than the optimization time threshold, so that different optimization measures may be taken according to the relationship between the time difference and the optimization threshold.

[0060] Step A221: If the time difference is less than the optimization time threshold, the client optimizes the submission time to be the difference between the normal submission time and 0.5 times the time difference.

[0061] If the time difference is less than the optimization time threshold, it means that there is a large room for optimization. At this time, the client can directly optimize based on the time difference. Specifically, the optimization submission time can be determined according to the following formula:

[0062] T op =T n -0.5T Δ

[0063] Among them, T op To optimize submission time, T n is the normal submission time, T Δ For the time difference.

[0064] Step A221: If the time difference is greater than or equal to the optimization time threshold, the client optimizes the submission time to be the difference between the normal submission time and 0.5 times the optimization time threshold.

[0065] If the time difference is less than the optimization time threshold, it means that the optimization space is limited. In order to avoid unreasonable optimization submission time, the client can use the optimization time threshold T th Optimization. Specifically, the following formula determines the optimization submission time:

[0066] T op =T n -0.5T th

[0067] In some embodiments, after submitting the current frame for decoding based on the optimized submission time, it also includes: when the new current frame of the streaming media screen is not an abnormal frame, submitting the new current frame for decoding based on the new optimized submission time, and the difference between the normal submission time of the new current frame and the new optimized submission time is 0.5 times the time difference or 0.5 times the optimization time threshold.

[0068] Combined with the previous embodiment, the optimized submission time of the current frame is earlier than its normal submission time, and the optimized submission time of the new current frame is later than its normal submission time, which is equivalent to breaking up the optimizable fluctuation time and allocating it to the current frame and the new current frame, thereby eliminating the fluctuation of the acquisition interval as much as possible.

[0069] For example, Figure 3 As shown in the figure, assuming the buffering time is 2 / fps, the interval threshold is 1.5 / fps, and assuming that the frame interval acceptable to users is 1.5 / fps, the optimization time threshold is 1 / fps, which is half of the buffering time. The third frame is the current frame, and the fourth frame is the new current frame. The acquisition interval between the first and second frames, and between the third and fourth frames is normal, both 1 / fps. The third frame has experienced acquisition fluctuations, and the acquisition interval becomes larger, 2 / fps.

[0070] When the third frame is received, according to the original processing method, the third frame should be submitted at the time of the dotted line, so the decoding interval changes from the original 1 / fps to 2 / fps, and the delay from the second frame to the third frame is present. In order to avoid the jamming, when the third frame is received, the client can determine that the acquisition interval of the current frame is greater than the interval threshold, and the third frame will be determined as an abnormal frame first; then the optimized submission time is determined according to the normal submission time and the time difference. After calculation, it can be known that the time difference is 1 / fps, which is equal to the optimization time threshold. It can be processed based on the optimization time threshold, that is, the 0.5 times optimization time threshold is directly distributed to the third frame and the fourth frame respectively, so the third frame is submitted at 1.5 / fps after the third frame is received, which is 0.5 / fps earlier than the original submission time. The fourth frame will also be submitted 1.5 / fps after the reception, which is 0.5 / fps later than the original submission time. Given that the frame interval of 1.5 / fps is acceptable to users, the smoothness of the streaming content can be guaranteed through the dispersed processing operation of the acquisition interval fluctuation.

[0071] For example, the above example is that the time difference is exactly equal to the optimization time threshold. If the time difference is greater than the optimization time threshold, for example, the acquisition interval becomes larger than 2.2 / fps, then the client will distribute 0.5 times the optimization time threshold to the third frame and the fourth frame respectively. The 0.2 / fps part that exceeds the time difference threshold cannot be eliminated and will still affect the decoding interval of the third frame, that is, the client will submit the third frame at 1.7 / fps after receiving it. Although the decoding interval of the third frame has increased, compared with submitting the third frame at 2.2 / fps after receiving it, the fluctuation of its decoding interval has improved, which can make the jamming less obvious, so as to improve the fluency of the streaming content.

[0072] For example, if the acquisition interval increases to 1.8 / fps, the time difference is 0.8 / fps, which is less than the optimization time threshold. At this time, processing can be performed directly based on the time difference, and the client distributes 0.5 times the time difference to the third and fourth frames respectively, so the client submits the third frame 1.4 / fps after the third frame is received, and the fourth frame will also be submitted 1.4 / fps after it is received. The frame interval of 1.4 / fps is less than the 1.5 / fps accepted by the user, so after the distributed processing operation of the acquisition interval fluctuation, the smoothness of the streaming content can be guaranteed.

[0073] In an embodiment of the present application, the fluctuations of the true interval collected by the server are respectively allocated to the current frame and the new current frame by the receiving end, so that the fluctuations of the frame interval originally appearing in the frame submitted for decoding can be eliminated, which can effectively ensure the smoothness of the client when presenting the streaming media content, thereby ensuring the user experience.

[0074] In some embodiments, after the aforementioned step 220, the method further includes: if the current frame is not an abnormal frame, the client submits the current frame based on a normal submission time.

[0075] If the acquisition interval is less than or equal to the interval threshold, it means that the acquisition interval between the current frame and the previous frame is normal or its fluctuation is within a controllable range. In other words, even if the acquisition interval fluctuates, the fluctuation is small and will not have a significant impact on the streaming media content presented by the client, so the fluctuation can be ignored. Accordingly, the client can determine that the current frame is not an abnormal frame, and can regard the current frame as a normal frame and directly submit the current frame based on the normal submission time.

[0076] In some embodiments, in order to allow the buffer algorithm to effectively cover the frame size fluctuation and network fluctuation during the period of submission for decoding, another data processing method of the client for streaming media is as follows Figure 4 As shown (the steps involved in the above embodiments will not be explained repeatedly), including:

[0077] Step 410: The client receives the current frame of the streaming media picture.

[0078] Step 420: The client calculates the normal submission time based on the expected receiving time and the buffering time.

[0079] Step 430: The client determines whether the normal submission time is greater than the current time.

[0080] Step 440: When the normal submission time is less than or equal to the current time, the client submits the current frame for decoding according to the normal submission.

[0081] If the current frame is large or network fluctuations occur during transmission, the client may receive the current frame later than the normal submission time. To solve this problem, the normal submission time can be compared with the current time to determine whether it is such a situation. If it is such a situation, the client can immediately submit the current frame for decoding to reduce the impact of frame size and network fluctuations. If it is not such a situation, step 450 can be executed.

[0082] Step 450: When the normal submission time is greater than the current time, the client determines whether the current frame is an abnormal frame according to the collection interval between the previous frame and the current frame.

[0083] Step 460: When the previous frame is not an abnormal frame, the client submits the current frame for decoding according to normal submission.

[0084] Step 470: When the previous frame is an abnormal frame, the client determines the optimized submission time of the current frame according to the collection interval and the normal submission time.

[0085] Step 480: The client submits the current frame for decoding according to the optimized submission time.

[0086] In this embodiment, the client can effectively cover the impact of frame size and network fluctuations through the buffer algorithm (steps 410 to 440); the collection interval fluctuation hiding algorithm (steps 450 to 480) can effectively reduce the impact of frame interval fluctuations, so that the client can improve the smoothness of the streaming media images presented by the client from different aspects, thereby ensuring the user experience.

[0087] After applying this embodiment, the client can reduce the decoding interval fluctuation from different aspects, such as Figure 5 As shown in the figure, the decoding interval is uniform and the fluctuation is small. Especially when a large acquisition interval occurs on the server side, the client can distribute the fluctuation caused by the abnormal acquisition interval to the current frame and the new current frame, thereby avoiding a large decoding interval of the current frame, effectively improving the reliability of client decoding, improving the fluency of streaming media content, and ensuring the user experience.

[0088] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0089] Corresponding to the streaming media data processing method of the above embodiment, Figure 6 A structural block diagram of a streaming media data processing device 6 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0090] Reference Figure 6 , the streaming media data processing device 6 comprises:

[0091] A first determination module 61 is used to determine whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture; when the acquisition interval is greater than the interval threshold, the current frame is an abnormal frame;

[0092] A second determination module 62 is used to determine the optimized submission time of the current frame according to the collection interval and the preset buffer time if the current frame is an abnormal frame;

[0093] The submission module 63 is used to submit the current frame for decoding according to the optimized submission time.

[0094] Optionally, the second determining module 62 includes:

[0095] A first determining unit is used to determine a normal submission time of the current frame according to a receiving time and a buffering time of the current frame; the optimized submission time is earlier than the normal submission time, and the difference between the optimized submission time and the normal submission time is determined according to a collection interval;

[0096] The second determining unit is configured to determine the optimized submission time based on the normal submission time and the collection interval.

[0097] Optionally, the second determining unit is specifically configured to:

[0098] Determine whether the time difference is less than an optimization time threshold, where the optimization time threshold is set based on the buffer time;

[0099] If the time difference is less than the optimization time threshold, the optimized submission time is the difference between the normal submission time and 0.5 times the time difference;

[0100] If the time difference is greater than or equal to the optimization time threshold, the optimized submission time is the difference between the normal submission time and 0.5 times the optimization time threshold.

[0101] Optionally, the submission module 63 may also be used to:

[0102] After the current frame is submitted for decoding based on the optimized submission time, if the new current frame of the streaming media screen is not an abnormal frame, the new current frame is submitted for decoding based on the new optimized submission time, and the difference between the normal submission time of the new current frame and the new optimized submission time is 0.5 times the time difference or 0.5 times the optimization time threshold.

[0103] Optionally, the data processing device 6 further includes:

[0104] A third determination module is used to determine whether the normal submission time is less than or equal to the reception time of the current frame before determining whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture;

[0105] The submission module 63 may also be configured to: submit the current frame for decoding if the normal submission time is less than or equal to the receiving time.

[0106] Optionally, the interval threshold is dynamically determined based on network information.

[0107] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0108] Figure 7 This is a schematic diagram of the physical structure of a client provided in an embodiment of the present application. Figure 7 As shown, the client 7 of this embodiment includes: at least one processor 70 ( Figure 7Only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on at least one processor 70. When the processor 70 executes the computer program 72, the steps in any of the above-mentioned streaming media data processing method embodiments are implemented, for example Figure 2 Steps 210-240 are shown.

[0109] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0110] In some embodiments, the memory 71 may be an internal storage unit of the client 7, such as a hard disk or memory of the client 7. In other embodiments, the memory 71 may also be an external storage device of the client 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the client 7.

[0111] Furthermore, the memory 71 may also include both an internal storage unit of the client 7 and an external storage device. The memory 71 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0112] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0113] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0114] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing streaming media data, characterized in that: Applied to the client, the data processing method includes: Determine whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture; if the acquisition interval is greater than the interval threshold, the current frame is the abnormal frame; If the current frame is the abnormal frame, determining the optimized submission time of the current frame according to the acquisition interval and the preset buffer time; The current frame is submitted for decoding according to the optimized submission time.

2. The method for processing streaming media data according to claim 1, characterized in that: Determining the optimized submission time of the current frame according to the acquisition interval and the preset buffer time includes: Determine the normal submission time of the current frame according to the reception time of the current frame and the buffering time; the optimized submission time is earlier than the normal submission time, and the difference between the optimized submission time and the normal submission time is determined according to the acquisition interval; The optimized submission time is determined based on the normal submission time and the collection interval.

3. The data processing method according to claim 2, characterized in that: The determining the optimized submission time based on the normal submission time and the collection interval includes: Determine the time difference of abnormal fluctuation based on the collection interval and a preset normal collection interval; The optimized commit time is determined based on the time difference and the normal commit time.

4. The data processing method according to claim 3, characterized in that: Determining the optimized submission time based on the time difference and the normal submission time includes: determining whether the time difference is less than an optimization time threshold, the optimization time threshold being set based on the buffer time; If the time difference is less than the optimization time threshold, the optimized submission time is the difference between the normal submission time and 0.5 times the time difference; If the time difference is greater than or equal to the optimization time threshold, the optimized submission time is the difference between the normal submission time and 0.5 times the optimization time threshold.

5. The data processing method according to claim 4, characterized in that: After submitting the current frame for decoding based on the optimized submission time, the method further includes: When the new current frame of the streaming media screen is not the abnormal frame, the new current frame is submitted for decoding based on the new optimized submission time, and the difference between the normal submission time of the new current frame and the new optimized submission time is 0.5 times the time difference or 0.5 times the optimization time threshold.

6. The data processing method according to any one of claims 2 to 5, characterized in that: Before determining whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture, the method further includes: Determining whether the normal submission time is less than or equal to the reception time of the current frame; If the normal submission time is less than or equal to the receiving time, the current frame is submitted for decoding.

7. The data processing method according to any one of claims 1 to 4, characterized in that: The interval threshold is dynamically determined based on network information.

8. A streaming media data processing device, characterized in that: include: A first determination module is used to determine whether the current frame is an abnormal frame according to the acquisition interval between the previous frame and the current frame of the received streaming media picture; In the case where the acquisition interval is greater than the interval threshold, the current frame is the abnormal frame; A second determination module, configured to determine, if the current frame is the abnormal frame, an optimized submission time of the current frame according to the acquisition interval and a preset buffer time; A submission module is used to submit the current frame for decoding according to the optimized submission time.

9. A client comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the streaming media data processing method according to any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by one or more processors, the streaming media data processing method according to any one of claims 1 to 7 is implemented.

Images