Multimedia data processing method, related equipment and storage medium

By obtaining the decoding performance test information of the terminal device and creating matching target rendering threads, the rendering delay problem caused by decoding performance fluctuations during multimedia data decoding and rendering is solved, and a smoother multimedia data output is achieved.

CN118900345APending Publication Date: 2024-11-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310494816.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-04
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In the process of multimedia data decoding and rendering, the decoding performance of terminal devices fluctuates greatly, resulting in rendering delays, affecting the output effect.

Method used

By obtaining the decoding performance test information of the terminal device, creating matching target rendering threads, processing decoding and rendering threads separately, avoiding the impact of single-threaded waiting and decoding time-consuming fluctuations, and controlling the maximum cached frame number of rendering threads to optimize the rendering effect.

Benefits of technology

Reduce rendering delay, improve the output effect of multimedia data, avoid excessive discarding of rendering threads or cache excessive data frames, and improve the rendering performance of terminal devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118900345A_ABST
    Figure CN118900345A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a multimedia data processing method, related equipment and a storage medium. The method comprises the following steps: acquiring decoding performance test information of terminal equipment for decoding and rendering multimedia application data; calling a decoding thread to sequentially decode each frame of application data to obtain decoded data corresponding to each frame of application data; in the process of sequentially sending the decoded data corresponding to each frame of application data to the target rendering thread, if the frame number of the decoded data cached in the target rendering thread is smaller than or equal to the maximum cache frame number, calling the target rendering thread to cache and render the corresponding decoded data so as to obtain corresponding rendering data, and if the frame number of the decoded data cached in the target rendering thread is smaller than or equal to the maximum cache frame number, sending the corresponding rendering data to the target rendering thread. Otherwise, performing rejection processing on the corresponding decoded data. The technical scheme of the embodiment of the invention can adapt to the decoding performance of the terminal equipment so as to reduce the rendering delay.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a method for processing multimedia data, a device for processing multimedia data, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] After acquiring multimedia data, the terminal device needs to decode and render the multimedia data, and finally output the rendered multimedia data. At present, multimedia data is mainly decoded and rendered by synchronous collaboration of decoding and rendering (that is, decoding and rendering the same frame of data in the same thread). However, this method has certain requirements on the decoding performance of the terminal device. Once the decoding time of the terminal device fluctuates greatly (that is, the decoding performance is poor), it will cause a large delay for the subsequent rendering process, resulting in poor output effect of the final multimedia data. Therefore, how to adapt the decoding performance of the terminal device to reduce the rendering delay is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a method for processing multimedia data, a device for processing multimedia data, an electronic device, a computer-readable storage medium, and a computer program product, which can adapt the decoding performance of a terminal device to reduce rendering delay.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a method for processing multimedia data is provided, including:

[0006] Obtaining decoding performance test information of a terminal device that performs decoding and rendering on multimedia application data; wherein the multimedia application data includes multiple frames of application data, and the decoding performance test information includes a maximum number of cached frames of a target rendering thread that matches the decoding performance of the terminal device;

[0007] Call the decoding thread to decode each frame of application data in turn to obtain the decoded data corresponding to each frame of application data;

[0008] In the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames, the target rendering thread is called to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected.

[0009] According to one aspect of an embodiment of the present application, a multimedia data processing device is provided, the device comprising an acquisition unit and a decoding and rendering unit, wherein:

[0010] An acquisition unit, configured to acquire decoding performance test information of a terminal device for decoding and rendering multimedia application data; wherein the multimedia application data includes multiple frames of application data, and the decoding performance test information includes a maximum cache frame number of a target rendering thread that matches the decoding performance of the terminal device;

[0011] The decoding and rendering unit is used to call the decoding thread to sequentially decode the application data of each frame to obtain the decoded data corresponding to the application data of each frame;

[0012] The decoding and rendering unit is also used to call the target rendering thread to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data if the number of frames of the decoded data cached in the target rendering thread is less than or equal to the maximum cached frame number during the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, otherwise the corresponding decoded data is rejected.

[0013] In one embodiment of the present application, when acquiring decoding performance test information of a terminal device that decodes and renders multimedia application data, the acquisition unit can be specifically used to: acquire multimedia reference data and a reference decoding time required for the terminal device to decode each frame of data; wherein the reference decoding time is obtained based on the display frame rate of the terminal device; decode each frame reference data in the multimedia reference data to obtain a decoding time corresponding to each frame reference data; and obtain the decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame reference data, and the total number of frames of reference data contained in the multimedia reference data.

[0014] In one embodiment of the present application, when the acquisition unit obtains the decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data contained in the multimedia reference data, it can be specifically used to: obtain the sum of the number of reference data frames whose decoding time is greater than the reference decoding time; and obtain the decoding performance test information based on the sum of the number of frames and the sum of the number of reference data frames.

[0015] In one embodiment of the present application, when the acquisition unit obtains the decoding performance test information based on the total number of frames and the sum of the reference data frame numbers, it can be specifically used to: determine the proportion of the sum of the reference data frame numbers to the total number of frames; if the proportion is greater than a preset proportion threshold, determine the first preset number of frames as the maximum cache frame number; if the proportion is less than or equal to the preset proportion threshold, determine the second preset number of frames as the maximum cache frame number; the second preset number of frames is less than the first preset number of frames; and generate the decoding performance test information based on the determined maximum cache frame number.

[0016] In one embodiment of the present application, when the acquisition unit obtains the decoding performance test information based on the total number of frames and the sum of the reference data frame numbers, it can be specifically used to: determine the proportion of the sum of the reference data frame numbers to the total number of frames; based on the correspondence between the preset number of frames and the proportion, obtain the number of frames corresponding to the determined proportion as the maximum cache frame number; and generate the decoding performance test information based on the obtained maximum cache frame number.

[0017] In one embodiment of the present application, the multimedia application data is multimedia data in a specified application; when acquiring multimedia reference data, the acquisition unit can be specifically used to: acquire the data volume of the installation package of the specified application; if the acquired data volume is greater than the data volume of the multimedia reference data, acquire the multimedia reference data from the local storage space of the terminal device; if the acquired data volume is less than or equal to the data volume of the multimedia reference data, send a reference data acquisition request to the server corresponding to the specified application, and receive the multimedia reference data returned by the server.

[0018] In one embodiment of the present application, before obtaining the decoding performance test information of the terminal device for decoding and rendering multimedia application data, the acquisition unit can be further used to: obtain the generation time of the historical performance test information stored in the terminal device; if the time difference between the current time and the generation time is less than or equal to a preset time threshold, render the multimedia application data based on the historical performance test information; if the time difference between the current time and the generation time is greater than the preset time threshold, obtain the decoding performance test information.

[0019] In one embodiment of the present application, the terminal device sets corresponding decoding performance test information for different applications respectively; when the acquisition unit acquires the decoding performance test information of the terminal device for decoding and rendering multimedia application data, it can be specifically used to: if a startup operation for a specified application is detected, then the decoding performance test information corresponding to the specified application is acquired.

[0020] In one embodiment of the present application, the multimedia data processing device also includes a sending and receiving unit and a display unit; the sending and receiving unit is used to send an application data acquisition request to the server corresponding to the specified application if a display request for the multimedia application data is detected, so that the server responds to the application data acquisition request and sends the multimedia application data to the terminal device; the sending and receiving unit is also used to receive multimedia application data; the display unit is used to display the rendering data after obtaining the corresponding rendering data.

[0021] According to one aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the multimedia data processing method as described above.

[0022] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the method for processing multimedia data as described above.

[0023] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the method for processing multimedia data as described above.

[0024] In the technical solution provided in the embodiment of the present application, by creating a decoding thread and a target rendering thread respectively, the decoding process and the rendering process are processed by two threads respectively. It can be seen that the embodiment of the present application avoids the situation in which the next frame can only be decoded after waiting for decoding and rendering of a frame of data during single-threaded processing, which is beneficial to reduce the impact of large fluctuations in decoding time on the decoding process and rendering process, thereby achieving the purpose of reducing processing delays. At the same time, the embodiment of the present application will also obtain decoding performance test information to create a target rendering thread that matches the decoding performance of the terminal device, thereby avoiding the situation in which the rendering thread excessively discards data frames or caches too many data frames, which is beneficial to improving rendering effects and reducing rendering delays.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 This is a schematic diagram of a decoding and rendering synchronization process provided by an embodiment of the present application.

[0028] Figure 2 is a structural diagram of a multimedia data processing system provided in an embodiment of the present application;

[0029] Figure 3 is a flowchart of a method for processing multimedia data shown in an exemplary embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a rendering frame loss process shown by an exemplary embodiment of the present application;

[0031] Figure 5 is a schematic diagram of a rendering delay process shown by an exemplary embodiment of the present application;

[0032] Figure 6 is a flowchart of another method for processing multimedia data shown in an exemplary embodiment of the present application;

[0033] Figure 7 is a schematic diagram of a process for determining a maximum number of cache frames shown in an exemplary embodiment of the present application;

[0034] Figure 8 is a schematic diagram of a processing process of a decoding performance detection module shown in an exemplary embodiment of the present application;

[0035] Fig. 9 is a structural block diagram of a multimedia data processing device shown in an exemplary embodiment of the present application;

[0036] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

[0040] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0041] Multimedia data encoding refers to the method of converting multimedia data files in the original data format into multimedia data files in another data format through compression technology, wherein the converted multimedia data can also be called a data stream. Multimedia data decoding refers to the reverse process of multimedia data encoding. At the same time, multimedia data rendering is a process of converting multimedia data into specific sensory information such as images, videos, and audio.

[0042] Specifically, multimedia data encoding, decoding and rendering can be applied to real-time interactive products, such as cloud applications such as cloud games and cloud conferences, as well as interactive live broadcasts and video calls. Among them, cloud applications refer to applications that interact between terminal devices and servers (also known as the cloud). Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. At the same time, cloud conferencing is an efficient, convenient and low-cost conference format based on cloud computing technology. Users only need to use the Internet interface and perform simple and easy-to-use operations to quickly and efficiently share voice, data files and videos with teams and customers around the world.

[0043] In related technologies, multimedia data is usually decoded and rendered in a synchronous and collaborative way. Specifically, synchronous and collaborative decoding and rendering refers to decoding and rendering the same frame of data in the same thread, that is, after decoding a frame of data, the frame of data will be rendered in the same thread; if the rendering of the frame of data is not completed, the thread cannot decode the next frame of data.

[0044] In actual application, please refer to the attached Figure 1 , shows a schematic diagram of a decoding and rendering synchronization process. Figure 1 As shown, the electronic device can create a decoding and rendering thread through the decoder to decode and render the multimedia data. The black arrow is used to indicate "sending frames to the decoder"; when the multimedia data can be obtained normally, the frame sending interval is relatively stable. For example, the frame sending interval can be in milliseconds (ms); then, the frame sending interval = 1000 / frame sending frame rate, where the frame sending frame rate (frame_rate) is in fps, which refers to the number of frames output per second.

[0045] In addition, if Figure 1 As shown in the figure, the gray square is the time taken to decode the nth frame data, and the white square is the time taken to render the nth frame, where n is a positive integer. Figure 1 It can be seen that the decoding and rendering of a frame of data must be completed before the next frame of data of the frame of data can be decoded and rendered. Figure 1 The 5th frame has been sent to the decoder, but the decoding and rendering of the 4th frame has not been completed, so the 5th frame cannot be decoded and rendered. Figure 1As shown in the figure, when the 8th frame data is input, the rendering of the 6th frame data has not yet been completed, and there will be a processing delay of more than 2 frames. It can be seen that when decoding and rendering multimedia data in a synchronous and coordinated manner, a large processing delay will be caused due to the large fluctuation in decoding time.

[0046] Based on this, the embodiment of the present application provides a multimedia data processing solution, which first obtains decoding performance test information to determine the maximum number of cached frames of the target rendering thread that should be created. Then, a decoding thread and a corresponding target rendering thread are created to complete the decoding and rendering of multimedia application data through the decoding thread and the target rendering thread respectively.

[0047] The decoding thread is used to sequentially decode each frame of the multimedia application data to obtain the decoded data corresponding to each frame of the application data; at the same time, the decoded data corresponding to each frame of the application data is sent to the target rendering thread. The target rendering thread is used to cache and render the corresponding decoded data to obtain the corresponding rendering data only when the number of frames of the decoded data cached by it is less than or equal to the maximum number of cached frames, otherwise it refuses to process the corresponding decoded data.

[0048] In addition, multimedia application data refers to multimedia data in an application; multimedia application data may include multiple frames of application data. The decoding performance test information includes the maximum cache frame number of the target rendering thread that matches the decoding performance of the terminal device.

[0049] It is not difficult to see that this solution creates a decoding thread and a target rendering thread separately, so that the decoding process and the rendering process are handled by two threads respectively. After the decoding thread completes decoding of a frame of data, it can directly decode the next frame, and the target rendering thread performs subsequent rendering processing. It can be seen that this solution avoids the situation in which a single-threaded processing needs to wait for a frame of data to be decoded and rendered before decoding the next frame, which is conducive to reducing the impact of large fluctuations in decoding time on decoding and rendering processes, thereby achieving the purpose of reducing processing delays.

[0050] At the same time, when the maximum number of cached frames of the rendering thread does not match the decoding performance of the terminal device, the rendering thread will discard data frames excessively, resulting in a decrease in the rendering frame rate, which will affect the rendering effect; or the rendering thread will cache too many data frames, increasing the hardware burden of the terminal device, resulting in a decrease in the rendering performance of the terminal device, which will increase the fluctuation of subsequent rendering time and increase rendering delay. Therefore, this solution avoids the situation where the rendering thread discards data frames excessively or caches too many data frames by creating a target rendering thread that matches the decoding performance of the terminal device, which is conducive to improving rendering effects and reducing rendering delays.

[0051] Based on the above multimedia data processing solution, the present application embodiment provides a multimedia data processing system, which can be seen in Figure 2 , Figure 2 The processing system of multimedia data shown may include multiple terminal devices 201 and multiple servers 202. Among them, a communication connection is established between any terminal device and any server. For example, the terminal device 201 may include any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car-mounted device, and a smart wearable device. Multimedia playback clients, cloud game clients, social clients, browser clients, information stream clients, educational clients, etc. can be run in the terminal device 201. Server 202 can be a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Any terminal device 201 and any server 202 can be directly or indirectly connected in communication via wired or wireless communication, and this application is not limited here.

[0052] In some embodiments, the above-mentioned method for processing multimedia data may be performed by only Figure 2 The terminal device 201 in the processing system of the multimedia data shown is executed, and the specific execution process is: the terminal device 201 can obtain the decoding performance test information of the terminal device 201, and the decoding performance test information includes the maximum cache frame number of the target rendering thread that matches the decoding performance of the terminal device 201. Afterwards, the terminal device 201 can call the decoding thread to decode each frame of application data in turn to obtain the decoded data corresponding to each frame of application data. Finally, in the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in turn, if the frame number of the decoded data cached in the target rendering thread is less than or equal to the maximum cache frame number, the terminal device 201 can call the target rendering thread to cache and render the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected. Optionally, after obtaining the corresponding rendering data, the terminal device 201 can also output the rendering data.

[0053] In other embodiments, the multimedia data processing method can be run in a multimedia data processing system, which can include a terminal device and a server. Figure 2The multimedia data processing system shown in the figure is jointly completed by the terminal device 201 and the server 202, and the specific execution process is: if the terminal device 201 detects a display request for multimedia application data, it sends an application data acquisition request to the server 202; then, the server 202 responds to the received application data acquisition request and acquires the corresponding multimedia application data. After that, the server 202 can send the multimedia application data to the terminal device 201. Optionally, the display request can carry the identification information of the multimedia application data, and the server 202 will store the identification information and the multimedia application data in correspondence, so that the server 202 can find the corresponding multimedia application data according to the identification information in the display request.

[0054] In the process of receiving the aforementioned multimedia application data, the terminal device 201 can obtain the decoding performance test information of the terminal device 201, and the decoding performance test information includes the maximum cache frame number of the target rendering thread that matches the decoding performance of the terminal device 201. Afterwards, the terminal device 201 can call the decoding thread to decode each frame of application data in the multimedia application data in turn, and obtain the decoded data corresponding to each frame of application data. Finally, in the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in turn, if the frame number of the decoded data cached in the target rendering thread is less than or equal to the maximum cache frame number, the terminal device 201 can call the target rendering thread to cache and render the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected. Optionally, after obtaining the corresponding rendering data, the terminal device 201 can also output the rendering data.

[0055] It should be noted that the embodiments of the present application can process various multimedia data such as images, videos, audios, etc., and can be applied to various scenarios, including but not limited to cloud technology, AI (Artificial Intelligence), smart home, assisted driving and other scenarios. It can also be used to process multimedia data in various applications such as cloud gaming applications, video applications, live broadcast applications, conference applications, etc., without limitation.

[0056] Specifically, the technical solution of the embodiment of the present application can be applied to smart home scenarios, such as the terminal device can be set as a smart TV, and the server can be a server that provides services for the smart TV. For example, a video application is running in the smart TV, and the smart TV can respond to the selection operation of a TV series in the video application and send a display request for the video data of the TV series to the server; then, the server responds to the display request for the video data of the TV series and obtains the corresponding video data. Finally, the server sends the video data to the smart TV. At the same time, the smart TV can obtain its own decoding performance test information to create a video rendering thread that matches the decoding performance test information. In addition, the smart TV can also create a video decoding thread.

[0057] After the smart TV receives the video data, the smart TV can call the video decoding thread to decode each frame of the video data in turn to obtain the decoded data corresponding to each frame of the video data. In the process of sending the decoded data corresponding to each frame of the video data to the video rendering thread in turn, if the number of frames of the decoded data cached in the video rendering thread is less than or equal to the maximum number of cached frames, the smart TV can call the video rendering thread to cache and render the corresponding decoded data to obtain the corresponding video rendering data, otherwise the corresponding decoded data is rejected. Finally, the smart TV displays the video rendering data.

[0058] For another example, the technical solution of the embodiment of the present application is applied to a live broadcast application, a live broadcast client may be running in the terminal device, the server is a cloud server of the live broadcast client, and the multimedia application data may be the live broadcast data in the live broadcast client. After the live broadcast client receives the live broadcast data sent by the cloud server, the live broadcast client may obtain the decoding performance test information of the terminal device and create a live broadcast rendering thread that is consistent with the decoding performance test information; at the same time, the live broadcast client may also create a live broadcast decoding thread.

[0059] Then, the live broadcast client can call the live broadcast decoding thread to decode the live broadcast data in sequence to obtain the decoded data corresponding to each frame of the live broadcast data. In the process of sending the decoded data corresponding to each frame of the live broadcast data to the live broadcast rendering thread in sequence, if the number of frames of the decoded data cached in the live broadcast rendering thread is less than or equal to the maximum number of cached frames, the live broadcast client can call the live broadcast rendering thread to cache and render the corresponding decoded data to obtain the corresponding live broadcast rendering data, otherwise the corresponding decoded data is rejected. Finally, the live broadcast client displays the live broadcast rendering data.

[0060] It should be noted that in the specific implementation of this application, if multimedia data, multimedia application data and other related data or information are related to objects, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data or information need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0061] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0062] like Figure 3 As shown, Figure 3 is a flowchart of a method for processing multimedia data according to an embodiment of the present application, which can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by the terminal device, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method executed by the terminal device is taken as an example for description, wherein the multimedia data processing method may include S301 to S303, which are described in detail as follows:

[0063] S301: Obtain decoding performance test information of a terminal device that performs decoding and rendering on multimedia application data.

[0064] In the embodiment of the present application, multimedia application data refers to multimedia data in an application; multimedia application data may include multiple frames of application data. Specifically, since multimedia data is played in a frame-by-frame manner, multiple frames of application data in the multimedia application data may be arranged in the order in which each frame of application data is played.

[0065] In addition, the decoding performance test information includes the maximum number of cached frames of the target rendering thread that matches the decoding performance of the terminal device. Specifically, if the decoding performance of the terminal device is normal (i.e., the decoding time fluctuation is small, and the decoding time of each frame of data is similar), then the maximum number of cached frames can be relatively small; if the decoding performance of the terminal device is poor (i.e., the decoding time fluctuation is large), then the maximum number of cached frames can be relatively large.

[0066] In one embodiment, a specific method for obtaining decoding performance test information of a terminal device may be: the terminal device obtains multimedia reference data containing multiple frames of reference data, and the terminal device decodes each frame of reference data by calling a decoding thread to obtain a decoding time corresponding to each frame of reference data; the terminal device may analyze the degree of fluctuation in decoding time of the terminal device according to the decoding time corresponding to each frame of reference data; finally, the terminal device may use the number of frames corresponding to the analyzed degree of fluctuation in decoding time as the maximum number of cached frames based on a pre-set correspondence between the degree of fluctuation in decoding time and the number of frames.

[0067] The decoding time fluctuation degree is used to characterize the average difference between the decoding times corresponding to adjacent reference data. In addition, the predetermined corresponding relationship between the decoding time fluctuation degree and the number of frames can be set by the R&D personnel based on their own experience in the field of data decoding and rendering; or it can be set by the terminal device or server in the above multimedia data processing system based on the historical decoding and rendering of the terminal device, and there is no limitation on this.

[0068] Exemplarily, it can be set that when the average difference between the decoding times corresponding to adjacent reference data is greater than the preset fluctuation difference, the fluctuation degree of the decoding time is large; and when the average difference between the decoding times corresponding to adjacent reference data is less than or equal to the preset fluctuation difference, the fluctuation degree of the decoding time is small. The preset fluctuation difference can be set manually or by the terminal device or server in the multimedia data processing system, which is not limited to this. At the same time, the number of frames corresponding to the large fluctuation degree of the decoding time can be set to 2, and the number of frames corresponding to the small fluctuation degree of the decoding time can be set to 1.

[0069] Optionally, the decoding time fluctuation degree can also be classified into normal and abnormal; it can also be classified into multiple degree categories such as relatively small, small, medium, relatively large, and large, without limitation. Optionally, the decoding time fluctuation degree can also be directly represented by the average difference between the decoding times corresponding to adjacent reference data.

[0070] In a possible implementation, since the decoding performance of the terminal device is also affected by the memory resources, the more memory resources the terminal device has, the better the decoding performance of the terminal device; however, the memory resources of the terminal device are usually limited. If the terminal device has already run multiple applications in the background before running a certain application, the memory resources that the terminal device can allocate to the application will be relatively small. In other words, the terminal device will allocate corresponding memory resources to different applications. Since the memory resources allocated to different applications are not exactly the same, the terminal device can set corresponding decoding performance test information for different applications; wherein the decoding performance test information corresponding to different applications is not exactly the same.

[0071] Then, the specific method of obtaining the decoding performance test information of the terminal device that decodes and renders multimedia application data may be: if a startup operation for a specified application is detected, the decoding performance test information corresponding to the specified application is obtained. The specified application refers to any one of the multiple applications running in the terminal device. The specific process of obtaining the decoding performance test information corresponding to the specified application can be found in the above-mentioned specific process of obtaining the decoding performance test information of the terminal device, which will not be repeated here. Specifically, each time the terminal device starts an application, it is necessary to obtain a decoding performance test information as the decoding performance test information corresponding to the application started this time by repeating the above-mentioned specific process of obtaining the decoding performance test information of the terminal device.

[0072] S302 , calling a decoding thread to sequentially decode each frame of application data to obtain decoded data corresponding to each frame of application data.

[0073] In the embodiment of the present application, the specific method of calling the decoding thread to sequentially decode each frame of application data to obtain the decoded data corresponding to each frame of application data can be: the terminal device creates a decoding thread, and the terminal device calls the decoding thread to sequentially decode each frame of application data to obtain the decoded data corresponding to each frame of application data. It should be noted that since the process of decoding data is a technical means commonly used by those skilled in the art, the specific decoding process will not be described in detail here.

[0074] In one embodiment, since the application data of each frame in the multimedia application data is arranged in a playback order as mentioned in step S301, the specific method of calling the decoding thread to decode each frame of application data in sequence and obtaining the decoded data corresponding to each frame of application data may also be: in the process of traversing multiple frames of application data according to the arrangement order of each frame of application data in the multimedia application data, calling the decoding thread to decode the current frame of application data currently traversed to obtain the decoded data corresponding to the current frame of application data. Among them, only after obtaining the decoded data corresponding to the current frame of application data, will the decoding thread be called to decode the application data of the next frame of traversal.

[0075] Optionally, the multimedia application data may be multimedia data in a designated application. The designated application may request the server corresponding to the designated application to obtain the multimedia application data in real time only when it detects that the user wants to watch or listen to the multimedia application data. Therefore, before calling the decoding thread to sequentially decode each frame of application data in the multimedia application data, if the terminal device detects a display request for the multimedia application data, the terminal device may send an application data acquisition request to the server corresponding to the designated application, so that the server corresponding to the designated application responds to the application data acquisition request and sends the multimedia application data to the terminal device.

[0076] Optionally, a request to display multimedia application data may be generated by a terminal device when it detects that a user performs a click operation, a selection operation, or an edit operation on information about multimedia application data (such as data identifiers and icons of multimedia application data) in an application interface of a specified application.

[0077] Furthermore, since multiple frames of data or only one frame of data may be transmitted at a time during real-time transmission, the server corresponding to the designated application may send each frame of application data in the multimedia application data to the terminal device in sequence, and the terminal device may immediately call the decoding thread to decode each frame of application data in sequence after receiving each frame of application data to obtain the decoded data corresponding to each frame of application data.

[0078] S303. In the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames, the target rendering thread is called to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected.

[0079] In the embodiment of the present application, the rejection process may be that the target rendering thread refuses to cache the decoded data, or the target rendering thread deletes the decoded data after refusing to cache the decoded data.

[0080] In one embodiment, since the decoding process of the decoding thread mentioned in step S302 is performed in accordance with the arrangement order of each frame of application data in the multimedia application data, the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence is: after the terminal device calls the decoding thread to obtain the decoded data corresponding to the current frame of application data, the decoded data corresponding to the current frame of application data is sent to the target rendering thread.

[0081] Optionally, the terminal device mentioned in step S302 may start decoding and rendering the multimedia application data only after responding to a request for displaying the multimedia application data; therefore, after obtaining the corresponding rendering data, the terminal device may also display the obtained rendering data.

[0082] In the embodiment of the present application, by creating a decoding thread and a target rendering thread respectively, the decoding process and the rendering process are processed by two threads respectively. After the decoding thread completes decoding of a frame of data, the next frame can be directly decoded, and the target rendering thread performs subsequent rendering processing. It can be seen that the embodiment of the present application avoids the situation in which the next frame can only be decoded after waiting for a frame of data to be decoded and rendered during single-thread processing, which is conducive to reducing the impact of large fluctuations in decoding time on decoding and rendering, thereby achieving the purpose of reducing processing delays.

[0083] However, in the specific implementation, please refer to the attached Figure 4 , showing a schematic diagram of a rendering frame loss process. The decoder of the terminal device includes a decoding thread 401 and a rendering thread 402; Figure 4 The maximum number of cached frames of the rendering thread 402 in is 1. Figure 4 As shown, the terminal device can create a decoding and rendering thread through the decoder to decode and render the multimedia data. Among them, the black arrow is used to indicate "sending frames to the decoder"; the dotted arrow is used to indicate that the data has been "decoded and sent for rendering". When each frame of the multimedia data can be obtained normally, the frame sending interval is relatively stable. In addition, the gray square is the decoding time required to decode the nth frame of data, and the white square is the rendering time required to render the nth frame of data, and n is a positive integer.

[0084] like Figure 4 As shown, after decoding a frame of data, the decoding thread 401 can decode the next frame of data without waiting for rendering to be completed. Figure 4 When the second frame data is sent to the decoding thread 401 in the decoder, the first frame data in the rendering thread 402 has not been rendered yet. However, the decoding thread 401 can decode the second frame immediately after receiving the second frame data without waiting for the first frame data to be rendered. This is very different from the synchronous coordination of decoding and rendering, which can reduce the rendering delay caused by the decoding delay, thereby reducing the overall processing delay.

[0085] At the same time, if Figure 4 As shown in the figure, when the decoding thread 401 finishes decoding the 4th frame data and sends it to the rendering thread 402 for rendering, the rendering thread 402 has not finished rendering the 3rd frame data, and the maximum number of cached frames of the rendering thread 402 is 1; therefore, the rendering thread 402 will refuse to cache the 4th frame data, so that the 4th frame data is discarded, that is, the "rendering frame loss" situation occurs. Similarly, the subsequent 6th frame data also has a rendering frame loss, which will not be described in detail here.

[0086] In addition, by Figure 4 It can be seen that the rendering process of the 3rd frame data has not been completed yet, mainly because the decoding time of the 3rd frame data is too long relative to the decoding time of the adjacent frame data, that is, the decoding time jitter is large, and the decoding performance of the terminal device is poor. Similarly, the reason why the rendering process of the 5th frame data has not been completed yet is the same as that of the 3rd frame data, which will not be repeated here. Since multimedia application data is played frame by frame, once the rendering frame is lost, the rendering frame rate will drop; and if the rendering frame rate drops too much, the multimedia data output will be not smooth.

[0087] Therefore, it can be Figure 4 Consider whether the maximum number of cached frames of the rendering thread can be set larger for terminal devices with poor decoding performance to avoid excessive rendering frame losses. For example, the maximum number of cached frames of the rendering thread 402 can be set to 2. When the decoding of the 4th frame data is completed, the rendering of the 3rd frame data has not yet ended, but because the rendering thread 402 can save a total of two frames of data, the rendering thread 402 can save the 4th frame data and wait until the rendering of the 3rd frame data is completed before immediately rendering the 4th frame data. At this time, the rendering frame rate will increase compared to when the maximum number of cached frames of the rendering thread 402 is 1.

[0088] Also, see the attached Figure 5 , showing a schematic diagram of a rendering delay process. The decoder of the terminal device includes a decoding thread 501 and a rendering thread 502; Figure 5 The maximum number of cached frames of the rendering thread 502 in is 2. Figure 5 As shown, the terminal device can create a decoding and rendering thread through the decoder to decode and render the multimedia data. Among them, the black arrow is used to indicate "sending frames to the decoder"; the dotted arrow is used to indicate that the data has been "decoded and sent for rendering". When the data of each frame in the multimedia data can be obtained normally, the frame sending interval is relatively stable. In addition, the gray square is the decoding time required to decode the nth frame data, and the white square is the rendering time required to render the nth frame, and n is a positive integer.

[0089] like Figure 5 As shown, the decoding time jitter of the decoding thread 501 is small, indicating that the decoding performance of the terminal device is normal; however, the rendering time of the rendering thread 502 for rendering each frame of data is long, and there is a situation of rendering time jitter. Figure 5 As shown, when the decoding of the nth frame data is completed, the rendering of the n-1th frame data has not yet been completed; where n is a positive integer. At this time, since the maximum number of cache frames of the rendering thread 502 is 2, the rendering thread 502 will retain the nth frame data and wait until the rendering of the n-1th frame data is completed before rendering the nth frame data. In this case, the rendering of each frame of data will be delayed by 1 frame or more, resulting in an increase in rendering delay.

[0090] In addition, according to engineering experience, Figure 5 As shown, the rendering thread 502 is almost always rendering without any rest, which will increase the burden on the hardware and lead to worse rendering performance of subsequent terminal devices, that is, the probability of rendering time jitter increases, resulting in a vicious cycle.

[0091] Therefore, it can be Figure 5Consider whether, for a terminal device with normal decoding performance, the maximum cache frame number of the rendering thread can be set to a smaller value to avoid the rendering thread caching too many data frames for rendering, thereby increasing the hardware burden of the terminal device and ultimately causing the rendering performance of the terminal device to deteriorate. Figure 5 In the example, the maximum number of cached frames of the rendering thread 502 is set to 1. Then, when the decoding of the second frame data is completed, the rendering of the first frame data is not completed. At this time, the frame loss can be appropriately rendered, that is, the rendering thread 502 refuses to cache the second frame data. Then, after the rendering thread 502 completes the decoding of the third frame data, the third frame data can be rendered immediately. It can be seen that in some cases, a smaller maximum number of cached frames of the rendering thread will significantly reduce the rendering delay, and at the same time allow the rendering thread to have idle time, and the probability of subsequent terminal device rendering jitter will also be reduced, and the rendering performance will not be further deteriorated.

[0092] Depend on Figure 4 and Figure 5 It can be seen that when the maximum number of cached frames of the rendering thread does not match the decoding performance of the terminal device, the rendering thread will excessively discard data frames, resulting in a decrease in the rendering frame rate, thereby affecting the rendering output effect; or the rendering thread will cache too many data frames, increasing the hardware burden of the terminal device, resulting in a decrease in the rendering performance of the terminal device, thereby increasing the fluctuation of subsequent rendering time and increasing the rendering delay. Therefore, in an embodiment of the present application, by obtaining decoding performance test information, a target rendering thread that matches the decoding performance of the terminal device is created to avoid the situation where the rendering thread excessively discards data frames or caches too many data frames, which is conducive to improving the rendering effect and reducing the rendering delay.

[0093] In one embodiment of the present application, another method for processing multimedia data is provided. The method for processing multimedia data can be applied to Figure 1 The multimedia data processing system shown in the figure can be executed by the terminal device, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is described by taking the method executed by the terminal device as an example. Figure 6 FIG. 1 is a flow chart showing another method for processing multimedia data. The method for processing multimedia data is Figure 3 This paper expands on the method shown in .

[0094] Among them, S601 to S605 are described in detail as follows:

[0095] S601, obtaining multimedia reference data and a reference decoding time required by a terminal device to decode each frame of data.

[0096] In an embodiment of the present application, the multimedia reference data is multimedia data used for decoding processing, and the multimedia reference data includes multiple frames of reference data.

[0097] In addition, the reference decoding time is obtained based on the display frame rate of the terminal device. Specifically, the display frame rate is the same as the above-mentioned frame rate, and the reference decoding time can be the same as the above-mentioned frame interval. For example, if the display frame rate of the terminal device is set to 60fps, then the reference decoding time can be 1000 / 60, which is about 16.67ms.

[0098] In one embodiment, since it is difficult for the decoding performance of a terminal device to suddenly deteriorate or improve in a short period of time, the decoding performance of the terminal device is usually changed due to hardware damage, long-term non-use of the terminal device, etc. Therefore, if the terminal device has previously tested the decoding performance of the terminal device and obtained historical performance test information of the terminal device, the terminal device can store the historical performance test information of the terminal device in a local storage space so that it can be used continuously for a period of time later.

[0099] The historical performance test information includes the maximum cached frame number of the historical rendering thread that matches the historical decoding performance of the terminal device. In addition, since the decoding performance of the terminal device mentioned above is variable, the historical decoding performance of the terminal device refers to the decoding performance of the terminal device at the time point when the historical performance test information is generated.

[0100] Therefore, before obtaining the multimedia reference data and reference decoding time, you can also: obtain the generation time of the historical performance test information stored in the terminal device; if the time difference between the current time and the generation time is less than or equal to the preset time threshold, render the multimedia application data based on the historical performance test information; if the time difference between the current time and the generation time is greater than the preset time threshold, obtain the decoding performance test information.

[0101] The preset time threshold may be set by a developer according to the variation of the decoding performance of the terminal device, or may be set by the terminal device or server in the multimedia data processing system according to the variation of the decoding performance of the terminal device, without limitation. For example, the preset time threshold may be 2 days, one week, half a month, etc., without limitation.

[0102] Optionally, referring to steps S302 to S303, the specific process of rendering multimedia application data based on historical performance test information may be: the terminal device creates a historical rendering thread based on the historical performance test information; the terminal device calls the decoding thread to decode each frame of application data in turn to obtain decoded data corresponding to each frame of application data; in the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in turn, if the number of frames of decoded data cached in the historical rendering thread is less than or equal to the maximum cached frame number of the historical rendering thread, the terminal device calls the historical rendering thread to cache and render the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected.

[0103] Optionally, after acquiring the decoding performance test information of the terminal device, the terminal device may store the acquired decoding performance test information as historical performance test information in the local memory space. At the same time, the terminal device may also store the generation time of the decoding performance test information as the generation time of the historical performance test information in the local memory space.

[0104] S602: Decode each frame of reference data in the multimedia reference data to obtain a decoding time corresponding to each frame of reference data.

[0105] In an embodiment of the present application, the specific method of decoding each frame reference data in the multimedia reference data to obtain the decoding time corresponding to each frame reference data can be: the terminal device creates a test decoding thread; the terminal device calls the test decoding thread to decode each frame reference data in the multimedia reference data to obtain the decoding time corresponding to each frame reference data.

[0106] Optionally, after the test decoding thread completes the decoding processing of all reference data in the multimedia reference data, the terminal device may delete the test decoding thread from the memory resources.

[0107] S603: Obtain decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data included in the multimedia reference data.

[0108] In an embodiment of the present application, based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data contained in the multimedia reference data, a specific method for obtaining decoding performance test information may be: obtaining the sum of the number of reference data frames whose decoding time is greater than the reference decoding time; and obtaining decoding performance test information based on the sum of the number of frames and the sum of the number of reference data frames.

[0109] In one embodiment, a specific method for obtaining decoding performance test information based on the total number of frames and the sum of the number of reference data frames may be: determining the proportion of the sum of the number of reference data frames to the total number of frames; if the proportion is greater than a preset proportion threshold, determining the first preset number of frames as the maximum number of cached frames; if the proportion is less than or equal to the preset proportion threshold, determining the second preset number of frames as the maximum number of cached frames; and generating decoding performance test information based on the determined maximum number of cached frames.

[0110] The larger the reference data frame number and the proportion of the total frame number, the longer the actual time for the terminal device to decode a frame of application data is, that is, the worse the decoding performance of the terminal device is. Figure 4 The example mentioned that for a terminal device with poor decoding performance, the maximum cache frame number can be set to be larger. Therefore, the second preset frame number is smaller than the first preset frame number.

[0111] Specifically, the first preset frame number can be set by a human or a terminal device or server in a multimedia data processing system to determine the maximum cache frame number of a rendering thread required for a terminal device with poor decoding performance through experimental testing; similarly, the second preset frame number can be set by a human or a terminal device or server to determine the maximum cache frame number of a rendering thread required for a terminal device with normal or good decoding performance through experimental testing. For example, the first preset frame number can be 2, and the second preset frame number can be 1; for another example, the first preset frame number can be 3, and the second preset frame number can be 2, etc., without limitation.

[0112] In addition, the preset proportion threshold is a dividing line for dividing the decoding performance of the terminal device. Therefore, the first terminal device and the second terminal device can be selected to respectively decode the reference multimedia data to obtain the decoding time of each frame reference data corresponding to the first terminal device and the decoding time of each frame reference data corresponding to the second terminal device; wherein the decoding performance of the first terminal device is better than the decoding performance of the second terminal device. Afterwards, the first number of multiple decoding times corresponding to the first terminal device that is greater than the reference decoding time and the second number of multiple decoding times corresponding to the second terminal device that is greater than the reference decoding time can be determined. Finally, the preset proportion threshold can be determined by the proportion of the first number to the total number of frames and the proportion of the second number to the total number of frames. Among them, the preset proportion threshold can be greater than or equal to the proportion of the first number to the total number of frames and less than or equal to the proportion of the second number to the total number of frames.

[0113] Optionally, the preset percentage threshold may be set manually based on one's own experience in decoding using a terminal device, or may be set by a terminal device or a server in the multimedia data processing system, without limitation. For example, the preset percentage threshold may be 20%, 0.1 or 35%.

[0114] In one embodiment, a specific method for obtaining decoding performance test information based on the total number of frames and the sum of the number of reference data frames may also be: determining the proportion of the sum of the number of reference data frames to the total number of frames; based on the correspondence between the preset number of frames and the proportion, obtaining the number of frames corresponding to the determined proportion as the maximum cache frame number; and generating decoding performance test information based on the obtained maximum cache frame number.

[0115] Specifically, the correspondence between the number of frames and the proportion can be set manually based on one's own experience of using the terminal device for decoding, or can be set by the terminal device or server in the above-mentioned multimedia data processing system, and there is no limitation on this. For example, the corresponding number of frames when the proportion is 0-20% can be set to 1, the corresponding number of frames when the proportion is 21-40% can be set to 2, the corresponding number of frames when the proportion is 41-60% can be set to 3, the corresponding number of frames when the proportion is 61-80% can be set to 4, and the corresponding number of frames when the proportion is 81-100% can be set to 5. For another example, the corresponding number of frames when the proportion is 0-10% can be set to 1, the corresponding number of frames when the proportion is 10-25% can be set to 2, and the corresponding number of frames when the proportion is 26-100% can be set to 3, and there is no limitation on the correspondence between the number of frames and the proportion.

[0116] In one possible implementation, based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data contained in the multimedia reference data, a specific method for obtaining decoding performance test information may also be: obtaining the total number of frames of reference data whose decoding time is less than or equal to the reference decoding time; and obtaining decoding performance test information based on the total number of frames and the total number of frames.

[0117] Furthermore, the specific method of obtaining the decoding performance test information based on the sum of the number of frames and the total number of frames can refer to the specific implementation method of obtaining the decoding performance test information based on the sum of the number of frames and the sum of the number of frames of the reference data; it is just that the larger the proportion of the total number of frames to the total number of frames, the shorter the time it takes for the terminal device to actually decode a frame of application data, that is, the better the decoding performance of the terminal device. Therefore, the correspondence between the proportion and the number of frames when obtaining the decoding performance test information based on the sum of the number of frames and the total number of frames is opposite to the correspondence between the proportion and the number of frames when obtaining the decoding performance test information based on the sum of the number of frames and the sum of the number of frames of the reference data, which will not be elaborated here.

[0118] In a possible implementation, the multimedia application data may be multimedia data in a specified application. Then, the specific method of obtaining the multimedia reference data may be: obtaining the data volume of the installation package of the specified application; if the obtained data volume is greater than the data volume of the multimedia reference data, obtaining the multimedia reference data from the local storage space of the terminal device; if the obtained data volume is less than or equal to the data volume of the multimedia reference data, sending a reference data acquisition request to the server corresponding to the specified application, and receiving the multimedia reference data returned by the server.

[0119] Specifically, since the data volume of the installation package of some applications is relatively small, such applications will not carry multimedia reference data with a data volume greater than that of their own installation package and store it in the local storage space of the terminal device; therefore, such applications need to obtain multimedia reference data by sending a reference data acquisition request to the corresponding server. Similarly, if the data volume of the installation package of some applications is relatively large, greater than the data volume of the multimedia reference data, then carrying multimedia reference data is feasible for such applications; therefore, the multimedia reference data of such applications can be carried through the installation package and stored in the local storage space of the terminal device.

[0120] Furthermore, since the multimedia reference data is not important data required by the specified application during runtime, the multimedia reference data may be carried in the installation package only when the data volume of the installation package of the specified application is much larger than the data volume of the multimedia reference data, that is, when the data volume of the multimedia reference data will not cause a significant increase in the data volume of the installation package.

[0121] Then, the specific method of obtaining multimedia reference data can be: obtaining the data volume of the installation package of the specified application; if the data volume difference between the obtained data volume and the data volume of the multimedia reference data is greater than the preset data volume, then obtaining the multimedia reference data from the local storage space of the terminal device; if the data volume difference between the obtained data volume and the data volume of the multimedia reference data is less than or equal to the preset data volume, then sending a reference data acquisition request to the server corresponding to the specified application, and receiving the multimedia reference data returned by the server. Among them, the preset data volume can be set manually, or it can be set by the terminal device or server in the above-mentioned multimedia processing system, which is not limited to this.

[0122] Optionally, the specific process of obtaining the decoding performance test information corresponding to the specified application mentioned in step S301 can also refer to the specific process of obtaining the decoding performance test information in steps S601 to S603, which is not repeated here.

[0123] For specific implementation, please refer to the attached Figure 7 , shows a schematic diagram of a process for determining the maximum number of cache frames. Figure 7 As shown, the terminal device will execute step 701 to determine whether the decoding performance detection module has been called within n days, where n is a positive integer, such as 7, 15, etc. The decoding performance detection module is used to generate decoding performance test information, so the specific way to determine whether the decoding performance detection module has been entered within n days can be to determine whether the time difference between the generation time of the last decoding performance test information (i.e., historical performance test information) generated by the terminal device and the current time exceeds n days.

[0124] like Figure 7 As shown in FIG. 7 , if the decoding performance detection module has not been called within n days, the terminal device will execute step 702 to obtain multimedia reference data; then as shown in steps 703 and 704, the terminal device will input the multimedia reference data into the decoding performance detection module, and the decoding performance detection module will process the multimedia reference data to obtain decoding performance test information. Finally, as shown in step 705, the terminal device can create a rendering thread based on the maximum number of cached frames included in the decoding performance test information obtained in step 704.

[0125] like Figure 7 As shown, if the decoding performance detection module has been called within n days, the terminal device will execute step 706 to obtain historical performance test information from the local storage space; then, as shown in step 708, the terminal device will further create a rendering thread based on the maximum number of cached frames included in the historical performance test information.

[0126] Finally, as shown in step 708, the terminal device performs subsequent rendering processing based on the created rendering thread.

[0127] For further details, please refer to the attached Figure 8 , showing a schematic diagram of the processing process of a decoding performance detection module. Wherein, i indicates that the currently decoded data frame is the i-th frame reference data in the multimedia reference data, and i is a positive integer; cnt_slow is a decoding slow frame counter, which is used to record the number of reference data frames whose current decoding time is greater than the reference decoding time; decode_time_i is used to indicate the decoding time of the i-th frame reference data; THR is used to indicate the preset proportion threshold, and in this example, the preset proportion threshold can be set to 0.2. In addition, size_pro is used to indicate the maximum number of cached frames. In this example, the first preset proportion can be set to 2, and the second preset proportion can be set to 1.

[0128] As shown in steps 801 and 802, while obtaining the multimedia reference data, the decoding performance detection module also determines the number of reference data frames included in the multimedia reference data; then, the decoding performance detection module further obtains the display frame rate of the terminal device (i.e., frame_rate), and calculates the reference decoding time according to the display frame rate, where the reference decoding time can be equal to 1000 / frame_rate.

[0129] Then, as shown in step 803, the decoding performance detection module initializes i to 1 and initializes cnt_slow to 0. Then, as shown in steps 804 to 805, when i < n, the decoding performance detection module decodes the i-th frame of reference data and records decode_time_i.

[0130] Furthermore, as shown in steps 806 to 808, if decode_time_i is greater than the reference decoding time, the decoding performance detection module will increment cnt_slow by 1; for example, if the decoding time of the first frame of reference data is greater than the reference decoding time, since cnt_slow is initialized to 0, then cnt_slow++ = 0 + 1 = 1. At the same time, if decode_time_i is less than or equal to the reference decoding time, the decoding performance detection module will directly increment i by 1 to decode the next frame of reference data.

[0131] As shown in steps 804 and 809, when i < n, it means that all frames of reference data in the multimedia reference data have been decoded. Then, the decoding performance detection module will determine whether cnt_slow / n is greater than THR, that is, determine whether the proportion of the sum of reference frames to the total number of frames is greater than the preset proportion threshold.

[0132] As shown in steps 810 and 811, if cnt_slow / n is greater than THR, the decoding performance detection module will assign size_pro to 2; if cnt_slow / n is less than or equal to THR, the decoding performance detection module will assign size_pro to 1. Finally, as shown in step S812, the decoding performance detection module will generate decoding performance test information according to the determined size_pro.

[0133] S604. Call the decoding thread to decode each frame of application data in the multimedia application data in sequence to obtain the decoded data corresponding to each frame of application data.

[0134] In the embodiment of the present application, the specific implementation manner of step S604 can refer to the specific implementation manner of step S302 in the above embodiment, and will not be elaborated here.

[0135] S605. In the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames, the target rendering thread is called to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected.

[0136] In the embodiment of the present application, the specific implementation of step S605 can refer to the specific implementation of step S303 in the above embodiment, which will not be repeated here.

[0137] In the embodiment of the present application, by creating a decoding thread and a target rendering thread respectively, the situation in which the next frame can be decoded only after waiting for decoding and rendering of a frame of data during single-thread processing is avoided, which is conducive to reducing the impact of large fluctuations in decoding time on decoding and rendering processing, thereby achieving the purpose of reducing processing delays. In addition, the embodiment of the present application also obtains decoding performance test information to create a target rendering thread that matches the decoding performance of the terminal device, thereby avoiding the situation in which the rendering thread excessively discards data frames or caches too many data frames, which is conducive to improving rendering effects and reducing rendering delays.

[0138] In addition, the embodiment of the present application can obtain the decoding time corresponding to each frame reference data by decoding the multimedia reference data, and the decoding time corresponding to each frame reference data can directly and intuitively reflect the decoding performance of the current terminal device; at the same time, the reference decoding time can be used as a standard for measuring the terminal device with normal decoding performance. Therefore, the embodiment of the present application can compare the decoding performance of the current terminal device with the terminal device with normal decoding performance by comparing the reference decoding time and the decoding time corresponding to each frame reference data, so as to judge whether the decoding performance of the current terminal device is normal or poor. Then, the decoding performance test information generated based on the reference decoding time and the decoding time corresponding to each frame reference data will be affected by the decoding performance of the terminal device and is matched with the decoding performance of the terminal device. It can be seen that in the embodiment of the present application, by comparing the reference decoding time and the decoding time corresponding to each frame reference data, it is possible to achieve simple, intuitive and rapid acquisition of decoding performance test information that matches the decoding performance of the terminal device.

[0139] Here, the device embodiment of the present application is introduced, which can be used to execute the method for processing multimedia data in the above-mentioned embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the embodiment of the method for processing multimedia data in the above-mentioned embodiment of the present application.

[0140] The present application embodiment provides a multimedia data processing device, such as Fig. 9As shown, the device includes an acquisition unit 901 and a decoding and rendering unit 902, wherein:

[0141] The acquisition unit 901 is used to acquire decoding performance test information of a terminal device for decoding and rendering multimedia application data; wherein the multimedia application data includes multiple frames of application data, and the decoding performance test information includes a maximum cache frame number of a target rendering thread that matches the decoding performance of the terminal device;

[0142] The decoding and rendering unit 902 is used to call the decoding thread to sequentially decode each frame of application data to obtain decoded data corresponding to each frame of application data;

[0143] The decoding and rendering unit 902 is also used to call the target rendering thread to cache and render the corresponding decoded data to obtain the corresponding rendering data if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames during the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, otherwise the corresponding decoded data is rejected.

[0144] In one embodiment of the present application, based on the aforementioned scheme, when acquiring decoding performance test information of a terminal device that decodes and renders multimedia application data, the acquisition unit 901 can be specifically used to: acquire multimedia reference data and a reference decoding time required for the terminal device to decode each frame of data; wherein the reference decoding time is obtained based on a display frame rate of the terminal device; decode each frame of reference data in the multimedia reference data to obtain a decoding time corresponding to each frame of reference data; and obtain decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data contained in the multimedia reference data.

[0145] In one embodiment of the present application, based on the aforementioned scheme, when the acquisition unit 901 obtains the decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of the reference data contained in the multimedia reference data, it can be specifically used to: obtain the sum of the number of reference data frames whose decoding time is greater than the reference decoding time; and obtain the decoding performance test information based on the total number of frames and the sum of the number of reference data frames.

[0146] In one embodiment of the present application, based on the aforementioned scheme, when the acquisition unit 901 obtains the decoding performance test information based on the total number of frames and the sum of the number of reference data frames, it can be specifically used to: determine the proportion of the sum of the number of reference data frames to the total number of frames; if the proportion is greater than a preset proportion threshold, determine the first preset number of frames as the maximum cache frame number; if the proportion is less than or equal to the preset proportion threshold, determine the second preset number of frames as the maximum cache frame number; the second preset number of frames is less than the first preset number of frames; and generate decoding performance test information based on the determined maximum cache frame number.

[0147] In one embodiment of the present application, based on the aforementioned scheme, when the acquisition unit 901 obtains the decoding performance test information based on the total number of frames and the sum of the number of reference data frames, it can be specifically used to: determine the proportion of the sum of the number of reference data frames to the total number of frames; based on the correspondence between the preset number of frames and the proportion, obtain the number of frames corresponding to the determined proportion as the maximum number of cached frames; and generate the decoding performance test information based on the obtained maximum number of cached frames.

[0148] In one embodiment of the present application, the multimedia application data is the multimedia data in the specified application; based on the aforementioned scheme, when acquiring the multimedia reference data, the acquisition unit 901 can be specifically used to: acquire the data volume of the installation package of the specified application; if the acquired data volume is greater than the data volume of the multimedia reference data, acquire the multimedia reference data from the local storage space of the terminal device; if the acquired data volume is less than or equal to the data volume of the multimedia reference data, send a reference data acquisition request to the server corresponding to the specified application, and receive the multimedia reference data returned by the server.

[0149] In one embodiment of the present application, based on the aforementioned scheme, before obtaining the decoding performance test information of the terminal device for decoding and rendering multimedia application data, the acquisition unit 901 can be further used to: obtain the generation time of the historical performance test information stored in the terminal device; if the time difference between the current time and the generation time is less than or equal to the preset time threshold, render the multimedia application data based on the historical performance test information; if the time difference between the current time and the generation time is greater than the preset time threshold, obtain the decoding performance test information.

[0150] In one embodiment of the present application, the terminal device sets corresponding decoding performance test information for different applications; based on the above-mentioned scheme, when the acquisition unit 901 acquires the decoding performance test information of the terminal device for decoding and rendering multimedia application data, it can be specifically used to: if a startup operation for a specified application is detected, then the decoding performance test information corresponding to the specified application is acquired.

[0151] In one embodiment of the present application, the multimedia data processing device also includes a sending and receiving unit 903 and a display unit 904; based on the aforementioned scheme, the sending and receiving unit 903 is used to send an application data acquisition request to a server corresponding to a specified application if a display request for multimedia application data is detected, so that the server responds to the application data acquisition request and sends the multimedia application data to the terminal device; the sending and receiving unit 903 is also used to receive multimedia application data; the display unit 904 is used to display the rendering data after obtaining the corresponding rendering data.

[0152] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0153] The device provided in the above-mentioned embodiment can be arranged in the terminal device, and can also be arranged in the server. The device provided in the embodiment of the present application can create a decoding thread and a target rendering thread respectively, so that the decoding process and the rendering process are processed by two threads respectively. It can be seen that the embodiment of the present application avoids the situation that the next frame can only be decoded after waiting for the decoding and rendering of a frame of data during single-threaded processing, which is beneficial to reduce the impact of large fluctuations in decoding time on the decoding process and the rendering process, thereby achieving the purpose of reducing processing delays. At the same time, the embodiment of the present application will also obtain decoding performance test information to create a target rendering thread that matches the decoding performance of the terminal device, so as to avoid the situation where the rendering thread excessively discards data frames or caches too many data frames, which is beneficial to improving rendering effects and reducing rendering delays.

[0154] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above-mentioned method for processing multimedia data.

[0155] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0156] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0157] like Fig.10As shown, the computer system 1000 includes a processor (Central Processing Unit, CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (Random Access Memory, RAM) 1003, such as executing the method in the above embodiment. In RAM 1003, various programs and data required for system operation are also stored. CPU 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Input / output (Input / Output, I / O) interface 1005 is also connected to bus 1004.

[0158] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0159] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a processor (CPU) 1001, various functions defined in the system of the present application are executed.

[0160] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0162] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0163] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method for processing multimedia data as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0164] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the multimedia data processing method as described above in the above embodiments.

[0165] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0166] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0167] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for processing multimedia data, characterized in that: include: Obtaining decoding performance test information of a terminal device that performs decoding and rendering on multimedia application data; wherein the multimedia application data includes multiple frames of application data, and the decoding performance test information includes a maximum number of cached frames of a target rendering thread that matches the decoding performance of the terminal device; Call the decoding thread to decode each frame of application data in turn to obtain the decoded data corresponding to each frame of application data; In the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames, the target rendering thread is called to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data, otherwise the corresponding decoded data is rejected.

2. The method according to claim 1, characterized in that The obtaining decoding performance test information of a terminal device for decoding and rendering multimedia application data includes: Acquire multimedia reference data and a reference decoding time required for the terminal device to decode each frame of data; wherein the reference decoding time is obtained based on a display frame rate of the terminal device; Decoding each frame of reference data in the multimedia reference data to obtain a decoding time corresponding to each frame of reference data; The decoding performance test information is obtained based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data included in the multimedia reference data.

3. The method according to claim 2, characterized in that The obtaining the decoding performance test information based on the reference decoding time, the decoding time corresponding to each frame of reference data, and the total number of frames of reference data included in the multimedia reference data includes: Obtaining a sum of reference data frames whose decoding time is greater than the reference decoding time; The decoding performance test information is obtained according to the total number of frames and the total number of reference data frames.

4. The method according to claim 3, characterized in that The obtaining the decoding performance test information according to the sum of the number of frames and the sum of the number of reference data frames includes: Determining the number of reference data frames and the proportion of the reference data frames to the total number of frames; If the proportion is greater than a preset proportion threshold, determining the first preset number of frames as the maximum cached number of frames; If the proportion is less than or equal to the preset proportion threshold, determining the second preset frame number as the maximum cache frame number; the second preset frame number is less than the first preset frame number; The decoding performance test information is generated according to the determined maximum number of cache frames.

5. The method according to claim 3, characterized in that: The obtaining the decoding performance test information according to the sum of the number of frames and the sum of the number of reference data frames includes: Determining the number of reference data frames and the proportion of the reference data frames to the total number of frames; Based on the correspondence between the preset frame number and the proportion, the frame number corresponding to the determined proportion is obtained as the maximum cache frame number; The decoding performance test information is generated according to the acquired maximum number of cache frames.

6. The method according to claim 2, characterized in that The multimedia application data is multimedia data in a specified application; The acquiring of multimedia reference data comprises: Obtaining the data volume of the installation package of the specified application; If the amount of data obtained is greater than the amount of data of the multimedia reference data, obtaining the multimedia reference data from the local storage space of the terminal device; If the acquired data volume is less than or equal to the data volume of the multimedia reference data, a reference data acquisition request is sent to the server corresponding to the designated application, and the multimedia reference data returned by the server is received.

7. The method according to claim 1, characterized in that The multimedia application data is multimedia data in a specified application; the method further comprises: If a display request for the multimedia application data is detected, sending an application data acquisition request to the server corresponding to the specified application, so that the server responds to the application data acquisition request and sends the multimedia application data to the terminal device; receiving the multimedia application data; After obtaining the corresponding rendering data, the method further includes: The rendering data is displayed.

8. The method according to any one of claims 1 to 7, characterized in that: Before obtaining the decoding performance test information of the terminal device for decoding and rendering the multimedia application data, the method further includes: Obtaining the generation time of the historical performance test information stored in the terminal device; If the time difference between the current time and the generation time is less than or equal to a preset time threshold, rendering the multimedia application data based on the historical performance test information; If the time difference between the current time and the generated time is greater than the preset time threshold, the decoding performance test information is obtained.

9. The method according to any one of claims 1 to 7, characterized in that: The terminal device sets corresponding decoding performance test information for different applications; The obtaining decoding performance test information of a terminal device for decoding and rendering multimedia application data includes: If a startup operation of a designated application is detected, decoding performance test information corresponding to the designated application is obtained.

10. A multimedia data processing device, characterized in that: It includes an acquisition unit and a decoding and rendering unit, wherein: The acquisition unit is used to acquire decoding performance test information of a terminal device that performs decoding and rendering on multimedia application data; wherein the multimedia application data includes multiple frames of application data, and the decoding performance test information includes a maximum cache frame number of a target rendering thread that matches the decoding performance of the terminal device; The decoding and rendering unit is used to call the decoding thread to sequentially decode each frame of application data to obtain decoded data corresponding to each frame of application data; The decoding and rendering unit is further configured to, during the process of sending the decoded data corresponding to each frame of application data to the target rendering thread in sequence, if the number of frames of decoded data cached in the target rendering thread is less than or equal to the maximum number of cached frames, call the target rendering thread to perform caching and rendering processing of the corresponding decoded data to obtain the corresponding rendering data; otherwise, reject the corresponding decoded data.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing multimedia data according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the multimedia data processing method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is suitable for being loaded by a processor and executing the method for processing multimedia data according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Video processing method, terminal device and machine-readable storage medium

    CN110832875A

  • Low-delay video rendering method and device for Android terminal

    CN113923507A

  • Low-delay video rendering method and device at iOS end

    CN114205662A

  • Video playing method and device and electronic equipment

    CN115209216A

  • Display processing method and device, computer equipment and readable storage medium

    CN115278288A