Terminal multimedia data processing method, device, computer equipment, and storage medium

Abstract

The present application relates to a method, apparatus, computer device, storage medium, and computer program product for processing multimedia data at a terminal. The method comprises: obtaining attribute information of multimedia data and analyzing the attribute information; based on the analysis results, reading the multimedia data to obtain read content; monitoring the network conditions of the terminal; if the monitored network conditions meet preset network conditions, adjusting the initial decoding mode, decoding the read content based on the adjusted decoding mode to obtain first decoded content; and displaying the first decoded content. This method enables the optimal decoding scheme to be used for decoding when the terminal is in abnormal network conditions to ensure the smoothness of the decoded content.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a terminal multimedia data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the development of multimedia technology, Ffmpeg (a multimedia framework) live broadcast technology has emerged. It is currently one of the most widely used open source programs in the field of computer audio and video streaming media processing. It has very powerful functions, including video acquisition, video format conversion, video capture, video watermarking, etc.

[0003] However, the current Ffmpeg live broadcast technology still has some defects, especially when the network conditions of the terminal are poor, the smoothness of the video will be greatly affected. Summary of the Invention

[0004] Based on this, it is necessary to provide a terminal multimedia data processing method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0005] In order to solve the above technical problems, the present application provides a terminal multimedia data processing method. The method includes:

[0006] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0007] Based on the analysis result, the multimedia data is read to obtain read content;

[0008] Monitor the network conditions of the terminal;

[0009] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, and decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0010] The first decoded content is displayed.

[0011] In one embodiment, a set scene parameter is obtained, wherein the scene parameter is used to adjust the number of key frames detected in the multimedia data;

[0012] The acquiring attribute information of the multimedia data and analyzing the attribute information includes:

[0013] Acquire attribute information of the multimedia data, and analyze the attribute information based on the scene parameters.

[0014] In one embodiment, if the monitored network condition meets a preset network condition, adjusting the initial decoding mode includes:

[0015] If the monitored network condition meets the preset network condition, at least one of the decoding method and the decoding rate in the initial decoding mode is adjusted.

[0016] In one embodiment, if the monitored network condition satisfies a preset network condition, adjusting at least one of a decoding mode and a decoding rate in an initial decoding mode includes:

[0017] If the monitored network condition meets the preset network condition, determining whether the current decoding mode is a hard decoding mode;

[0018] If the current decoding mode is a hard decoding mode, the decoding rate of the hard decoding mode is changed until the display smoothness of the first decoded content meets the preset smoothness.

[0019] In one embodiment, if the monitored network condition satisfies a preset network condition, adjusting at least one of a decoding mode and a decoding rate in an initial decoding mode includes:

[0020] If the monitored network condition meets the preset network condition, determining whether the current decoding mode is a hard decoding mode;

[0021] If the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

[0022] In one embodiment, if the monitored network condition does not meet the preset network condition, the read content is decoded based on the initial decoding mode to obtain second decoded content; and the second decoded content is displayed.

[0023] In order to solve the above technical problems, the present application also provides a terminal multimedia data processing device.

[0024] The device comprises:

[0025] An analysis module, configured to obtain attribute information of multimedia data and analyze the attribute information;

[0026] A reading module, configured to read the multimedia data based on the analysis result to obtain read content;

[0027] A monitoring module is used to monitor the network conditions of the terminal;

[0028] a decoding module, configured to adjust an initial decoding mode if the monitored network condition satisfies a preset network condition, and decode the read content based on the adjusted decoding mode to obtain first decoded content;

[0029] A display module is configured to display the first decoded content.

[0030] In some embodiments of the present application, a computer device may be provided. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0031] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0032] Based on the analysis result, the multimedia data is read to obtain read content;

[0033] Monitor the network conditions of the terminal;

[0034] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, and decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0035] The first decoded content is displayed.

[0036] In some embodiments of the present application, a computer-readable storage medium may be provided. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0038] Based on the analysis result, the multimedia data is read to obtain read content;

[0039] Monitor the network conditions of the terminal;

[0040] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, and decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0041] The first decoded content is displayed.

[0042] In this application, by monitoring the network conditions of the terminal, when it is detected that the network conditions of the terminal meet the preset network conditions, the initial decoding mode is automatically adjusted, and the read content is decoded using the adjusted decoding mode. Therefore, when the terminal is in an abnormal network condition, the optimal decoding scheme is used for decoding to ensure the fluency of the decoded content. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a diagram of an application environment of a terminal multimedia data processing method in one embodiment;

[0044] Figure 2 1 is a flow chart of a method for processing multimedia data of a terminal according to an embodiment;

[0045] Figure 3 1 is a flow chart of a method for processing multimedia data of a terminal according to another embodiment;

[0046] Figure 4 A schematic diagram of a video buffer in one embodiment;

[0047] Figure 5 Schematic diagram of expanding video support formats in the module-lite.sh file in one embodiment;

[0048] Figure 6 A schematic diagram of the underlying logic for adding recording and screenshot functions to an embodiment;

[0049] Figure 7 This is a schematic diagram showing the effect of adding recording and screenshot functions in one embodiment;

[0050] Figure 8 A schematic diagram of analyzing attribute information in one embodiment;

[0051] Figure 9 A schematic diagram of the interaction between the decoding layer and the display layer in one embodiment;

[0052] Figure 10 A bottom-level logic diagram of the entire computer device in one embodiment;

[0053] Figure 11 This is a flow chart of a terminal multimedia data processing device in one embodiment;

[0054] Figure 12 A flow chart of a terminal multimedia data processing device in another embodiment;

[0055] Figure 13 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] The terminal multimedia data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, Figure 1 This is a diagram of the application environment of the terminal multimedia data processing method. Terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablet computers. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0058] In one embodiment, reference Figure 2 , Figure 2 This is a flow chart of the terminal multimedia data processing method. Figure 2 As shown, a terminal multimedia data processing method is provided, comprising the following steps:

[0059] Step 202: Acquire attribute information of multimedia data and analyze the attribute information.

[0060] Multimedia refers to the integration of multiple media, generally encompassing text, sound, images, and other media formats. In computer systems, multimedia refers to a combination of two or more media types for interactive human-computer information exchange and dissemination. Optionally, multimedia data is a live network stream.

[0061] The attribute information refers to characteristic information related to the content of the multimedia data. Optionally, the attribute information may be format information of the multimedia data, or address information of the multimedia data.

[0062] In one embodiment, analyzing the attribute information is analyzing the multimedia data itself.

[0063] Specifically, the system analyzes the video data's format to see if it supports MediaCodec (a codec interface). If so, it initializes the MediaCodec module, binds a surface (an interface for producers and consumers to exchange buffers), and initializes the video cache queue. If not, it initializes the FFmpeg module's video decoder, opens a video cache queue, configures OpenGL (a cross-programming language, cross-platform graphics programming interface), and associates the surface. Different processing steps are then performed based on the analysis results.

[0064] In one embodiment, before obtaining attribute information of multimedia data and analyzing the attribute information, a multimedia request instruction of a user is first received, and multimedia data corresponding to the multimedia request instruction is obtained.

[0065] 204. Based on the analysis result, the multimedia data is read to obtain read content.

[0066] Reading refers to transferring data stored in the hard disk into the system.

[0067] Optionally, the video data or audio data is read. Since the video is composed of frames of images, reading the video data is essentially reading the image data.

[0068] Specifically, open the multimedia data, create an audio and video decoding thread, cyclically read the decapsulated Packet (data packet) and store it in the corresponding queue; read the Packet in the queue, soft-decode a frame (a communication frame) and store it in frame_queue (a data structure); create (audio playback thread) aout_thread and read data from frame_quque through audio_cblk (data control block) for playback.

[0069] 206. Monitor the network conditions of the terminal.

[0070] The term "terminal" refers to a device that inputs programs and data into a computer or receives computer output processing results via a communication facility. It is readily understood that this application does not explicitly limit the specific form of the terminal, and the terminal may be, but is not limited to, various personal computers, laptops, mobile terminals, tablet computers, and the like.

[0071] In this embodiment, there is no explicit limitation on the specific network conditions. It is easy to understand that any network factor that can affect the terminal can be used as the network condition in this embodiment.

[0072] Optionally, the terminal's network environment is monitored to determine whether it is a weak network environment. For real-time multimedia playback, any network environment characterized by network complexity, heterogeneity, protocol non-standardization, network anomalies, network errors, network jitter, network latency, network packet loss, and network bandwidth impairments can be considered a weak network environment.

[0073] In this embodiment, there is no explicit limitation on the specific method for monitoring the network environment in which the terminal is located.

[0074] Optionally, a comprehensive network environment monitoring system is established based on the user's wireless network and mobile terminal traffic. The ping function return time is used to monitor the current network environment and provide real-time feedback and real-time adjustments.

[0075] 208. If the monitored network condition meets the preset network condition, adjust the initial decoding mode, and decode the read content based on the adjusted decoding mode to obtain first decoded content.

[0076] Among them, the preset network conditions are network conditions pre-set by the network environment monitoring system that can trigger automatic adjustment of the decoding mode. Then, in the specific application process, the network conditions monitored in real time by the network environment monitoring system are compared with the preset network conditions to determine whether to trigger automatic adjustment of the decoding mode.

[0077] The initial decoding mode refers to the default decoding mode when decoding the read multimedia data. It is easy to understand that there is no clear limitation on the initial decoding mode. Optionally, the initial decoding mode is the default hard decoding mode.

[0078] In this embodiment, the first decoded content refers to the situation where the network conditions monitored by the current network environment monitoring system meet the preset network conditions, which triggers the automatic adjustment of the decoding mode. The read multimedia data is decoded based on the adjusted decoding mode, and the decoded content obtained is called the first decoded content.

[0079] Optionally, the first decoding content is that the ping value monitored by the current network environment monitoring system (the time from the PC sending data to the network server to receiving feedback data from the server) meets the preset ping value, triggering automatic adjustment of the decoding mode, and decoding the read multimedia data based on the adjusted decoding mode. The decoded content obtained is called the first decoding content.

[0080] Optionally, the first decoding content is that the delay value monitored by the current network environment monitoring system meets the preset delay value, triggering automatic adjustment of the decoding mode, and decoding the read multimedia data based on the adjusted decoding mode, so the obtained decoding content is called the first decoding content.

[0081] Optionally, the first decoding content is that the packet loss rate monitored by the current network environment monitoring system meets the preset packet loss rate, triggering automatic adjustment of the decoding mode, and decoding the read multimedia data based on the adjusted decoding mode. The obtained decoding content is called the first decoding content.

[0082] Optionally, the first decoding content is the bandwidth value monitored by the current network environment monitoring system meeting a preset bandwidth value, triggering automatic adjustment of the decoding mode, and decoding the read multimedia data based on the adjusted decoding mode. The decoded content thus obtained is referred to as the first decoding content. It is easy to understand that the specific monitoring object can be unique or diverse, for example, only the ping value can be monitored, or both the ping value and the packet loss rate can be monitored. When the monitored ping value and packet loss rate both meet the preset ping value and the preset packet loss rate, the decoding mode adjustment is successfully triggered.

[0083] In one embodiment, if the monitored network condition meets the preset network condition, a balance is made between fluency and real-time performance by adjusting the frame rate to achieve a certain degree of effect optimization.

[0084] Specifically, in multimedia preview scenarios, the larger the frame rate, the larger the packet size. It is understandable that a 40fps video will require more bandwidth than a 25fps (frames per second). In a weak network environment, more waiting time is required. The default frame rate control of ffplay (the simple player that comes with the FFmpeg project) does not have a frame chasing strategy, so if the network jitters, the delay will accumulate. The above performance can be improved by adjusting the frame rate. In a weak network environment, the decoding rate can be reduced to shorten the image acquisition time. The specific implementation method is as follows: The specific approach is to obtain the local time of the machine when the decoding thread is running. The first frame calculates the time difference between the machine time and the stream pts (local time and stream time are synchronized). When the stream time lags far behind the local time, fast play or consider packet loss. The principle is to strike a balance between smoothness and real-time performance. The principle is to strike a balance between smoothness and real-time performance.

[0085] In one embodiment, an initial time period and a buffer zone are set, and the frame rate of the displayed content is appropriately reduced during the initial time period to prioritize the smoothness of the displayed content. Subsequently, as the cached data in the buffer zone accumulates, the frame rate of the displayed content is gradually increased.

[0086] In one embodiment, a set scene parameter is obtained, and the scene parameter is used to adjust the number of detected key frames; the attribute information of the multimedia data is obtained and the attribute information is analyzed, including: obtaining the attribute information of the multimedia data and analyzing the attribute information based on the scene parameter.

[0087] Specifically, the computer device obtains a set probesize (scene parameter) parameter, which is used to adjust the number of detected keyframes. The computer device then obtains attribute information of the multimedia data and analyzes and separates the attribute information based on the probesize parameter. When the multimedia data is first opened, the vformat find stream info function consumes a significant amount of time during the analysis and separation of the multimedia data. However, setting the probesize parameter effectively reduces the time spent analyzing and separating the multimedia data. The probesize parameter is closely related to the number of detected keyframes.

[0088] Optionally, the rtmp (live streaming protocol) live stream opens slowly. In actual use, the avformat_find_stream_info function will take up most of the time. Its main function is to analyze and split the live stream. Setting the probesize parameter will effectively save the time spent on stream analysis. The probesize parameter is used to set the maximum number of bytes read from the source file in bytes. The default value is 5000000. Reducing the probesize value in a good network environment can reduce latency. The probesize parameter can be appropriately associated with the number of key frames detected.

[0089] Since the parameters that can affect the detection of key frames of multimedia data have been optimized, the time spent on analyzing and splitting multimedia data can be effectively saved when the multimedia is just opened. When the user's multimedia opening request is received, the multimedia can be opened quickly, which also makes the multimedia display smoother and saves the user's waiting time.

[0090] In one embodiment, if the monitored network condition meets the preset network condition, adjusting the initial decoding mode includes: if the monitored network condition meets the preset network condition, adjusting at least one of the decoding method and the decoding rate in the initial decoding mode.

[0091] Specifically, if the monitored network conditions meet the preset network conditions, the computer device optimizes the fluency by adjusting the initial decoding method. Adjusting the initial decoding method includes adjusting the selection between the hard decoding method and the soft decoding method and the decoding rate. It is easy to understand that this embodiment does not explicitly limit whether the selection between the hard decoding method and the soft decoding method and the decoding rate are adjusted simultaneously, or the order in which the selection between the hard decoding method and the soft decoding method and the decoding rate are adjusted. As long as the fluency requirement is ultimately achieved, it is sufficient.

[0092] By adjusting the decoding rate and decoding method in the decoding mode, the decoding mode adjustment is made more diversified. Therefore, this decoding mode adjustment can cope with more complex network conditions.

[0093] In one embodiment, if the monitored network condition meets the preset network condition, it is determined whether the current decoding method is a hard decoding method; if the current decoding method is a hard decoding method, the hard decoding method is changed to a soft decoding method, so that the display smoothness of the first decoded content meets the preset smoothness.

[0094] In one embodiment, if the monitored network condition meets the preset network condition, at least one of the decoding method and the decoding rate in the initial decoding mode is adjusted, including: if the monitored network condition meets the preset network condition, determining whether the current decoding method is a hard decoding method; if the current decoding method is a hard decoding method, changing the decoding rate of the hard decoding method until the display smoothness of the first decoded content meets the preset smoothness.

[0095] Specifically, if the monitored network conditions meet the preset network conditions, the computer device determines whether the current decoding method is a hard decoding method; if hard encoding is currently used for encoding, that is, the decoding method at this time is a hard decoding method, then under this network environment, the computer device changes the hard-coded decoding rate to make the multimedia display smoother. It is easy to think that when poor network conditions are detected and frame loss occurs, a strategy of reducing the decoding rate can also be adopted. When the Android sdk (software development kit) version is greater than or equal to 19, the decoding rate of the multimedia data output by the hard encoder can be changed by passing parameters to MediaCodec (the native codec interface of the mobile operating system).

[0096] By first determining whether the current decoding method is a hard decoding method, and after determining that it is a hard decoding method, adjusting the decoding rate of the hard decoding method, when the network conditions are poor and the decoding method is still a hard decoding method, the multimedia display is made smoother by adjusting the decoding rate, until the display smoothness of the first decoded content meets the preset smoothness requirement.

[0097] In one embodiment, if the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode; if the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched back to the hard decoding mode, and the decoding rate of the hard decoding mode is adjusted until the display smoothness of the first decoded content meets the preset smoothness requirement.

[0098] In one embodiment, if the monitored network condition meets the preset network condition, at least one of the decoding mode and the decoding rate in the initial decoding mode is adjusted, including: if the monitored network condition meets the preset network condition, determining whether the current decoding mode is a hard decoding mode; if the current decoding mode is a non-hard decoding mode, switching the non-hard decoding mode to a soft decoding mode.

[0099] Specifically, if the monitored network conditions meet the preset network conditions, the computer device will determine that the current decoding method is not a hard decoding method, and then the computer device will adjust the current decoding method in real time and automatically switch to a soft decoding method. By successfully automatically switching to the soft decoding method, the multimedia display becomes smoother and meets the preset fluency requirements.

[0100] By adjusting the current decoding method in real time and automatically switching to soft decoding, when the network conditions are poor and the current decoding method is not hard decoding, the multimedia display can still be guaranteed to be smoother and meet the preset fluency requirements.

[0101] 210. Display the first decoded content.

[0102] There is no explicit limitation on the device specifically used to display the first decoded content, and it may be various personal computers, laptops, mobile terminals, and tablet computers.

[0103] In one embodiment, if the monitored network condition does not meet the preset network condition, the read content is decoded based on the initial decoding mode to obtain second decoded content; and the second decoded content is displayed.

[0104] The second decoded content is the decoded content obtained by decoding the read content using the initial decoding mode if the monitored network condition does not meet the preset network condition, that is, the network condition of the current terminal is normal or good.

[0105] Specifically, if the currently monitored terminal's network condition does not meet the preset network condition, i.e., is in a normal or good network condition, the computer device decodes the read content using an initial default decoding mode to obtain decoded content. The initial default decoding mode is a default hard decoding mode, and the initial hard decoding rate is a preset fixed value. The preset fixed value can be pre-set based on specific application scenarios and terminal types.

[0106] The initial default hard decoding mode and hard decoding rate are used to decode the read content when the current network conditions are normal or good, that is, when the monitored network conditions do not meet the preset network conditions. This completes the response process when the monitoring result does not meet the preset network conditions, ensuring that the decoded content can be displayed smoothly regardless of the current terminal's network conditions.

[0107] refer to Figure 3 , Figure 3 This is a flow chart of a method for processing multimedia data in a terminal according to another embodiment. Figure 3 As shown, a terminal multimedia data processing method is provided, comprising the following steps:

[0108] Step 302, obtaining the set scene parameters, which are used to adjust the number of key frames detected by the multimedia data; obtaining the attribute information of the multimedia data and analyzing the attribute information, including: obtaining the attribute information of the multimedia data and analyzing the attribute information based on the scene parameters.

[0109] Step 304: Based on the analysis result, the multimedia data is read to obtain the read content;

[0110] Step 306: Monitor the network conditions of the terminal;

[0111] Step 308: If the monitored network condition meets the preset network condition, determine whether the current decoding mode is hard decoding mode; if the current decoding mode is hard decoding mode, change the decoding rate of the hard decoding mode until the display smoothness of the first decoded content meets the preset smoothness; decode the read content based on the adjusted decoding mode to obtain the first decoded content. Or

[0112] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode; if the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode; the read content is decoded based on the adjusted decoding mode to obtain the first decoded content. Or

[0113] If the monitored network condition does not meet the preset network condition, the read content is decoded based on the initial decoding mode to obtain second decoded content.

[0114] Step 310: Display the first decoded content. Or

[0115] The second decoded content is displayed.

[0116] In one embodiment, a network condition monitoring system is used to monitor network conditions in real time. A network condition monitoring feedback mechanism is used to establish multimedia buffer optimization logic. In a weak network environment, the probesize and analyzeduration parameters (playback delay parameters) are adjusted. The probesize parameter is the maximum number of bytes read from the source file, in bytes, with a default value of 5,000,000. In multimedia player live broadcast and preview scenarios, when network conditions are relatively good, this value can be configured to be reduced, thereby reducing latency. The analyzeduration parameter is the time required to parse the specified media, with a default value of 0. When the network environment is poor and the multimedia home screen is opened, multimedia data is obtained by calling the avformat_find_stream_info function. A longer analyzeduration parameter results in more complete information acquisition, but a slower opening time. Therefore, the user's network condition monitoring feedback mechanism is used to adjust the analyzeduration parameter value in real time to improve the opening speed of the multimedia home screen. When the user is in a weak network environment, the established buffer is opened, and a certain amount of data is cached before playback, reducing the number of freezes. When the user's network conditions are good, the buffer size is reduced or disabled to reduce video latency. Relying on the network condition monitoring feedback mechanism to flexibly adjust the buffer size, improving the multimedia player's multimedia preview performance in weak network environments.

[0117] In one embodiment, multimedia data is passed into a buffer, and the sender retrieves the multimedia data from the buffer and sends it, thus forming an asynchronous producer-consumer model. The producer only needs to push the collected and encoded multimedia data to the buffer, while the consumer is responsible for retrieving the multimedia data from the buffer and sending it.

[0118] In one embodiment, Figure 4The video buffer diagram shown in the figure shows the video buffer. When network conditions are good, the buffer size is minimized to ensure packet loss rate and streaming speed. When network conditions are poor, data is read from the buffer to ensure smoothness. After multimedia encoding, there are keyframes and non-keyframes. Keyframes are complete images, while non-keyframes describe relative changes in the image. Asynchrony can be resolved by modifying the value of framedrop (the number of input frames to be dropped during transcoding). Framedrop drops some frames to achieve synchronization when the video cannot be processed.

[0119] In one embodiment, Figure 5 The diagram below shows how to expand video format support in the module-lite.sh (compilation configuration) file. The module_lite.sh file also supports video format expansion. The open-source IJK player project only supports a few multimedia formats. However, production environments often encounter different live streams depending on the manufacturer and model, so the original compilation process must support as many live stream types as possible. Furthermore, to play encrypted videos using the HLS (streaming media protocol), OpenSSL (Secure Sockets Layer) cryptography library must be recompiled to enable IJK player to support HTTPS (Hypertext Transfer Protocol Security) playback.

[0120] In one of the embodiments, in a production environment, the live stream address is often encrypted and redirected for its security during transmission, so decryption / redirection operations need to be performed at the terminal, and the time-consuming operations are completed at the terminal, which can also partially optimize the streaming speed.

[0121] In one embodiment, reference Figure 6 Add recording and screenshot functions to the underlying logic diagram and Figure 7The following is a diagram of the effect after adding the recording and screenshot functions. After compilation, a so file is generated. Because the mapping method of jni (a programming framework) is added, the terminal layer method also needs to be added to jni. Import the ijkplayer-java (native interface for the so library) module. The code that calls the jni layer is in the ijkMediaPlayer class. In ijkMediaPlayer, add the corresponding native method for calling the mapping (an interface for Java to call non-Java code). The terminal calls the C-end code in ijkPlayer jni. The ijkPlayer jni can only call specific functions through ijkPlayer. The ijkPlayer jni needs to contain the mapping of all methods. The ff_ffplay.c (the player that comes with the FFmpeg source code) file is responsible for the underlying implementation of the recording and screenshot methods, and is responsible for interacting with the Ffmpeg layer data.

[0122] refer to Figure 8 , Figure 8 As a schematic diagram of analyzing attribute information, open the live stream, create an audio and video decoding thread, and loop through the decapsulated packets and store them in the corresponding queue. Read the packets from the queue, soft-decode a frame, and store it in the frame_queue.

[0123] refer to Figure 9 , Figure 9 This is a schematic diagram of the interaction between the decoding layer and the display layer. After the video or audio data is decoded at the decoding layer, the audio is displayed or played on the video display interface.

[0124] refer to Figure 10 , Figure 10 It is the underlying logic diagram of the entire computer device.

[0125] It should be understood that, although the various steps in the flowcharts involved in the various embodiments as above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments as above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0126] Based on the same inventive concept, embodiments of the present application further provide a terminal multimedia data processing device for implementing the terminal multimedia data processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiments of one or more terminal multimedia data processing devices provided below can be found in the above-mentioned limitations of the terminal multimedia data processing method and will not be repeated here.

[0127] refer to Figure 11 , Figure 11 This is a flow chart of a method for processing multimedia data in a terminal in one embodiment. Figure 11 As shown, a terminal multimedia data processing device 1100 is provided, comprising: an analysis module, a reading module, a monitoring module, a decoding module and a display module, wherein:

[0128] The analysis module 1102 is used to obtain attribute information of multimedia data and analyze the attribute information.

[0129] The reading module 1104 is configured to read the multimedia data based on the analysis result to obtain the read content.

[0130] The monitoring module 1106 is configured to monitor the network conditions of the terminal.

[0131] The decoding module 1108 is configured to adjust an initial decoding mode if the monitored network condition meets a preset network condition, and decode the read content based on the adjusted decoding mode to obtain first decoded content.

[0132] The display module 1110 is configured to display the first decoded content.

[0133] In some embodiments of the present application, the analysis module 1102 also includes an acquisition module 11021, which is used to obtain the set scene parameters, and the scene parameters are used to adjust the number of key frames detected by the multimedia data; the attribute information of the multimedia data is obtained and the attribute information is analyzed, including: obtaining the attribute information of the multimedia data and analyzing the attribute information based on the scene parameters.

[0134] In some embodiments of the present application, the decoding module 1108 is configured to adjust at least one of a decoding method and a decoding rate in an initial decoding mode if the monitored network condition meets a preset network condition.

[0135] In some embodiments of the present application, the decoding module 1108 is used to determine whether the current decoding method is a hard decoding method if the monitored network condition meets the preset network condition; if the current decoding method is a hard decoding method, the decoding rate of the hard decoding method is changed until the display smoothness of the first decoded content meets the preset smoothness.

[0136] In some embodiments of the present application, the decoding module 1108 is used to determine whether the current decoding mode is a hard decoding mode if the monitored network condition meets the preset network condition; if the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

[0137] In some embodiments of the present application, the terminal multimedia data processing device further includes a second display module 1112 for decoding the read content based on the initial decoding mode to obtain second decoded content if the monitored network condition does not meet the preset network condition; and displaying the second decoded content.

[0138] In the above embodiment, when the current network conditions are poor, i.e., when the currently monitored network conditions meet the preset network conditions, the read content is decoded in two ways: adjusting the hard decoding rate and switching to soft decoding, respectively, to ensure smooth display of the decoded content. Furthermore, when the current network conditions are normal or good, i.e., when the monitored network conditions do not meet the preset network conditions, the read content is decoded using the initial default hard decoding mode and the hard decoding rate to obtain the decoded content. This completes the handling process for when the monitoring result indicates that the preset network conditions are not met, ensuring smooth display of the decoded content regardless of the current terminal's network conditions.

[0139] Each module in the terminal multimedia data processing device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data related to terminal multimedia data processing. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a terminal multimedia data processing method is implemented.

[0141] Those skilled in the art will understand that Figure 13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0143] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0144] Based on the analysis result, the multimedia data is read to obtain the read content;

[0145] Monitor the network conditions of the terminal;

[0146] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0147] The first decoded content is displayed.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] Obtaining the set scene parameters, which are used to adjust the number of key frames detected by the multimedia data;

[0150] The acquiring attribute information of the multimedia data and analyzing the attribute information include:

[0151] Attribute information of the multimedia data is acquired, and the attribute information is analyzed based on the scene parameters.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] If the monitored network condition meets the preset network condition, at least one of the decoding method and the decoding rate in the initial decoding mode is adjusted.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0156] If the current decoding mode is a hard decoding mode, the decoding rate of the hard decoding mode is changed until the display smoothness of the first decoded content meets the preset smoothness.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0159] If the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] If the monitored network condition does not meet the preset network condition, decoding the read content based on the initial decoding mode to obtain second decoded content;

[0162] The second decoded content is displayed.

[0163] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the following steps:

[0164] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0165] Based on the analysis result, the multimedia data is read to obtain the read content;

[0166] Monitor the network conditions of the terminal;

[0167] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0168] The first decoded content is displayed.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0170] Obtaining the set scene parameters, which are used to adjust the number of key frames detected by the multimedia data;

[0171] The acquiring attribute information of the multimedia data and analyzing the attribute information include:

[0172] Acquire attribute information of the multimedia data, and analyze the attribute information based on the scene parameter. In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0173] If the monitored network condition meets the preset network condition, at least one of the decoding method and the decoding rate in the initial decoding mode is adjusted.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0175] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0176] If the current decoding mode is a hard decoding mode, the decoding rate of the hard decoding mode is changed until the display smoothness of the first decoded content meets the preset smoothness.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0178] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0179] If the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0181] If the monitored network condition does not meet the preset network condition, decoding the read content based on the initial decoding mode to obtain second decoded content;

[0182] The second decoded content is displayed.

[0183] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the following steps:

[0184] Acquiring attribute information of multimedia data and analyzing the attribute information;

[0185] Based on the analysis result, the multimedia data is read to obtain the read content;

[0186] Monitor the network conditions of the terminal;

[0187] If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, decoding the read content based on the adjusted decoding mode to obtain first decoded content;

[0188] The first decoded content is displayed.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0190] Obtaining the set scene parameters, which are used to adjust the number of key frames detected by the multimedia data;

[0191] The acquiring attribute information of the multimedia data and analyzing the attribute information include:

[0192] Attribute information of the multimedia data is acquired, and the attribute information is analyzed based on the scene parameters.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0194] If the monitored network condition meets the preset network condition, at least one of the decoding method and the decoding rate in the initial decoding mode is adjusted.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0196] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0197] If the current decoding mode is a hard decoding mode, the decoding rate of the hard decoding mode is changed until the display smoothness of the first decoded content meets the preset smoothness.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0199] If the monitored network condition meets the preset network condition, it is determined whether the current decoding mode is a hard decoding mode;

[0200] If the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0202] If the monitored network condition does not meet the preset network condition, decoding the read content based on the initial decoding mode to obtain second decoded content;

[0203] The second decoded content is displayed.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0205] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0206] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A terminal multimedia data processing method, characterized in that: The method comprises: Acquiring attribute information of multimedia data and analyzing the attribute information; Based on the analysis result, the multimedia data is read to obtain read content; Monitor the network conditions of the terminal; If the monitored network condition meets the preset network condition, adjusting the initial decoding mode, and decoding the read content based on the adjusted decoding mode to obtain first decoded content; displaying the first decoded content; If the monitored network condition meets the preset network condition, adjusting the initial decoding mode includes: If the monitored network condition meets the preset network condition, determining whether the current decoding mode is a hard decoding mode; If the current decoding mode is a hard decoding mode, changing the decoding rate of the hard decoding mode until the display smoothness of the first decoded content meets the preset smoothness; If the current decoding mode is a non-hard decoding mode, the non-hard decoding mode is switched to a soft decoding mode.

2. The method according to claim 1, characterized in that The method further comprises: Acquiring a set scene parameter, wherein the scene parameter is used to adjust the number of key frames detected by the multimedia data; The acquiring attribute information of the multimedia data and analyzing the attribute information includes: Acquire attribute information of the multimedia data, and analyze the attribute information based on the scene parameters.

3. The method according to claim 1, characterized in that The preset network condition includes one or more of a preset ping value, a preset packet loss rate, and a preset bandwidth value.

4. The method according to claim 1, wherein When the user is in a weak network environment, open the established buffer; when the user's network conditions are good, reduce the size of the buffer or close the buffer.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If the monitored network condition does not meet the preset network condition, decoding the read content based on the initial decoding mode to obtain second decoded content; The second decoded content is displayed.

6. A terminal multimedia data processing method and device, characterized in that: The device comprises: An analysis module, configured to obtain attribute information of multimedia data and analyze the attribute information; A reading module, configured to read the multimedia data based on the analysis result to obtain read content; A monitoring module is used to monitor the network conditions of the terminal; a decoding module configured to, if the monitored network condition satisfies a preset network condition, determine whether a current decoding mode is a hard decoding mode; if the current decoding mode is a hard decoding mode, change a decoding rate of the hard decoding mode until a display smoothness of the decoded content satisfies a preset smoothness; if the current decoding mode is a non-hard decoding mode, switch the non-hard decoding mode to a soft decoding mode; and decode the read content based on the adjusted decoding mode to obtain first decoded content; A display module is configured to display the first decoded content.

7. The device according to claim 6, characterized in that The device further comprises: The second display module is configured to decode the read content based on the initial decoding mode to obtain second decoded content if the monitored network condition does not meet the preset network condition; and display the second decoded content.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images