Multimedia data processing method, system and device

Abstract

The embodiment of the invention provides a multimedia data processing method, system and device, and the multimedia data processing method is applied to a client, and comprises the steps: obtaining a first loudness of first multimedia data and a data package of second multimedia data in response to a switching request for switching the first multimedia data to the second multimedia data; analyzing the data packet to obtain second multimedia data and second loudness of the second multimedia data; and playing the second multimedia data according to the first loudness and the second loudness. And multimedia data processing is carried out by the client, so that the comprehensiveness of multimedia data processing is improved. And in response to the switching request, playing the second multimedia data according to the first loudness and the second loudness, thereby realizing volume balanced preheating, and ensuring that loudness mutation does not occur during switching.

Description

[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on September 12, 2024, with application number "202411279325.8" and invention name "Multimedia Data Processing Method, System and Device", all contents of which are recorded in this application by reference. Technical Field

[0002] The embodiments of the present specification relate to the field of computer technology, and more particularly to a method, system and device for processing multimedia data. Background Art

[0003] With the continuous development of Internet technology, video websites have been increasingly widely used. Users can watch different audio and video files through video websites. However, when users watch audio and video files, there may be a problem of uneven loudness in the audio and video files, resulting in a poor user experience.

[0004] At present, the server can usually balance the volume of singing and accompaniment in a single audio to achieve a suitable loudness ratio between the human voice and the accompaniment. However, when the server does not perceive the content sliding up and down, it is impossible to preheat the volume balance in advance. In addition, a product is often connected to multiple servers, and there may be a situation where the complete audio and video content cannot be covered due to inconsistency in the servers. Summary of the invention

[0005] In view of this, the embodiments of this specification provide a multimedia data processing method. One or more embodiments of this specification also relate to a multimedia data processing system, a multimedia data processing apparatus, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a multimedia data processing method is provided, which is applied to a client and includes: In response to a switching request from the first multimedia data to the second multimedia data, obtaining a first loudness of the first multimedia data and a data packet of the second multimedia data; parsing the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; The second multimedia data is played according to the first loudness and the second loudness.

[0007] According to a second aspect of an embodiment of this specification, there is provided a multimedia data processing system, including a server and a client; The server is used to obtain multimedia data; measure the loudness of the multimedia data to obtain loudness information of the multimedia data; and group the multimedia data and the loudness information to obtain a data packet of the multimedia data; The client is used for obtaining a first loudness of the first multimedia data and a data packet of the second multimedia data in response to a switching request from the first multimedia data to the second multimedia data; parsing the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; and playing the second multimedia data according to the first loudness and the second loudness.

[0008] According to a third aspect of an embodiment of this specification, a multimedia data processing device is provided, which is applied to a client and includes: A first acquisition module is configured to acquire a first loudness of the first multimedia data and a data packet of the second multimedia data in response to a switching request from the first multimedia data to the second multimedia data; A parsing module configured to parse the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; The playing module is configured to play the second multimedia data according to the first loudness and the second loudness.

[0009] According to a fourth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method provided in the first aspect are implemented.

[0010] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0011] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method provided in the first aspect when executed by a processor.

[0012] A multimedia data processing method provided by an embodiment of the present specification is applied to a client, including: in response to a switching request from the first multimedia data to the second multimedia data, obtaining a first loudness of the first multimedia data and a data packet of the second multimedia data; parsing the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; playing the second multimedia data according to the first loudness and the second loudness. The multimedia data processing by the client avoids the problem of not being able to cover the multimedia content due to the inconsistency of the server, and improves the comprehensiveness of the multimedia data processing. In response to the switching request, the second multimedia data is played at the first loudness and the second loudness of the client, realizing the volume balance preheating when the multimedia data is switched, and ensuring that there is no sudden change in loudness when the multimedia data is switched. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is an architecture diagram of a multimedia data processing system provided by an embodiment of this specification; Figure 2 is a flowchart of a multimedia data processing method applied to a client provided by an embodiment of this specification; Figure 3 is a flowchart of a multimedia data processing method applied to a server provided by an embodiment of this specification; Figure 4 is an architecture diagram of another multimedia data processing system provided by an embodiment of this specification; Figure 5 is a processing flow chart of a multimedia data processing method applied to a client provided by an embodiment of the present specification; Figure 6 is a flow chart of a multimedia data processing method provided by an embodiment of this specification; Figure 7 It is a structural diagram of a multimedia data processing device applied to a client provided by an embodiment of this specification; Figure 8 It is a structural diagram of a multimedia data processing device applied to a server provided by an embodiment of this specification; Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0014] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0015] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0017] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0018] First, the terms involved in one or more embodiments of this specification are explained.

[0019] Audio loudness equalization: refers to adjusting the loudness of different audio clips or tracks in multimedia data processing so that they sound at similar volume levels. It can also be called volume equalization.

[0020] Streaming Media: refers to multimedia content, such as audio, video or other media files, transmitted in real time via the Internet or other network connections. Unlike the traditional method of downloading the entire file before playing it, streaming media allows users to start playing as soon as they receive part of the content, without having to wait for the entire file to be downloaded.

[0021] High-pass filter (HPF) is an electronic filter that allows signals above a predetermined cutoff frequency to pass through, while signals below this frequency will be significantly attenuated. High-pass filters are used to remove low-frequency components in signals and retain high-frequency components.

[0022] Audio loudness: refers to the human ear's perception of sound intensity or volume. In the field of audio engineering and technology, loudness is used to describe the perception of sound strength.

[0023] When users switch between different live broadcast rooms during live broadcast or on-demand broadcast, the volume of some audio and video in the live broadcast room is too loud, and the user needs to manually turn down the playback volume urgently; the volume of some audio and video is too low, and the user cannot hear the audio and video content clearly, and needs to manually turn up the playback volume, or because the user cannot hear the main content of the audio and video in time and is not attracted by the audio and video, he leaves the live broadcast room directly, resulting in a relatively poor experience; and in the same live broadcast content, the speaker switches, location switches, and device switches may also occur, resulting in uneven audio loudness.

[0024] At present, the audio loudness equalization of singing and accompaniment in a single audio can usually be performed on the server to achieve a suitable loudness ratio between the human voice and the accompaniment. However, when performing audio loudness equalization on singing and accompaniment in a single audio, the server does not perceive the content sliding up and down, and cannot preheat the volume equalization in advance. During the audio loudness equalization process, the music will be over-compressed according to the human voice, resulting in music distortion and poor accuracy in multimedia data processing. In addition, a product is often connected to multiple servers, and there may be a situation where the complete audio and video content cannot be covered due to inconsistency in the server.

[0025] In order to solve the above problems, the embodiments of this specification propose a multimedia data processing method that can achieve volume balance. The multimedia data processing method is applied to a client, and in response to a switching request from the first multimedia data to the second multimedia data, obtains the first loudness of the first multimedia data and the data packet of the second multimedia data; parses the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; and plays the second multimedia data according to the first loudness and the second loudness. The client performs multimedia data processing to avoid the problem of not being able to cover the multimedia content due to the inconsistency of the server, thereby improving the comprehensiveness of multimedia data processing. In response to the switching request, the second multimedia data is played at the first loudness and the second loudness of the client, thereby achieving volume balance preheating when the multimedia data is switched, and ensuring that there will be no sudden loudness changes when the multimedia data is switched.

[0026] In this specification, a multimedia data processing method is provided. This specification also relates to a multimedia data processing system, a multimedia data processing apparatus, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0027] See also Figure 1 , Figure 1 The following is an architecture diagram of a multimedia data processing system provided by an embodiment of the present specification. The multimedia data processing system may include a client 100 and a server 200; The server 200 is used to obtain multimedia data; measure the loudness of the multimedia data to obtain loudness information of the multimedia data; and package the multimedia data and the loudness information to obtain a data packet of the multimedia data; The client 100 is configured to obtain, in response to a switching request from the first multimedia data to the second multimedia data, a first loudness of the first multimedia data and a data packet of the second multimedia data from the server 200; parse the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; and play the second multimedia data according to the first loudness and the second loudness.

[0028] It should be noted that the client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being published to the server 200.

[0029] The client 100 can be a browser, an application (APP, Application), or a web application such as a HyperText Markup Language Version 5 (HTML5, HyperText Markup Language5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, mailbox clients, social platform software, etc.

[0030] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0031] By applying the solution of the embodiment of this specification, the client performs multimedia data processing, avoiding the problem of not being able to cover multimedia content due to inconsistency in the server, and improving the comprehensiveness of multimedia data processing. In response to the switching request, the client plays the second multimedia data according to the first loudness and the second loudness, realizing the balanced preheating of the volume when the multimedia data is switched, and ensuring that there will be no sudden loudness changes when the multimedia data is switched.

[0032] See also Figure 2 , Figure 2A flowchart of a multimedia data processing method applied to a client provided by an embodiment of the present specification is shown, which specifically includes the following steps: Step 202: In response to a switch request from first multimedia data to second multimedia data, obtain a first loudness of the first multimedia data and a data packet of the second multimedia data.

[0033] It should be noted that the first multimedia data refers to multimedia data played by the client before playing the second multimedia data, so the first multimedia data can be regarded as the last played multimedia data of the second multimedia data. For example, in a live broadcast scene, the video in the live broadcast room currently watched by the user can be regarded as the first multimedia data, and when the user slides from the live broadcast room currently watched to the next live broadcast room, the video in the next live broadcast room can be regarded as the second multimedia data. Multimedia data refers to a data set containing multiple different types of information, usually including text, images, audio, video and their combinations. Multimedia data can also include streaming media data. Streaming media data is a special type of multimedia data, which is characterized in that the data can be processed immediately by the receiver during the transmission process without waiting for the entire file to be downloaded. The second multimedia data can be multimedia data in different scenarios, such as live broadcast scenes, on-demand scenes, social media sharing scenes, etc. The first loudness of the first multimedia data is obtained based on the loudness measurement of the first multimedia data. The data packet of the second multimedia data may include the encoded data of the second multimedia data and the second loudness obtained by the loudness measurement of the second multimedia data, or may include the second multimedia data and the second loudness obtained by the loudness measurement of the second multimedia data.

[0034] In actual applications, during the playback of the first multimedia data, the user can perform a multimedia data switching operation to send a switching request to the client to switch the first multimedia data to the second multimedia data. In response to the switching request, the client obtains the first loudness of the first multimedia data and the data packet of the second multimedia data. The user performs the multimedia data switching operation in any manner including clicking, double-clicking, touching, hovering the mouse, sliding, long pressing, voice control, or shaking, etc., which is selected according to the actual situation, and the embodiments of this specification do not limit this in any way.

[0035] Furthermore, there are multiple ways for the client to obtain the first loudness of the first multimedia data and the data packet of the second multimedia data, which can be selected according to the actual situation, and the embodiments of this specification do not impose any restrictions on this. Regarding the first loudness of the first multimedia data: In a first possible implementation, the client can call the target player of the second multimedia data, and read the first loudness of the first multimedia data from the original player through the interface of the original player of the first multimedia data. In a second possible implementation, the client can obtain the data packet of the first multimedia data from the server through the target player, parse the data packet of the first multimedia data, and obtain the first loudness of the first multimedia data. In a third possible implementation, the client can measure the loudness of the first multimedia data through the target player to obtain the first loudness of the first multimedia data. Regarding the data packet of the second multimedia data: In one possible implementation, the client can call the target player to obtain the data packet of the second multimedia data from the server. In another possible implementation, the client can receive the data packet of the second multimedia data sent by the user.

[0036] In an optional embodiment of the present specification, the obtaining of the first loudness of the first multimedia data and the data packet of the second multimedia data may include the following steps: Creating a target player for the second multimedia data; The target player is called to obtain a data packet of the second multimedia data, and the first loudness of the first multimedia data is read from the original player using an interface of the original player, wherein the original player is used to play the first multimedia data.

[0037] The target player refers to a player used to play the second multimedia data. The player refers to a software application in the client that is used to play various types of multimedia content, including audio, video, images, and other media formats. For example, in a live broadcast scenario, when a viewer wants to switch from one live stream to another, the client software (such as a live broadcast application) responds to the switch request and creates a new player instance to prepare to play the new live stream.

[0038] Furthermore, there are many ways for the client to create a target player for the second multimedia data, which can be selected according to actual conditions, and the embodiments of this specification do not limit this. In one possible implementation of this specification, a target player can be directly created, that is, a new target player instance is created on the client to play the second multimedia data. At the same time, player properties can be set for the created target player, such as setting the target loudness to -13db. For example, in a Web environment, a new target player can be created in HTML. <video>or <audio>Element, bind multimedia data to it and set properties. Alternatively, use the media framework of the second multimedia data to create a target player for the second multimedia data, where the media framework is such as an extended player (ExoPlayer). In another possible implementation of this specification, in order to ensure that the target player can seamlessly replace the work of the original player of the first multimedia data while maintaining consistency in user experience, the state of the original player can be copied and applied to the initial player created for the second multimedia data to obtain the target player of the second multimedia data.

[0039] In actual applications, after the client creates a target player for the second multimedia data, the target player can obtain a data packet of the second multimedia data from the server. Since the original player is used to play the first multimedia data, the first loudness of the first multimedia data can be stored in the original player. At this time, the client can call the target player and read the first loudness of the first multimedia data from the original player using the interface of the original player.

[0040] By applying the solution of the embodiments of the present specification, the target player is called to read the first loudness of the first multimedia data from the interface of the original player, and the client no longer needs to measure the loudness of the first multimedia data or parse the data packet of the first multimedia data, thereby improving the efficiency and accuracy of obtaining the first loudness.

[0041] In an optional embodiment of the present specification, the above-mentioned creation of the target player of the second multimedia data may include the following steps: Creating an initial player for second multimedia data; Get the status information of the original player; According to the state information, the state of the initial player is configured to obtain a target player for the second multimedia data.

[0042] It should be noted that the state information of the original player refers to the key information that needs to be saved and restored in the new player when switching from one multimedia content to another. This state information helps to ensure that the user can get a seamless experience when switching content. The state information of the original player includes but is not limited to the full screen opening state, the playback speed, the subtitle opening state, and the subtitle language. The specific selection is based on the actual situation, and the embodiments of this specification do not make any restrictions on this.

[0043] In practical applications, there are many ways to obtain the status information of the original player, which can be selected according to the actual situation, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the status information of the original player can be obtained through the application programming interface of the original player. In another possible implementation of this specification, the status information of the original player can be parsed from the operation log of the original player.

[0044] By applying the solution of the embodiment of this specification, the initial player of the second multimedia data is configured in accordance with the status information of the original player of the first multimedia data, and the target player of the second multimedia data is obtained, so that the target player can seamlessly take over the work of the original player while maintaining consistency in the user experience.

[0045] Step 204: Parse the data packet to obtain second multimedia data and a second loudness of the second multimedia data.

[0046] It should be noted that the second loudness of the second multimedia data is obtained based on loudness measurement of the second multimedia data. In order to ensure the accuracy of the loudness measurement, the loudness measurement of the first multimedia data and the second multimedia data can be performed by the server. Therefore, the client can call the target player to obtain the first loudness of the first multimedia data and the data packet of the second multimedia data from the server.

[0047] In practical applications, there are many ways to parse the data packet and obtain the second multimedia data and the second loudness of the second multimedia data, which can be selected according to actual conditions, and the embodiments of this specification do not limit this. In a possible implementation of this specification, the data packet includes the encoded data of the second multimedia data and the second loudness obtained by measuring the loudness of the second multimedia data, and the second multimedia data and the second loudness can be directly extracted from the data packet.

[0048] In another optional embodiment of the present specification, the above-mentioned parsing of the data packet to obtain the second multimedia data and the second loudness of the second multimedia data may include the following steps: Parsing the data packet to determine the second loudness of the second multimedia data and the encoding data of the second multimedia data; The encoded data is decoded to obtain second multimedia data.

[0049] It should be noted that the encoded data of the second multimedia data is data obtained by encoding the second multimedia data, and these data are usually used to store or transmit the content of the second multimedia data. Coding is a process of converting multimedia data into a format more suitable for storage and transmission. The encoded data is compressed and optimized, thereby reducing the demand for storage space and reducing bandwidth consumption during transmission.

[0050] In practical applications, a multimedia analysis tool can be used to parse the data packet to determine the second loudness of the second multimedia data and the encoding data of the second multimedia data. Furthermore, after obtaining the encoding data of the second multimedia data, the encoded data can be decoded by a decoder to obtain the second multimedia data. The decoder refers to a software or hardware component used to convert the encoded multimedia data back to the original format for playback. The decoder can provide a choice of encoding format for the second multimedia data, and the decoder includes but is not limited to a Free Lossless Audio Codec (FLAC) decoder and an Advanced Audio Coding (AAC) decoder.

[0051] By applying the solution of the embodiment of this specification, the client locally decodes the encoded data of the second multimedia data to obtain the second multimedia data, thereby ensuring the integrity of the second multimedia data.

[0052] Step 206: Play the second multimedia data according to the first loudness and the second loudness.

[0053] It should be noted that when the client plays the second multimedia data according to the first loudness and the second loudness, it can call the target player to render the second multimedia data according to the first loudness and the second loudness, obtain the target multimedia data, and play the target multimedia data.

[0054] In an optional embodiment of the present specification, the playing of the second multimedia data according to the first loudness and the second loudness may include the following steps: Rendering the second multimedia data according to the first loudness and the second loudness to obtain initial multimedia data; Performing content detection on the initial multimedia data, and rendering the initial multimedia data according to the content detection result to obtain target multimedia data; Play the target multimedia data.

[0055] It should be noted that the target multimedia data refers to the second multimedia data obtained after loudness equalization processing. After the loudness equalization processing, there will be no sudden loudness change when the target player plays the target multimedia data. Since there are multiple content elements in the multimedia data, using the same rendering strategy for different content elements may cause the processed multimedia data to be distorted. Therefore, the rendering strategy of the initial multimedia data can be determined according to the content detection result of the initial multimedia data. The initial multimedia data refers to the second multimedia data obtained after compensating the second loudness using the first loudness. Content detection is used to detect the content elements of multimedia data. The content elements of multimedia data include human voice, music, ambient sound, etc. The content detection result refers to the information obtained after content detection of the multimedia data. The content detection result can help understand the specific content elements contained in the multimedia data.

[0056] By applying the solution of the embodiment of this specification, the initial multimedia data is rendered according to the content detection result to obtain the target multimedia data. On the basis of loudness compensation for the second multimedia data, the distortion of the target multimedia data is avoided, thereby ensuring the quality of the target multimedia data.

[0057] In an optional embodiment of the present specification, the rendering of the second multimedia data according to the first loudness and the second loudness to obtain the initial multimedia data may include the following steps: Determining a loudness gain of the second multimedia data according to the first loudness and the second loudness; The second multimedia data is rendered according to the loudness gain to obtain initial multimedia data.

[0058] It should be noted that the loudness gain refers to the gain amount that needs to be applied in order to make the second loudness of the second multimedia data match the first loudness of the first multimedia data. That is, the loudness gain is used to adjust the second loudness of the second multimedia data so that the adjusted loudness matches the first loudness of the first multimedia data.

[0059] In practical applications, there are multiple ways to determine the loudness gain of the second multimedia data based on the first loudness and the second loudness, which can be selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the difference between the first loudness and the second loudness can be calculated, and the difference can be determined as the loudness gain of the second multimedia data. In another possible implementation of this specification, on the basis of the first loudness and the second loudness, the target loudness set by the target player can be additionally introduced, and the loudness gain of the second multimedia data can be determined according to the second loudness, the first loudness and the target loudness, so that the loudness gain meets the target loudness set by the target player. For example, the larger loudness of the first loudness and the second loudness can be determined first, and then the difference between the target loudness and the larger loudness can be calculated. If the difference is greater than 0, the difference is determined as the loudness gain of the second multimedia data. If the difference is less than 0, 0 is determined as the loudness gain of the second multimedia data. That is, the loudness gain of the second multimedia data can be calculated by the following formula (1): (1) in, represents the loudness gain of the second multimedia data, represents a function that finds the largest element in a given set. Indicates the second loudness, Indicates the first loudness, Indicates the target loudness, such as -13db, which is set according to actual conditions, and the embodiments of this specification do not impose any limitation on this.

[0060] By applying the solution of the embodiment of this specification, the second multimedia data is processed according to the loudness gain of the second multimedia data to obtain the initial multimedia data of the second multimedia data, thereby achieving volume balance preheating when the multimedia data is switched, and ensuring that there will be no sudden change in loudness when the multimedia data is switched.

[0061] In another optional embodiment of the present specification, since some low-frequency components may exist in the initial multimedia data of the second multimedia data, and the user's perception of these low-frequency components is relatively weak, the multimedia data supplemented with the loudness gain may be subjected to low-frequency suppression, thereby reducing the data volume of the multimedia data. That is, the above-mentioned rendering of the second multimedia data according to the loudness gain to obtain the initial multimedia data may include the following steps: Rendering the second multimedia data according to the loudness gain to obtain intermediate multimedia data; The intermediate multimedia data is input into a high-pass filter to obtain initial multimedia data.

[0062] In practical applications, when rendering the second multimedia data according to the loudness gain, the loudness gain may be linearly converted, and the second multimedia data may be compensated with the loudness gain after the linear conversion to obtain the intermediate multimedia data.

[0063] Furthermore, after obtaining the intermediate multimedia data, a high-pass filter can be used to suppress low-frequency components in the intermediate multimedia data, such as suppressing components below 100 Hz, to obtain initial multimedia data, wherein the frequency of the low-frequency components is set specifically according to actual conditions, and the embodiments of this specification do not impose any limitations on this.

[0064] By applying the solution of the embodiment of the present specification, the second multimedia data is processed by using the loudness gain so that the loudness of the second multimedia data matches the first loudness, and no sudden loudness change occurs when switching from the first multimedia data to the second multimedia data. In addition, a high-pass filter is used to suppress low-frequency components in the intermediate multimedia data, thereby reducing the data volume of the multimedia data.

[0065] In an optional embodiment of the present specification, the above-mentioned performing content detection on the initial multimedia data, and rendering the initial multimedia data according to the content detection result to obtain the target multimedia data may include the following steps: Performing content detection on the initial multimedia data and determining compression parameters according to the content detection result; Using the compression parameter, compressing the initial multimedia data to obtain compressed multimedia data; Determining a compression gain of the compressed multimedia data according to the compression loudness of the compressed multimedia data; The compressed multimedia data is processed according to the compression gain to obtain target multimedia data.

[0066] It should be noted that compression parameters refer to a set of settings used to control the dynamic range of signals in multimedia data processing. Compression parameters are used to adjust the loudness of signals. Compression parameters include but are not limited to compression ratio, compression gain (also known as compression gain compensation), and compression threshold, which are selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. Among them, the compression ratio refers to the ratio between the input level change and the output level change. By adjusting the compression ratio, the dynamic range of the signal can be controlled so that the multimedia data sounds more consistent and smooth. Compression gain refers to the overall gain adjustment value of the multimedia data after the multimedia data is compressed. The compression gain is used to compensate for the level drop of the multimedia data caused by compression, ensuring that the overall volume of the output signal remains unchanged. The compression threshold is a set level value, expressed in decibels. When the level of the signal exceeds the compression threshold, the dynamic range of the signal begins to be compressed.

[0067] In practical applications, there are many ways to determine compression parameters based on content detection results, and the specific selection is based on actual conditions. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the compression parameters corresponding to the content detection results can be obtained from a parameter library, where the parameter library includes multiple groups of candidate compression parameters, each candidate compression parameter carries a content tag. In another possible implementation of this specification, the compression parameters can be determined based on the content detection results combined with prior knowledge.

[0068] It should be noted that when the initial multimedia data is compressed using the compression parameters, a compressor corresponding to the compression parameters can be determined, and the initial multimedia data is input into the compressor to obtain compressed multimedia data, wherein the compressor is an effector for controlling the loudness dynamic range of the signal. The compressor automatically adjusts the volume to make the loudness of the signal more consistent.

[0069] The compressed loudness of the compressed multimedia data is obtained based on the loudness measurement of the compressed multimedia data. When determining the compression gain of the compressed multimedia data according to the compressed loudness of the compressed multimedia data, in one possible implementation of the present specification, the compression gain can be directly determined according to the compressed loudness. In another possible implementation of the present specification, on the basis of the compressed loudness, a compression gain reference value and a target loudness set by the target player can be additionally introduced, and the loudness gain is determined according to the compressed loudness, the compression gain reference value and the target loudness, so that the compression gain meets the target loudness set by the target player software, wherein the compression gain reference value is such as 20, which is specifically set according to the actual situation, and the embodiments of the present specification do not impose any restrictions on this. For example, the difference between the target loudness and the compressed loudness can be calculated first, and then the compression gain of the compressed multimedia data can be determined according to the difference and the compression gain reference value. Specifically, taking the compression gain reference value of 20 as an example, the compression gain of the compressed multimedia data can be calculated by the following formula (2): (2) in, represents the compression gain of compressed multimedia data, represents a function that finds the smallest element in a given set. represents a function that finds the largest element in a given set. Indicates the compressed loudness of compressed multimedia data, Indicates the target loudness.

[0070] In practical applications, when the compressed multimedia data is processed according to the compression gain and the target multimedia data is obtained, the compression gain can be linearly converted and the compression gain after the linear conversion can be compensated to the compressed multimedia data to obtain the target multimedia data. For example, the compression gain can be linearly converted by the following formula (3) to obtain the compression gain after the linear conversion: (3) in, represents the compression gain after linear conversion, represents the compression gain of compressed multimedia data, It represents a linear conversion parameter, such as 0.05, which is set according to actual conditions, and the embodiments of this specification do not impose any limitation on this.

[0071] By applying the scheme of the embodiments of this specification, the compression gain of the compressed multimedia data is determined according to the compression loudness of the compressed multimedia data; the compressed multimedia data is processed according to the compression gain to obtain the target multimedia data, thereby avoiding over-compression of the multimedia data and improving the accuracy of the target multimedia data.

[0072] In an optional embodiment of the present specification, when processing the compressed multimedia data according to the compression gain, the compression gain can be linearly converted to obtain the compression gain after the linear conversion, and then the compression gain after the linear conversion is smoothed point by point using a smoothing factor to obtain an updated compression gain, and finally the compressed multimedia data is processed using the updated compression gain to obtain the target multimedia data. For example, the updated compression gain can be calculated by the following formula (4), and the target multimedia data can be determined by the following formula (5): (4) (5) in, represents the updated compression gain, represents the compression gain after linear conversion, represents the target multimedia data, Indicates compressed multimedia data. It represents a smoothing factor used to adjust the smoothness of the signal, which is set according to actual conditions and is not limited in any way in the embodiments of this specification.

[0073] By applying the solution of the embodiment of this specification, the compression gain after linear conversion is smoothed point by point using a smoothing factor, so that the signal change can be smoother and more natural, thereby improving the quality of the target multimedia data.

[0074] In an optional embodiment of the present specification, the compressed multimedia data is processed according to the compressed loudness of the compressed multimedia data, and before the target multimedia data is obtained, the compressed loudness of the compressed multimedia data can be obtained. In practical applications, there are many ways to obtain compressed multimedia data, which are selected according to actual conditions, and the embodiments of the present specification do not limit this. In a possible implementation of the present specification, the compressed loudness of the compressed multimedia data can be read from other data acquisition devices or databases.

[0075] In another possible implementation of the present specification, loudness measurement may be performed on the compressed multimedia data to obtain the compressed loudness of the compressed multimedia data. That is, before determining the compression gain of the compressed multimedia data according to the compressed loudness of the compressed multimedia data, the following steps may be further included: Performing loudness measurement on the compressed multimedia data to obtain a third loudness and a fourth loudness, wherein a measurement duration of the third loudness is shorter than a measurement duration of the fourth loudness; According to the content detection result, the compressed loudness of the compressed multimedia data is screened out from the third loudness and the fourth loudness.

[0076] It should be noted that, since the measurement duration of the third loudness is shorter than the measurement duration of the fourth loudness, the third loudness can be called short-term loudness and the fourth loudness can be called long-term loudness. For example, the average loudness of the compressed multimedia data in the past 3 seconds is taken as the third loudness, and the average loudness of the compressed multimedia data in the past 30 seconds is taken as the fourth loudness.

[0077] In practical applications, there are many ways to measure the loudness of compressed multimedia data, which can be selected according to actual conditions, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the loudness of compressed multimedia data can be measured using a loudness measurement standard (such as EBU R.128) to obtain the third loudness and the fourth loudness. In another possible implementation of this specification, the loudness of compressed multimedia data can be measured using a loudness measurement tool (such as Audacity) to obtain the third loudness and the fourth loudness.

[0078] By applying the solution of the embodiments of the present specification, the compressed loudness of the compressed multimedia data is screened out from the third loudness and the fourth loudness according to the content detection result, so that the compressed loudness of the compressed multimedia data can be flexibly determined, and the loudness adaptive adjustment during the multimedia data playback process is achieved, thereby avoiding the problem of inconsistent loudness caused by reasons such as speaker switching, location switching, and device switching during the multimedia data playback process.

[0079] In actual applications, there are multiple ways to select the compressed loudness of the compressed multimedia data from the third loudness and the fourth loudness according to the content detection result, which are selected according to the actual situation, and the embodiments of this specification do not limit this. In a possible implementation of this specification, when the content detection result indicates that the initial multimedia data includes music content, the fourth loudness is determined as the compressed loudness of the compressed multimedia data, and when the content detection result indicates that the initial multimedia data does not include music content, the compressed loudness is randomly selected from the third loudness and the fourth loudness.

[0080] In another possible implementation of the present specification, the above-mentioned filtering out the compressed loudness of the compressed multimedia data from the third loudness and the fourth loudness according to the content detection result may include the following steps: If the content detection result indicates that the initial multimedia data includes music content, determining the fourth loudness as a compressed loudness of the compressed multimedia data; When the content detection result indicates that the initial multimedia data does not include music content, a loudness difference is determined according to the third loudness and the fourth loudness, and the compressed loudness of the compressed multimedia data is screened out from the third loudness and the fourth loudness using the loudness difference.

[0081] In practical applications, when the content detection result indicates that the initial multimedia data includes music content, the compressed loudness of the compressed multimedia data can be determined by the following formula (6): (6) in, Indicates the compressed loudness of compressed multimedia data, Indicates the fourth loudness.

[0082] When the content detection result indicates that the initial multimedia data does not include music content, the loudness difference value may be determined by the following formula (7), and the compressed loudness of the compressed multimedia data may be determined by the following formula (8): (7) (8) Where R represents the loudness difference, abs represents the function of taking the absolute value, Indicates the fourth loudness, Indicates the third loudness, represents the compressed loudness of the compressed multimedia data, and b represents the loudness difference threshold, such as 9bd, which is set according to actual conditions and is not limited in any way in the embodiments of this specification.

[0083] By applying the solution of the embodiments of the present specification, when the initial multimedia data includes music content, the fourth loudness is directly determined as the compressed loudness of the compressed multimedia data, thereby ensuring the authenticity of the music content. When the initial multimedia data does not include music content, if the loudness difference is greater than the loudness difference threshold, indicating that a sudden change in gain occurs, the third loudness is used as the compressed loudness, thereby avoiding a sudden change in loudness.

[0084] In an optional embodiment of the present specification, performing content detection on the initial multimedia data and determining the compression parameter according to the content detection result may include the following steps: Performing content detection on the initial multimedia data to obtain a content detection result; In the case where the content detection result indicates that the initial multimedia data includes music content, determining the first compression parameter as the compression parameter; In a case where the content detection result indicates that the initial multimedia data does not include music content, a second compression parameter is determined as the compression parameter, wherein the first compression parameter is smaller than the second compression parameter.

[0085] In practical applications, there are many ways to perform content detection on the initial multimedia data and obtain content detection results, which can be selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, in the multimedia data, the spectrum ranges corresponding to different content elements are usually different, such as the spectrum range of music content is relatively wide, therefore, the initial multimedia data can be subjected to spectrum analysis, and the content detection results can be determined based on the spectrum analysis results. In another possible implementation of this specification, the initial multimedia data can be input into a content detection model to obtain content detection results, wherein the content detection model is a machine learning model that can be trained using a large number of sample detection results and sample initial multimedia data.

[0086] It should be noted that the first compression parameter and the second compression parameter are set according to the actual situation, and the embodiments of this specification do not limit this. For example, the first compression parameter may include a first compression ratio (1.2-1.6), a first compression gain (6db), and a first compression threshold (-30db). The second compression parameter may include a second compression ratio (2-3), a second compression gain (12db), and a second compression threshold (-40db).

[0087] By applying the scheme of the embodiments of the present specification, content detection is performed on the initial multimedia data to obtain content detection results, and different compression strategies are adopted for the initial multimedia data including music content and the initial multimedia data not including music content. A conservative compression strategy is adopted for the initial multimedia data including music content to suppress audio waveform glitches (the part in which the peak value is significantly greater than the average due to a sudden change in the waveform), and an aggressive strategy is adopted for the initial multimedia data not including music content to compress the dynamic range of the multimedia data in order to pursue a larger loudness adjustment space, thereby avoiding over-compression of the multimedia data and improving the accuracy of multimedia data processing.

[0088] See also Figure 3 , Figure 3 A flowchart of a multimedia data processing method applied to a server provided by an embodiment of the present specification is shown, which specifically includes the following steps: Step 302: Acquire multimedia data.

[0089] Step 304: loudness measurement is performed on the multimedia data to obtain loudness information of the multimedia data.

[0090] Step 306: Packaging the multimedia data and the loudness information to obtain a data packet of the multimedia data.

[0091] Step 308: Send the multimedia data packet to the client, so that the client generates target multimedia data using the multimedia data processing method.

[0092] It should be noted that multimedia data includes first multimedia data and second multimedia data, that is, the server can process any multimedia data. The loudness information of multimedia data refers to the auditory intensity or size of the audio signal in the multimedia data. The implementation method of "measuring the loudness of multimedia data" can refer to the implementation method of "measuring the loudness of compressed multimedia data" mentioned above; the implementation method of "the client generates target multimedia data using the multimedia data processing method" can refer to Figure 2 The implementation method of the multimedia data processing method shown in the figure will not be described in detail in the embodiments of this specification.

[0093] In practical applications, there are many ways to obtain multimedia data, which can be selected according to actual conditions, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the server can read multimedia data from other data acquisition devices or databases. In another possible implementation of this specification, the server can obtain the encoded data of the multimedia data, decode the encoded data of the multimedia data, and obtain the multimedia data.

[0094] In an optional embodiment of the present specification, after the server measures the loudness of multimedia data and obtains the loudness information of the multimedia data, the multimedia data and the loudness information can be packaged in the form of media enhancement information, thereby recording the loudness information of the multimedia data and assisting the client in obtaining the loudness of the multimedia data. The package includes but is not limited to the extension of the multimedia coding extension field and the extension of the multimedia data transmission protocol, which is selected according to the actual situation and is not limited in any way in the embodiments of the present specification.

[0095] By applying the solution of the embodiments of the present specification, loudness measurement is performed on multimedia data at the server end, multimedia data and loudness information are packaged, and the data packet of the multimedia data obtained by the packaged is sent to the client end, thereby reducing the processing pressure of the client end. At the same time, the client end processes the multimedia data to generate target multimedia data, thereby avoiding the problem of not being able to cover the audio content due to the inconsistency of the server end, and improving the comprehensiveness of the multimedia data processing.

[0096] In an optional embodiment of the present specification, the above-mentioned packetizing the multimedia data and the loudness information to obtain a data packet of the multimedia data may include the following steps: Encoding the multimedia data to obtain encoded data; The loudness information is embedded into the coded data by using the multimedia coding extension field to obtain a data packet of the multimedia data.

[0097] It should be noted that when encoding multimedia data, multimedia data can be collected and preprocessed, such as removing noise, equalizing, cropping, etc. Then, select an encoder suitable for the multimedia data, use the encoder to encode the multimedia data, compress the multimedia data, reduce the data volume of the multimedia data, and obtain the encoded data of the multimedia data. For example, for audio data, the original audio data can be obtained. The original audio data is usually in an uncompressed pulse code modulation format. According to the requirements, a suitable audio encoder is selected, the audio data is removed from the noise and equalized, and the audio data is encoded using the selected encoder. After the encoding is completed, the encoded data of the audio data is obtained. For video data, the original video data can be obtained, and a suitable video encoder for the original video data is selected according to the actual application scenario, such as H.264 / AVC, H.265 / HEVC, VP9, ​​etc. The original video data is preprocessed, such as adjusting the color space, cropping, etc. The video data is encoded using the selected encoder, and the encoded data of the video data is obtained after the encoding is completed.

[0098] Multimedia coding extension fields refer to some special fields reserved in the coded data stream in order to carry additional enhanced information during the multimedia data encoding process. These fields allow additional functions or information to be added without changing the basic coding structure. In audio coding, loudness information, as a kind of enhanced information, can be carried through multimedia coding extension fields. The advantage of this is that this information can be used during decoding to improve the user experience, for example, by dynamically adjusting the volume to maintain a relatively constant loudness level. The multimedia coding extension field is selected according to the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0099] In practical applications, when using multimedia coding extension fields to embed loudness information in encoded data, the appropriate location can be selected to embed the loudness information according to the support of the encoder. For example, in Advanced Audio Coding (AAC) encoding, it can be embedded in the Audio Data Transport Stream (ADTS) or the private data part, where ADTS is a synchronization header in the AAC file format, which is used to identify the start of the frame and contains basic information required to decode the frame, such as sampling rate, number of channels, protection bits, etc. ADTS is used in real-time transmission and file formats, which enables AAC data to be correctly decoded and played. When embedding loudness information, the loudness value can be encoded in binary form and inserted into the selected coding extension field. The encoded data with embedded loudness information is packaged into a data packet of multimedia data.

[0100] By applying the solution of the embodiments of this specification and using the multimedia coding extension field to embed loudness information into the coded data and obtain a data packet of multimedia data, the client can achieve loudness balance when playing different multimedia data, thereby reducing the processing pressure of the client.

[0101] In an optional embodiment of the present specification, the above-mentioned packetizing the multimedia data and the loudness information to obtain a data packet of the multimedia data may include the following steps: Encoding the multimedia data to obtain encoded data; The loudness information is embedded into the coded data by utilizing the extension field of the multimedia data transmission protocol to obtain a data packet of the multimedia data.

[0102] It should be noted that the extension field of the multimedia data transmission protocol refers to a specific field designed in the multimedia data transmission protocol to carry additional enhanced information. These fields are usually used to provide more control information, quality of service parameters, synchronization information or other types of data. Multimedia data transmission protocols include but are not limited to the Real-time Transport Protocol (RTP) and the Hypertext Transfer Protocol (HTTP). Taking the RTP protocol as an example, RTP is a network protocol for transmitting real-time multimedia data (such as audio and video). RTP includes an optional extension header (Extension Header) for carrying additional information. These extension fields can be used to transmit loudness information, as well as other types of data, such as timestamps, receiving end rendering time, etc. The extension header field of RTP includes the length of the extension field (Length), specific extension data (Payload), the type identifier of the extension field (Extension ID), etc., which are selected according to the actual situation, and the embodiments of this specification do not make any restrictions on this.

[0103] In practical applications, when using the extension field of the multimedia data transmission protocol to embed the loudness information into the coded data, the multimedia data transmission protocol can be selected, and according to the characteristics of the selected protocol, a suitable extension field can be selected to embed the loudness information, and the loudness information can be encoded into a binary form and embedded into the extension field of the multimedia data transmission protocol to generate the encoded data embedded with the loudness information. The encoded data embedded with the loudness information can be packaged into a data packet of the multimedia data.

[0104] By applying the solution of the embodiments of this specification and using the extended field of the multimedia data transmission protocol to embed loudness information into the encoded data and obtain a data packet of multimedia data, the client can achieve loudness balance when playing different multimedia data, thereby reducing the processing pressure of the client.

[0105] See also Figure 4 , Figure 4 An architecture diagram of another multimedia data processing system provided by an embodiment of the present specification is shown, and the multimedia data processing system includes a server and a client; the multimedia data processing system is described by taking the multimedia data processing system processing audio data in a live broadcast scene as an example.

[0106] The server is used to obtain audio encoding data; decode the audio encoding data to obtain audio data; measure the loudness of the audio data to obtain loudness information of the audio data; package the loudness information with the audio encoding data in the form of media enhancement information to obtain a data packet of audio data; and send the data packet of audio data to the client; The client is used to create a target player for the second audio data in response to a switching request from the first audio data to the second audio data; call the target player to obtain a data packet of the first loudness of the first audio data and the second audio data; parse the data packet to obtain the second audio encoding data and the second loudness of the second audio data; decode the second audio encoding data to obtain the second audio encoding data; render and perform loudness equalization processing on the second audio data according to the first loudness and the second loudness to obtain the target audio data; and play the target audio data.

[0107] See also Figure 5 , Figure 5 A flowchart of a processing process of a multimedia data processing method applied to a client provided by an embodiment of the present specification is shown. The processing process can be regarded as a volume equalization processing process. Taking the multimedia data as a live stream of a live scene as an example, the processing process specifically includes: In a live broadcast scenario, in response to a switch request from a first live broadcast stream to a second live broadcast stream, a target player for the second live broadcast stream is created; a first loudness of the first live broadcast stream and a data packet of the second live broadcast stream are obtained; the data packet is parsed to obtain the second live broadcast stream and the second loudness of the second live broadcast stream; a loudness gain of the second live broadcast stream is determined according to the first loudness and the second loudness; according to the loudness gain, the second live broadcast stream is rendered to obtain an intermediate live broadcast stream; the intermediate live broadcast stream is input into a high-pass filter to obtain an initial live broadcast stream; Perform music content detection on the initial live stream, and determine whether there is music in the initial live stream based on the content detection result. If so, input the initial live stream into compressor 1 for compression processing to obtain a compressed live stream; if not, input the initial live stream into compressor 2 for compression processing to obtain a compressed live stream, wherein the first compression parameters of compressor 1 may include a first compression ratio (1.2-1.6), a first compression gain (6db) and a first compression threshold (-30db). The second compression parameters of compressor 2 may include a second compression ratio (2-3), a second compression gain (12db) and a second compression threshold (-40db).

[0108] The loudness of the compressed live stream is measured to obtain a third loudness and a fourth loudness, wherein the measurement duration of the third loudness is shorter than the measurement duration of the fourth loudness; whether there is music in the initial live stream is determined according to the content detection result, and if so, the fourth loudness is determined as the compressed loudness of the compressed live stream; if not, a loudness difference is determined according to the third loudness and the fourth loudness, and the compressed loudness of the compressed live stream is screened out from the third loudness and the fourth loudness by using the loudness difference.

[0109] The compression gain of the compressed live stream is determined according to the compression loudness of the compressed live stream; the compressed live stream is processed according to the compression gain to obtain the target live stream, and the target live stream is played.

[0110] By applying the solution of the embodiments of the present specification, volume equalization and preheating are achieved when the live stream is switched, ensuring that there will be no sudden loudness changes when the live stream is switched, so that the live stream processing method is suitable for content sliding scenarios of streaming media players; by adopting the method of music content detection, compressors with different compression parameters are switched, and a compressor with a higher compression degree is applied to voice compression to provide a larger loudness adjustment space, and a compressor with a lower compression degree is applied to music to protect the audibility of the music; during the live stream playback, volume equalization is performed in real time with low latency, and the loudness stability of the music is taken into account. When music is detected, the fourth loudness is used as a measurement standard to ensure the loudness consistency of the played music. When no music is detected, the loudness difference between the third loudness and the fourth loudness is monitored in real time, and according to the size of the loudness difference and the loudness difference threshold, it is determined whether the audio loudness has suddenly changed, so that different loudnesses are used as measurement standards to quickly respond to changes in loudness.

[0111] See also Figure 6 , Figure 6 A flow chart of a multimedia data processing method provided by an embodiment of the present specification is shown. In the multimedia data processing process, the server, the client, the original player in the client, and the target player in the client perform data interaction, which specifically includes: The server side: obtains multimedia data; measures the loudness of the multimedia data to obtain loudness information of the multimedia data; packages the multimedia data and the loudness information to obtain a data packet of the multimedia data; wherein the multimedia data includes first multimedia data and second multimedia data; The original player in the client: plays the first multimedia data; Client: during the playback of the first multimedia data or after the playback of the first multimedia data is completed, the client responds to a switching request from the first multimedia data to the second multimedia data and creates a target player for the second multimedia data; The target player in the client: reads the first loudness of the first multimedia data from the original player; obtains the data packet of the second multimedia data from the server; parses the data packet to determine the second loudness of the second multimedia data and the encoding data of the second multimedia data; decodes the encoding data to obtain the second multimedia data; renders the second multimedia data according to the first loudness and the second loudness to obtain the initial multimedia data; performs content detection on the initial multimedia data, and renders the initial multimedia data according to the content detection result to obtain the target multimedia data; plays the target multimedia data.

[0112] Corresponding to the above-mentioned multimedia data processing method embodiment applied to the client, this specification also provides a multimedia data processing device embodiment applied to the client. Figure 7 FIG. 1 shows a schematic diagram of a multimedia data processing device applied to a client provided by an embodiment of the present specification. Figure 7 As shown, the device comprises: The first acquisition module 702 is configured to acquire a first loudness of the first multimedia data and a data packet of the second multimedia data in response to a switching request from the first multimedia data to the second multimedia data; The parsing module 704 is configured to parse the data packet to obtain the second multimedia data and the second loudness of the second multimedia data; The playing module 706 is configured to play the second multimedia data according to the first loudness and the second loudness.

[0113] Optionally, the parsing module 704 is further configured to parse the data packet to determine the second loudness of the second multimedia data and the encoding data of the second multimedia data; and decode the encoding data to obtain the second multimedia data.

[0114] Optionally, the playback module 706 is further configured to render the second multimedia data according to the first loudness and the second loudness to obtain initial multimedia data; perform content detection on the initial multimedia data and render the initial multimedia data according to the content detection result to obtain target multimedia data; and play the target multimedia data.

[0115] Optionally, the playing module 706 is further configured to determine a loudness gain of the second multimedia data according to the first loudness and the second loudness; and render the second multimedia data according to the loudness gain to obtain initial multimedia data.

[0116] Optionally, the playback module 706 is further configured to render the second multimedia data according to the loudness gain to obtain intermediate multimedia data; and input the intermediate multimedia data into a high-pass filter to obtain initial multimedia data.

[0117] Optionally, the playback module 706 is further configured to perform content detection on the initial multimedia data, and determine compression parameters based on the content detection results; compress the initial multimedia data using the compression parameters to obtain compressed multimedia data; determine the compression gain of the compressed multimedia data based on the compression loudness of the compressed multimedia data; and process the compressed multimedia data based on the compression gain to obtain target multimedia data.

[0118] Optionally, the playback module 706 is further configured to perform content detection on the initial multimedia data to obtain a content detection result; when the content detection result indicates that the initial multimedia data includes music content, a first compression parameter is determined as the compression parameter; when the content detection result indicates that the initial multimedia data does not include music content, a second compression parameter is determined as the compression parameter, wherein the first compression parameter is less than the second compression parameter.

[0119] Optionally, the device further includes: a screening module configured to perform loudness measurement on the compressed multimedia data to obtain a third loudness and a fourth loudness, wherein a measurement duration of the third loudness is shorter than a measurement duration of the fourth loudness; and screen out the compressed loudness of the compressed multimedia data from the third loudness and the fourth loudness according to the content detection result.

[0120] Optionally, the screening module is further configured to, when the content detection result indicates that the initial multimedia data includes music content, determine the fourth loudness as the compressed loudness of the compressed multimedia data; when the content detection result indicates that the initial multimedia data does not include music content, determine a loudness difference based on the third loudness and the fourth loudness, and use the loudness difference to screen out the compressed loudness of the compressed multimedia data from the third loudness and the fourth loudness.

[0121] Optionally, the first acquisition module 702 is further configured to create a target player for the second multimedia data; call the target player to obtain a data packet of the second multimedia data, and use an interface of the original player to read the first loudness of the first multimedia data from the original player, wherein the original player is used to play the first multimedia data.

[0122] Optionally, the first acquisition module 702 is further configured to create an initial player for the second multimedia data; obtain status information of the original player; and perform status configuration on the initial player according to the status information to obtain a target player for the second multimedia data.

[0123] By applying the solution of the embodiment of this specification, the client performs multimedia data processing, avoiding the problem of not being able to cover multimedia content due to inconsistency in the server, and improving the comprehensiveness of multimedia data processing. In response to the switching request, the client plays the second multimedia data according to the first loudness and the second loudness, realizing the balanced preheating of the volume when the multimedia data is switched, and ensuring that there will be no sudden loudness changes when the multimedia data is switched.

[0124] The above is a schematic scheme of a multimedia data processing device applied to a client of this embodiment. It should be noted that the technical scheme of the multimedia data processing device applied to the client and the technical scheme of the multimedia data processing method applied to the client belong to the same concept, and the details of the technical scheme of the multimedia data processing device applied to the client that are not described in detail can all be referred to the description of the technical scheme of the multimedia data processing method applied to the client.

[0125] Corresponding to the above-mentioned multimedia data processing method embodiment applied to the server, this specification also provides a multimedia data processing device embodiment applied to the server. Figure 8 FIG. 1 shows a schematic diagram of a multimedia data processing device applied to a server provided by an embodiment of the present specification. Figure 8 As shown, the device comprises: The second acquisition module 802 is configured to acquire multimedia data; The measuring module 804 is configured to perform loudness measurement on the multimedia data to obtain loudness information of the multimedia data; The packetizing module 806 is configured to packetize the multimedia data and the loudness information to obtain a data packet of the multimedia data; The sending module 808 is configured to send the data packet of multimedia data to the client, so that the client generates target multimedia data by using the multimedia data processing method.

[0126] Optionally, the packet assembly module 806 is further configured to encode the multimedia data to obtain encoded data; and embed the loudness information into the encoded data using a multimedia encoding extension field to obtain a data packet of the multimedia data.

[0127] Optionally, the packet assembly module 806 is further configured to encode the multimedia data to obtain encoded data; and embed the loudness information into the encoded data using an extended field of the multimedia data transmission protocol to obtain a data packet of the multimedia data.

[0128] By applying the solution of the embodiments of the present specification, loudness measurement is performed on multimedia data at the server end, multimedia data and loudness information are packaged, and the data packet of the multimedia data obtained by the packaged is sent to the client end, thereby reducing the processing pressure of the client end. At the same time, the client end processes the multimedia data to generate target multimedia data, thereby avoiding the problem of not being able to cover the audio content due to the inconsistency of the server end, and improving the comprehensiveness of the multimedia data processing.

[0129] The above is a schematic scheme of a multimedia data processing device applied to a server in this embodiment. It should be noted that the technical scheme of the multimedia data processing device applied to the server and the technical scheme of the multimedia data processing method applied to the server belong to the same concept, and the details of the technical scheme of the multimedia data processing device applied to the server that are not described in detail can all be referred to the description of the technical scheme of the multimedia data processing method applied to the server.

[0130] Fig. 9 The structure block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and the database 950 is used to store data.

[0131] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world microwave interconnection access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0132] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0133] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0134] The processor 920 is used to execute computer programs / instructions, which implement the steps of the above-mentioned multimedia data processing method when executed by the processor.

[0135] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multimedia data processing method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the multimedia data processing method described above.

[0136] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned multimedia data processing method when executed by a processor.

[0137] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimedia data processing method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the multimedia data processing method described above.

[0138] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned multimedia data processing method when executed by a processor.

[0139] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the multimedia data processing method described above are of the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the multimedia data processing method described above.

[0140] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0141] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0142] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.< / audio> < / video>

Claims

1. A multimedia data processing method, characterized in that: Applied to the client, including: In response to a switching request from first multimedia data to second multimedia data, acquiring a first loudness of the first multimedia data and a data packet of the second multimedia data; parsing the data packet to obtain the second multimedia data and a second loudness of the second multimedia data; The second multimedia data is played according to the first loudness and the second loudness.

2. The method according to claim 1, characterized in that The step of parsing the data packet to obtain the second multimedia data and a second loudness of the second multimedia data includes: parsing the data packet to determine a second loudness of the second multimedia data and encoding data of the second multimedia data; The encoded data is decoded to obtain the second multimedia data.

3. The method according to claim 1, characterized in that The playing the second multimedia data according to the first loudness and the second loudness includes: Rendering the second multimedia data according to the first loudness and the second loudness to obtain initial multimedia data; Performing content detection on the initial multimedia data, and rendering the initial multimedia data according to the content detection result to obtain target multimedia data; The target multimedia data is played.

4. The method according to claim 3, characterized in that The rendering of the second multimedia data according to the first loudness and the second loudness to obtain initial multimedia data includes: determining a loudness gain of the second multimedia data according to the first loudness and the second loudness; The second multimedia data is rendered according to the loudness gain to obtain initial multimedia data.

5. The method according to claim 4, characterized in that The step of rendering the second multimedia data according to the loudness gain to obtain initial multimedia data includes: Rendering the second multimedia data according to the loudness gain to obtain intermediate multimedia data; The intermediate multimedia data is input into a high-pass filter to obtain the initial multimedia data.

6. The method according to claim 3, characterized in that The performing content detection on the initial multimedia data and rendering the initial multimedia data according to the content detection result to obtain target multimedia data includes: Performing content detection on the initial multimedia data, and determining compression parameters according to the content detection result; Using the compression parameters, compressing the initial multimedia data to obtain compressed multimedia data; Determining a compression gain of the compressed multimedia data according to the compression loudness of the compressed multimedia data; The compressed multimedia data is processed according to the compression gain to obtain target multimedia data.

7. The method according to claim 6, characterized in that The performing content detection on the initial multimedia data and determining the compression parameter according to the content detection result includes: Performing content detection on the initial multimedia data to obtain a content detection result; In a case where the content detection result indicates that the initial multimedia data includes music content, determining a first compression parameter as the compression parameter; In a case where the content detection result indicates that the initial multimedia data does not include the music content, a second compression parameter is determined as the compression parameter, wherein the first compression parameter is smaller than the second compression parameter.

8. The method according to claim 6, characterized in that Before determining the compression gain of the compressed multimedia data according to the compression loudness of the compressed multimedia data, the method further includes: Performing loudness measurement on the compressed multimedia data to obtain a third loudness and a fourth loudness, wherein a measurement duration of the third loudness is shorter than a measurement duration of the fourth loudness; The compressed loudness of the compressed multimedia data is filtered out from the third loudness and the fourth loudness according to the content detection result.

9. The method according to claim 8, characterized in that The step of selecting the compressed loudness of the compressed multimedia data from the third loudness and the fourth loudness according to the content detection result includes: If the content detection result indicates that the initial multimedia data includes music content, determining the fourth loudness as a compressed loudness of the compressed multimedia data; When the content detection result indicates that the initial multimedia data does not include music content, a loudness difference is determined according to the third loudness and the fourth loudness, and the compressed loudness of the compressed multimedia data is screened out from the third loudness and the fourth loudness by using the loudness difference.

10. The method according to claim 1, characterized in that The obtaining the first loudness of the first multimedia data and the data packet of the second multimedia data includes: Creating a target player for the second multimedia data; The target player is called to obtain a data packet of the second multimedia data, and the first loudness of the first multimedia data is read from the original player using an interface of the original player, wherein the original player is used to play the first multimedia data.

11. The method according to claim 10, characterized in that The step of creating a target player for the second multimedia data comprises: Creating an initial player for the second multimedia data; Obtaining status information of the original player; According to the state information, the state of the initial player is configured to obtain a target player for the second multimedia data.

12. A multimedia data processing system, characterized in that: Including server and client; The server is used to obtain multimedia data; measure the loudness of the multimedia data to obtain loudness information of the multimedia data; and group the multimedia data and the loudness information to obtain a data packet of the multimedia data; The client is configured to obtain, from the server, a first loudness of the first multimedia data and a data packet of the second multimedia data in response to a switching request from the first multimedia data to the second multimedia data; The data packet is parsed to obtain the second multimedia data and a second loudness of the second multimedia data; and the second multimedia data is played according to the first loudness and the second loudness.

13. A multimedia data processing device, characterized in that: Applied to the client, including: A first acquisition module, configured to acquire a first loudness of the first multimedia data and a data packet of the second multimedia data in response to a switching request from the first multimedia data to the second multimedia data; a parsing module configured to parse the data packet to obtain the second multimedia data and a second loudness of the second multimedia data; The playing module is configured to play the second multimedia data according to the first loudness and the second loudness.

14. A computing device, characterized in that include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium, characterized in that: It stores a computer program / instruction, which implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

16. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images