Correction parameter generation method and apparatus

Abstract

The application discloses a correction parameter generation method and device, and relates to the technical field of audio processing. A specific embodiment of the method comprises the following steps: in response to determining that an original channel layout of original audio data does not match a target channel layout of a playback device, determining a target correction model based on the quantity relationship between each channel of the original channel layout and each corresponding channel of the target channel layout; obtaining the scene category of the original audio data; and inputting first specified data into the target correction model to obtain a correction parameter. This embodiment effectively improves the accuracy and reliability of the generated correction parameter for correcting the audio data.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of audio processing technology, and in particular to a method and apparatus for generating correction parameters. Background Technology

[0002] With the continuous development of the audio industry, the demand and user base at different levels and in different fields have surged. At the same time, the devices used for online audio have become increasingly diversified. Due to this diversification, the standards for audio production and playback on these devices are not uniform, resulting in poor compatibility between different devices. The same audio can perform very differently on different devices, leading to a poor user experience.

[0003] For most audio playback systems: if the number of channels in the audio channel layout exceeds the maximum number of channels that the audio playback device can support, the device will discard the excess channels and only retain the channel data that the device supports for playback. This method will destroy the authenticity of the audio. If the number of channels in the audio channel layout is less than the maximum number of channels that the audio playback device can support, the audio playback device will usually choose to play the audio directly or fill the remaining channels with empty data. This method will lead to excess or idle device performance and a poor user experience. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for generating correction parameters.

[0005] According to a first aspect, embodiments of this application provide a method for generating correction parameters. The method includes: in response to determining that the original channel layout of the original audio data does not match the target channel layout of the playback device, determining a target correction model based on the quantity relationship between each channel of the original channel layout and each corresponding channel of the target channel layout; obtaining the scene category of the original audio data; and inputting first specified data into the target correction model to obtain correction parameters.

[0006] According to a second aspect, embodiments of this application provide a correction parameter generation apparatus, comprising: a determining module configured to determine a target correction model based on the quantity relationship between each channel of the original channel layout and each corresponding channel of the target channel layout in response to determining that the original channel layout of the original audio data does not match the target channel layout of the playback device; an acquiring module configured to acquire the scene category of the original audio data; and a generating module configured to input first specified data into the target correction model to obtain correction parameters.

[0007] According to a third aspect, embodiments of this application provide an electronic device including one or more processors; a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the correction parameter generation method as described in any embodiment of the first aspect.

[0008] According to a fourth aspect, embodiments of this application provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the correction parameter generation method as described in any embodiment of the first aspect.

[0009] This application, in response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, determines a target correction model based on the quantitative relationship between each channel in the original channel layout and each corresponding channel in the target channel layout; obtains the scene category of the original audio data; inputs the first specified data into the target correction model to obtain correction parameters, fully considering the impact of the scene classification of the audio data on audio processing, and improving the accuracy and reliability of the generated correction parameters for correcting the audio data.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0012] Figure 2 This is a flowchart of an embodiment of the modified parameter generation method according to this application;

[0013] Figure 3 This is a schematic diagram of an application scenario of the modified parameter generation method according to this application;

[0014] Figure 4a This is a flowchart of yet another embodiment of the modified parameter generation method according to this application;

[0015] Figure 4b This is a schematic diagram of yet another embodiment of the modified parameter generation method according to this application;

[0016] Figure 5 This is a schematic diagram of one embodiment of the modified parameter generation apparatus according to this application;

[0017] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing the server embodiments of this application. Detailed Implementation

[0018] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 An exemplary system architecture 100 is shown, in which the modified parameter generation method of this application can be applied.

[0021] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0022] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Client application software, such as playback application software or communication application software, can be installed on terminal devices 101, 102, and 103.

[0023] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, desktop computers, projectors, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0024] Server 105 can be a server that provides various services. For example, in response to determining that the original channel layout of the original audio data does not match the target channel layout of the playback device, a target correction model is determined based on the number relationship between each channel of the original channel layout and each corresponding channel of the target channel layout; the scene category of the original audio data is obtained; and the first specified data is input into the target correction model to obtain correction parameters.

[0025] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide correction parameter generation services), or as a single software program or software module. No specific limitations are made here.

[0026] It should be noted that the modified parameter generation method provided in the embodiments of this disclosure can be executed by server 105, by terminal devices 101, 102, and 103, or by server 105 and terminal devices 101, 102, and 103 in cooperation with each other. Accordingly, the various parts (e.g., various units, sub-units, modules, and sub-modules) included in the modified parameter generation device can all be set in server 105, all of them can be set in terminal devices 101, 102, and 103, or they can be set in server 105 and terminal devices 101, 102, and 103 respectively.

[0027] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0028] Figure 2 A flow 200 is shown, illustrating an embodiment of the modified parameter generation method applicable to this application. The modified parameter generation method includes the following steps:

[0029] Step 201: In response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, a target correction model is determined based on the number relationship between each channel in the original channel layout and each corresponding channel in the target channel layout.

[0030] In this embodiment, the execution entity (e.g., Figure 1 The terminal devices 101, 102, 103 or the server 105 can acquire raw audio data, such as game audio data, movie audio data, dialogue audio data, etc., which are audio data to be played on the playback device, through wired or wireless connection. They determine whether the original channel layout of the raw audio data matches the target channel layout of the playback device. If they do not match, they further determine the quantity relationship between each channel of the original channel layout and each corresponding channel of the target channel layout, and determine the target correction model based on the quantity relationship between each channel of the original channel layout and each corresponding channel of the target channel layout.

[0031] Here, wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods that are currently known or will be developed in the future.

[0032] The target correction model can be either an upper mixing model or a lower mixing model.

[0033] Here, the channels in the channel layout typically include surround channels, bass channels, and height channels.

[0034] Specifically, if the number of each channel in the original channel layout (surround, bass, and height) is greater than or equal to the number of corresponding channels in the target channel layout, then the target correction model can be determined as a downmixing model; if the number of each channel in the original channel layout (surround, bass, and height) is less than or equal to the number of corresponding channels in the target channel layout, then the target correction model can be determined as an upmixing model.

[0035] It should be noted that if the original channel layout of the original audio data does not match the target channel layout of the playback device, and if at least one of the surround channels, bass channels, and height channels in the original channel layout has a smaller number of channels than the corresponding number of channels in the target channel layout, and at least one of the channels has a larger number of channels than the corresponding number of channels in the target channel layout, then the target correction model can be either an upmixing model or an downmixing model.

[0036] For example, the original channel layout is 5.1.4, and the target channel layout is 7.1. The number of height channels in the original channel layout (4) is greater than the number of height channels in the target channel layout (0), and the number of surround channels in the original channel layout (5) is less than the number of surround channels in the target channel layout (7). The target correction model can be either an upmixing model or an downmixing model.

[0037] The upmixing model can be trained on specified data samples labeled with upmixing parameters. The specified data samples labeled with upmixing parameters include a pair of different channel layouts (front channel layout and rear channel layout, where the front channel layout has at least one channel with a smaller number of channels than the corresponding channel in the rear channel layout), the loudness of the audio data corresponding to the front channel layout, and the scene category. The downmixing model can be trained on specified data samples labeled with downmixing parameters. The specified data samples labeled with downmixing parameters include a pair of different channel layouts (front channel layout and rear channel layout, where the front channel layout has at least one channel with a larger number of channels than the corresponding channel in the rear channel layout), the loudness of the audio data corresponding to the front channel layout, and the scene category.

[0038] Furthermore, it should be noted that the number of correction parameters, i.e., upmixing or downmixing parameters, is usually determined by the difference between the channel layout before and after correction. For example, downmixing from a 7.1.4 channel layout to a 5.1 channel layout requires 6 parameters, of which 3 are applied to the surround channels and the remaining 3 to the height channels. The values ​​of the correction parameters are usually determined by the scene category and loudness. For example, during downmixing, for dialogue scenes, the parameter values ​​for the height channels are smaller, and the parameter values ​​for the surround channels are larger, which helps to make the dialogue clearer; for game scenes, the parameter values ​​for the height channels are larger, and the parameter values ​​for the surround channels are smaller, which helps to enhance the directionality and spatiality of the audio. During upmixing, for game scenes, the parameter values ​​enhance the environmental signal; for dialogue scenes, the parameter values ​​attenuate the environmental signal.

[0039] In some alternative approaches, a target correction model is determined based on the number relationship between each channel in the original channel layout and each corresponding channel in the target channel layout. This includes: in response to the determination that the number of each channel in the surround channel, bass channel, and height channel of the original channel layout is greater than or equal to the number of corresponding channels in the target channel layout, the target correction model is determined to be a downmixing model.

[0040] In this implementation, if the original channel layout of the original audio data does not match the target channel layout of the playback device, and the number of each channel in the surround channel, bass channel, and height channel of the original channel layout is greater than or equal to the number of corresponding channels in the target channel layout, then the target correction model can be determined as the downmixing model.

[0041] Specifically, the original channel layout is 7.1.4, and the target channel layout is 5.1. The number of surround channels in the original channel layout (7) is greater than the number of surround channels in the target channel layout (5), and the number of height channels in the original channel layout (4) is greater than the number of height channels in the target channel layout (0). Therefore, the target correction model can be determined to be a submixing model.

[0042] This implementation improves the accuracy of determining the target correction model by ensuring that the number of each channel in the original channel layout's surround channel, bass channel, and height channel is greater than or equal to the number of channels corresponding to the target channel layout, thus identifying the target correction model as a submixing model.

[0043] In some alternative approaches, a target correction model is determined based on the number relationship between each channel in the original channel layout and each corresponding channel in the target channel layout. This includes determining the target correction model as an upmixing model in response to the fact that the number of each channel in the surround channel, bass channel, and height channel of the original channel layout is less than or equal to the number of corresponding channels in the target channel layout.

[0044] In this implementation, if the original channel layout of the original audio data does not match the target channel layout of the playback device, and the number of each channel in the surround channel, bass channel and height channel of the original channel layout is less than or equal to the number of corresponding channels in the target channel layout, then the target correction model can be determined as an upmixing model.

[0045] Specifically, the original channel layout is 5.1, and the target channel layout is 5.1.4. The number of height channels in the original channel layout is 0, which is less than the number of height channels in the target channel layout is 4. Therefore, the target correction model can be determined to be an upmixing model.

[0046] This implementation improves the accuracy of determining the target correction model by responding to the fact that the number of each channel in the surround channel, bass channel, and height channel of the original channel layout is less than or equal to the number of channels corresponding to the target channel layout, thus determining the target correction model as an upmixing model.

[0047] Step 202: Obtain the scene category of the original audio data.

[0048] In this embodiment, the executing entity can determine the scene category of the original audio data based on the original audio data and a preset audio data and scene category comparison table, or it can input the original audio data into a preset classification model to obtain the scene category of the original audio data. This application does not limit this.

[0049] The pre-defined classification model can be trained based on audio data samples labeled with scene categories.

[0050] In some optional ways, the scene category of the raw audio data can be obtained, including: inputting the raw audio data into a preset classification model to obtain the scene category of the raw audio data.

[0051] In this implementation, the executing entity can input the original audio data into a preset classification model to obtain the scene category of the original audio data.

[0052] The scene category can be movie scene, game scene, dialogue scene, etc., and this application does not limit it.

[0053] Here, the pre-defined classification model includes a feature extractor, a time-related vector extractor, and a predictor.

[0054] The feature extractor is used to obtain audio features from the original audio data, the time correlation vector extractor is used to extract the audio features output by the feature extractor and the time correlation vector of the previously obtained features, and the predictor is used to determine the scene category based on the audio features and the time correlation vector.

[0055] Here, the feature extractor is usually composed of convolutional layers and fully connected layers, and the predictor, i.e. the classifier, can be composed of multiple fully connected layers. Specifically, the feature extractor can be composed of three convolutional layers and one fully connected layer, and the predictor can be composed of two fully connected layers.

[0056] This implementation improves the accuracy of the determined scene categories by inputting the raw audio data into a preset classification model.

[0057] Step 203: Input the first specified data into the target correction model to obtain the correction parameters.

[0058] In this embodiment, the executing entity can input the first specified data into the target correction model determined above to obtain correction parameters, wherein the correction parameters can be the first upper mixing parameter corresponding to the upper mixing model or the first lower mixing parameter corresponding to the lower mixing model.

[0059] Here, the first specified data includes: the loudness of the original audio data, the scene category, the original channel layout, and the target channel layout.

[0060] Specifically: In response to determining that the target modified model is a lower mixing model, specified data is input into the lower mixing model to obtain the first lower mixing parameters. In response to determining that the target model is an upper mixing model, specified data is input into the upper mixing model to obtain the first upper mixing parameters.

[0061] See also Figure 3 , Figure 3 This is a schematic diagram of an application scenario of the correction parameter generation method according to this embodiment.

[0062] The execution entity 301 obtains the original audio data, such as game audio. The channel layout of the original audio data is 7.1.4. The target channel layout of the playback device, such as a VR / AR device, is 5.1. In response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, a target correction model is determined based on the number relationship between each channel of the original channel layout and each corresponding channel of the target channel layout, as shown in the following mixing model; the scene category of the original audio data is obtained, such as a game scene; the first specified data 302 is input into the target correction model 303 to obtain the correction parameters 304, as shown in the following mixing parameters.

[0063] The correction parameter generation method provided in the embodiments of this disclosure, in response to determining that the original channel layout of the original audio data does not match the target channel layout of the playback device, determines a target correction model based on the quantitative relationship between each channel of the original channel layout and each corresponding channel of the target channel layout; obtains the scene category of the original audio data; inputs the first specified data into the target correction model to obtain correction parameters, fully considering the impact of the scene classification of the audio data on audio processing, and improving the accuracy and reliability of the generated correction parameters for correcting audio data.

[0064] Further reference Figure 4a This illustrates a flow 400 of another embodiment of the modified parameter generation method. In this embodiment, the flow 400 of the modified parameter generation method may include the following steps:

[0065] Step 401: In response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, a target correction model is determined based on the number relationship between each channel in the original channel layout and each corresponding channel in the target channel layout.

[0066] In this embodiment, the implementation details and technical effects of step 401 can be found in the description of step 201, and will not be repeated here.

[0067] Step 402: Obtain the scene category of the raw audio data.

[0068] In this embodiment, the implementation details and technical effects of step 402 can be found in the description of step 202, and will not be repeated here.

[0069] Step 403: Input the specified data into the target correction model to obtain the correction parameters.

[0070] In this embodiment, the implementation details and technical effects of step 403 can be found in the description of step 203, and will not be repeated here.

[0071] Step 404: Correct the original audio data based on the correction parameters to obtain the first audio data.

[0072] In this embodiment, after obtaining the correction parameters (first upmixing parameters or first downmixing parameters), the execution entity can correct the original audio data according to the correction parameters to obtain the corrected audio data, namely the first audio data, and play the first audio data on the playback device with the corresponding target channel layout.

[0073] Specifically, in response to determining that the target correction model is a downmixing model, all or part of the channel data in the original audio data can be weighted based on the first downmixing parameter, and the weighted data can be merged into the corresponding channel of the target channel layout to obtain the first audio data; in response to determining that the target correction model is an upmixing model, the original audio data can be weighted based on the first upmixing parameter to obtain the first audio data.

[0074] Furthermore, the channel layout of the first audio data may or may not match the target channel layout; this application does not impose any limitations on this.

[0075] If the channel layout of the first audio data matches the target channel layout, the first audio data can be identified as the target audio data.

[0076] In some alternative approaches, the original audio data is modified based on the modification parameters to obtain the first audio data, including: in response to determining that the target modification model is a downmixing model, weighting the data of each channel of the original audio data based on the first downmixing parameters, and merging the weighted data into the corresponding channel of the target channel layout to obtain the first audio data.

[0077] In this implementation, if the target correction model is a downmixing model, the execution entity can perform weighted processing on the original audio data based on the first downmixing parameter, and merge the weighted data into the corresponding channel of the target channel layout to obtain the first audio data.

[0078] Specifically, such as Figure 4b As shown, the original channel layout of the original audio data (such as game audio) is 7.1.4. The number of surround channels, bass channels, and height channels are all greater than or equal to the corresponding number of channels in the target channel layout 5.1 of the playback device (such as a TV). Therefore, the target correction model is a downmixing model. The first specified data, namely the original channel layout, the target channel layout, the scene category, and the loudness of the original audio data, are input into the downmixing model to obtain the downmixing parameters. Specifically, the downmixing parameters are a = 0.18366132, b = 0.46504679, c = 0.90630779, d = 0.96155926, e = 0.45315389, and f = 0.27459750. Here, a, c, and d are applied to the surround channels, and b, e, and f are applied to the height channels. Because b and e are relatively large, the height channels can be largely retained in the surround channels, avoiding the adverse effects of directly discarding channels.

[0079] Furthermore, the L, R, C, Lfe, Ls, and Rs audio data from the game are directly copied to the corresponding channels on the TV. Additionally, the Ls and Rs channel data from the game audio are multiplied by a, and the Hfl and Hfr channel data are multiplied by b, then merged into the L and R channels of the TV; the Ls and Rs channel data are multiplied by c, the Lb and Rb channel data by d, the Hfl and Hfr channel data by e, and the Hbl and Hbr channel data by f, then merged into the Ls and Rs channels of the TV. Finally, the corrected 5.1 channel game audio, i.e., the first audio data, is played on the TV.

[0080] This implementation responds to the determination that the target correction model is a downmixing model, performs weighted processing on the channel data of the original audio data based on the first downmixing parameter, and merges the weighted data into the corresponding channel of the target channel layout to obtain the first audio data. This avoids discarding all or part of the redundant channel data, making the generated audio data more consistent with the original audio content and effectively enhancing the sound quality.

[0081] In some alternative approaches, the original audio data is modified based on the modification parameters to obtain the first audio data, including: in response to determining that the target modification model is an upmixing model, extracting the left and right channel data of the original audio data and separating the sound environment signal from the left and right channel data; performing weighted processing on the left and right channel data and the sound environment signal based on the first upmixing parameters, and filling the weighted data into the channels missing from the original audio data relative to the target audio data to obtain the first audio data.

[0082] In this implementation, if the target correction model is an upmixing model, the execution entity can first extract the left and right channel data of the original audio data, and separate the sound environment signal from the left and right channel data. Based on the first upmixing parameter, the left and right channel data and the sound environment signal are weighted and processed, and the weighted data is filled into the channels of the original audio data that are missing relative to the target audio data.

[0083] Here, the ambient sound signal separated from the left and right channel data can be achieved in the following way:

[0084] 1. Establish a signal model. The left and right channels of the audio signal can be described as a weighted sum of multiple source signals and additively uncorrelated ambient signals in the left and right channels. The weight of a single source is called a translation coefficient. The translation coefficients are between 0 and 1, and their sum of squares should equal 1.

[0085] To invert the signal model and approximately recover the source signal and its translation parameters, two simplifications are required.

[0086] i) At a certain instant and in a certain frequency band, there is only one dominant source, and the contributions of other sources are close to zero.

[0087] ii) The environmental signals on the left and right sound very similar, but they are unrelated and originate from a single environmental signal.

[0088] Combining these two assumptions, we obtain a simplified signal model.

[0089] 2. Source direction prediction

[0090] The amplitude of the mixed ambient signal can be assumed to be much smaller than that of the direct signal. This also means that the left and right forces mainly depend on the weighted direct signal power, which can be estimated from the power of the left and right stereo channels. A simple estimate of the sub-band signal power can be determined by recursive averaging of the coefficients. Thus, based on the power of the left and right stereo channels, estimates of the position index and angle can be obtained. Using the translation coefficients derived above and the estimated source direction, a mathematical transformation is performed on the signal model to obtain the direct signal and ambient signal components. The ambient signal serves as the source to supplement the missing channels.

[0091] This implementation method, in response to determining the target correction model as an upmixing model, extracts the left and right channel data of the original audio data and separates the sound environment signal from the left and right channel data; based on the first upmixing parameter, it performs weighted processing on the left and right channel data and the sound environment signal, and fills the missing channels of the original audio data relative to the target audio data with the weighted processed data to obtain the first audio data, which expands the original audio channels, avoids excessively low audio quality and excessive performance of user devices, and effectively improves the user experience.

[0092] In some alternative approaches, the method further includes: in response to determining that the target correction model is a downmixing model and that the channel layout of the first audio data does not match the target channel layout, inputting second specified data into the upmixing model to obtain second upmixing parameters; and correcting the first audio data based on the second upmixing parameters to obtain the target audio data.

[0093] In this implementation, in response to determining that the target correction model is a downmixing model and that the channel layout of the first audio data does not match the target channel layout, the executing entity can further input the second specified data into the upmixing model to obtain the second upmixing parameters; and correct the first audio data based on the second upmixing parameters to obtain the target audio data.

[0094] Here, the second specified data includes: the loudness of the first audio data, the channel layout of the first audio data, the scene category, and the target channel layout.

[0095] The target audio data's channel layout is matched to the target channel layout. The second upmixing parameter is used to indicate the parameters for filling the missing channels in the first audio data's channel layout relative to the target channel layout with the ambient sound signal separated from the first audio data.

[0096] Specifically, the original channel layout of the raw audio data (such as movie audio) is 7.1, and its number of surround channels is greater than the number of height channels is less than the number of channels corresponding to the channel layout of the playback device (such as a user's home theater) of 5.1.2, so it needs to be downmixed and upmixed.

[0097] The specific downmixing parameters are a = 0.3426864, b = 0.92672619, and c = 0.91518348. 'a' is used as a coefficient to merge the surround channels into left and right channels, while 'b' and 'c' are used to merge the front and rear surround channels into left and right surround channels. Due to the cinematic setting, the downmixing parameter values ​​are relatively large to ensure a good user experience.

[0098] Furthermore, the L, R, C, Lfe, Ls, and Rs audio data from the movie are directly copied to the corresponding channels in the home theater. In addition, the Ls and Rs channel data from the movie audio are multiplied by a and then merged into the L and R channels of the TV; the Ls and Rs channel data are multiplied by b, and the Lb and Rb channel data are multiplied by c and merged into the Ls and Rs channels of the home theater.

[0099] The downmixed audio data, i.e., the first audio data, has channels L, R, C, Lfe, Ls, and Rs, and is missing two height channels relative to the target channel layout. The loudness, channel layout, target channel layout, and scene category of the second specified data (i.e., the first audio data's loudness, channel layout, and scene category) are input into the upmixing model to obtain the second upmixing parameters, such as a = 0.73242931 and b = 0.43523491. A signal model is built using the L and R channels of the first audio data to separate the left and right ambient signals. The first audio data is then used to:

[0100] The L channel data is multiplied by a, the left ambient signal is multiplied by b, and then merged into the H1 channel of the home theater.

[0101] The R channel data is multiplied by a, the right ambient signal is multiplied by b, and then merged into the HR channel of the home theater.

[0102] After upmixing, the newly expanded channels are Hl and Hr. The Hl and Hr channels are mixed with the 5.1 channel audio of the previous first audio data to generate the 5.1.2 channel audio, which is the target audio data.

[0103] Finally, play the 5.1.2 channel movie audio on the home theater speakers.

[0104] This implementation responds to the determination that the target correction model is a downmixing model and that the channel layout of the first audio data does not match the target channel layout by inputting the second specified data into the upmixing model to obtain the second upmixing parameters; the first audio data is corrected based on the second upmixing parameters to obtain the target audio data, which further improves the effectiveness of restoring the original audio data to the target channel and enhances the user experience.

[0105] In some alternative approaches, the method further includes: in response to determining that the target correction model is an upmixing model and that the channel layout of the first audio data does not match the target channel layout, inputting second specified data into the downmixing model to obtain second downmixing parameters; and correcting the first audio data based on the second downmixing parameters to obtain the target audio data.

[0106] In this implementation, in response to determining that the target correction model is an upmixing model and that the channel layout of the first audio data does not match the target channel layout, the executing entity can further input the second specified data into the downmixing model to obtain the second downmixing parameters; and correct the first audio data based on the second downmixing parameters to obtain the target audio data.

[0107] The second specified data includes the channel layout of the first audio data, the loudness of the first audio data, the target channel layout, and the scene category.

[0108] Here, the channel layout of the target audio data is matched with the target channel layout, and the second downmixing parameter is used to indicate the parameters for merging the first audio data into the corresponding channel of the target channel layout.

[0109] This implementation responds to the determination that the target correction model is an upmixing model and that the channel layout of the first audio data does not match the target channel layout by inputting the second specified data into the downmixing model to obtain the second downmixing parameters; the first audio data is corrected based on the second downmixing parameters to obtain the target audio data, which further improves the effectiveness of restoring the original audio data to the target channel and enhances the user experience.

[0110] The above embodiments of this application, and Figure 2 Compared with the corresponding embodiments, the process 400 of the correction parameter generation method in this embodiment reflects the correction of the original audio data based on the correction parameters to obtain the first audio data. This realizes the correction of the original audio data based on the correction parameters, improves the authenticity of the generated audio data for playback, and enhances the user experience.

[0111] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a correction parameter generation apparatus, which is similar to... Figure 2Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0112] like Figure 5 As shown, the correction parameter generation device 500 of this embodiment includes: a determination module 501, an acquisition module 502, and a generation module 503.

[0113] The determining module 501 can be configured to determine a target correction model in response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, based on the number relationship between each channel of the original channel layout and each corresponding channel of the target channel layout.

[0114] The acquisition module 502 can be configured to acquire scene categories of raw audio data.

[0115] The generation module 503 can be configured to input the first specified data into the target correction model to obtain correction parameters.

[0116] In some optional embodiments of this invention, the apparatus further includes a correction module configured to correct the original audio data based on correction parameters to obtain first audio data.

[0117] In some optional embodiments of this example, the correction module is further configured to, in response to determining that the target correction model is a downmixing model, perform weighted processing on the channel data of the original audio data based on the first downmixing parameter, and merge the weighted data into the corresponding channel of the target channel layout to obtain the first audio data.

[0118] In some optional embodiments of this example, the correction module is further configured to, in response to determining that the target correction model is an upmixing model, extract the left and right channel data of the original audio data and separate the sound environment signal from the left and right channel data; based on the first upmixing parameter, perform weighted processing on the left and right channel data and the sound environment signal, and fill the weighted data into the channels of the original audio data that are missing relative to the target audio data to obtain the first audio data.

[0119] In some optional embodiments of this invention, the apparatus further includes: an upmixing module configured to, in response to determining that the target correction model is a downmixing model and that the channel layout of the first audio data does not match the target channel layout, input second specified data into the upmixing model to obtain second upmixing parameters; and correct the first audio data based on the second upmixing parameters to obtain the target audio data.

[0120] In some optional embodiments of this invention, the apparatus further includes: a downmixing module configured to, in response to determining that the target correction model is an upmixing model and that the channel layout of the first audio data does not match the target channel layout, input second specified data into the downmixing model to obtain second downmixing parameters; and correct the first audio data based on the second downmixing parameters to obtain the target audio data.

[0121] In some optional embodiments of this example, the acquisition module is further configured to: input the original audio data into a preset classification model to obtain the scene category of the original audio data.

[0122] In some optional embodiments of this example, the determining module is further configured to: determine the target correction model as a downmixing model in response to the determination that the number of each channel in the surround channel, bass channel and height channel of the original channel layout is greater than or equal to the number of channels corresponding to the target channel layout.

[0123] In some optional embodiments of this example, the determining module is further configured to: determine the target correction model as an upmixing model in response to the determination that the number of each channel in the surround channel, bass channel and height channel of the original channel layout is less than or equal to the number of channels corresponding to the target channel layout.

[0124] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0125] like Figure 6 The diagram shown is a block diagram of an electronic device according to an embodiment of the modified parameter generation method of this application.

[0126] 600 is a block diagram of an electronic device according to an embodiment of the modified parameter generation method of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0127] like Figure 6As shown, the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take the 601 processor as an example.

[0128] The memory 602 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the correction parameter generation method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the correction parameter generation method provided in this application.

[0129] Memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the correction parameter generation method in the embodiments of this application (e.g., appendix). Figure 5 The processor 601 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 602, thereby implementing the correction parameter generation method in the above method embodiments.

[0130] Memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device that generates the correction parameters. Furthermore, memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories can be connected to the electronic device that generates the correction parameters via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The electronic device for modifying the parameter generation method may further include an input device 603 and an output device 604. The processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0132] Input device 603 can receive input digital or character information, as well as key signal inputs related to user settings and function control of electronic devices for quality monitoring of live video streams, such as touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 604 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0133] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0138] According to the technical solution of the embodiments of this application, the impact of scene classification of audio data on audio processing is fully considered, thereby improving the accuracy and reliability of the generated correction parameters for correcting audio data.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating correction parameters, the method comprising: In response to the determination that the original channel layout of the original audio data does not match the target channel layout of the playback device, a target correction model is determined based on the number relationship between each channel of the original channel layout and each corresponding channel of the target channel layout. The target correction model is an upmixing model or an downmixing model. Obtain the scene category of the raw audio data; The first specified data is input into the target correction model to obtain correction parameters. The correction parameters are either the first upmixing parameter corresponding to the upmixing model or the first downmixing parameter corresponding to the downmixing model. The first specified data includes the loudness of the original audio data, the scene category, the original channel layout, and the target channel layout. The original channel layout and the target channel layout are used to determine the number of correction parameters, and the loudness of the original audio data and the scene category are used to determine the value of the correction parameters.

2. The method according to claim 1, further comprising: The original audio data is corrected based on the correction parameters to obtain the first audio data.

3. The method according to claim 2, wherein, The step of correcting the original audio data based on the correction parameters to obtain the first audio data includes: In response to determining that the target correction model is a downmixing model, the original audio data for each channel is weighted based on the first downmixing parameter, and the weighted data is merged into the corresponding channel of the target channel layout to obtain the first audio data.

4. The method according to claim 2, wherein, The step of correcting the original audio data based on the correction parameters to obtain the first audio data includes: In response to determining that the target correction model is an overmixing model, the left and right channel data of the original audio data are extracted and the sound environment signal is separated from the left and right channel data; Based on the first upmixing parameter, the left and right channel data and the sound environment signal are weighted and processed, and the weighted data is filled into the channels where the original audio data is missing relative to the target audio data to obtain the first audio data.

5. The method according to claim 2, further comprising: In response to determining that the target correction model is a downmixing model and that the channel layout of the first audio data does not match the target channel layout, the second specified data is input into the upmixing model to obtain the second upmixing parameters, wherein the second specified data includes: the loudness of the first audio data, the scene category, the channel layout of the first audio data, and the target channel layout; The first audio data is corrected based on the second upmixing parameter to obtain target audio data, wherein the channel layout of the target audio data matches the target channel layout.

6. The method according to claim 2, further comprising: In response to determining that the target correction model is an upmixing model and that the channel layout of the first audio data does not match the target channel layout, the second specified data is input into the downmixing model to obtain the second downmixing parameters, wherein the second specified data includes: the loudness of the first audio data, the scene category, the channel layout of the first audio data, and the target channel layout; The first audio data is corrected based on the second downmixing parameter to obtain target audio data, wherein the channel layout of the target audio data matches the target channel layout.

7. The method according to claim 1, wherein, The scenarios for acquiring the raw audio data include: The original audio data is input into a preset classification model to obtain the scene category of the original audio data. The preset classification model includes a feature extractor, a time-related vector extractor, and a predictor. The feature extractor is used to extract the audio features of the original audio data. The time-related vector extractor is used to extract the audio features and the time-related vector of the previously obtained features. The predictor is used to determine the scene category based on the audio features and the time-related vector.

8. The method according to claim 1, wherein, The determination of the target correction model based on the quantitative relationship between each channel of the original channel layout and each corresponding channel of the target channel layout includes: In response to the fact that the number of each channel in the original channel layout, including the surround channel, bass channel, and height channel, is greater than or equal to the number of channels corresponding to the target channel layout, the target correction model is determined to be the undermix model.

9. The method according to claim 1, wherein, The determination of the target correction model based on the quantitative relationship between each channel of the original channel layout and each corresponding channel of the target channel layout includes: In response to the fact that the number of each channel in the original channel layout, including the surround channel, bass channel, and height channel, is less than or equal to the number of channels corresponding to the target channel layout, the target correction model is determined to be an upmixing model.

10. A correction parameter generation apparatus, the apparatus comprising: The determination module is configured to, in response to determining that the original channel layout of the original audio data does not match the target channel layout of the playback device, determine a target correction model based on the number relationship between each channel of the original channel layout and each corresponding channel of the target channel layout, wherein the target correction model is an upmixing model or an downmixing model. The acquisition module is configured to acquire the scene category of the raw audio data; The generation module is configured to input first specified data into the target correction model to obtain correction parameters. The correction parameters are first upmixing parameters corresponding to the upmixing model or first downmixing parameters corresponding to the downmixing model. The first specified data includes the loudness of the original audio data, the scene category, the original channel layout, and the target channel layout. The original channel layout and the target channel layout are used to determine the number of correction parameters, and the loudness of the original audio data and the scene category are used to determine the value of the correction parameters.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores information that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

Images