Audio data processing method, computer device and storage medium

Abstract

The application discloses an audio data processing method, a computer device and a storage medium. The method can be applied to various scenes such as cloud technology and artificial intelligence. The method comprises the following steps: performing feature extraction on to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, wherein the feature information of each audio segment comprises one or more of chord information, pitch information and envelope information; then performing relaxation processing on the feature information of each audio segment to determine a sleep-aiding audio structure file, wherein the sleep-aiding audio structure file is used for indicating the music characteristics of sleep-aiding audio data; and finally performing encoding processing on the sleep-aiding audio structure file to obtain sleep-aiding audio data. Through the method, a user can customize sleep-aiding music with various styles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an audio data processing method, a computer device and a storage medium. BACKGROUND

[0002] In the fast-paced era, sleep-aid music is very popular because it can help people relieve stress and fall asleep quickly. Currently, most sleep-aid music is displayed in the form of finished products. If a user wants to listen to it, he or she can directly open the application and click to play. However, since these sleep-aid music are finished products, the style is single and the quantity is limited. Therefore, how to customize and generate sleep-aid music with variable styles becomes a technical problem to be solved. SUMMARY

[0003] The embodiments of the present application provide an audio data processing method, a computer device and a storage medium. Through the method, a user can customize and generate sleep-aid music with variable styles.

[0004] In a first aspect, the embodiments of the present application disclose an audio data processing method, which comprises the following steps.

[0005] extracting features of the audio data to be processed to obtain feature information of each audio segment included in the audio data to be processed, wherein the feature information of each audio segment comprises one or more of chord information, pitch information and envelope information;

[0006] performing relaxation processing on the feature information of each audio segment to determine a sleep-aid audio structure file, wherein the sleep-aid audio structure file is used to indicate music characteristics of sleep-aid audio data;

[0007] performing encoding processing on the sleep-aid audio structure file to obtain sleep-aid audio data.

[0008] In a second aspect, the embodiments of the present application disclose an audio data processing device, which comprises the following steps.

[0009] a processing unit configured to extract features of the audio data to be processed to obtain feature information of each audio segment included in the audio data to be processed, wherein the feature information of each audio segment comprises one or more of chord information, pitch information and envelope information;

[0010] the processing unit is further configured to perform relaxation processing on the feature information of each audio segment to determine a sleep-aid audio structure file, wherein the sleep-aid audio structure file is used to indicate music characteristics of sleep-aid audio data;

[0011] the processing unit is further configured to perform encoding processing on the sleep-aid audio structure file to obtain sleep-aid audio data.

[0012] In a third aspect, the embodiments of the present application disclose a computer device, which comprises a processor adapted to implement one or more computer programs; and a computer storage medium storing the one or more computer programs, the one or more computer programs being adapted to be loaded and executed by the processor to implement the audio data processing method described above.

[0013] In a fourth aspect, the embodiments of the present application disclose a computer readable storage medium storing one or more computer programs, the one or more computer programs being adapted to be loaded and executed by a processor to implement the audio data processing method described above.

[0014] In a fifth aspect, the embodiments of the present application disclose a computer program product comprising a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to cause the computer device to implement the audio data processing method described above.

[0015] In the embodiments of the present application, the audio data processing method mainly comprises: performing feature extraction on to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, the feature information of each audio segment comprising one or more of chord information, pitch information and envelope information; performing relaxation processing on the feature information of each audio segment to determine a sleep-aiding audio structure file, wherein the sleep-aiding audio structure file is used to indicate the music characteristics of sleep-aiding audio data; and finally performing encoding processing on the sleep-aiding audio structure file to obtain sleep-aiding audio data. The method can process any audio data, and the sleep-aiding audio data generated after processing all has the effect of sleep aiding. Since the types of any audio data are various and can be adjusted differently, the audio data processing method of the present application can enable a user to generate sleep-aiding music with variable styles. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is an architecture diagram of an audio data processing system disclosed by the embodiments of the present application;

[0018] Figure 2 is a flowchart of an audio data processing method disclosed by the embodiments of the present application;

[0019] Figure 3 is a flowchart of another audio data processing method disclosed by an embodiment of the present application;

[0020] Figure 4 is an interface display diagram for displaying audio data to be processed disclosed by an embodiment of the present application;

[0021] Figure 5 is a structural diagram of an audio data processing device disclosed by an embodiment of the present application;

[0022] Figure 6 is a structural diagram of a computer device disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] In order to effectively improve the diversity of sleep aid music, the present application provides an audio data processing method, which can generate sleep aid music from any song according to user selection. In the implementation process, the audio data processing method also involves artificial intelligence and machine learning technologies.

[0025] Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0026] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0027] The present application relates to machine learning under artificial intelligence technology. Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0028] Please refer to Figure 1 , Figure 1 is an architecture diagram of an audio data processing system disclosed in the application embodiment, as Figure 1 shown, the audio data processing system 100 can at least include a terminal device 101 and a computer device 102, wherein the terminal device 101 and the computer device 102 can be communicatively connected, and the connection mode can include wired connection and wireless connection, which is not limited herein. In the specific implementation process, the terminal device 101 is mainly used for user to perform related operations, such as clicking to start the operation of sleep-aiding music, and can also be used for displaying results, in the present application, such as displaying the to-be-processed audio data and the sleep-aiding audio data on the display screen of the terminal device 101, under the operation of the user, the sleep-aiding audio data and the to-be-processed audio data can be played through the terminal device 101; the computer device 102 is mainly used for obtaining the to-be-processed audio data and processing the to-be-processed audio data, mainly adjusting the feature information of the to-be-processed audio data, so as to obtain the sleep-aiding audio data. In addition, the computer device 102 can also be used for feeding back the obtained sleep-aiding audio data to the terminal device 101.

[0029] In a possible implementation manner, the terminal device 101 mentioned above includes but is not limited to a mobile phone, a computer, a smart voice interactive device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc.; the computer device 102 mentioned above can be a server, which can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. Figure 1 The architecture diagram of the audio data processing system is only exemplary and is not limited thereto. For example, Figure 1The computer device 102 in the above embodiment can be deployed as a node in a blockchain network, or the computer device 102 can access the blockchain network, so that the computer device 102 can upload the sleep-aiding audio data to the blockchain network for storage to prevent internal data from being tampered with, thereby ensuring data security.

[0030] In combination with the above audio data processing system, the audio data processing method of the embodiment of the present application can generally include: the computer device 102 performs feature extraction on the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, and the feature information of each audio segment includes one or more of chord information, pitch information, and envelope information; then the feature information of each audio segment is processed to determine a sleep-aiding audio structure file, where the sleep-aiding audio structure file is used to indicate the music characteristics of the sleep-aiding audio data; finally, the sleep-aiding audio structure file is encoded to obtain the sleep-aiding audio data. The method can process any audio data, and the sleep-aiding audio data generated after processing has a sleep-aiding effect. Since there are various types of audio data and different adjustments can be made, the audio data processing method of the present application can enable users to generate sleep-aiding music with various styles.

[0031] The embodiment of the present application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation, and auxiliary driving, and the audio data processing system can also be deployed on a node on a blockchain. At the same time, the audio data processing method in the embodiment of the present application can not only be applied to the generation of sleep-aiding music, but also be applied to the generation of other types of music, such as rock and electronic music.

[0032] It can be understood that in the specific embodiments of the present application, data related to the to-be-processed audio data and the tone library is involved, and when the above embodiments of the present application are applied to specific products or technologies, the collection, use, and processing of the related data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0033] Based on the above description of the architecture of the audio data processing system, the embodiment of the present application discloses an audio data processing method, please see Figure 2 The audio data processing method disclosed in the embodiment of the present application can be executed by a computer device, and the audio data processing method specifically can include steps S201-S203:

[0034] S201: performing feature extraction on the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, and the feature information of each audio segment includes one or more of chord information, pitch information, and envelope information.

[0035] The characteristic information of each audio segment includes one or more of chord information, pitch information, and envelope information, which is defined in one or more ways to include some special cases. For example, a to-be-processed audio data containing a solo voice does not include chord information, but includes pitch information and envelope information. The audio data processing method provided by the embodiments of the present application is also applicable to this case.

[0036] In a specific implementation process, the to-be-processed audio data can be any type of song. The to-be-processed audio data can be local audio data downloaded by the user and stored in the terminal device, or audio data obtained in real time from the Internet by the user opening an application. In the present application, the to-be-processed audio data can be a music file containing a human voice for subsequent processing.

[0037] In a possible implementation, after obtaining the to-be-processed audio file, the first method can first divide the to-be-processed audio file according to the tempo information of the to-be-processed audio file, that is, divide it into multiple audio segments, which can also be understood as "sections". "Section" is a term in music theory. Then, for each audio segment, the AI technology is used to extract the chord information, pitch information, and envelope information of each audio segment. The second method can be to first use the AI technology to extract the chord information, pitch information, and envelope information of the entire to-be-processed audio file, and then divide the audio segments based on the tempo information of the to-be-processed audio data in sequence, thereby obtaining the chord information, pitch information, and envelope information of each audio segment. The results of the above two methods are the chord information, pitch information, and envelope information of each audio segment, so the present application does not limit which method to use.

[0038] Taking the first method as an example, in the process of obtaining the chord information, pitch information, and envelope information of each audio segment, the BLSTM (Bidirectional Long Short-Term Memory Network) model and the TFLite (TFLite is a toolkit for deploying deep learning models on mobile devices and embedded devices. A trained TF model can be converted, deployed, and optimized in three steps to improve operation speed and reduce memory and video memory usage) inference framework can be used to obtain the tempo information. The tempo is a symbol used in the music score, which is expressed in the form of a fraction. Each music score has a tempo in front of it, and if the rhythm changes in the middle, the changed tempo is marked, such as 2 / 4, 3 / 4, etc. The denominator represents the time value of the beat, that is, how many notes are used as a beat. For example, 2 / 4 represents using a quarter note as a beat, and each section has two beats. The numerator represents the number of beats in each section. As mentioned earlier, 2 / 4 beats are a quarter note as a beat, and each section has two beats. 3 / 4 is a quarter note as a beat, and each section has three beats.

[0039] For chord information, a Deep Chromagram CNN (a kind of deep neural network related to chromagram) model and a TFLite inference framework can be used. Chord refers to a group of sounds with certain interval relationship, that is, three or more sounds are combined vertically according to the superimposed relationship of three or non-three, which is called chord. Three-degree chord: usually has three chords (three-voice chord), seven chords (four-voice chord), nine chords (five-voice), eleven chords (six-voice), and thirteen chords (seven-voice). Non-three-degree chord: usually has hanging chord, powerful chord (i.e. five chord), etc.

[0040] For pitch information, Spleeter (a kind of audio track separation software) can be used to separate the human voice, and then PYIN (pitch cycle estimation algorithm) algorithm can be used to extract the pitch and the time stamp and time value (time interval, i.e. start time to end time) corresponding to the pitch of the separated human voice. Pitch refers to the sound of different tones, i.e. the height of the sound, which is a basic feature of the sound.

[0041] For envelope information, it can be extracted by time domain spectrum extraction method or frequency domain extraction method. Specific processing includes convolution, Fourier transform, Laplace transform, etc. The envelope information of audio data refers to the connecting line between the peaks of each amplitude (frequency) period in the sound (or musical sound), which represents the energy of the song. The emotion of a song can be determined based on the envelope information, such as slow emotion or intense emotion.

[0042] S202: Slow processing of the feature information of each audio segment to determine a sleep-aiding audio structure file, which is used to indicate the music characteristics of the sleep-aiding audio data.

[0043] Connecting the feature information of each audio segment together constitutes the feature information of the to-be-processed audio data. For example, connecting the chord information of each audio segment together obtains the chord information of the to-be-processed audio data.

[0044] In a possible implementation, after the chord information, the pitch information and the envelope information of each audio segment are determined, the accompaniment information, the melody information and the adjusted envelope information of the sleep-aiding audio structure file are determined based on the chord information, the pitch information and the envelope information of each audio segment, and finally the accompaniment information, the melody information and the adjusted envelope information are combined to obtain the sleep-aiding audio structure file. The sleep-aiding audio structure file is used to indicate the music characteristics of the sleep-aiding audio data, and can also be understood as a musical score, which can clearly indicate how many notes each measure includes, the melody and the duration of each note and the like. The characteristic information of the sleep-aiding audio structure file is changed, and is adjusted on the basis of the original information to achieve the sleep-aiding effect. In this application, the sleep-aiding audio structure file can be a midi (Musical Instrument Digital Interface, for music device digital interface) file. The midi file is a descriptive music language, which describes the music information to be played by using bytes. For example, what instrument is used at a certain time, what note is started with, what pitch is ended with, what accompaniment is added and the like. That is, the midi file itself does not contain waveform data, so the midi file is very small. The midi file is not a recorded sound, but information of the sound, and is a set of instructions for indicating how the sound card (combined with a tone library) reproduces music.

[0045] In a possible implementation, the sleep-aiding audio structure file mainly includes the accompaniment information, the melody information and the adjusted envelope information, and obtaining the sleep-aiding audio structure file mainly can include: determining the accompaniment information of the sleep-aiding audio structure file according to the chord information of each audio segment, determining the melody information of the sleep-aiding audio structure file according to the pitch information of each audio segment, determining a note intensity adjustment rule according to the time signature information of the audio data to be processed, adjusting the envelope information of each audio segment according to the note intensity adjustment rule to obtain the adjusted envelope information, and finally determining the sleep-aiding audio structure file based on the accompaniment information, the melody information and the envelope information of the sleep-aiding audio structure file. The processing processes of the accompaniment information, the melody information and the adjusted envelope information are not in sequence, and can be processed simultaneously.

[0046] In a specific implementation process, the accompaniment information of the sleep-aiding audio structure file is determined according to the chord information of each audio segment, which specifically can include: determining a chord adjustment strategy for each audio segment according to the chord information of each audio segment, then adjusting the chord information of each audio segment according to the chord adjustment strategy, adjusting the chord information of each audio segment, and combining the adjusted chord information to obtain initial accompaniment information of the sleep-aiding audio structure file; then adding a chord root note to the initial accompaniment information to obtain the accompaniment information of the sleep-aiding audio structure file. For different chord information, the corresponding chord adjustment strategy is different. Specifically, the chord adjustment strategy can be determined by the chord constituent sound and the playing technique. For example, the constituent sound of a C chord is 1, 3 and 5. Considering the sleep-aiding scene, the full decomposition playing technique of a piano is referred to, such as technique 1: playing 1 and 3 in the first beat, and playing quarter note 5 in the second beat, technique 2: playing 1 and 3 in the first beat, and playing quarter note 1 and 5 in the second beat, and so on. In order to make the accompaniment more varied, the technique can also be randomly changed every four bars. The present application does not make specific limitation on this. The chord root note is to better highlight the sleep-aiding scene. Specifically, the root note can be lowered to the C1 sound area (a sound range, a more lyrical sound), and the duration is two beats. Thus, a low-frequency continuous pulse can be generated, which is beneficial to sleep-aiding.

[0047] In a specific implementation process, the melody information of the sleep-aiding audio structure file is determined according to the pitch information of each audio segment, which specifically can include: determining the note information of each audio segment and the time interval (i.e. time value) of each note according to the pitch information of each audio segment; adjusting the time interval of the notes included in each audio segment according to a time adjustment strategy to obtain the melody information of the sleep-aiding audio structure file. The time adjustment strategy includes: if a first note is connected with a second note in any audio segment, the time interval of the second note is shortened; if the first note is connected with a rest, the time value of the rest is lengthened. The time adjustment strategy is to make the rhythm of the generated target audio file not mechanical. In a specific example, if the time value of a note is connected with the time value of another note, the time value of the other note is randomly shortened, and the shortened value is in the interval of 0 to 32 note time values. If a note is followed by a rest, the time value of the rest is lengthened, and the lengthened value is in the interval of 0 to 32 note time values.

[0048] In a specific implementation process, the envelope information of each audio segment is adjusted according to the note intensity adjustment rule to obtain adjusted envelope information of the sleep-aiding audio structure file. Specifically, the envelope information of each audio segment is adjusted according to the note intensity adjustment rule to obtain initial adjustment envelope information of each audio segment; average envelope values of each audio segment are determined according to the beat information of the to-be-processed audio data and the envelope information of the to-be-processed audio data; and the envelope information of each audio segment is adjusted according to the initial adjustment envelope information of each audio segment and the average envelope values of each audio segment to obtain adjusted envelope information of the sleep-aiding audio structure file. For example, for to-be-processed audio data with a time signature of 4 / 4, the intensity of each beat is adjusted according to the knowledge of strong, weak, and secondary strong beats. Specifically, the intensity of a strong beat is A, the intensity of a weak beat is A*80%, and the intensity of a secondary strong beat is A*90%. In order to restore the emotional color of the to-be-processed audio data, the intensity is preliminarily adjusted to obtain initial adjustment envelope information A1. Then, the intensity is adjusted again according to the envelope value of each beat of the to-be-processed audio data. An average value is calculated for the envelope value of each beat of the to-be-processed audio data to obtain an average envelope value A2. Finally, the envelope information of each audio segment is adjusted based on the initial adjustment envelope information and the average envelope value to obtain adjusted envelope information = A1*(envelope value of a beat / A2). Other adjustment methods can also be used, which are not listed here.

[0049] Optionally, based on the sleep-aiding audio structure file determined above, a sustain pedal control word can be added to each interval of the sleep-aiding audio structure file to make the generated target audio file more soothing and more sleep-aiding. The sustain pedal is mainly used to prolong the vibration time of the string. It can be understood that the sustain pedal is used to push a layer of felt originally close to the string to prolong the vibration time of the string.

[0050] The accompaniment information, melody information, and envelope information of each audio segment can be adjusted in sequence to obtain the sleep-aiding audio structure file. In this way, each information is adjusted in sequence, and the accompaniment information, melody information, and envelope information are not sorted, and any one can be adjusted first. Alternatively, the accompaniment information, melody information, and envelope information of each audio segment can be adjusted, and then the adjusted audio segments are combined to obtain the sleep-aiding audio structure file. In this way, each audio segment is processed in segments, and then combined. The two methods are not essentially different. One is a whole processing method, and the other is a partial processing method followed by a whole combination. The present application does not make a specific limitation on this.

[0051] S203: encode the sleep-aiding audio structure file to obtain sleep-aiding audio data.

[0052] In the present application, the sleep-aiding audio data can be understood as a sleep-aiding music file, which can be directly played by the user, or downloaded and stored in the local database of the terminal device.

[0053] In a possible implementation, the sleep-aiding audio structure file is converted into pulse code modulation data, which can be converted from a midi file into PCM data. In audio and video, PCM is a method for representing a sampled analog signal by a digital signal. PCM data can be directly stored on a medium, or stored or transmitted after being processed by a codec. In actual implementation, the midi file can be set into an open source library fluidsynth, and a timbre library (set in the form of a path) is set for fluidsynth, so that fluidsynth converts the midi file into PCM data. In other words, the midi file and the timbre library are written into the open source library at the same time, to realize the conversion of the midi file into PCM data. The timbre library is a sound corresponding to a note in the midi file, for example, the "1 (do)" sound in the midi file can be found in the corresponding sound in the timbre library. Then, the pulse code modulation data is preprocessed to obtain processed pulse code modulation data, which includes one or both of sound mixing and environmental sound effect processing. Finally, the processed pulse code modulation data is encoded to obtain sleep-aiding audio data. Since the obtained data is encoded data, it needs to be encoded to obtain sleep-aiding audio data. The sound mixing processing specifically refers to mixing the PCM data corresponding to white noise and the PCM data obtained based on the midi file to obtain the processed pulse code modulation data. The white noise can be mountain, stream, etc. The environmental sound effect processing specifically refers to mixing the PCM data corresponding to different environmental sound effects and the PCM data obtained based on the midi file to obtain the processed pulse code modulation data. For example, the surround sound processing is performed to realize the empty and spiritual feeling, and the sleep-aiding rhythm is added, which is more conducive to sleep-aiding. The environmental sound effects can also include panoramic surround sound effects, super-low heavy sound effects, melodious and nostalgic effects, etc. In the present application, they are not listed one by one. It should be noted that the timbre library (i.e., the sample library) includes a collection of various timbres, including the performance of various musical instruments, the singing of various human voices, the narration, and the recording of various natural and artificial sounds. In summary, it is all kinds of sounds that can be heard and created by humans.

[0054] Optionally, the sleep-aiding audio structure file can be format-converted to obtain pulse code modulation data, and then the pulse code modulation data is directly encoded to obtain sleep-aiding audio data. That is, the pulse code modulation data can not be preset processed (one or both of the mixing processing and the environmental sound effect processing), and sleep-aiding audio data with a sleep-aiding effect can also be obtained.

[0055] In the embodiment of the present application, the computer device extracts features of the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, and the feature information of each audio segment includes one or more of chord information, pitch information and envelope information. Then, the feature information of each audio segment is adjusted to determine a sleep-aiding audio structure file. Finally, sleep-aiding audio data is generated based on the sleep-aiding audio structure file and a timbre library. The audio data processing method of the present application can process any audio data, and the sleep-aiding audio data generated after processing has a sleep-aiding effect. Since the music type of any audio data is not limited, the audio data processing method of the present application can flexibly generate sleep-aiding music.

[0056] Based on the above-described method embodiment, the present application discloses another flowchart of an audio data processing method, please refer to Figure 3 Take a to-be-processed audio data (a song) as an example for illustration. Please refer to steps S301-S310:

[0057] S301: Obtain the to-be-processed audio data.

[0058] The to-be-processed audio data can be understood as a song, and the subject of obtaining refers to a computer device. Before that, the to-be-processed audio data can be selected by a user, please refer to Figure 4 is an interface diagram for displaying to-be-processed audio data disclosed by the embodiment of the present application, wherein the to-be-processed audio data is "AA song", Figure 4 A sleep-aiding button 401 is displayed in the interface, and if the user clicks to open the button, the computer device obtains the "AA song" and executes step S302, and the corresponding Figure 4 The interface also displays related information of the song, such as an image 402 representing a singer, lyrics 403 and a song playing progress display 404, etc.

[0059] S302: Extract features of the to-be-processed audio data to obtain feature information of the to-be-processed audio data, and the feature information of the to-be-processed audio data includes beat information, chord information, pitch information and envelope information.

[0060] In step S302, the entire to-be-processed audio data is processed to obtain beat information, chord information, pitch information, and envelope information of the entire to-be-processed audio data.

[0061] S303: Determine feature information of each audio segment according to the beat information of the to-be-processed audio data, the feature information of each audio segment including chord information, pitch information, and envelope information.

[0062] In a specific implementation, the beat information of the to-be-processed audio data includes start time and end time of each audio segment, and the chord information, pitch information, and envelope information of each audio segment are determined from the chord information, pitch information, and envelope information of the entire to-be-processed audio data based on the start time and end time of each audio segment.

[0063] S304: Determine a chord adjustment strategy for each audio segment according to the chord information of each audio segment, and adjust the chord information of each audio segment according to the chord adjustment strategy, add a chord root note to the initial accompaniment information obtained after adjustment to obtain accompaniment information of the sleep-aiding audio structure file.

[0064] S305: Determine note information of each audio segment and a time interval of each note according to the pitch information of each audio segment, and adjust the time interval of the notes included in each audio segment according to a time adjustment strategy to obtain melody information of the sleep-aiding audio structure file.

[0065] S306: Adjust the envelope information of each audio segment according to a note intensity adjustment rule to obtain initial tuning envelope information of the audio segment, determine an average envelope value of each audio segment according to the beat information of the to-be-processed audio data and the envelope information of the to-be-processed audio data, and adjust the envelope information of each audio segment according to the initial tuning envelope information of each audio segment and the average envelope value of each audio segment to obtain adjusted envelope information of the sleep-aiding audio structure file.

[0066] S307: Determine the sleep-aiding audio structure file based on the accompaniment information, the melody information, and the adjusted envelope information.

[0067] S308: Convert the sleep-aiding audio structure file to obtain pulse code modulation data.

[0068] S309: Perform preset processing on the pulse code modulation data to obtain processed pulse code modulation data, the preset processing including one or both of sound mixing processing and environmental sound effect processing.

[0069] The sound mixing processing and the environmental sound effect processing can be one or both of sound mixing processing and environmental sound effect processing selected by the user in the sleep-aiding audio structure file. Figure 4The interface in the interface triggers the corresponding option to select the instruction to the computer device, and the computer device processes according to the instruction. When the computer device does not receive the selection instruction of the user, the computer device can adopt a default way to process the mixing and environmental sound effect. The default way can be no processing, or a fixed processing method, for example, the default method of mixing processing is rain sound, and the environmental sound effect is stereo sound. Here, the default way is not limited, and different settings can be made according to needs.

[0070] S310: Match the processed pulse coded modulation data with the timbre library to obtain the sleep aid audio data.

[0071] Among them, steps S304-S310 are Figure 2 The embodiments shown have been described in detail, which will not be described here.

[0072] The embodiments of the present application mainly describe the processing steps of a song, wherein the user's operable steps are introduced, so that the audio data processing method is more autonomous, and the user can synthesize sleep aid music according to his own selection.

[0073] Based on the above method embodiments, the embodiments of the present application also provide a structural diagram of an audio data processing device. Referring to Figure 5 , a structural diagram of an audio data processing device provided by the embodiments of the present application. Figure 5 The audio data processing device 500 shown can run a processing unit 501, a determination unit 502 and an acquisition unit 503, specifically:

[0074] The processing unit 501 is configured to extract features from the audio data to be processed to obtain feature information of each audio segment included in the audio data to be processed, and the feature information of each audio segment includes one or more of chord information, pitch information and envelope information.

[0075] The processing unit 501 is also configured to perform relaxation processing on the feature information of each audio segment to determine a sleep aid audio structure file, and the sleep aid audio structure file is used to indicate the music characteristics of sleep aid audio data.

[0076] The processing unit 501 is also configured to perform encoding processing on the sleep aid audio structure file to obtain sleep aid audio data.

[0077] In a possible implementation, the feature information of each audio segment includes the chord information, the pitch information and the envelope information, and when the processing unit 501 performs relaxation processing on the feature information of each audio segment to determine a sleep aid audio structure file, it is specifically configured to:

[0078] determine accompaniment information of the sleep-aiding audio structure file according to chord information of the audio segments;

[0079] determine melody information of the sleep-aiding audio structure file according to pitch information of the audio segments;

[0080] determine note intensity adjustment rules according to tempo information of the audio data to be processed, and adjust envelope information of the audio segments according to the note intensity adjustment rules to obtain adjusted envelope information of the sleep-aiding audio structure file;

[0081] determine the sleep-aiding audio structure file based on the accompaniment information, the melody information, and the envelope information of the sleep-aiding audio structure file.

[0082] In a possible implementation, when the determining unit 502 determines the accompaniment information of the sleep-aiding audio structure file according to chord information of the audio segments, the determining unit 502 is specifically configured to:

[0083] determine a chord adjustment strategy for each audio segment according to the chord information of the audio segments;

[0084] adjust the chord information of the audio segments according to the chord adjustment strategy to obtain initial accompaniment information of the sleep-aiding audio structure file;

[0085] add a chord root note to the initial accompaniment information to obtain the accompaniment information of the sleep-aiding audio structure file.

[0086] In a possible implementation, when the determining unit 502 determines the melody information of the sleep-aiding audio structure file according to pitch information of the audio segments, the determining unit 502 is specifically configured to:

[0087] determine a time interval of each note according to the pitch information of the audio segments;

[0088] adjust the time interval of the notes included in the audio segments according to a time adjustment strategy to obtain the melody information of the sleep-aiding audio structure file, the time adjustment strategy including: shortening a time interval of a second note if the second note is connected to a first note in any audio segment; and lengthening a time value of a rest if the rest is connected to the first note.

[0089] In a possible implementation, when the processing unit 501 adjusts the envelope information of the audio segments according to the note intensity adjustment rules to obtain adjusted envelope information of the sleep-aiding audio structure file, the processing unit 501 is specifically configured to:

[0090] adjust the envelope information of the audio segments according to the note dynamics adjustment rule, to obtain initial adjustment envelope information of the audio segments;

[0091] determine average envelope values of the audio segments according to the beat information of the to-be-processed audio data and the envelope information of the to-be-processed audio data;

[0092] adjust the envelope information of the audio segments according to the initial adjustment envelope information of the audio segments and the average envelope values of the audio segments, to obtain adjusted envelope information of the sleep-aiding audio structure file.

[0093] In a possible implementation, the processing unit 501 encodes the sleep-aiding audio structure file to obtain sleep-aiding audio data, and specifically is configured to:

[0094] convert the sleep-aiding audio structure file into pulse code modulation data;

[0095] perform preset processing on the pulse code modulation data to obtain processed pulse code modulation data, the preset processing including one or both of sound mixing processing and environmental sound effect processing;

[0096] encode the processed pulse code modulation data to obtain sleep-aiding audio data.

[0097] In a possible implementation, the feature extraction on the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data includes:

[0098] The obtaining unit 503 is configured to obtain to-be-processed audio data.

[0099] The processing unit 501 is configured to perform feature extraction on the to-be-processed audio data to obtain feature information of the to-be-processed audio data, the feature information of the to-be-processed audio data including beat information, chord information, pitch information, and envelope information.

[0100] The determining unit 502 is configured to determine feature information of each audio segment according to the feature information of the to-be-processed audio data, the feature information of each audio segment including chord information, pitch information, and envelope information.

[0101] In a possible implementation, the beat information of the to-be-processed audio data includes start time and end time of each audio segment, and the determining unit 502, when determining the feature information of each audio segment according to the feature information of the to-be-processed audio data, is specifically configured to:

[0102] The chord information, the pitch information and the envelope information of the audio segment are determined from the chord information of the to-be-processed audio data, the pitch information of the to-be-processed audio data and the envelope information of the to-be-processed audio data based on the start time and the end time of the audio segment.

[0103] According to one embodiment of the present application, Figure 2 and Figure 3 The steps involved in the audio data processing method shown in Figure 5 may be performed by the units in the data processing apparatus shown in Figure 2 For example, the steps S201-S203 in the audio data processing method shown in Figure 5 may be performed by the processing unit 501 in the audio data processing apparatus shown in Figure 3 For another example, the step S301 in the audio data processing method shown in Figure 5 may be performed by the acquisition unit 503 in the audio data processing apparatus shown in Figure 5 The steps S302-S306 and the steps S308-S310 may be performed by the processing unit 501 in the audio data processing apparatus shown in Figure 5 The step S307 may be performed by the determination unit 502 in the audio data processing apparatus shown in

[0104] According to another embodiment of the present application, Figure 5 The units in the audio data processing apparatus shown in may be combined into one or several other units respectively or all, or some of the units can be further split into multiple units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above units are logically divided, and in actual application, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the audio data processing apparatus can also include other units, and in actual application, these functions can also be assisted by other units, and can be implemented by multiple units.

[0105] According to another embodiment of the present application, the steps involved in the corresponding method shown in Figure 2 and Figure 3 may be constructed by running the computer program (including program code) capable of performing the steps on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), etc. Figure 5The audio data processing apparatus and the audio data processing method are used for realizing the embodiments of the present application. The computer program can be recorded on a computer readable storage medium, loaded into the computer device mentioned above through the computer readable storage medium, and run in the computer device.

[0106] In the embodiments of the present application, the processing unit 501 of the audio data processing apparatus extracts features of the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, the feature information of each audio segment includes one or more of chord information, pitch information and envelope information; then the feature information of each audio segment is processed to determine a sleep-aiding audio structure file, the sleep-aiding audio structure file is used to indicate the music characteristics of the sleep-aiding audio data; finally, the sleep-aiding audio structure file is encoded to obtain the sleep-aiding audio data. The method can process any audio data, and the sleep-aiding audio data generated after processing has a sleep-aiding effect. Therefore, the audio data processing method of the present application can increase the style types of the generated sleep-aiding music.

[0107] Based on the above method and apparatus embodiments, the embodiments of the present application provide a computer device, as shown in Figure 6 , which is a structural schematic diagram of a computer device provided in the embodiments of the present application. Figure 6 The computer device 600 shown in the figure at least includes a processor 601, an input interface 602, an output interface 603, a computer storage medium 604 and a memory 605. The processor 601, the input interface 602, the output interface 603, the computer storage medium 604 and the memory 605 can be connected through a bus or other means.

[0108] The computer storage medium 604 can be stored in the memory 605 of the computer device 600, and the computer storage medium 604 is used to store a computer program, the computer program includes program instructions, and the processor 601 is used to execute the program instructions stored in the computer storage medium 604. The processor 601 (or CPU (Central Processing Unit, Central Processor)) is the computing core and control core of the computer device 600, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more computer instructions to realize the corresponding method flow or corresponding function.

[0109] The embodiment of the present application further provides a computer storage medium (Memory), which is a memory device in the computer device 600, used for storing programs and data. It can be understood that the computer storage medium herein can include an internal storage medium in the computer device 600, and of course can also include an extended storage medium supported by the computer device 600. The computer storage medium provides a storage space, which stores an operating system of the computer device 600. In addition, one or more computer programs (including program codes) suitable for being loaded and executed by the processor 601 are also stored in the storage space. It should be noted that the computer storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory; and optionally can also be at least one computer storage medium located away from the aforementioned processor.

[0110] In one embodiment, the computer storage medium can be loaded and executed by the processor 601 to implement the corresponding steps of the audio data processing method described above with regard to Figure 2 and Figure 3 . In a specific implementation, one or more instructions in the computer storage medium are loaded and executed by the processor 601 to implement the following steps:

[0111] perform feature extraction on the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, the feature information of each audio segment including one or more of chord information, pitch information and envelope information;

[0112] perform soothing processing on the feature information of each audio segment to determine a sleep-aiding audio structure file, the sleep-aiding audio structure file being used to indicate music characteristics of sleep-aiding audio data;

[0113] perform encoding processing on the sleep-aiding audio structure file to obtain sleep-aiding audio data.

[0114] In a possible implementation, the feature information of each audio segment includes the chord information, the pitch information and the envelope information, and when the processor 601 performs soothing processing on the feature information of each audio segment to determine the sleep-aiding audio structure file, the soothing processing is specifically used for:

[0115] determining accompaniment information of the sleep-aiding audio structure file according to the chord information of each audio segment;

[0116] determining melody information of the sleep-aiding audio structure file according to the pitch information of each audio segment;

[0117] determine a note intensity adjustment rule according to the beat information of the to-be-processed audio data, and adjust envelope information of the audio segments according to the note intensity adjustment rule to obtain adjusted envelope information of the sleep-aiding audio structure file;

[0118] determine the sleep-aiding audio structure file based on accompaniment information, melody information, and envelope information of the sleep-aiding audio structure file.

[0119] In a possible implementation, when the processor 601 determines the accompaniment information of the sleep-aiding audio structure file according to chord information of the audio segments, the processor 601 is specifically configured to:

[0120] determine a chord adjustment strategy for each audio segment according to the chord information of the audio segments;

[0121] adjust the chord information of the audio segments according to the chord adjustment strategy to obtain initial accompaniment information of the sleep-aiding audio structure file;

[0122] add a chord root note to the initial accompaniment information to obtain the accompaniment information of the sleep-aiding audio structure file.

[0123] In a possible implementation, when the processor 601 determines the melody information of the sleep-aiding audio structure file according to pitch information of the audio segments, the processor 601 is specifically configured to:

[0124] determine a time interval of each note according to the pitch information of the audio segments;

[0125] adjust the time interval of the notes included in the audio segments according to a time adjustment strategy to obtain the melody information of the sleep-aiding audio structure file, the time adjustment strategy including: shortening a time interval of a second note if a first note is connected to the second note in any audio segment; and lengthening a time value of a rest if the first note is connected to the rest.

[0126] In a possible implementation, when the processor 601 adjusts the envelope information of the audio segments according to the note intensity adjustment rule to obtain adjusted envelope information of the sleep-aiding audio structure file, the processor 601 is specifically configured to:

[0127] adjust the envelope information of the audio segments according to the note intensity adjustment rule to obtain initial envelope information of the audio segments;

[0128] determine an average envelope value of the audio segments according to beat information of the to-be-processed audio data and envelope information of the to-be-processed audio data;

[0129] Adjust the envelope information of each audio segment according to the initial key envelope information of each audio segment and the average envelope value of each audio segment, to obtain adjusted envelope information of the sleep-aiding audio structure file.

[0130] In a possible implementation, the processor 601 encodes the sleep-aiding audio structure file to obtain sleep-aiding audio data, and specifically is configured to:

[0131] convert the sleep-aiding audio structure file into pulse code modulation data;

[0132] perform preset processing on the pulse code modulation data to obtain processed pulse code modulation data, the preset processing including one or both of sound mixing processing and environmental sound effect processing;

[0133] encode the processed pulse code modulation data to obtain sleep-aiding audio data.

[0134] In a possible implementation, the processor 601 extracts features from the to-be-processed audio data to obtain feature information of each audio segment included in the to-be-processed audio data, including:

[0135] obtain to-be-processed audio data;

[0136] extract features from the to-be-processed audio data to obtain feature information of the to-be-processed audio data, the feature information of the to-be-processed audio data including beat information, chord information, pitch information, and envelope information;

[0137] determine feature information of each audio segment according to the feature information of the to-be-processed audio data, the feature information of each audio segment including chord information, pitch information, and envelope information.

[0138] In a possible implementation, the beat information of the to-be-processed audio data includes start time and end time of each audio segment, and the processor 601 determines feature information of each audio segment according to the feature information of the to-be-processed audio data, and specifically is configured to:

[0139] determine, based on the start time and the end time of each audio segment, the chord information, the pitch information, and the envelope information of each audio segment from the chord information of the to-be-processed audio data, the pitch information of the to-be-processed audio data, and the envelope information of the to-be-processed audio data.

[0140] According to an aspect of the present application, the embodiments of the present application further provide a computer program product, which comprises a computer program stored in a computer readable storage medium. The computer program is read by the processor 601 from the computer readable storage medium, and the processor 601 executes the computer program, so that the computer device 600 executes the audio data processing method of Figure 2 and Figure 3 .

[0141] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, some steps can be performed in other sequences or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0142] In several embodiments provided by the present application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of the above modules is only a logical function division. In actual implementation, another division manner can be adopted. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0143] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Performing feature extraction on the audio data to be processed to obtain feature information of each audio segment included in the audio data to be processed, wherein the feature information of each audio segment includes chord information, pitch information, and envelope information; performing soothing processing on the characteristic information of each audio clip to determine a sleep-aiding audio structure file, wherein the sleep-aiding audio structure file is used to indicate the musical characteristics of the sleep-aiding audio data; Encoding the sleep-aiding audio structure file to obtain sleep-aiding audio data; The soothing processing of the characteristic information of each audio clip to determine the sleep-aiding audio structure file includes: Determining the accompaniment information of the sleep-aiding audio structure file according to the chord information of each audio clip; Determining melody information of the sleep-aiding audio structure file according to the pitch information of each audio clip; Determining a note velocity adjustment rule according to the time signature information of the audio data to be processed, and adjusting the envelope information of each audio clip according to the note velocity adjustment rule to obtain the envelope information of the sleep-aid audio structure file; The sleep-aiding audio structure file is determined based on the accompaniment information, melody information, and envelope information of the sleep-aiding audio structure file.

2. The method according to claim 1, characterized in that The determining of the accompaniment information of the sleep-aiding audio structure file according to the chord information of each audio segment includes: determining a chord adjustment strategy for each audio segment according to the chord information of each audio segment; Adjusting the chord information of each audio clip according to the chord adjustment strategy to obtain initial accompaniment information of the sleep-aiding audio structure file; A chord root note is added to the initial accompaniment information to obtain the accompaniment information of the sleep-aiding audio structure file.

3. The method according to claim 1, characterized in that The determining of the melody information of the sleep-aiding audio structure file according to the pitch information of each audio segment includes: Determining the time interval of each note according to the pitch information of each audio segment; The time intervals of the notes included in each audio clip are adjusted according to a time adjustment strategy to obtain melody information of the sleep-aid audio structure file. The time adjustment strategy includes: if the first note in any audio clip is connected to a second note, the time interval of the second note is shortened; if the first note is connected to a blank beat, the duration of the blank beat is extended, and the first note is any note in any audio clip.

4. The method according to claim 1, wherein The step of adjusting the envelope information of each audio segment according to the note velocity adjustment rule to obtain the adjusted envelope information of the sleep-aiding audio structure file includes: Adjusting the envelope information of each audio segment according to the note velocity adjustment rule to obtain initial envelope information of each audio segment; determining an average envelope value of each audio segment according to the beat information of the audio data to be processed and the envelope information of the audio data to be processed; The envelope information of each audio segment is adjusted according to the initially adjusted envelope information of each audio segment and the average envelope value of each audio segment to obtain the envelope information of the sleep-aiding audio structure file.

5. The method according to claim 1, wherein The encoding process of the sleep-aiding audio structure file to obtain the sleep-aiding audio data includes: Convert the sleep-aid audio structure file into a format to obtain pulse code modulation data; Performing preset processing on the pulse code modulation data to obtain processed pulse code modulation data, wherein the preset processing includes one or both of mixing processing and ambient sound effect processing; The processed pulse code modulation data is encoded to obtain sleep-aiding audio data.

6. The method according to claim 1, characterized in that The feature extraction of the audio data to be processed to obtain feature information of each audio segment included in the audio data to be processed includes: Extracting features from the audio data to be processed to obtain feature information of the audio data to be processed, wherein the feature information of the audio data to be processed includes beat information, chord information, pitch information, and envelope information; The characteristic information of each audio segment is determined according to the characteristic information of the audio data to be processed, where the characteristic information of each audio segment includes chord information, pitch information, and envelope information.

7. The method according to claim 6, characterized in that The beat information of the audio data to be processed includes the start time and the end time of each audio segment, and determining the feature information of each audio segment according to the feature information of the audio data to be processed includes: Based on the start time and end time of each audio segment, the chord information, pitch information and envelope information of each audio segment are determined from the chord information of the audio data to be processed, the pitch information of the audio data to be processed and the envelope information of the audio data to be processed.

8. A computer device, characterized in that: The computer device comprises: a processor adapted to execute one or more computer programs; and A computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded by the processor and executing the audio data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by a processor and executing the audio data processing method according to any one of claims 1 to 7.

Images