Audio generation method and apparatus

Abstract

An audio generation method and apparatus. The audio generation method (300) comprises: acquiring a hummed audio (301), the hummed audio being inputted by a user by means of a human-machine interaction interface; performing MIR on the hummed audio to obtain musical information of the hummed audio (302), the musical information comprising meter information, pitch information, and tone information; and generating a target audio on the basis of the musical information of the hummed audio (303), the target audio at least comprising an accompaniment audio corresponding to the hummed audio. A highly adaptable accompaniment or song can be obtained, thereby improving the robustness of humming-based accompaniment generation.

Description

Audio generation method and device TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to an audio generation method and device. BACKGROUND

[0002] The humming song generation task aims to automatically generate a music piece based on the humming content of a user, which can be highly matched with the tonality, rhythm, style, etc. hummed by the user. The humming song generation task can have various application scenarios, such as music creation scenarios, karaoke scenarios, personal customization scenarios, etc.

[0003] The related art can generate accompaniment audio tracks based on humming semantic information, limited templates, audio track adjustment, etc. to achieve the purpose of humming song generation. However, the accompaniment of these technologies has poor adaptability and cannot achieve automatic recognition and adaptation, and the generation effect of humming music is low in robustness. SUMMARY

[0004] The present application provides an audio generation method and device to obtain accompaniment or songs with high adaptability and improve the robustness of the generation effect of humming music.

[0005] In a first aspect, the present application provides an audio generation method, comprising: obtaining a humming audio, the humming audio being input by a user through a human-computer interaction interface; performing music information extraction MIR on the humming audio to obtain music information of the humming audio, the music information comprising beat information, pitch information and tone information; generating a target audio according to the music information of the humming audio, the target audio at least comprising accompaniment audio corresponding to the humming audio.

[0006] In the present application embodiment, the music information of the humming audio, such as beat, tone, pitch, etc. is obtained through MIR, so as to calculate the matching information of the accompaniment audio and the humming of the user, such as song key points, the role and emotion of the accompaniment in different music sections, chord progressions, etc. and then obtain the corresponding target audio based on the music information. It can be seen that both ordinary users and professional singers can obtain accompaniment or songs with high adaptability through the present application embodiment, and the robustness of the generation effect of humming music is improved.

[0007] The humming audio can be input by a user through a human-computer interaction interface.

[0008] The music information can include meter information, pitch information, and tone information. Optionally, the music information can also include section information. The meter (also known as Time) refers to regularly repeated accents; the pitch refers to sounds of different heights, i.e., the height of the sound; the tone refers to an attribute or state of whether certain sounds can be perceived as a central tone; and the section refers to the smallest structure of a complete musical thought or a relatively complete musical thought in a musical work, i.e., the smallest structure of an independent paragraph.

[0009] In the embodiments of the present application, the humming audio can be subjected to music information retrieval (MIR) to obtain the above-mentioned music information. After MIR, the meter information, pitch information, tone information, and section information in the music information can be displayed on the human-computer interaction interface, so that the user can intuitively see the meter, pitch, tone, and section of the humming audio.

[0010] In a possible implementation, the MIR can be implemented in the form of an AI model, i.e., the humming audio is input into the AI model, and the music information of the humming audio is output.

[0011] In a possible implementation, the MIR can be implemented in the form of cooperation of multiple AI models, i.e., the humming audio is input into a first AI model, and the meter information is output; the meter information and the humming audio are input into a second AI model, and the pitch information is output; the pitch information of the humming audio is input into a third AI model, and the tone information is output; and the humming audio is input into a fourth AI model, and the section information is output.

[0012] In a possible implementation, the MIR can be implemented in the form of matching with an original score and cooperation of multiple AI models, i.e., the humming audio is subjected to automatic speech recognition (ASR) to obtain the lyrics of the humming audio; the lyrics of the humming audio are matched with an original score to obtain the meter information, where the original score is the score of an original song corresponding to the humming audio; the meter information and the humming audio are input into a second AI model, and the pitch information is output; and the pitch information of the humming audio is input into a third AI model, and the tone information is output.

[0013] Whether one AI model or multiple AI models are used, the AI models can be pre-trained to implement the above-mentioned functions required by the MIR.

[0014] In a possible implementation, the target audio at least includes accompaniment audio corresponding to the humming audio.

[0015] The accompaniment score can be generated according to the music information of the humming audio, and the accompaniment score includes a multi-track MIDI (Musical Instrument Digital Interface) score corresponding to multiple instruments. In this step, the chord progression can be generated according to the pitch information and the tone information in the music information, and the minimum unit is two beats. The main logic is to calculate the chord that matches the most pitches and functions best (for example, the popular music rules such as termination) in each chord unit. Optionally, in order to facilitate the representation of the function and reusability of different chords, all chord progressions can be represented by Roman numerals; then the accompaniment score is generated according to the music information of the humming audio and the chord progression, that is, the style input by the user through the human-computer interaction interface is obtained (refer to the description above); then the multiple instruments are determined according to the style and the chord progression; and then the accompaniment score is generated according to the multiple instruments and the music information of the humming audio. Among them,

[0016] 1. Piano / Guitar / Bass: The main function of these parts is to provide chord support for the song and serve as the main accompaniment instrument, so the generation method usually follows the two-beat unit. Then the system will refer to the chord sound and rhythm pattern library to generate the corresponding accompaniment pattern for each unit.

[0017] 2. Drums are generated in two-beat or four-beat units. However, it relies more on the timing arrangement of the bars within the structure, for example, at the end of a section, a fill-in pattern can be filled. In addition, due to the characteristics of the drum instrument number in the MIDI format, specific mapping to the corresponding drum instrument is required when calling each sound source.

[0018] 3. Strings (Cello Strings): Strings follow a non-linear generation method, which is more complex than the beat-based logic of other instruments. Strings provide more counterpoint melodies and layered roles. The counterpoint melody is generated according to the major scale pattern, and the chord progression is referred to to ensure that the generated variable-length notes fit the chord progression within the time frame.

[0019] 4. Finally, it also includes miscellaneous generation logic for other parts, such as motive ideas, doubled melodies, fill-in patterns for specific small numbers of bars, etc. In addition, the configuration of the piano instrument sound velocity is also included, which makes the overall arrangement more like a real performance.

[0020] The mixing processing based on the above accompaniment score can obtain the target audio, that is, the virtual instrument rendering and single-track sound effect processing of the corresponding instruments are performed according to the multi-track MIDI score corresponding to the multiple instruments, to obtain multiple single-track audios; and the multi-track stereo mixing processing is performed on the multiple single-track audios to obtain the target audio. In this way, the timbres of multiple instruments can be mixed in the target audio, and the melody is played in the form of chords, which is a accompaniment song highly adapted to the humming audio of the user.

[0021] In a possible implementation, the target audio includes accompaniment audio corresponding to the humming audio and human voice audio in the humming audio.

[0022] The humming audio can be preprocessed to obtain human voice single-track audio; an accompaniment score is generated according to music information of the humming audio, the accompaniment score including multi-track MIDI scores corresponding to multiple instruments, virtual instrument rendering and single-track sound effect processing of the corresponding instruments are respectively performed according to the multi-track MIDI scores corresponding to the multiple instruments to obtain multiple accompaniment single-track audios; multiple single-track audios are obtained according to the human voice single-track audio and the multiple accompaniment single-track audios; and multi-track stereo mixing processing is performed on the multiple single-track audios to obtain the target audio. Compared with the target audio generated in the above manner, the user's voice is added in the target audio, that is, the target audio includes human voice and accompaniment, and is a song with high fusion of human voice and accompaniment.

[0023] In a second aspect, the present application provides an audio generation device, comprising: a human-computer interaction module, configured to obtain humming audio, the humming audio being input by a user through a human-computer interaction interface; a MIR module, configured to perform music information extraction MIR on the humming audio to obtain music information of the humming audio, the music information including beat information, pitch information and tone information; and an audio generation module, configured to generate target audio according to the music information of the humming audio, the target audio at least including accompaniment audio corresponding to the humming audio.

[0024] In a possible implementation, the MIR module is specifically configured to input the humming audio into an artificial intelligence AI model, and output the music information of the humming audio.

[0025] In a possible implementation, the MIR module is specifically configured to input the humming audio into a first AI model, and output the beat information; input the beat information and the humming audio into a second AI model, and output the pitch information; and input the pitch information of the humming audio into a third AI model, and output the tone information.

[0026] In a possible implementation, the MIR module is specifically configured to perform automatic speech recognition ASR on the humming audio to obtain lyrics of the humming audio; match the lyrics of the humming audio with an original score to obtain the beat information, the original score being a score of an original music corresponding to the humming audio; input the beat information and the humming audio into a second AI model, and output the pitch information; and input the pitch information of the humming audio into a third AI model, and output the tone information.

[0027] In a possible implementation, the music information of the humming audio further includes section information; and the MIR module is further configured to input the humming audio into a fourth AI model, and output the section information.

[0028] In a possible implementation, the audio generation module is specifically configured to generate an accompaniment score according to the music information of the humming audio, the accompaniment score including a multi-track MIDI score corresponding to a plurality of instruments; and perform mixing processing on the accompaniment score to obtain the target audio.

[0029] In a possible implementation, the audio generation module is specifically configured to generate a chord progression according to the pitch information and the tone information; and generate the accompaniment score according to the music information of the humming audio and the chord progression.

[0030] In a possible implementation, the audio generation module is specifically configured to obtain a style input by a user through a human-computer interaction interface; determine the plurality of instruments according to the style and the chord progression; and generate the accompaniment score according to the plurality of instruments and the music information of the humming audio.

[0031] In a possible implementation, the audio generation module is specifically configured to perform virtual instrument rendering and single-track sound effect processing on a corresponding instrument according to the multi-track MIDI score corresponding to the plurality of instruments, to obtain a plurality of single-track audios; and perform multi-track stereo mixing processing on the plurality of single-track audios, to obtain the target audio.

[0032] In a possible implementation, the target audio further includes a vocal audio in the humming audio; and the audio generation module is further configured to perform vocal preprocessing on the humming audio, to obtain a vocal single-track audio; perform virtual instrument rendering and single-track sound effect processing on a corresponding instrument according to the multi-track MIDI score corresponding to the plurality of instruments, to obtain a plurality of accompaniment single-track audios; obtain a plurality of single-track audios according to the vocal single-track audio and the plurality of accompaniment single-track audios; and perform multi-track stereo mixing processing on the plurality of single-track audios, to obtain the target audio.

[0033] In a third aspect, the present application provides a terminal device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method in any one of the above first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, including a computer program, when the computer program is executed on a computer, the computer program causes the computer to execute the method in any one of the above first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, characterized in that the computer program product comprises computer program codes, when the computer program codes are run on a computer, the computer program codes cause the computer to execute the method of any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] FIG. 1 shows a structural schematic diagram of a terminal device 100;

[0037] FIG. 2 is a software structure block diagram of the terminal device 100 of the present application;

[0038] FIG. 3 is a flowchart of a process 300 of an audio generation method provided by an embodiment of the present application;

[0039] FIG. 4 is a schematic diagram of a human-computer interaction interface according to an embodiment of the present application;

[0040] FIG. 5 is a schematic diagram of music information according to an embodiment of the present application;

[0041] FIG. 6 is an architecture schematic diagram of an MIR system 600 according to an embodiment of the present application;

[0042] FIG. 7 is a structural schematic diagram of a CNN according to an embodiment of the present application;

[0043] FIG. 8 is a schematic diagram of an accompaniment score according to an embodiment of the present application;

[0044] FIG. 9 is an architecture diagram of an audio generation system according to an embodiment of the present application;

[0045] FIG. 10 is a flowchart of an audio generation method according to an embodiment of the present application;

[0046] FIG. 11 is a visualization diagram of an MIR module according to an embodiment of the present application;

[0047] FIG. 12 is a generation flowchart of an accompaniment MIDI score according to an embodiment of the present application;

[0048] FIG. 13 is a mixing flowchart of automatic mixing according to an embodiment of the present application;

[0049] FIG. 14 is a flowchart of an audio generation method according to an embodiment of the present application;

[0050] FIG. 15 is a structural schematic diagram of an audio generation apparatus 1500 according to an embodiment of the present application;

[0051] FIG. 16 shows a schematic block diagram of an apparatus 1600 according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0053] The terms "first", "second", etc. in the description embodiments of the present application and claims and drawings are only used for distinguishing the purposes of description, and cannot be understood as indicating or implying relative importance, nor can be understood as indicating or implying sequence. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, comprising a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0054] It should be understood that in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0055] Before the technical solutions of the embodiments of the present application are described, the application scenarios of the embodiments of the present application will be described first in combination with the drawings. The present application can be widely applied to scenarios with humming song generation needs, for example, music creation scenarios (including professional singer recording or inspiration capture), karaoke scenarios (including existing song remixing), personal customization scenarios (including ordinary user recording), etc. The foregoing scenarios can be implemented as system services on terminal devices with recording functions, for example, mobile phones, tablets, earphones, etc. Optionally, they can also be matched with cloud servers; or, as application programming interfaces (API) interfaces, they can be added to application programs (APPs) with recording functions, for example, virtual assistants, humming song generation APPs, etc.

[0056] FIG. 1 shows a structural schematic diagram of a terminal device 100. It should be understood that the terminal device 100 shown in FIG. 1 is only an example, and the terminal device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in FIG. 1 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0057] The terminal device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0058] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors.

[0059] The controller can be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions.

[0060] The processor 110 can also have a memory that stores instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0061] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0062] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C buses. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the terminal device 100.

[0063] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S buses to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface to enable the function of answering a phone call through a Bluetooth earphone.

[0064] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 can be coupled with the wireless communication module 160 through a PCM bus interface. In some embodiments, the audio module 170 can also deliver audio signals to the wireless communication module 160 through the PCM interface to enable the function of playing music through a Bluetooth earphone. Both the I2S interface and the PCM interface can be used for audio communication.

[0065] The UART interface is a universal serial bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to enable Bluetooth functionality. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the UART interface to enable the function of playing music through a Bluetooth earphone.

[0066] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to enable the camera function of the terminal device 100. The processor 110 and the display screen 194 communicate through the DSI interface to enable the display function of the terminal device 100.

[0067] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0068] The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0069] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the terminal device 100. In some other embodiments of the present application, the terminal device 100 can also use different interface connection modes or combinations of multiple interface connection modes in the above embodiments.

[0070] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the terminal device 100. The charging management module 140 can charge the battery 142 while also supplying power to the terminal device through the power management module 141.

[0071] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0072] The wireless communication function of the terminal device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0073] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0074] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the terminal device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer the processed signals to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0075] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module 150 or other functional modules.

[0076] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the terminal device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0077] In some embodiments, the antenna 1 and the mobile communication module 150 of the terminal device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the terminal device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0078] The terminal device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0079] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the terminal device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0080] The terminal device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0081] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0082] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the terminal device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0083] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0084] The video codec is used to compress or decompress digital video. The terminal device 100 can support one or more video codecs. In this way, the terminal device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0085] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also continuously self-learn. Through the NPU, intelligent cognitive applications of the terminal device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0086] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, etc. Files are saved in the external memory card.

[0087] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phonebook, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0088] The terminal device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0089] The audio module 170 is configured to convert digital audio information into an analog audio signal output, and to convert an analog audio input into a digital audio signal. The audio module 170 can also be configured to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0090] The speaker 170A, also referred to as a “loudspeaker”, is configured to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0091] The receiver 170B, also referred to as a “earpiece”, is configured to convert an audio electrical signal into a sound signal. When the terminal device 100 answers a call or a voice message, the user can listen to the voice by holding the receiver 170B close to the ear.

[0092] The microphone 170C, also referred to as a “microphone”, “sound transducer”, is configured to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak into the microphone 170C by holding the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, in addition to collecting sound signals, the noise reduction function can also be realized. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C, in addition to collecting sound signals and noise reduction, the sound source can also be identified, and the directional recording function can also be realized.

[0093] The earphone interface 170D is configured to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0094] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 194, the terminal device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The terminal device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0095] The gyroscope sensor 180B can be configured to determine the motion attitude of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the terminal device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the terminal device 100 by reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0096] The barometric pressure sensor 180C is configured to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude, assists positioning and navigation by using the air pressure value measured by the barometric pressure sensor 180C.

[0097] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can detect the opening and closing of a flip cover by using the magnetic sensor 180D. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the cover or the flip cover, the terminal device 100 can set a feature such as automatic unlocking of the flip cover.

[0098] The acceleration sensor 180E can detect the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the acceleration sensor 180E can detect the magnitude and direction of gravity. The acceleration sensor 180E can also be used to identify the attitude of the terminal device, and can be applied to landscape / portrait screen switching and pedometer applications.

[0099] Distance sensor 180F is configured to measure distance. Terminal device 100 can measure distance by infrared or laser. In some embodiments, terminal device 100 can utilize distance sensor 180F to measure distance for fast focusing when taking a picture.

[0100] Proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode can be an infrared light emitting diode. Terminal device 100 emits infrared light outwardly through the light emitting diode. Terminal device 100 detects infrared reflected light from nearby objects using the photodiode. When sufficient reflected light is detected, terminal device 100 can determine that there is an object near terminal device 100. When insufficient reflected light is detected, terminal device 100 can determine that there is no object near terminal device 100. Terminal device 100 can utilize proximity light sensor 180G to detect when a user is holding terminal device 100 close to the ear for a phone call, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used for automatic unlocking and locking of the screen in a holster mode or a pocket mode.

[0101] Ambient light sensor 180L is configured to sense ambient light brightness. Terminal device 100 can adaptively adjust the brightness of display screen 194 according to the sensed ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking a picture. Ambient light sensor 180L can also cooperate with proximity light sensor 180G to detect whether terminal device 100 is in a pocket to prevent accidental touch.

[0102] Fingerprint sensor 180H is configured to acquire a fingerprint. Terminal device 100 can utilize the acquired fingerprint characteristics to implement fingerprint unlocking, access application lock, fingerprint picture taking, fingerprint call answering, and the like.

[0103] Temperature sensor 180J is configured to detect temperature. In some embodiments, terminal device 100 utilizes the temperature detected by temperature sensor 180J to implement temperature handling strategies. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal device 100 implements performance reduction of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal device 100 heats battery 142 to avoid abnormal shutdown of terminal device 100 caused by low temperature. In yet other embodiments, when the temperature is below yet another threshold, terminal device 100 implements voltage boosting of the output voltage of battery 142 to avoid abnormal shutdown caused by low temperature.

[0104] Touch sensor 180K, also referred to as "touch panel". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 together form a touch screen, also referred to as "touch panel". Touch sensor 180K is configured to detect touch operations applied to or near the touch sensor 180K. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K can also be disposed on the surface of terminal device 100, which is different from the position of display screen 194.

[0105] Bone conduction sensor 180M can obtain vibration signals. In some embodiments, bone conduction sensor 180M can obtain vibration signals of the human body's vocal vibration bone block. Bone conduction sensor 180M can also contact the human body pulse to receive blood pressure pulsation signals. In some embodiments, bone conduction sensor 180M can also be disposed in a headset to form a bone conduction headset. Audio module 170 can analyze voice signals based on the vibration signals of the vocal vibration bone block obtained by the bone conduction sensor 180M to realize voice functions. The application processor can analyze heart rate information based on the blood pressure pulsation signals obtained by the bone conduction sensor 180M to realize heart rate detection functions.

[0106] Keys 190 include power on / off keys, volume keys, and the like. Keys 190 can be mechanical keys. They can also be touch keys. Terminal device 100 can receive key input and generate key signal input related to user settings and function control of terminal device 100.

[0107] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations applied to different regions of display screen 194 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. Touch vibration feedback effects can also be customizable.

[0108] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, and also to indicate messages, missed calls, notifications, and the like.

[0109] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and disconnection with the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. The same SIM card interface 195 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The terminal device 100 interacts with a network through the SIM card to achieve functions such as call and data communication. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0110] The software system of the terminal device 100 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application exemplarily illustrate the software structure of the terminal device 100 by taking an Android system with a layered architecture as an example.

[0111] FIG. 2 is a software structure block diagram of the terminal device 100 of the present application. As shown in FIG. 2, the layered architecture of the terminal device 100 divides the software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Android runtime and system library, and a kernel layer.

[0112] The application layer can include a series of application packages.

[0113] As shown in FIG. 2, the application packages can include applications such as a camera, a gallery, a calendar, a call, a map, navigation, WLAN, Bluetooth, music, video, short message, humming, and the like.

[0114] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0115] As shown in FIG. 2, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0116] The window manager is used to manage windows programs. The window manager can acquire the display screen size, determine whether there is a status bar, lock the screen, and capture the screen, etc.

[0117] The content provider is used to store and acquire data, and make the data accessible to the application program. The data can include video, image, audio, dialed and received phone, browsing history and bookmark, phone book, etc.

[0118] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0119] The telephony manager is used to provide the communication function of the terminal device 100. For example, the management of the call state (including call connection, call hang-up, etc.).

[0120] The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc.

[0121] The notification manager makes the application program display notification information in the status bar, which can be used to convey a type of message, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of downloading, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application program running in the background, and can also be a notification in the form of a dialogue window appearing on the screen. For example, prompting text information in the status bar, issuing a prompt sound, terminal device vibration, indicator light blinking, etc.

[0122] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0123] The core library contains two parts: one part is the function function called by the java language, and the other part is the core library of Android.

[0124] The application program layer and the application program framework layer run in the virtual machine. The virtual machine executes the java file of the application program layer and the application program framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0125] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and the like.

[0126] The surface manager is used to manage a display subsystem and provides fusion of 2D and 3D layers for a plurality of applications.

[0127] The media libraries support playback and recording of a plurality of commonly used audio, video formats, and still image files. The media libraries can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0128] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, composition, and layer processing, and the like.

[0129] The 2D graphics engine is a drawing engine for 2D drawing.

[0130] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, audio drivers, and sensor drivers.

[0131] It can be understood that the components included in the system framework layer, the system library, and the runtime layer shown in FIG. 2 do not constitute a specific limitation on the terminal device 100. In some other embodiments of the present application, the terminal device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements.

[0132] Since the present application relates to the application of artificial intelligence (AI), in order to facilitate understanding, some related terms or terms are explained below.

[0133] (1) Neural network

[0134] A neural network (NN) is a machine learning model. The neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0135] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units described above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0136] (2) Deep neural network

[0137] Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers, where "many" has no special measurement standard. From the position of DNN according to different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: where, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Since DNN has many layers, the number of coefficients W and offset vectors is also large. These parameters in DNN are defined as follows: taking the coefficient W as an example: assuming in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as Note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex situations in the real world. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity" to perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrices, and the ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (the weight matrices formed by the vectors W of many layers).

[0138] (3) Convolutional neural network

[0139] A convolutional neural network (CNN) is a deep neural network with a convolutional structure, and is a deep learning architecture that learns at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which each neuron can respond to an input image. The CNN includes a feature extractor composed of convolutional layers and pooling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or a convolutional feature map.

[0140] The convolution layer refers to a neuron layer in a convolutional neural network that performs convolution processing on an input signal. The convolution layer can include a plurality of convolution operators, also referred to as kernels, which function as filters that extract specific information from an input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix produces a single-depth-dimension convolution output, but most cases do not use a single weight matrix, but apply a plurality of weight matrices of the same size (rows x columns), i.e., a plurality of homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "plurality" described above. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image, etc. The plurality of weight matrices are of the same size (rows x columns), and the feature maps extracted by the plurality of weight matrices of the same size are also of the same size. The plurality of feature maps of the same size are combined to form the output of the convolution operation. The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the trained weight values can be used to extract information from the input image, thereby enabling the convolutional neural network to make correct predictions. When the convolutional neural network has multiple convolution layers, the initial convolution layer often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network increases, the features extracted by the later convolution layers become more and more complex, such as high-level semantic features, etc. The higher the semantic level, the more suitable the features are for the problem to be solved.

[0141] Because it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after a convolution layer, which can be a convolution layer followed by a pooling layer, or a plurality of convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the average value of the pixel values in the image within a certain range as the result of the average pooling. The maximum pooling operator can take the maximum value of the pixels in the range as the result of the maximum pooling. In addition, just like the size of the weight matrix in the convolution layer should be related to the size of the image, the operator in the pooling layer should also be related to the size of the image. The size of the image output by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0142] After the convolution layer / pooling layer processing, the convolutional neural network is not enough to output the required output information. Because, as mentioned earlier, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate an output of one or a group of the number of required classes. Therefore, the neural network layer can include a plurality of hidden layers, and the parameters contained in the plurality of hidden layers can be pre-trained according to the related data of a specific task type, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc.

[0143] Optionally, after the plurality of hidden layers in the neural network layer, an output layer of the entire convolutional neural network is further included, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation starts to update the weight values and biases of the layers mentioned above to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0144] (4) Recurrent neural network

[0145] Recurrent neural networks (RNN) are used to process sequential data. In traditional neural network models, it is from the input layer to the hidden layer and then to the output layer, and the layers are fully connected, while the nodes between each layer are not connected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, you want to predict the next word of a sentence, you usually need to use the previous words, because the previous and next words in a sentence are not independent. RNN is called recurrent neural network because the current output of a sequence is also related to the previous output. The specific form of expression is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer unconnected but connected, and the input of the hidden layer not only includes the output of the input layer but also the output of the previous hidden layer. In theory, RNN can process sequential data of any length. The training of RNN is the same as the training of traditional CNN or DNN. Similarly, the error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded into a network, the parameters such as W are shared; but the traditional neural network as described above is not. And in the use of gradient descent algorithm, the output of each step depends not only on the current network but also on the state of the previous several steps of network. This learning algorithm is called back propagation through time (BPTT).

[0146] Since there is a convolutional neural network, why do we still need a recurrent neural network? The reason is simple: in a convolutional neural network, there is a premise assumption that elements are independent of each other, and input and output are also independent, such as cats and dogs. But in the real world, many elements are connected to each other, such as the change of stock over time, and again, a person said: I like traveling, and the most favorite place is Yunnan, and I will definitely go there if I have the opportunity. Fill in the blank, humans should all know to fill in "Yunnan". Because humans will infer according to the content of the context, but how can machines do this? RNN was born at the right moment. RNN aims to give machines the ability to remember like humans. Therefore, the output of RNN needs to depend on the current input information and historical memory information.

[0147] (5) Loss function

[0148] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the truly intended target value by comparing the predicted value of the current network with the truly intended target value (of course, there is usually an initialization process before the first update, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss.

[0149] (6) Back propagation algorithm

[0150] The convolutional neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model in the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output will produce an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0151] (7) Generative adversarial network

[0152] A generative adversarial network (GAN) is a kind of deep learning model. The model includes at least two modules: one module is a generative model (Generative Model), and the other module is a discriminative model (Discriminative Model), which learn from each other through the game to produce better output. The generative model and the discriminative model can both be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: taking a GAN for generating pictures as an example, there are two networks, G (Generator) and D (Discriminator), where G is a network for generating pictures, which receives a random noise z and generates a picture through the noise, denoted as G(z); D is a discriminative network for discriminating whether a picture is real or not. Its input parameter is x, which represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means 100% real picture, and if it is 0, it means it is not a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate real pictures as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the pictures generated by G from real pictures as much as possible. In this way, G and D constitute a dynamic "game" process, that is, the "adversarial" in the "generative adversarial network". Finally, the result of the game is that, in an ideal state, G can generate pictures G(z) that are difficult to distinguish from real pictures, and D cannot determine whether the pictures generated by G are real or not, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate pictures.

[0153] Based on this, the application provides an audio generation method to improve the adaptability of humming song generation.

[0154] FIG. 3 is a flowchart of a process 300 of an audio generation method provided by an embodiment of the application. The process 300 can be performed by the terminal device 100 described above, especially the system service provided by the terminal device 100 or the humming song generation APP installed thereon. The process 300 is described as a series of steps or operations, and it should be understood that the process 300 can be performed in various orders and / or simultaneously, without being limited to the execution order shown in FIG. 3.

[0155] The process 300 can include:

[0156] Step 301, obtaining humming audio.

[0157] The humming audio can be input by a user through a human-computer interaction interface.

[0158] For example, FIG. 4 is a schematic diagram of a human-computer interaction interface according to an embodiment of the present application. As shown in FIG. 4, the main interface of the humming song creation APP includes two pages, i.e., a start creation page and an all works page. The start creation page includes a style selection control key (for example, pure piano, pure guitar, and orchestra) and a recording control key. The user can first click the control key of the desired style and then click the recording control key. Then, the user can start humming to record the humming audio.

[0159] It should be noted that FIG. 4 only shows one possible implementation of the human-computer interaction interface, but this does not limit the human-computer interaction interface. The human-computer interaction interface can be implemented in any manner and can include more or less content. The embodiments of the present application do not limit this.

[0160] Step 302: performing MIR on the humming audio to obtain music information of the humming audio.

[0161] The music information can include beat information, pitch information, and tone information. Optionally, the music information can also include section information. The beat (Meter, commonly known as Time) refers to regularly repeated accents; the pitch (Pitch) refers to sounds of different heights, i.e., the height of the sound; the tone refers to a property or state in which certain sounds can be felt as a central tone; and the section refers to the smallest structure that expresses a complete musical thought or a relatively complete musical thought in a musical work, i.e., the smallest structure that constitutes an independent paragraph.

[0162] In the embodiments of the present application, the humming audio can be subjected to music information retrieval (MIR) to obtain the above-mentioned music information. For example, as shown in FIG. 5 (FIG. 5 is a schematic diagram of music information according to an embodiment of the present application), the beat information, pitch information, tone information, and section information in the music information obtained after MIR can be displayed on the human-computer interaction interface, so that the user can intuitively see the beat, pitch, tone, and section of the humming audio.

[0163] In one possible implementation, MIR can be implemented in the form of an AI model, i.e., the humming audio is input into the AI model, and the music information of the humming audio is output.

[0164] In one possible implementation, MIR can be implemented in the form of multiple AI models cooperating, i.e., the humming audio is input into a first AI model, and the beat information is output; the beat information and the humming audio are input into a second AI model, and the pitch information is output; the pitch information of the humming audio is input into a third AI model, and the tone information is output; and the humming audio is input into a fourth AI model, and the section information is output.

[0165] In a possible implementation, the MIR can be implemented in a manner matched with the original score and multiple AI models, that is, performing automatic speech recognition (ASR) on the humming audio to obtain the lyrics of the humming audio; matching the lyrics of the humming audio with the original score to obtain the beat information, the original score being the score of the original music corresponding to the humming audio; inputting the beat information and the humming audio into a second AI model to output the pitch information; inputting the pitch information of the humming audio into a third AI model to output the tone information.

[0166] Whether one AI model or multiple AI models are used, the AI models can be pre-trained to implement the above functions required by the MIR. For example, FIG. 6 is a schematic diagram of an architecture of an MIR system 600 according to an embodiment of the present application. As shown in FIG. 6, a data collection device 660 is configured to collect data (for example, multiple sets of humming audio) and store the data in a database 630, and a training device 620 is configured to generate a target model / rule 601 based on the data maintained in the database 630. The following will describe in more detail how the training device 620 obtains the target model / rule 601 based on the data, and the target model / rule 601 can implement the functions of the above one AI model or multiple AI models and output corresponding information.

[0167] The work of each layer in the deep neural network can be described by a mathematical expression The work of each layer in the deep neural network can be understood from a physical perspective as completing the transformation of the input space (a set of input vectors) to the output space (a set of output vectors) through five operations on the input space, which include: 1, dimensionality increase / decrease; 2, magnification / reduction; 3, rotation; 4, translation; and 5, "bending". The operations of 1, 2, and 3 are completed by The operation of 4 is completed by +b, and the operation of 5 is implemented by a(). The reason why the word "space" is used here is that the objects to be classified are not single things, but a class of things, and the space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the layer of the neural network. The vector W determines the spatial transformation of the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training the deep neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, and more specifically, learning the weight matrix.

[0168] Because the output of the deep neural network is expected to be as close as possible to the value that is really wanted to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network and the value that is really wanted to be predicted, and then adjusting the weight vector according to the difference between the two, for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the neural network can predict the value that is really wanted to be predicted. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss.

[0169] The target model / rule 601 obtained by the training device 620 can be applied in different systems or devices.

[0170] The execution device 610 is configured with an I / O interface 612 to interact with external devices, and a "user" can input data to the I / O interface 612 through a terminal device 640.

[0171] The execution device 610 can call data, code, etc. in the data storage system 650, and also can store data, instructions, etc. in the data storage system 650.

[0172] The computing module 611 uses the target model / rule 601 to process the input data to realize the MIR of the embodiments of the present application.

[0173] The correlation function module 613 and the correlation function module 614 can respectively realize the related functions in the training process, for example, preprocessing, filtering processing, etc.

[0174] Finally, the I / O interface 612 returns the processing result to the terminal device 640 to provide for the user.

[0175] More deeply, the training device 620 can generate corresponding target model / rule 601 based on different data for different targets to provide better results for the user.

[0176] In the case shown in FIG. 6, the user can manually specify the data input into the execution device 610, for example, by operating in the interface provided by the I / O interface 612. In another case, the terminal device 640 can automatically input data into the I / O interface 612 and obtain the result, and if the terminal device 640 needs to obtain the authorization of the user to automatically input data, the user can set the corresponding permission in the terminal device 640. The user can view the result output by the execution device 610 in the terminal device 640, and the specific presentation form can be a specific way such as display, sound, action, etc. The terminal device 640 can also act as a data collection end to store the collected data into the database 630.

[0177] It is worth noting that FIG. 6 is only a schematic diagram of a MIR system provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 6, the data storage system 650 is an external memory relative to the execution device 610, and in other cases, the data storage system 650 can also be placed in the execution device 610. For another example, in FIG. 6, the terminal device 640 and the execution device 610 are two devices, and in other cases, the terminal device 640 and the execution device 610 can also be integrated into one device.

[0178] A convolutional neural network (CNN) is a deep neural network with a convolutional structure, and is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, the CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network responds to overlapping regions in an image input thereto.

[0179] For example, FIG. 7 is a structural schematic diagram of a CNN according to an embodiment of the present application. As shown in FIG. 7, the CNN 700 can include an input layer 710, a convolutional layer / pooling layer 720, and a neural network layer 730, wherein the pooling layer is optional.

[0180] The convolutional layer / pooling layer 720:

[0181] The convolutional layer:

[0182] As shown in FIG. 7, the convolutional / pooling layers 720 can include layers such as examples 721-726, in one implementation, the 721 layer is a convolutional layer, the 722 layer is a pooling layer, the 723 layer is a convolutional layer, the 724 layer is a pooling layer, the 725 is a convolutional layer, and the 726 is a pooling layer; in another implementation, the 721 and 722 are convolutional layers, the 723 is a pooling layer, the 724 and 725 are convolutional layers, and the 726 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0183] Taking the convolutional layer 721 as an example, the convolutional layer 721 can include a plurality of convolutional operators, also known as kernels, which function as filters that extract specific information from an input image matrix in image processing. The convolutional operator can be essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix will produce a single-depth-dimension convolution output, but most cases do not use a single weight matrix, but apply multiple weight matrices of the same dimension. The output of each weight matrix is stacked to form the depth dimension of the convolution image. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image. The multiple weight matrices are of the same dimension, and the feature maps extracted by the multiple weight matrices of the same dimension are also of the same dimension. The extracted multiple feature maps of the same dimension are combined to form the output of the convolution operation.

[0184] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the trained weight values can extract information from the input image, thereby helping the CNN 700 to make correct predictions.

[0185] When CNN 700 has multiple convolution layers, the initial convolution layers (e.g. 721) tend to extract more general features, which can also be referred to as low-level features; as the depth of CNN 700 increases, the convolution layers at the back (e.g. 726) tend to extract more complex features, such as high-level semantic features, and the higher the semantic level, the more suitable the features are for the problem to be solved.

[0186] Pooling layer:

[0187] Due to the need to reduce the number of training parameters, a pooling layer is often periodically introduced after a convolution layer, i.e. each of 721-726 in FIG. 7 can be a convolution layer followed by a pooling layer, or a plurality of convolution layers followed by one or more pooling layers. In image processing, the only purpose of a pooling layer is to reduce the spatial size of an image. A pooling layer can include an average pooling operator and / or a max pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the average value of the pixel values in a certain range in the image. The max pooling operator can take the pixel with the maximum value in a certain range as the result of max pooling. In addition, just as the size of the weight matrix in the convolution layer should be related to the size of the image, the operator in the pooling layer should also be related to the size of the image. The size of the image output by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0188] Neural network layer 730:

[0189] After processing by convolution / pooling layer 720, CNN 700 is still not sufficient to output the required output information. As mentioned above, convolution / pooling layer 720 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), CNN 700 needs to use neural network layer 730 to generate an output of the number of required classes. Therefore, neural network layer 730 can include multiple hidden layers (e.g. 731, 732,..., 73n) and output layer 740, and the parameters contained in the multiple hidden layers can be pre-trained according to the related data of a specific task type, e.g. image recognition, image classification, image super-resolution reconstruction, etc.

[0190] After the multiple hidden layers in the neural network layer 730, that is, the last layer of the entire CNN 700 is the output layer 740, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire CNN 700 (as shown in FIG. 7, the propagation from 710 to 740 is forward propagation) is completed, the back propagation (as shown in FIG. 7, the propagation from 740 to 710 is back propagation) starts to update the weight values and biases of the layers mentioned above, so as to reduce the loss of the CNN 700 and the error between the result output by the CNN 700 through the output layer and the ideal result.

[0191] It should be noted that the CNN 700 shown in FIG. 7 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, as shown in FIG. 7, the multiple convolutional layers / pooling layers are parallel, and the features extracted respectively are input to the full neural network layer 730 for processing.

[0192] Step 303: generating target audio according to the music information of the humming audio.

[0193] In a possible implementation, the target audio at least includes accompaniment audio corresponding to the humming audio.

[0194] The accompaniment score can be generated according to the music information of the humming audio, and the accompaniment score includes a multi-track MIDI (Musical Instrument Digital Interface) score corresponding to multiple instruments. In this step, the chord progression can be generated according to the pitch information and the tone information in the music information, and the minimum unit is two beats. The main logic is to calculate the chord that matches the most pitches and functions most in each chord unit (for example, the popular music rules such as termination). Optionally, in order to facilitate the representation of the functions of different chords and reusability, all chord progressions can be represented by Roman numerals; then the accompaniment score is generated according to the music information of the humming audio and the chord progression, that is, the style input by the user through the human-computer interaction interface is obtained (refer to the description above); then the multiple instruments are determined according to the style and the chord progression; and then the accompaniment score is generated according to the multiple instruments and the music information of the humming audio. Wherein,

[0195] 1. Piano / guitar / bass: The main function of these parts is to provide chord support and serve as the main accompaniment instrument for the song, so the generation method usually follows the two-beat unit as the chord progression. Then the system will refer to the chord sound and rhythm pattern library to generate the corresponding accompaniment pattern for each unit.

[0196] 2、Drum is generated in units of two or four beats. However, it is more dependent on the timing of the bars within the structure, for example, fill-in patterns at the end of a section. In addition, due to the characteristics of the drum instrument number in the MIDI format, specific mapping to the corresponding drum instrument is required when calling each sound source.

[0197] 3、Strings (Cello and Strings): Strings follow a non-linear generation method, which is more complex than the logic of other instruments in units of beats. Strings provide more counterpoint melodies and layered roles, and counterpoint melodies are generated according to the major scale mode, while referring to the chord progression to ensure that the generated variable-length notes fit the chord progression within the time frame.

[0198] 4、Finally, it also includes other section miscellaneous generation logic, such as motive ideas, doubled melodies, fill-in patterns of specific small numbers, etc., and also includes the configuration of the velocity of the piano instrument, so that the overall arrangement is more like a live performance.

[0199] For example, as shown in FIG. 8 (FIG. 8 is a schematic diagram of the accompaniment score of the embodiment of the present application), the accompaniment score generated based on the foregoing music information can include various music styles, such as pure guitar, pure piano style of instruments, or multi-instrument orchestra style, including piano, electric guitar, bass, cello, violin, drums, synthesizer, etc. In the embodiment of the present application, FIG. 8 can be displayed on the human-computer interaction interface, so that the user can intuitively see the accompaniment score generated for the humming audio, and then can use the score for more professional arrangement, production, etc.

[0200] The mixing processing based on the above accompaniment score can obtain the target audio, that is, the virtual instrument rendering and single-track sound effect processing of the corresponding instrument according to the multi-track MIDI score of the corresponding instrument to obtain a plurality of single-track audios; and multi-track stereo mixing processing is performed on the plurality of single-track audios to obtain the target audio. In this way, the timbre of multiple instruments can be mixed in the target audio to play the melody in the form of chords, which is a piece of accompaniment music highly adapted to the humming audio of the user.

[0201] In one possible implementation, the target audio includes accompaniment audio corresponding to the humming audio and human voice audio in the humming audio.

[0202] The humming audio can be pre-processed to obtain a vocal single-track audio; a accompaniment score is generated according to music information of the humming audio, the accompaniment score includes multi-track MIDI scores corresponding to multiple instruments, virtual instrument rendering and single-track sound effect processing of the corresponding instruments are performed according to the multi-track MIDI scores corresponding to the multiple instruments, to obtain multiple accompaniment single-track audios; multiple single-track audios are obtained according to the vocal single-track audio and the multiple accompaniment single-track audios; and multi-track stereo mixing processing is performed on the multiple single-track audios to obtain a target audio. Compared with the target audio generated in the above manner, the user's voice is added in the target audio, that is, the target audio includes vocals and accompaniment, and is a song with high fusion of vocals and accompaniment.

[0203] In the embodiments of the present application, the music information of the humming audio, such as beat, pitch, tone, etc., is obtained by MIR, so that the matching information of the accompaniment audio and the humming of the user is calculated, such as song key points, the role and emotion of the accompaniment in different musical phrases, and chord progressions, and then the corresponding target audio is obtained based on the music information. It can be seen that even ordinary users without professional singing skills can also obtain complete and reasonable accompaniment or songs through the embodiments of the present application, and realize the singing and writing effect that can only be provided by professional music production teams; professional singer-songwriters can also convert their inspiration into complete music in a short time through the embodiments of the present application, providing reference value; the embodiments of the present application can also provide new-style accompaniment generation for existing songs, as secondary creation or to avoid potential infringement of the copyright of the original version.

[0204] The technical solutions of the method embodiment shown in FIG. 3 will be described in detail below by using several specific embodiments.

[0205] FIG. 9 is an architecture diagram of an audio generation system according to an embodiment of the present application. As shown in FIG. 9, the system includes four functional modules: a user interface (UI) module, an MIR module, an automatic accompaniment module, and an automatic mixing module. Among them,

[0206] The purpose of the UI module is to guide the user to select the song style and record through simple and clear instructions. The UI module can provide multiple recording modes, so that ordinary users, advanced users, and professional users can record songs with specific beat per minute (BPM) according to their individual needs. Based on the human-computer interaction interface provided by the UI module, the humming audio of the user can be obtained.

[0207] The MIR module includes vocal transcription, beat recognition, tonality recognition, and musical phrase recognition, and can extract music information of the humming audio, such as beat, pitch, tonality, musical phrase, etc., by AI technology, and convert the music information into digital Json information for use by other modules.

[0208] The automatic accompaniment module includes chord programming and MIDI generation, which can generate a symbolic domain accompaniment MIDI score according to the music information extracted by the MIR module. Different numbers of tracks, different types of instruments (orchestration), and chords can also be configured according to the length and style of the song to create different effects. Finally, the humming audio is aligned to ensure that it can completely match the accompaniment audio without producing a sense of discord. That is, first, the chord with the highest matching degree is selected based on the pitch of the humming audio, and then a multi-track accompaniment MIDI score with different orchestrations is generated for each section based on the user-specified style and the identified section structure. Optionally, the melody of the chorus part can be extracted as the prelude and coda of the song to create a sense of level in the song.

[0209] The automatic mixing module includes sound source rendering, vocal mixing, accompaniment stereo mixing, and mastering, which can render high-quality target audio as the final product according to the accompaniment MIDI score generated by the automatic accompaniment module. During the process, the appropriate sound source files will be selected according to the song style and the instruments used in the track to generate dry audio for the corresponding instruments, and the humming audio will be mixed to make the instrument sound and the voice completely integrated, reaching the level of commercial songs. Optionally, in addition to the complete song, the system can also generate pure accompaniment audio for various scene purposes (such as pure accompaniment for KTV use).

[0210] FIG. 10 is a flowchart of the audio generation method according to an embodiment of the present application. As shown in FIG. 10, the following functions are implemented by each module in the system shown in FIG. 9:

[0211] UI module

[0212] The user can record through the human-computer interaction interface provided by the UI module. The user can choose a professional mode with a metronome or a normal mode without a metronome to record according to their own music expertise. After recording is complete, the user can choose different music styles, including orchestra, pure piano, pure guitar, etc. The user's humming audio and the aforementioned information can be packaged in the form of an HTTP request and transmitted to the subsequent module.

[0213] MIR module

[0214] The MIR module performs MIR on the humming audio to obtain music information such as beat, section, pitch transcription, and pitch, and performs post-processing. As shown in FIG. 11 (which is a visualization diagram of the MIR module according to an embodiment of the present application), the processing process is as follows:

[0215] 1. Beat information: the beat point of humming audio, including the time point of the heavy beat and the split beat, stored in the form of timestamp. If the user selects the normal recording mode, i.e. without the metronome, the humming will not follow an absolutely fixed BPM, but will produce a certain degree of time swing. At this time, the beat point will become the most core information for the complete alignment of humming and accompaniment.

[0216] 2. Section structure: section analysis is performed on the humming audio, and the music is automatically divided into six popular music sections according to the time node (paragraph start and end), i.e. prelude, main song, chorus, interlude, bridge, and tail. The subsequent accompaniment module will prepare different energy levels of accompaniment according to different sections, for example, the main song is lower in energy and more calm in emotion than the chorus, and the prelude and tail should reproduce the melody motive of the music. A complete popular song is usually composed of multiple main songs and choruses, but in actual cases, the user usually records a shorter length, which may not even constitute a complete paragraph. At this time, the post-processing algorithm will treat it as an independent section.

[0217] 3. Voice pitch transcription and tonality: according to the beat information, the system will divide the humming audio into 16-beat units and find the pitch of each 16-beat unit, and judge the tonality according to the pitch and prepare the chord in the subsequent module. The pitch is also a key component of extracting the melody motive, which may be used in the composition.

[0218] Automatic accompaniment module

[0219] According to the extracted music information, a symbolic domain accompaniment MIDI score is generated, in which different numbers of audio tracks, different types of instruments, and chords are arranged according to the length and style of the song to create different effects. Finally, the user's song input is aligned to ensure that it can completely match the accompaniment audio without producing a sense of discord.

[0220] For example, as shown in FIG. 12 (FIG. 12 is a generation flowchart of the accompaniment MIDI score of the embodiment of the present application), the corresponding chord is generated according to the pitch and tone, and the smallest unit is two beats. The main logic is to calculate the chord that matches the most pitches and functions best (such as the termination type commonly used in popular music) in each chord unit. Optionally, in order to facilitate the representation of the function and reusability of different chords, all chord progressions can be represented by Roman numerals. The tone information is a necessary condition for generating chords, based on which chords that conform to the tone can be generated, and based on which the parts of all instruments can be generated.

[0221] Then, according to the pitch, chord progression and other corresponding music information, the following MIDI scores of different arrangements are generated:

[0222] 1. Piano / Guitar / Bass: These parts are mainly responsible for providing chord support and as the main backing instrument for the song, so the generation method usually follows the same pattern as the chords, which is two beats per unit. Then the system will refer to the chord sound and rhythm pattern library to generate the corresponding accompaniment pattern for each unit.

[0223] 2. Drums are generated in units of two or four beats. However, it relies more on the timing of the bars within the structure, for example, at the end of a section, it may fill in the pattern. In addition, due to the characteristics of the drum instrument number in the MIDI format, specific mapping to the corresponding drum instrument is required when calling each sound source.

[0224] 3. Strings (Cello and Strings): Strings follow a non-linear generation method, which is more complex than the beat-based logic of other instruments. Strings provide more counterpoint melodies and layered roles, and counterpoint melodies are generated according to the major scale pattern, while referring to the chord progression to ensure that the generated variable-length notes fit the chord progression within the time frame.

[0225] 4. Finally, it also includes miscellaneous generation logic for other parts, such as motive ideas, doubled melodies, fill-in patterns for specific bar numbers, etc., and also includes the configuration of the piano's instrument velocity to make the overall arrangement more like a live performance.

[0226] Automatic mixing module

[0227] According to the MIDI score of different arrangements, it is rendered into high-quality target audio as the final product. During the process, according to the song style, the instruments used in the audio track, the appropriate sound source file is selected to generate the dry sound of the corresponding instrument, and the mixing process is combined with the user's voice input, so that the instrument sound and the voice can be completely integrated, reaching the level of commercial songs.

[0228] For example, as shown in Figure 13 (Figure 13 is the mixing process of the automatic mixing of the embodiment of the present application), the processing process is as follows:

[0229] 1. Based on the information of each track in the MIDI score, first, through the "virtual instrument rendering module", the sound source of the virtual instrument is selected according to the genre, and the audio is rendered;

[0230] 2. After the voice is denoised, repaired, and balanced by the "voice preprocessing module", like the rendered audio of each instrument, it is optimized for single-track sound effects, including adjusting its balance, compression ratio, etc.

[0231] 3. After passing through the "reverb space rendering module", add reverb, delay, etc.

[0232] 4. After passing through the "dry-wet balance module", adjust the overall dry-wet ratio;

[0233] 5. The "multi-track mixing module" is used to analyze the vocals, drum group instruments, and non-drum group instruments respectively, and a ducking adaptive volume control module is called, which is more conducive to the integration of the accompaniment and the vocals.

[0234] 6. A "mastering module" is called, and a multi-section compression sound effect is used to adaptively process the final mix, so as to ensure that the output product audio meets the signal objective indicators of commercial music.

[0235] After obtaining the final product, the audio can be sent to the UI module in the form of an HTTP request for the user to listen to and spread.

[0236] In this embodiment, the user records the chorus of the popular song "Love Story" by humming, and the music information is accurately captured by the MIR module, so that the subsequent accompaniment mixing module can generate accompaniment audio that perfectly matches the humming audio on the basis of preserving the user's humming information.

[0237] In terms of accompaniment MIDI score generation, the chord progression can meet the needs of ordinary users as long as it is consistent with the vocals, and there will be no conflict between the vocals and the instrument pitch. In terms of orchestration, this embodiment selects the orchestra style, which has most of the instrument parts, so it can better simulate the integrity of a real popular song.

[0238] In terms of automatic mixing, since there are many types of instruments in the orchestra style, the sound image distribution of different instruments is performed: the piano has a wide stereo width and is located in the middle left, the violin and the cello are separated left and right, the celesta has a far depth, and the vocals are in the center and have a sense of space. In terms of spectral energy, the bass has enough elasticity; the drum group instrument has a natural full-frequency energy distribution and a sense of space; and the string instrument highlights the characteristic high frequency. Compared with micro-singing, the balance and integration of the self-developed vocal and accompaniment are better; compared with Ripple, the final product also has a larger volume and more full-frequency energy, and the listening experience is more professional.

[0239] FIG. 14 is a flowchart of an audio generation method according to an embodiment of the present application. As shown in FIG. 14, the embodiment and the embodiment shown in FIG. 10 can both be applied to recording humming accompaniment by a general mobile phone or tablet device, and the flow is basically the same, the difference being that the humming audio recorded in this embodiment has an original score (i.e., an existing song), so that more accurate music information can be obtained by comparing it with the original song using automatic speech recognition (ASR), thereby achieving better accompaniment effect. The input and automatic accompaniment and automatic mixing parts of this embodiment are basically the same as those of the embodiment shown in FIG. 10, but the MIR module is integrated with the ASR:

[0240] 1. Prepare a song template set: For the use of this application in different scenarios (such as holiday scenarios, customized scenarios), we can find the corresponding existing songs to transcribe, obtain the score template, lyrics template, and chord template, and organize them into a template set to align with user's singing input.

[0241] 2. User humming lyrics recognition: Use ASR technology to extract the lyrics of the humming audio and match them with the songs in the template set to find the original song corresponding to the humming audio. If no song is matched, the MIR module of the embodiment shown in FIG. 10 will be used.

[0242] 3. Lyrics timestamp matching: After matching the original song, the word-level timestamp of the humming audio lyrics will be matched with the original song to find the length of each lyric beat, thereby calculating the beat information to ensure that the accompaniment can be completely matched with the lyrics. Considering that the humming audio will produce a certain degree of time swing, to prevent the swing from being too large, the system will normalize the final beat to ensure that the beat fluctuation is within a controllable range.

[0243] 4. Voice pitch transcription and tonality: The original song's pitch will be considered when obtaining the pitch to increase the accuracy of tonality recognition, while ensuring that the user's voice can also find the closest tonality even if there is a certain degree of pitch change, maintaining audibility.

[0244] After the ASR and MIR modules, the system will run the automatic accompaniment module to generate the accompaniment MIDI score and perform mixing processing. The chord progression of the MIDI accompaniment will use the original chord template in the song template to adapt to the original song effect and ensure that the user can recognize the similarity to the original song in terms of listening experience.

[0245] Based on the technical effects of the embodiment shown in FIG. 10, since the original song score is used as a reference, the accuracy of music information extraction is further increased, and the original song chord progression can be directly reused to ensure that the user can recognize the similarity to the original song in terms of listening experience, and to produce emotional resonance in specific scenarios. For example, in the Mother's Day scenario we have tried, we have tried to add 9 Mother's Day song templates to the process to quickly align the music information and generate accurate emotional and musical information when the humming audio corresponds to the song.

[0246] In addition, considering the copyright issues that may arise from this embodiment, the system will only reuse the original song chord progression, and will not use materials in the arrangement and other mixing levels, and can also achieve a completely different style from the original song in terms of music style, thereby avoiding copyright issues.

[0247] FIG. 15 is a structural schematic diagram of an audio generation apparatus 1500 according to an embodiment of the present application. As shown in FIG. 15, the audio generation apparatus 1500 can be applied to the terminal device described above. The audio generation apparatus 1500 can include a human-computer interaction module 1501, a MIR module 1502, and an audio generation module 1503. Wherein,

[0248] The human-computer interaction module 1501 is configured to obtain a humming audio, the humming audio being input by a user through a human-computer interaction interface. The MIR module 1502 is configured to perform music information extraction MIR on the humming audio to obtain music information of the humming audio, the music information including beat information, pitch information, and tone information. The audio generation module 1503 is configured to generate a target audio according to the music information of the humming audio, the target audio including at least accompaniment audio corresponding to the humming audio.

[0249] In a possible implementation, the MIR module 1502 is specifically configured to input the humming audio into an artificial intelligence AI model, and output the music information of the humming audio.

[0250] In a possible implementation, the MIR module 1502 is specifically configured to input the humming audio into a first AI model, and output the beat information; input the beat information and the humming audio into a second AI model, and output the pitch information; and input the pitch information of the humming audio into a third AI model, and output the tone information.

[0251] In a possible implementation, the MIR module 1502 is specifically configured to perform automatic speech recognition ASR on the humming audio to obtain lyrics of the humming audio; match the lyrics of the humming audio with an original score to obtain the beat information, the original score being a score of an original music corresponding to the humming audio; input the beat information and the humming audio into a second AI model, and output the pitch information; and input the pitch information of the humming audio into a third AI model, and output the tone information.

[0252] In a possible implementation, the music information of the humming audio further includes section information; and the MIR module 1502 is further configured to input the humming audio into a fourth AI model, and output the section information.

[0253] In a possible implementation, the audio generation module 1503 is specifically configured to generate an accompaniment score according to the music information of the humming audio, the accompaniment score including a multi-track musical instrument digital interface MIDI score corresponding to a plurality of arrangers; and perform mixing processing according to the accompaniment score to obtain the target audio.

[0254] In a possible implementation, the audio generation module 1503 is specifically configured to generate a chord progression according to the pitch information and the tone information; and generate the accompaniment score according to the music information of the humming audio and the chord progression.

[0255] In a possible implementation, the audio generation module 1503 is specifically configured to obtain a style input by a user through a human-computer interaction interface; determine the plurality of instruments according to the style and the chord progression; and generate the accompaniment score according to the plurality of instruments and the music information of the humming audio.

[0256] In a possible implementation, the audio generation module 1503 is specifically configured to perform virtual instrument rendering and single-track sound effect processing on a corresponding instrument according to the multi-track MIDI score of the corresponding plurality of instruments, to obtain a plurality of single-track audios; and perform multi-track stereo mixing processing on the plurality of single-track audios, to obtain the target audio.

[0257] In a possible implementation, the target audio further includes human voice audio in the humming audio; and the audio generation module 1503 is further configured to perform human voice preprocessing on the humming audio, to obtain a human voice single-track audio; perform virtual instrument rendering and single-track sound effect processing on a corresponding instrument according to the multi-track MIDI score of the corresponding plurality of instruments, to obtain a plurality of accompaniment single-track audios; obtain a plurality of single-track audios according to the human voice single-track audio and the plurality of accompaniment single-track audios; and perform multi-track stereo mixing processing on the plurality of single-track audios, to obtain the target audio.

[0258] The apparatus of the embodiment can be used to execute the technical solutions of the method embodiment shown in FIG. 3, and has similar implementation principles and technical effects, which will not be described here.

[0259] It can be understood that, in order to implement the above functions, the terminal device contains hardware and / or software modules corresponding to each function. The algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solutions. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0260] In one example, FIG. 16 shows a schematic block diagram of an apparatus 1600 according to an embodiment of the present application. The apparatus 1600 can include a processor 1601 and a transceiver / transceiver pin 1602, and optionally further include a memory 1603.

[0261] The various components of the apparatus 1600 are coupled together by a bus system 1604, which can include a data bus, a power bus, a control bus, and a state line bus. For the sake of clarity, the various buses are illustrated in Figure 1604 as the single bus system 1604. The bus system 1604 can be implemented using any one or more buses, including industry standard Architecture (ISA), Micro Channel Architecture (MCA), Peripheral Component Interconnect (PCI), Low Pin Count (LPC), Serial Advanced Technology Attachment (SATA), and / or any other bus or bus type known in the art.

[0262] Optionally, the memory 1603 can be configured to store instructions of the foregoing method embodiments. The processor 1601 can be configured to execute the instructions in the memory 1603, and control the receiving pin to receive signals and the sending pin to send signals.

[0263] The apparatus 1600 can be a terminal device or a chip of a terminal device in the foregoing method embodiments.

[0264] Wherein, all the related content of each step of the above method embodiments can be cited to the function description of the corresponding function module, and will not be repeated here.

[0265] The embodiment also provides a computer storage medium, which stores computer instructions, and when the computer instructions run on a terminal device, the terminal device executes the related method steps to implement the audio generation method in the above embodiment.

[0266] The embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the related steps to implement the audio generation method in the above embodiment.

[0267] In addition, the embodiment of the present application also provides an apparatus, which can be a chip, a component or a module. The apparatus can include a processor and a memory connected to each other. The memory is configured to store computer execution instructions. When the apparatus runs, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the audio generation method in the above method embodiments.

[0268] Wherein, the terminal device, the computer storage medium, the computer program product or the chip provided by the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects that can be achieved are referred to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0269] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0270] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the modules or units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0271] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or a plurality of physical units, that is, can be located in one place, or can be distributed to a plurality of different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0272] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0273] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above is within the scope of the present application.

[0274] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions to make a device (which can be a single chip, a chip, etc.) or a processor execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0275] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, and all of them belong to the protection of the present application.

[0276] The steps of methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in software or in a combination of software and hardware. Software embodied under the present application can be stored in a computer-readable medium, which can include but is not limited to random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), electrically EPROM (EEPROM), registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0277] Those skilled in the art should be aware that the functions described in the above one or more examples can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0278] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, and all of them belong to the protection of the present application.

Claims

An audio generation method, characterized by The method comprises: acquiring a humming audio input by a user through a human-computer interaction interface; performing music information extraction (MIR) on the humming audio to obtain music information of the humming audio, the music information comprising beat information, pitch information and tone information; generating a target audio according to the music information of the humming audio, the target audio comprising at least accompaniment audio corresponding to the humming audio. The method of claim 1, wherein The MIR performed on the humming audio to obtain the music information of the humming audio comprises: inputting the humming audio into an artificial intelligence (AI) model to output the music information of the humming audio. The method according to claim 1 or 2, characterized in that The MIR performed on the humming audio to obtain the music information of the humming audio comprises: inputting the humming audio into a first AI model to output the beat information; inputting the beat information and the humming audio into a second AI model to output the pitch information; inputting the pitch information of the humming audio into a third AI model to output the tone information. The method according to claim 1 or 2, characterized in that The MIR performed on the humming audio to obtain the music information of the humming audio comprises: performing automatic speech recognition (ASR) on the humming audio to obtain lyrics of the humming audio; matching the lyrics of the humming audio with an original score to obtain the beat information, the original score being a score of an original music corresponding to the humming audio; inputting the beat information and the humming audio into a second AI model to output the pitch information; inputting the pitch information of the humming audio into a third AI model to output the tone information. The method according to any one of claims 1-4, characterized in that The music information of the humming audio further comprises section information; the MIR performed on the humming audio to obtain the music information of the humming audio comprises: inputting the humming audio into a fourth AI model to output the section information. The method according to any one of claims 1-5, characterized in that The generation of the target audio according to the music information of the humming audio comprises: generating accompaniment scores according to the music information of the humming audio, the accompaniment scores comprising multi-track musical instrument digital interface (MIDI) scores corresponding to a plurality of instruments; performing mixing processing on the accompaniment scores to obtain the target audio. The method according to claim 6, characterized in that The generation of the accompaniment scores according to the music information of the humming audio comprises: generating a chord progression according to the pitch information and the tone information; generating the accompaniment scores according to the music information of the humming audio and the chord progression. The method of claim 7, wherein The generation of the accompaniment scores according to the music information of the humming audio and the chord progression comprises: acquiring a style input by a user through a human-computer interaction interface; determining the plurality of instruments according to the style and the chord progression; generating the accompaniment scores according to the plurality of instruments and the music information of the humming audio. The method according to any one of claims 6-8, characterized in that The mixing processing performed on the accompaniment scores to obtain the target audio comprises: performing virtual instrument rendering and single-track sound effect processing on the multi-track MIDI scores corresponding to the plurality of instruments respectively to obtain a plurality of single-track audios; performing multi-track stereo mixing processing on the plurality of single-track audios to obtain the target audio. The method according to any one of claims 6-8, characterized in that The target audio further comprises human voice audio in the humming audio; the method comprises: The human voice pre-processing is performed on the humming audio to obtain a human voice single-track audio; Virtual instrument rendering and single-track sound effect processing are performed on the corresponding multiple instruments according to the multi-track MIDI music scores of the corresponding multiple instruments to obtain multiple accompaniment single-track audios; Multiple single-track audios are obtained according to the human voice single-track audio and the multiple accompaniment single-track audios; Multi-track stereo mixing processing is performed on the multiple single-track audios to obtain the target audio. An audio generating device, characterized by Comprise: An interactive module configured to obtain a humming audio, the humming audio being input by a user through an interactive interface; An MIR module configured to perform music information extraction (MIR) on the humming audio to obtain music information of the humming audio, the music information comprising beat information, pitch information and tone information; An audio generation module configured to generate a target audio according to the music information of the humming audio, the target audio comprising at least an accompaniment audio corresponding to the humming audio. The apparatus of claim 11, wherein The MIR module is specifically configured to input the humming audio into an artificial intelligence (AI) model and output the music information of the humming audio. The apparatus according to claim 11 or 12, characterized in that The MIR module is specifically configured to input the humming audio into a first AI model and output the beat information, input the beat information and the humming audio into a second AI model and output the pitch information, and input the pitch information of the humming audio into a third AI model and output the tone information. The apparatus according to claim 11 or 12, characterized in that The MIR module is specifically configured to perform automatic speech recognition (ASR) on the humming audio to obtain lyrics of the humming audio, match the lyrics of the humming audio with an original score to obtain the beat information, and input the beat information and the humming audio into a second AI model and output the pitch information, and input the pitch information of the humming audio into a third AI model and output the tone information, wherein the original score is a score of an original music corresponding to the humming audio. The apparatus of any one of claims 11-14, wherein The music information of the humming audio further comprises section information, and the MIR module is further configured to input the humming audio into a fourth AI model and output the section information. The apparatus of any one of claims 11-15, wherein The audio generation module is specifically configured to generate an accompaniment score according to the music information of the humming audio, the accompaniment score comprising multi-track MIDI music scores of corresponding multiple instruments, and perform mixing processing according to the accompaniment score to obtain the target audio. The apparatus of claim 16, wherein The audio generation module is specifically configured to generate a chord progression according to the pitch information and the tone information, and generate the accompaniment score according to the music information of the humming audio and the chord progression. The apparatus of claim 17, wherein The audio generation module is specifically configured to obtain a style input by a user through an interactive interface, determine the multiple instruments according to the style and the chord progression, and generate the accompaniment score according to the multiple instruments and the music information of the humming audio. The apparatus of any one of claims 16-18, wherein The audio generation module is specifically configured to perform virtual instrument rendering and single-track sound effect processing on the corresponding multiple instruments according to the multi-track MIDI music scores of the corresponding multiple instruments to obtain multiple single-track audios; Multi-track stereo mixing processing is performed on the multiple single-track audios to obtain the target audio. The apparatus of any one of claims 16-18, wherein The target audio further comprises human voice audio in the humming audio; the audio generation module is further configured to perform human voice preprocessing on the humming audio to obtain human voice single-track audio; perform virtual instrument rendering and single-track sound effect processing on the corresponding instrument according to the multi-track MIDI score of the corresponding instrument to obtain a plurality of accompaniment single-track audios; obtain a plurality of single-track audios according to the human voice single-track audio and the plurality of accompaniment single-track audios; perform multi-track stereo mixing processing on the plurality of single-track audios to obtain the target audio. A terminal device, characterized by comprising: comprise: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-10. A computer-readable storage medium, characterized by, The computer program comprises a computer program which, when executed on a computer, causes the computer to perform the method of any one of claims 1-10. A computer program product, characterized in that The computer program product comprises computer program code which, when executed on a computer, causes the computer to perform the method of any one of claims 1-10.

Images