Electronic equipment, server and sound processing method

Abstract

The embodiment of the invention relates to the technical field of intelligent terminals, and discloses electronic equipment, a server and a sound processing method.The electronic equipment can comprise a sound collector which is configured to receive sound data input by a user; the controller is coupled with the sound collector and is configured to respond to the sound data and send the sound data to a server which establishes communication connection with the electronic equipment; receiving feature information sent by the server and song information of at least one candidate song, wherein the feature information and the at least one candidate song are determined by performing voiceprint recognition on the sound data by the server; and based on the feature information and / or the song information, determining a to-be-played song in the at least one candidate song, and playing the to-be-played song. By applying the technical scheme of the invention, the song switching convenience of the user can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent terminals, and in particular to an electronic device, a server and a sound processing method. Background Art

[0002] With the development of electronic devices, the functions and services they can provide to users are becoming more and more abundant. For example, users can sing karaoke songs through karaoke applications installed on devices such as televisions.

[0003] Currently, when singing karaoke, users need to use a remote control to open the search box of the karaoke application, and then enter the song name to search for the song they want to sing. This method is rather cumbersome, causing users to spend a long time switching songs, and will interrupt the user's karaoke process, affecting the user's karaoke experience. Summary of the invention

[0004] The embodiments of the present invention provide an electronic device, a server and a sound processing method, which are used to solve the problem that the song switching process of the electronic device in the prior art is relatively complicated.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0006] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a sound collector, configured to receive sound data input by a user; a controller, coupled to the sound collector and configured to: in response to the sound data, send the sound data to a server that establishes a communication connection with the electronic device; receive feature information and song information of at least one candidate song sent by the server, the feature information being determined by the server performing voiceprint recognition on the sound data, and the at least one candidate song being determined by the server converting the sound data into text data and using the text data; based on the feature information and / or the song information, determine a song to be played from the at least one candidate song, and play the song to be played.

[0007] The above technical solution has the following advantages or beneficial effects: through the interaction between the electronic device and the server, real-time response to the user's voice data can be achieved, the process of electronic device playing songs is simplified, and convenience is provided for users to play and switch songs. At the same time, by playing the songs to be played in the candidate songs according to the feature information and song information, the corresponding version can be selected for playback according to the user's characteristics, which can provide users with more personalized search services to meet the user's personal needs.

[0008] In some embodiments, the controller is further configured to: when the automatic song switching function of the electronic device is in an on state, and the song to be played is different from the song currently being played by the electronic device, trigger an operation of playing the song to be played, and the automatic song switching function is used to indicate whether the electronic device can automatically switch songs; when the automatic song switching function of the electronic device is in an off state, and the song to be played is different from the song currently being played by the electronic device, display song prompt information, and the song prompt information is used to prompt the user to stop playing the song currently being played by the electronic device and play the song to be played.

[0009] The above technical solution has the following advantages or beneficial effects: through this method, songs can be selected and played in different ways through the automatic song switching function of the electronic device, which can improve the flexibility of the electronic device in playing songs and meet the different operation requirements of different users.

[0010] In some embodiments, the controller determines the song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played. Specifically, the controller is configured as follows: if the at least one candidate song includes multiple candidate songs, then based on the feature information and the song information, the song to be played from the multiple candidate songs is determined, and the song to be played is played; if the at least one candidate song includes only one candidate song, then based on the song information, the one candidate song is determined as the song to be played, and the song to be played is played.

[0011] The above technical solution has the following advantages or beneficial effects: through the above method, appropriate songs to be played can be selected according to the number of candidate songs returned by the server. This process does not require user participation, so it can improve the efficiency of electronic devices in playing songs and reduce the impact of interrupting song playback on user experience.

[0012] In some embodiments, the song information includes text data corresponding to the sound data, and the controller executes playing of the song to be played, and is specifically configured to: determine the playing progress of the song to be played based on the text data, and play the song to be played according to the playing progress.

[0013] The above technical solution has the following advantages or beneficial effects: Through the above method, the time for users to wait for songs to be played to the current progress can be reduced, thereby improving the user's karaoke experience.

[0014] According to another aspect of an embodiment of the present invention, a server is provided, wherein the server is configured to: receive sound data sent by an electronic device that establishes a communication connection with the server; perform voiceprint recognition on the sound data to determine feature information corresponding to the sound data, wherein the feature information is used to indicate user features corresponding to the sound data; and convert the sound data into text data, and use the text data to determine at least one candidate song corresponding to the sound data; send the feature information and song information of the at least one candidate song to the electronic device, so that the electronic device determines a song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played.

[0015] The above technical solution has the following advantages or beneficial effects: the server can receive sound data transmitted by the electronic device through interaction with the electronic device, process and analyze the sound data, determine feature information and song information of candidate songs, and return the feature information and song information to the electronic device, so that it can select the corresponding song to be played for playing, which can improve the convenience of electronic devices in searching and playing songs, simplify user operation processes, and improve the efficiency of song playback.

[0016] In some embodiments, the server performs voiceprint recognition on the sound data to determine feature information corresponding to the sound data, and is specifically configured to: extract voiceprint features of the sound data, wherein the voiceprint features include any one or more of Mel-frequency cepstral coefficients, linear prediction coefficients, and perceptual linear prediction features; and input the voiceprint features into a feature extraction model to obtain the feature information.

[0017] The above technical solution has the following advantages or beneficial effects: the server can extract characteristic information of the sound data to provide support for the subsequent determination of candidate songs and recommended songs.

[0018] In some embodiments, before converting the sound data into text data, the server is further configured to: process the sound data using audio separation technology to obtain sound data after removing background sound, wherein the background sound includes other sounds in the sound data except human voice; the server executes the conversion of the sound data into text data, and uses the text data to determine at least one candidate song corresponding to the sound data, and is specifically configured to: convert the sound data after removing the background sound into text data, and match the text data with the song text in the song database to determine the at least one candidate song.

[0019] The above technical solution has the following advantages or beneficial effects: by removing the background sound in the sound data, the sound data can be optimized and the accuracy of determining the candidate songs can be improved.

[0020] In some embodiments, the server is further configured to: input the feature information and the historically played songs of the user corresponding to the electronic device into a pre-trained song recommendation model to obtain song recommendation information, wherein the song recommendation model is composed of any one of a collaborative filtering recommendation algorithm, a content filtering recommendation algorithm and a hybrid recommendation algorithm; and send the song recommendation information to the electronic device so that the electronic device selects any one of the songs to be played in the song recommendation information for playback.

[0021] The above technical solution has the following advantages or beneficial effects: Through the above method, the user can be provided with recommendation information matching his / her age, gender and other characteristics, so as to facilitate the user to select the corresponding song for playing.

[0022] According to another aspect of an embodiment of the present invention, there is provided a sound processing method, which is applied to an electronic device, the method comprising: in response to sound data input by a user, sending the sound data to a server that establishes a communication connection with the electronic device; receiving feature information and song information of at least one candidate song sent by the server, the feature information being determined by the server performing voiceprint recognition on the sound data, and the at least one candidate song being determined by the server converting the sound data into text data and using the text data; determining a song to be played from the at least one candidate song based on the feature information and / or the song information, and playing the song to be played.

[0023] The above technical solution has the following advantages or beneficial effects: by detecting the user's voice data and obtaining the candidate songs and feature information corresponding to the sound data through interaction with the server, the songs to be played in the candidate songs are played, which can simplify the process of playing songs by electronic devices and provide convenience for users to play and switch songs. At the same time, by playing the songs to be played in the candidate songs based on the feature information and song information, the corresponding version can be selected for playback according to the user's characteristics, which can provide users with more personalized search services to meet the user's personal needs.

[0024] According to another aspect of an embodiment of the present invention, there is provided a sound processing method, which is applied to a server, and the method comprises: receiving sound data sent by an electronic device that establishes a communication connection with the server; performing voiceprint recognition on the sound data to determine feature information corresponding to the sound data, wherein the feature information is used to indicate user features corresponding to the sound data; and converting the sound data into text data, and using the text data to determine at least one candidate song corresponding to the sound data; sending the feature information and song information of the at least one candidate song to the electronic device, so that the electronic device determines a song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played.

[0025] The above technical solution has the following advantages or beneficial effects: the server can receive sound data transmitted by the electronic device through interaction with the electronic device, process and analyze the sound data, determine feature information and song information of candidate songs, and return the feature information and song information to the electronic device, so that it can select the corresponding song to be played for playing, which can improve the convenience of electronic devices in searching and playing songs, simplify user operation processes, and improve the efficiency of song playback.

[0026] According to the sound processing method provided by an embodiment of the present invention, an electronic device can receive sound data input by a user, and in response to the sound data, send the sound data to a server so that the server performs voiceprint recognition on the sound data, determines feature information corresponding to the sound data, and determines at least one candidate song corresponding to the sound data, then receives the feature information and song information of at least one candidate song sent by the server, and plays a song to be played from the at least one candidate song based on the feature information and / or song information.

[0027] By applying this solution, the user's voice data can be detected, and the candidate songs and feature information corresponding to the voice data can be obtained through interaction with the server, so as to play the songs to be played among the candidate songs. This can simplify the process of playing songs by electronic devices and provide convenience for users to play and switch songs. At the same time, by playing the songs to be played among the candidate songs based on the feature information and song information, the corresponding version can be selected for playback based on the user's characteristics, which can provide users with more personalized search services to meet their personal needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of an operation scenario between an electronic device and a control device provided in some embodiments of the present application;

[0029] Figure 2 A schematic diagram of the hardware configuration of an electronic device provided in some embodiments of the present application;

[0030] Figure 3 A schematic diagram of software configuration of an electronic device provided in some embodiments of the present application;

[0031] Figure 4 A flowchart of a sound processing method provided in some embodiments of the present application;

[0032] Figure 5 A sub-flow chart of a sound processing method provided in some embodiments of the present application;

[0033] Figure 6 A sub-flow chart of another sound processing method provided in some embodiments of the present application;

[0034] Figure 7 A flowchart of another sound processing method provided for some embodiments of the present application;

[0035] Figure 8 A sub-flow chart of another sound processing method provided in some embodiments of the present application;

[0036] Fig. 9 A sub-flow chart of another sound processing method provided in some embodiments of the present application;

[0037] Fig.10 A sub-flow chart of another sound processing method provided for some embodiments of the present application. DETAILED DESCRIPTION

[0038] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0039] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0040] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0041] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0042] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0043] In the embodiment of the present application, the electronic device 200 generally refers to a device with image display and data processing capabilities. For example, the electronic device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0044] Figure 1 Schematic diagram of operation scenarios between electronic devices and control devices provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the electronic device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the electronic device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0045] The mobile terminal 300 can be used as a control device for performing human-computer interaction between a user and the electronic device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the electronic device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the electronic device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the electronic device 200 to achieve a synchronous display function.

[0046] In some embodiments, the mobile terminal 300 or other electronic devices can also simulate the functions of the control device 100 by running an application program for controlling the electronic device 200 .

[0047] like Figure 1 As also shown in FIG. 4 , the electronic device 200 also communicates data with the server 400 through various communication methods. The electronic device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0048] The electronic device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0049] Figure 2 Some embodiments of the present application provide Figure 1 2 is a hardware configuration block diagram of the electronic device 200.

[0050] In some embodiments, the electronic device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The sound collector is a detection device in the detector 230.

[0051] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures.

[0052] In some embodiments, the sound collector may be a microphone, etc., for receiving sound data input by the user. The microphone may include a USB microphone, a Bluetooth microphone, a sound receiving device installed on a remote controller, a far-field microphone of the electronic device 200, etc.

[0053] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0054] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The electronic device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the electronic device 200 supports wireless network communication, the electronic device 200 may be provided with a communication device 220 including a WiFi function. When the electronic device 200 supports Bluetooth connection communication, the electronic device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0055] The communication device 220 can enable the electronic device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the electronic device 200 to the external device through components such as data cables and interfaces. The wireless connection can connect the electronic device 200 to the external device through wireless signals or wireless networks. The electronic device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0056] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the electronic device 200 and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the electronic device 200.

[0057] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0058] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0059] In some embodiments, the audio output interface 270 may be a local speaker of the electronic device 200, or an external audio output device of the electronic device 200. For the external audio output device of the electronic device 200, the electronic device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the electronic device 200 through the external audio output terminal to output the sound of the electronic device 200.

[0060] In some embodiments, the user interface 280 may be used to receive instructions input from a user, such as a sliding operation instruction input from a user.

[0061] In order to perform user interaction, in some embodiments, the electronic device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the electronic device 200. The operating system can control the electronic device to provide a user interface, for example, the operating system can directly control the electronic device to provide a user interface, or can provide a user interface by running an application. The operating system also allows the user to interact with the electronic device 200.

[0062] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for electronic devices.

[0063] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system runtime layer and the kernel layer.

[0064] In some embodiments, the application layer is used to provide services and interfaces for applications so that the electronic device 200 can run applications and interact with users based on applications. At least one application can be run in the application layer, and these applications can be karaoke applications, window programs that come with the operating system, system settings programs, or clock programs, etc.; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0065] In some embodiments, the application layer may also provide functional modules such as a sound input module to receive sound data input by the user through different devices and perform preliminary processing.

[0066] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0067] like Figure 3 As shown, in the embodiment of the present application, the framework layer may include a speech processing framework, a machine learning framework, a window management system (Window Management System), an activity management system (Activity Management System), a package management system (Package Management System), etc. Among them, the speech processing framework is used to provide basic support for operations related to sound recording and processing, and the machine learning framework is used to support machine learning related operations, such as model training and prediction. The window management system is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0068] The activity management system is used to interact with all activities running in the system, such as managing the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, shrinking the display window, shaking the display, distorting the display, etc.

[0069] In some embodiments, the framework layer may also include a view system, managers, content providers, etc., wherein the view system may design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager.

[0070] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer. For example, the system library layer can provide audio management services and permission management services, among which the audio management service is responsible for managing audio-related operations, such as playback and recording, and the permission management service is used to manage the permissions of applications and systems, such as handling the microphone, USB, Bluetooth and other permissions of the karaoke application to ensure user privacy and system security.

[0071] In some embodiments, the kernel layer is a functional layer between the hardware and software of the electronic device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), power driver, device driver, etc.

[0072] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the electronic device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0073] The embodiment of the present invention provides a sound processing method. Figure 4 A flow chart of the sound processing method is shown, which can be applied to Figure 1 An electronic device 200 is shown.

[0074] The electronic device 200 may include a sound collector and a controller 250 coupled to the sound collector. The sound collector may be used to receive sound data input by a user. The sound collector may be a microphone that may receive sound data when a user sings, voice commands input by a user, and the like.

[0075] According to the sound processing method provided by an embodiment of the present invention, an electronic device can receive sound data input by a user, and in response to the sound data, send the sound data to a server that establishes a communication connection with the electronic device, and then receive feature information and song information of at least one candidate song sent by the server, and based on the feature information and / or song information, determine a song to be played from the at least one candidate song, and play the song to be played.

[0076] By applying this solution, the user's voice data can be detected, and the candidate songs and feature information corresponding to the voice data can be obtained through interaction with the server, so as to play the songs to be played among the candidate songs. This can simplify the process of playing songs by electronic devices and provide convenience for users to play and switch songs. At the same time, by playing the songs to be played among the candidate songs based on the feature information and song information, the corresponding version can be selected for playback based on the user's characteristics, which can provide users with more personalized search services to meet their personal needs.

[0077] like Figure 4 As shown, the controller 250 (hereinafter referred to as the controller) is configured to perform the following steps S410 to S430:

[0078] Step S410: In response to the sound data input by the user, the sound data is sent to a server that has established a communication connection with the electronic device.

[0079] For example, when a user uses a karaoke application installed on an electronic device, the sound data of the user can be continuously collected by the sound collector of the electronic device. Figure 4The step S411 shown sends the sound data to the server, so that the server performs voiceprint recognition on the sound data through step S412, determines the feature information corresponding to the sound data, converts the sound data into text data, and uses the text data to determine at least one candidate song corresponding to the sound data.

[0080] In order to reduce the waste of resources caused by the electronic device continuously receiving and sending sound data to the server, in some embodiments, the controller can start the continuous monitoring function of the sound data when it detects that the user starts the karaoke application, so as to continuously receive the sound data input by the user and send the sound data to the server for the server to analyze and process.

[0081] In some embodiments, the controller can turn off the continuous monitoring function of the sound data when it detects that the user closes the karaoke application. At this time, the electronic device will only receive the sound data input by the user when the user wakes up the voice input function of the electronic device through a wake-up command.

[0082] In some embodiments, when receiving the user's voice data, the voice data can be sent to the server as a data stream. For example, when the electronic device receives the voice data continuously input by the user, the voice data can be sent to the server in real time at a time interval of every 2 seconds so that the server can process and analyze it.

[0083] Step S420: Receive the feature information and song information of at least one candidate song sent by the server.

[0084] The characteristic information is determined by the server performing voiceprint recognition on the sound data, and the at least one candidate song is determined by the server converting the sound data into text data and using the text data. The characteristic information includes user characteristics of the user. The user characteristics may include the user's age, gender, mood, etc. The song information may include the song name, song type, and song content of each candidate song in the at least one candidate song.

[0085] When receiving the characteristic information and song information of at least one candidate song sent by the server, the controller can store the characteristic information and song information. For example, the characteristic information and song information can be stored in a pre-allocated cache address so as to play the corresponding song later according to the characteristic information.

[0086] Step S430: Based on the feature information and / or song information, determine a song to be played from the at least one candidate song, and play the song to be played.

[0087] Exemplarily, when determining a song to be played from at least one candidate song and playing the song to be played, if the characteristic information indicates that the user is a child, a children's version of the candidate song can be selected from the candidate songs as the song to be played and played; if the characteristic information indicates that the user is a girl, a girl's version of the candidate song can be selected from the candidate songs as the song to be played and played.

[0088] Through the above steps S410 to S430, the electronic device can receive the sound data input by the user and send the sound data to the server, thereby obtaining the characteristic information and song information of at least one candidate song returned by the server after processing the sound data, and then select the song to be played based on the characteristic information and song information, thereby realizing song playback control based on sound data and improving the convenience of user control and switching song playback.

[0089] At the same time, due to the differences in age, gender and other characteristics of different users, by determining the characteristic information, suitable songs can be selected for users and played directly, without the need for users to manually enter the song name in the search box for search. For users who only remember the lyrics of the song, it can also improve the convenience and accuracy of search. In addition, users do not need to select the desired song to play from all the searched songs, so it can also reduce the user's operation steps and improve the efficiency of switching songs, especially during the karaoke process, without interrupting the user's karaoke experience, so it can improve the service quality of karaoke.

[0090] In some embodiments, in step S430, reference Figure 5 As shown, the controller can execute the following methods:

[0091] Step S510: If at least one candidate song includes multiple candidate songs, a song to be played is determined from the multiple candidate songs based on the feature information and the song information, and the song to be played is played.

[0092] For example, candidate songs that match the user's age, gender, etc. may be determined as songs to be played based on the feature information.

[0093] In some embodiments, corresponding priorities can be set for user characteristics, such as setting the user's age characteristic as the first priority and the gender characteristic as the second priority. Among at least one candidate song, the candidate song that matches the user's age characteristic can be first determined as the song to be played and played.

[0094] If a command for switching versions input by the user is received, the currently playing song is stopped, and a candidate song matching the gender feature of the user is determined as the song to be played, so that the song to be played is played.

[0095] Step S520: If the at least one candidate song includes only one candidate song, based on the song information, one candidate song is determined as the song to be played, and the song to be played is played.

[0096] When at least one candidate song includes only one candidate song, the controller can directly play the candidate song as the song to be played. This process also does not require user selection, but is executed synchronously when the electronic device receives the song information of the candidate song returned by the server.

[0097] Therefore, through the above method, appropriate songs to be played can be selected according to the number of candidate songs returned by the server. This process does not require user participation, so it can improve the efficiency of electronic devices in playing songs and reduce the impact of interrupting song playback on user experience.

[0098] In some embodiments, reference Figure 6 As shown, the controller can also execute the following methods:

[0099] Step S610: When the automatic song switching function of the electronic device is turned on and the song to be played is different from the song currently being played by the electronic device, an operation of playing the song to be played is triggered.

[0100] Among them, the automatic song switching function is used to indicate whether the electronic device can automatically switch songs.

[0101] When the automatic song switching function is turned on, it means that the electronic device currently supports automatic switching of songs, so the controller can determine whether the song to be played is the same as the song currently being played by the electronic device. If they are different, it can trigger the operation of playing the song to be played.

[0102] Step S620: When the automatic song switching function of the electronic device is in a turned-off state and the song to be played is different from the song currently being played by the electronic device, song prompt information is displayed.

[0103] The song prompt information is used to prompt the user to stop playing the song currently being played by the electronic device and play the song to be played.

[0104] When the automatic song switching function is turned off, the electronic device currently does not support automatic switching of songs, so the controller can control the electronic device to display song prompt information to prompt the user to switch songs, that is, stop playing the song being played and play the song to be played.

[0105] In some embodiments, the song prompt information may include the song title of each candidate song in at least one candidate song. When receiving an operation in which the user selects the song title of any candidate song, the controller may respond to the operation and directly play the candidate song.

[0106] Through this method, songs can be selected and played in different ways through the automatic song switching function of the electronic device, which can improve the flexibility of the electronic device in playing songs and meet the different operation requirements of different users.

[0107] In some embodiments, when the user controls the song switching function to be turned on or off, the following method may be executed:

[0108] In response to a first operation by a user to start the automatic song switching function, the automatic song switching function is set to an on state.

[0109] In response to the user's second operation of turning off the automatic song switching function, the automatic song switching function is set to an off state.

[0110] The first operation and the second operation may be operation instructions in the form of voice, or operation instructions triggered by corresponding controls on the device interface of the electronic device.

[0111] For example, when the electronic device displays a setting interface, if the controller receives a first operation from the user to turn on the control of the automatic song switching function, the controller can set the automatic song switching function to the on state; if the controller receives a second operation from the user to turn off the control of the automatic song switching function, the controller will set the automatic song switching function to the off state.

[0112] For another example, when receiving a voice command of "turn on the automatic song switching function" input by the user, the controller determines that a first operation is received and sets the automatic song switching function to an on state. When receiving a voice command of "turn off the automatic song switching function" input by the user, the controller determines that a second operation is received and sets the automatic song switching function to an off state.

[0113] Through the above method, users can choose to turn on or off the automatic song switching function, thereby controlling song playback in different ways.

[0114] In some embodiments, the song information may include text data corresponding to the sound data. When playing the song to be played, the controller may determine the playing progress of the song to be played based on the text data, and play the song to be played according to the playing progress.

[0115] For example, the text data corresponding to the sound data can be matched with the lyrics of the song to be played to determine the playing progress of the song to be played, so that the song to be played is played according to the playing progress, which can reduce the time the user waits for the song to be played to the current progress and improve the user's karaoke experience.

[0116] This embodiment also provides a sound processing method. Figure 7 A flow chart of the sound processing method is shown, which can be applied to Figure 1 Server 400 is shown.

[0117] The server 400 may be a single server, or a server cluster consisting of a plurality of servers, or a cloud server, which is not specifically limited in this embodiment.

[0118] According to the sound processing method provided by an embodiment of the present invention, the server can receive sound data sent by an electronic device, perform voiceprint recognition on the sound data, determine feature information corresponding to the sound data, convert the sound data into text data, and use the text data to determine at least one candidate song corresponding to the sound data, and then send the feature information and song information of at least one candidate song to the electronic device, so that the electronic device determines a song to be played from at least one candidate song based on the feature information and / or song information, and plays the song to be played.

[0119] By applying this solution, the server can receive sound data transmitted by the electronic device through interaction with the electronic device, process and analyze the sound data, determine feature information and song information of candidate songs, and return the feature information and song information to the electronic device so that it can select the corresponding song to be played for playing. This can improve the convenience of searching and playing songs by the electronic device, simplify the user operation process, and improve the efficiency of song playback.

[0120] like Figure 7 As shown, the server 400 (hereinafter referred to as the server) is configured to perform the following steps S710 to S740:

[0121] Step S710: receiving sound data sent by an electronic device that has established a communication connection with a server.

[0122] Exemplarily, the electronic device may receive voice data input by the user through a configured microphone, etc., and synchronously transmit the voice data to the server.

[0123] Step S720: Perform voiceprint recognition on the sound data to determine feature information corresponding to the sound data.

[0124] The characteristic information is used to indicate the user characteristics corresponding to the sound data, such as the user's age, gender, mood, etc.

[0125] Voiceprint recognition is a technology that determines the identity of a speaker by analyzing sound data. For example, the sound data can be subjected to noise reduction processing, and then endpoint detection can be performed to determine the valid speech segments in the sound data, and then the sound features in the sound data can be extracted, and the sound features can be matched with the pre-stored voiceprint template to determine the feature information corresponding to the sound data.

[0126] In some embodiments, reference Figure 8 As shown, the server may also determine the characteristic information corresponding to the sound data by the following method:

[0127] Step S810: extracting voiceprint features of sound data.

[0128] The voiceprint features may include any one or more of Mel-frequency cepstral coefficients, linear prediction coefficients, and perceptual linear prediction features.

[0129] Exemplarily, when extracting the Mel-frequency cepstral coefficients of sound data, the sound data can first be Fourier transformed to obtain the spectrum of the sound data, and then the spectrum can be filtered through a Mel-frequency filter group to convert the linear frequency into Mel-frequency, and then the logarithm is taken to obtain the Mel-frequency cepstral coefficients. Finally, a discrete cosine transform is performed to obtain the Mel-frequency cepstral coefficients.

[0130] For another example, by performing signal preprocessing on sound data, such as converting the sound data into a digital signal through an analog-to-digital converter to obtain a discrete sound sample sequence, then using a first-order high-pass filter to pre-emphasize the sound signal, then dividing the sound signal into short frames, performing windowing on each frame signal, and then calculating its autocorrelation function, and then solving a set of linear equations to obtain the linear prediction coefficients of the sound data.

[0131] Step S820: Input the voiceprint feature into the feature extraction model to obtain feature information.

[0132] Among them, the feature extraction model may include any one or more feature extraction models, such as MFCC model, PLP model, deep neural network model, convolutional neural network model, long short-term memory network model, attention mechanism model, etc.

[0133] When extracting voiceprint features through a feature extraction model, the feature extraction model can be first trained using a sample data set, and the parameters of the model can be adjusted through a back propagation algorithm, etc., so that the feature information output by the model is as close as possible to the true label or target feature, thereby completing the training of the feature extraction model.

[0134] Then, the voiceprint features are input into the trained feature extraction model to obtain the feature information corresponding to the sound data.

[0135] Through the above method, the server can extract characteristic information of the sound data to provide support for the subsequent determination of candidate songs and recommended songs.

[0136] Step S730: Convert the sound data into text data, and use the text data to determine at least one candidate song corresponding to the sound data.

[0137] When converting sound data into text data, the sound data can be preprocessed and acoustic features can be extracted. Then, a suitable speech recognition model, such as a hidden Markov model, a Gaussian mixture model, etc., is selected to analyze the acoustic features of the sound data and finally output the corresponding text data.

[0138] When determining candidate songs, a pre-built song database that stores a large number of song files and corresponding text descriptions can be used to process the sound data, such as natural language processing operations such as word segmentation and stop word removal, and then a text matching algorithm is used to calculate the similarity between the processed text data and the text descriptions of each song in the song database, such as using cosine similarity, edit distance and other methods to measure the degree of matching between texts, and screen out song files corresponding to texts with higher similarity as candidate songs.

[0139] In order to improve the accuracy of determining candidate songs, in some embodiments, reference Fig. 9 As shown, the server can also execute the following methods:

[0140] Step S910: Process the sound data using audio separation technology to obtain sound data after removing background sound.

[0141] The background sound includes other sounds in the sound data except human voice. For example, the background sound may include the sound of the electronic device playing songs, the sound of other people talking and the noise of the environment during the user's karaoke.

[0142] Exemplarily, a song separation model based on a convolutional neural network can be used to take the time-frequency graph of the sound data as input and send it into a pre-trained convolutional neural network model. After the convolutional neural network model is trained with a large amount of labeled mixed songs and corresponding pure vocal song data, it can separate the vocal part from the mixed song and output the corresponding time-frequency graph containing only the vocals, which is then converted back into the time domain sound data through operations such as inverse short-time Fourier transform.

[0143] Step S920: Convert the sound data after removing the background sound into text data, and match the text data with the song text in the song database to determine at least one candidate song.

[0144] After obtaining the sound data after removing the background sound, the sound data can be converted into text data using a corresponding speech recognition model, and at least one candidate song can be determined using a song database or the like.

[0145] Through the above method, the sound data can be optimized and processed to improve the accuracy of determining candidate songs.

[0146] Step S740: Sending feature information and song information of at least one candidate song to the electronic device, so that the electronic device determines a song to be played from at least one candidate song based on the feature information and / or the song information, and plays the song to be played.

[0147] By sending feature information and song information to the electronic device, the electronic device can select the corresponding song to be played, that is, the user can interact with the electronic device through sound. During the karaoke process, there is no need to operate a remote control to control the playback and switching of songs, so the efficiency of the user's playback and switching of songs can be improved.

[0148] In some embodiments, reference Fig.10 As shown, the server can also execute the following methods:

[0149] Step S1010: input the feature information and the historically played songs of the user corresponding to the electronic device into a pre-trained song recommendation model to obtain song recommendation information.

[0150] Among them, the song recommendation model is composed of any one of a collaborative filtering recommendation algorithm, a content filtering recommendation algorithm and a hybrid recommendation algorithm; the user's historically played songs can be searched in a database that stores the song data played at the user's historical moments, using the user name as an index.

[0151] By inputting the feature information and the user's historically played songs into a pre-trained song recommendation model, the song content related to the user's historical preferences can be selected based on the user's feature information as song recommendation information.

[0152] Step S1020: Send the song recommendation information to the electronic device, so that the electronic device selects any song to be played in the song recommendation information for playing.

[0153] Exemplarily, the song information and song recommendation information of the candidate songs can be sent to the electronic device at the same time, so that the electronic device displays the song recommendation information while playing the songs to be played among the candidate songs for the user to select.

[0154] Through the above method, the user can be provided with recommendation information matching his / her age, gender and other characteristics, so as to facilitate the user to select the corresponding song to play.

[0155] According to the sound processing method provided in this embodiment, the user's voice data can be detected in real time by an electronic device, and the sound data can be processed by a server to determine the characteristic information corresponding to the sound data and the song information of the candidate songs, so that the electronic device can directly play the song content that matches the user's voice data. Whether it is the first time to play a song or when switching songs, no manual operation by the user is required, so the efficiency of the user's playing and switching songs can be improved. At the same time, songs that are adapted to the user's personal characteristics can be pushed to the user, making it easier for the user to select the songs to be played.

[0156] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on an electronic device, the electronic device executes the sound processing method in any of the above method embodiments.

[0157] The executable instructions can be specifically used to enable the electronic device to execute the above-mentioned sound processing method.

[0158] In this embodiment, computer readable storage media may be any tangible media that can contain or store a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] More specific examples (a non-exhaustive list) of computer-readable storage media may include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0161] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0162] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. An electronic device, characterized in that: include: A sound collector is configured to receive sound data input by a user; A controller is coupled to the sound collector and is configured to: In response to the sound data, sending the sound data to a server that establishes a communication connection with the electronic device; Receiving feature information and song information of at least one candidate song sent by the server, wherein the feature information is determined by the server performing voiceprint recognition on the sound data, and the at least one candidate song is determined by the server converting the sound data into text data and using the text data; Based on the feature information and / or the song information, a song to be played is determined from the at least one candidate song, and the song to be played is played.

2. The electronic device according to claim 1, characterized in that: The controller is further configured to: When the automatic song switching function of the electronic device is turned on and the song to be played is different from the song currently being played by the electronic device, triggering an operation of playing the song to be played, wherein the automatic song switching function is used to indicate whether the electronic device can automatically switch songs; When the automatic song switching function of the electronic device is in a turned-off state and the song to be played is different from the song currently being played by the electronic device, song prompt information is displayed, and the song prompt information is used to prompt the user to stop playing the song currently being played by the electronic device and play the song to be played.

3. The electronic device according to claim 1, characterized in that: The controller determines a song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played, and is specifically configured as follows: If the at least one candidate song includes a plurality of candidate songs, determining the song to be played from the plurality of candidate songs based on the feature information and the song information, and playing the song to be played; If the at least one candidate song includes only one candidate song, based on the song information, the one candidate song is determined as the song to be played, and the song to be played is played.

4. The electronic device according to any one of claims 1 to 3, characterized in that: The song information includes text data corresponding to the sound data, and the controller plays the song to be played, and is specifically configured as follows: Based on the text data, the playing progress of the song to be played is determined, and the song to be played is played according to the playing progress.

5. A server, characterized in that: The server is configured to: Receiving sound data sent by an electronic device that has established a communication connection with the server; Performing voiceprint recognition on the voice data to determine feature information corresponding to the voice data, where the feature information is used to indicate user features corresponding to the voice data; as well as, Converting the sound data into text data, and using the text data to determine at least one candidate song corresponding to the sound data; The feature information and the song information of the at least one candidate song are sent to the electronic device, so that the electronic device determines a song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played.

6. The server according to claim 5, characterized in that: The server performs voiceprint recognition on the sound data to determine feature information corresponding to the sound data, and is specifically configured as follows: Extracting voiceprint features of the sound data, wherein the voiceprint features include any one or more of Mel-frequency cepstrum coefficients, linear prediction coefficients, and perceptual linear prediction features; The voiceprint feature is input into a feature extraction model to obtain the feature information.

7. The server according to claim 5, characterized in that: Before converting the sound data into text data, the server is further configured to: Processing the sound data using an audio separation technology to obtain sound data after background sound is removed, wherein the background sound includes other sounds in the sound data except for human voice; The server converts the sound data into text data, and uses the text data to determine at least one candidate song corresponding to the sound data, and is specifically configured as follows: The sound data after background sound removal is converted into text data, and the text data is matched with song texts in a song database to determine the at least one candidate song.

8. The server according to any one of claims 5 to 7, characterized in that: The server is further configured to: Input the feature information and the historically played songs of the user corresponding to the electronic device into a pre-trained song recommendation model to obtain song recommendation information, wherein the song recommendation model is composed of any one of a collaborative filtering recommendation algorithm, a content filtering recommendation algorithm and a hybrid recommendation algorithm; The song recommendation information is sent to the electronic device, so that the electronic device selects any one of the songs to be played in the song recommendation information for playing.

9. A sound processing method, characterized in that: Applied to electronic equipment, the method comprises: In response to the sound data input by the user, sending the sound data to a server that has established a communication connection with the electronic device; Receiving feature information and song information of at least one candidate song sent by the server, wherein the feature information is determined by the server performing voiceprint recognition on the sound data, and the at least one candidate song is determined by the server converting the sound data into text data and using the text data; Based on the feature information and / or the song information, a song to be played is determined from the at least one candidate song, and the song to be played is played.

10. A sound processing method, characterized in that: Applied to a server, the method comprises: Receiving sound data sent by an electronic device that has established a communication connection with the server; Performing voiceprint recognition on the voice data to determine feature information corresponding to the voice data, where the feature information is used to indicate user features corresponding to the voice data; and Converting the sound data into text data, and using the text data to determine at least one candidate song corresponding to the sound data; The feature information and the song information of the at least one candidate song are sent to the electronic device, so that the electronic device determines a song to be played from the at least one candidate song based on the feature information and / or the song information, and plays the song to be played.

Images