Display device and sound synthesis method

Abstract

The invention provides a display device and a sound synthesis method, and the method comprises the steps: receiving sound data collected by a sound collector in response to an access instruction of the sound collector, and storing the sound data and corresponding collection time at a preset position; according to an input playing instruction, accompaniment audio data and original sound audio data of a target song are decoded, a start time identifier contained in the accompaniment audio data is detected, and after the start time identifier is detected, sound data are periodically read according to time corresponding to the start time identifier and collection time; when the sound data is read, generating target audio data according to the accompaniment audio data and the read sound data; and when the sound data is not read, generating target audio data according to the accompaniment audio data and the replacement audio data. According to the method, the missing sound data of the user can be filled in a mode of replacing the audio data, the integrity of the target audio is ensured, and the problem that the audio is incomplete in the audio recording process is solved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a sound synthesis method. Background Art

[0002] In addition to presenting traditional TV programs, display devices can also host various entertainment activities, such as games, movies, music, recording songs, karaoke, etc. Take karaoke as an example. As a popular form of entertainment, karaoke is widely used on large screens. Through the karaoke application of smart TVs, the user's voice can be synthesized with the accompaniment music to form a complete karaoke work.

[0003] Most existing karaoke applications use real-time recording of user singing voices to synthesize works, and can also be equipped with sound beautification functions, such as reverberation, equalizer, etc., to achieve sound beautification. When recording karaoke works through karaoke applications, that is, recording audio, the karaoke application will process the user's actual singing voice in real time and synthesize the final karaoke work.

[0004] Although the above method can meet the user's recording needs to a certain extent, when processing the incomplete singing of the user, the sound will be intermittent or missing. For example, when the user does not sing a certain sentence or a certain paragraph, the audio synthesized by the display device will be incomplete, affecting the integrity of the final audio. Summary of the invention

[0005] Some embodiments of the present application provide a display device and a sound synthesis method to solve the problem of incomplete audio during audio recording.

[0006] In a first aspect, some embodiments of the present application provide a display device, including:

[0007] a display configured to display a user interface;

[0008] The controller is configured as:

[0009] In response to an access instruction from a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset location;

[0010] Decoding the accompaniment audio data and the original sound audio data of the target song according to the inputted playing instruction, wherein the playing instruction is used to instruct to play the target song;

[0011] Detecting a start time mark included in the accompaniment audio data, wherein the start time mark is used to indicate the start of the vocal data in the original audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the acquisition time;

[0012] When the sound data is read, target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, target audio data is generated according to the accompaniment audio data and replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

[0013] The above technical solution has the following advantages or beneficial effects: if the accompaniment audio data requires human voice input, but the sound collector such as a microphone does not collect the human voice, that is, the sound data, the display device will supplement the missing sound data with the original sound or simulated user voice, so that the recorded audio data maintains integrity and continuity, and fills the missing sound data of the user by replacing the audio data, thereby solving the problem of incomplete audio during the audio recording process.

[0014] In some embodiments, before the step of periodically reading the sound data, the controller is further configured to:

[0015] At a preset time point before the time point corresponding to the start time identifier, the sound data whose collection time is before the time point corresponding to the start time identifier and stored in the preset position is cleared.

[0016] The above technical solution has the following advantages or beneficial effects: the display device can automatically clear outdated or invalid data before formally collecting or processing sound data, avoiding problems such as inefficient processing or inaccurate results caused by redundant or outdated data, thereby improving the data quality of the sound data.

[0017] In some embodiments, after the step of periodically reading the sound data, the controller is further configured to:

[0018] Detecting an end time mark corresponding to the start time mark; the end time mark is used to indicate the end of the human voice data in the original audio data;

[0019] After the end time mark is detected, the reading of the sound data is stopped.

[0020] The above technical solution has the following advantages or beneficial effects: by detecting the end time mark, the display device can avoid unnecessary reading and analysis of sound data, and only read the interval where the accompaniment audio data exists in the original audio data, thereby improving the processing efficiency of the sound data. At the same time, after stopping reading the sound data, the display device can release related resources to improve the sorting efficiency of the display device.

[0021] In some embodiments, the controller periodically reads the sound data and is specifically configured to:

[0022] After identifying the start time mark, generating a sound reading instruction;

[0023] In response to the sound reading instruction, the sound data is periodically read.

[0024] The above technical solution has the following advantages or beneficial effects: by periodically reading the sound data, the display device can determine the real-time and continuity of the sound data, and provide a basis for subsequent analysis and synthesis of the sound data.

[0025] In some embodiments, the controller stops reading the sound data and is specifically configured to:

[0026] After identifying the end time mark, generating a stop reading instruction;

[0027] In response to the stop reading instruction, the reading of the sound data is stopped.

[0028] The above technical solution has the following advantages or beneficial effects: the display device will immediately stop the current sound data reading operation according to the stop reading instruction to ensure that the reading of the sound data can be stopped accurately and timely, thereby improving the processing efficiency of the sound data.

[0029] In some embodiments, the replacement audio data further includes analog audio data, and before the step of generating target audio data according to the accompaniment audio data and the replacement audio data, the controller is further configured to:

[0030] Analyzing the voice characteristics of the user according to the voice data;

[0031] Acquire the original audio data at the time point corresponding to the sound data that is not read;

[0032] The simulated audio data of the user at the time point is generated according to the sound feature and the target original audio data.

[0033] The above technical solution has the following advantages or beneficial effects: the gaps in the sound data can be filled by generating simulated audio data, and since the simulated audio data comprehensively considers the voice characteristics of the user, and combines the voice characteristics of the user and the voice characteristics of the missing part, the simulated audio is equivalent to the actual voice data of the user. In this way, the simulated audio data generated based on the voice characteristics of the user and the voice characteristics of the missing part can fill in the segments that the user has not sung, thereby ensuring the integrity and continuity of the audio.

[0034] In some embodiments, the controller generates target audio data and is specifically configured to:

[0035] Calculating a sound energy value corresponding to the sound data;

[0036] comparing the sound energy value with an energy threshold;

[0037] If the sound energy value is greater than or equal to the energy threshold, determining that the sound data is valid, and generating target audio data based on the accompaniment audio data and the sound data;

[0038] If the sound energy value is less than the energy threshold, the sound data is determined to be invalid, and target audio data is generated based on the accompaniment audio data and the replacement audio data.

[0039] The above technical solution has the following advantages or beneficial effects: by calculating the energy value of the sound data and comparing it with a preset energy threshold, the display device can automatically determine the validity of the sound data. When the sound data is valid, the target audio data is generated based on the accompaniment audio data and the sound data; when the sound data is invalid, the target audio data is generated based on the accompaniment audio data and the replacement audio data. In this way, invalid or low-quality audio data can be avoided from being used to generate target audio data, thereby improving the continuity, integrity and accuracy of audio processing.

[0040] In some embodiments, before the step of decoding the accompaniment audio data and the original audio data of the target song, the controller is further configured to:

[0041] Receive an initial audio and video stream file corresponding to the target song;

[0042] The initial audio and video stream file is parsed to separate the accompaniment audio data and the original sound audio data in the initial audio and video stream file.

[0043] The above technical solution has the following advantages or beneficial effects: the display device separates the accompaniment audio data and the original audio data, which is helpful for subsequent audio recording, synthesis and other processing steps, and provides a data basis for subsequent processing.

[0044] In some embodiments, before the step of receiving the sound data collected by the sound collector, the controller is further configured to:

[0045] After the sound collector is connected, a start collection instruction is generated;

[0046] In response to the start collection instruction of the sound collector, the sound collector is controlled to be in a continuous recording state.

[0047] The above technical solution has the following advantages or beneficial effects: after collecting the sound data, the display device can save the recorded sound data in real time to a pre-set storage location. In this way, by storing the sound data, subsequent data processing and analysis can be facilitated, and the security and integrity of the sound data can also be guaranteed.

[0048] In a second aspect, some embodiments of the present application provide a method for synthesizing sound, which can be applied to the display device of the first aspect, the display device comprising a display, a sound collector, and a controller, the method comprising:

[0049] In response to an access instruction from a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset location;

[0050] Decoding the accompaniment audio data and the original sound audio data of the target song according to the inputted playing instruction, wherein the playing instruction is used to instruct to play the target song;

[0051] Detecting a start time mark included in the accompaniment audio data, wherein the start time mark is used to indicate the start of the vocal data in the original audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the acquisition time;

[0052] When the sound data is read, target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, target audio data is generated according to the accompaniment audio data and replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

[0053] The above technical solution has the following advantages or beneficial effects: if the accompaniment audio data requires human voice input, but the sound collector such as a microphone does not collect the human voice, that is, the sound data, the display device will supplement the missing sound data with the original sound or simulated user voice, so that the recorded audio data maintains integrity and continuity, and fills the missing sound data of the user by replacing the audio data, thereby solving the problem of incomplete audio during the audio recording process.

[0054] As can be seen from the above technical scheme, some embodiments of the present application provide a display device and a method for synthesizing sound, the method comprising: in response to an access instruction of a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset position; according to the input play instruction, decoding the accompaniment audio data and the original sound audio data of the target song, wherein the play instruction is used to indicate the play of the target song; detecting the start time mark contained in the accompaniment audio data, the start time mark is used to characterize the start of the human voice data in the original sound audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the collection time; when the sound data is read, generating the target audio data according to the accompaniment audio data and the read sound data; when the sound data is not read, generating the target audio data according to the accompaniment audio data and the replacement audio data, wherein the replacement audio data includes the original sound audio data of the time corresponding to the sound data that is not read. The method can fill the missing sound data of the user by replacing the audio data, ensure the integrity of the target audio, and solve the problem of incomplete audio in the audio recording process. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate some embodiments of the present application or technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0057] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0058] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0059] Figure 4 A schematic flow chart of a method for synthesizing sound performed by a display device provided in some embodiments of the present application;

[0060] Figure 5 A schematic diagram of a flow chart of separating accompaniment audio data and original sound audio data by a display device provided in some embodiments of the present application;

[0061] Figure 6 A schematic diagram of a scenario in which a display device detects an end time mark corresponding to a start time mark provided in some embodiments of the present application;

[0062] Figure 7 A schematic diagram of a process for generating replacement audio data corresponding to an unreadable state provided in some embodiments of the present application;

[0063] Figure 8 A schematic diagram of a process for generating target audio data by a display device provided in some embodiments of the present application;

[0064] Fig. 9 A schematic diagram of the effect of a display device generating target audio data provided by some embodiments of the present application;

[0065] Fig.10 A timing diagram of a method for synthesizing sound performed by a display device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0066] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0067] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0068] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0069] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0070] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0071] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0072] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0073] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0074] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0075] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0076] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0077] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0078] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0079] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0080] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0081] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0082] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0083] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0084] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0085] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0086] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0087] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0088] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0089] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0090] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0091] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0092] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0093] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0094] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0095] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0096] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0097] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0098] The above embodiments illustrate the hardware / software architecture and functional implementation of a display device 200. In addition to traditional TV viewing, the display device 200 can also carry various entertainment activities, such as games, movies, music, song recording, karaoke, etc. Take karaoke as an example. As a popular form of entertainment, karaoke is widely used on large screens. Through the karaoke application of smart TVs, the user's voice can be synthesized with the accompaniment music to form a complete karaoke work.

[0099] Most existing karaoke applications use real-time recording of user singing voices to synthesize works, and can also be equipped with sound beautification functions, such as reverberation, equalizer, etc., to achieve sound beautification. When recording karaoke works, that is, recording audio, through a karaoke application, the karaoke application will process in real time according to the actual singing voice of the user, and synthesize the final karaoke work. Although this method can meet the user's recording needs to a certain extent, when processing the passages that the user has not sung completely, there will be intermittent or missing sounds. For example, when the user does not sing a certain sentence or a certain passage, the audio synthesized by the display device 200 will be incomplete, affecting the integrity of the final audio.

[0100] That is, when recording songs or karaoke on TV, the display device 200 can record the user's karaoke works through the karaoke application. However, in the process of recording audio, some lyrics or paragraphs may not be sung, so that the display device 200 will have obvious blanks or discontinuous sounds when generating the final audio, affecting the integrity of the audio.

[0101] In order to solve the problem of incomplete audio during audio recording, some embodiments of the present application provide a display device 200, which includes a display 260 and a controller 250. The display 260 is configured to display a user interface, and the controller 250 enables the display device 200 to execute a sound synthesis method by running an application. The display device 200 can fill in the segments that the user has not sung with the original audio or the user's voice, and splice the segments that have not been sung with the sung parts to ensure the integrity of the audio.

[0102] In order to facilitate the understanding of the technical solutions in some embodiments of the present application, each step is described in detail below in conjunction with some specific embodiments and drawings. Figure 4 A flow chart of a method for synthesizing sound performed by a display device provided in some embodiments of the present application, such as Figure 4 As shown, in some embodiments, when the display device 200 executes the sound synthesis method, it can include the following steps S1-S4, the specific contents are as follows:

[0103] Step S1: In response to an access instruction from a sound collector, the display device 200 receives sound data collected by the sound collector, and stores the sound data and the corresponding collection time in a preset location.

[0104] In some embodiments, before acquiring the sound data collected by the sound collector, the display device 200 may also execute the following process: after the sound collector is connected, generate a start collection instruction; in response to the start collection instruction of the sound collector, control the sound collector to be in a continuous recording state.

[0105] Exemplarily, after a sound collection device such as a microphone is connected to the display device 200, sound data can be collected through the sound collection device. When the sound collector is ready and issues an instruction to start collecting, the display device 200 can maintain the sound collector in a state of continuous recording until an instruction to stop recording is received. After collecting the sound data, the display device 200 can save the recorded sound data to a pre-set storage location in real time. In this way, by storing the sound data, subsequent data processing and analysis can be facilitated, and the security and integrity of the sound data can also be guaranteed.

[0106] In some embodiments, when a sound collector (such as a microphone) is connected to the display device 200 through a physical connection or wirelessly, after the display device 200 detects that the sound collector is connected, it can generate a preset access instruction. Through the access instruction, the display device 200 can start monitoring and receiving real-time sound data collected from the sound collector.

[0107] In order to ensure the traceability and synchronization of the sound data, the display device 200 can store the sound data and the corresponding collection time (such as collecting in the form of a timestamp) together in a preset storage location, for example, it can be stored in a specific buffer in the memory or a dedicated folder on the hard disk, etc. The storage in the preset location can facilitate the access and management of the sound data and improve the processing effect of the sound data. The display device 200 can receive and store the sound data and its collection time in real time, providing an accurate time synchronization basis for the subsequent processing of the sound data. After step S1 is executed, the following step S2 may be included.

[0108] Step S2: the display device 200 decodes the accompaniment audio data and the original audio data of the target song according to the input play instruction.

[0109] In some embodiments, the display device 200 needs to separate the accompaniment audio data and the original audio data before decoding the accompaniment audio data and the original audio data of the target song. Figure 5 A schematic diagram of a flow chart of separating accompaniment audio data and original sound audio data from a display device provided in some embodiments of the present application, such as Figure 5 As shown, the display device 200 can first receive the initial audio and video stream file corresponding to the target song, and then parse the received initial audio and video stream file, the purpose of which is to separate the accompaniment audio data and the original sound audio data in the initial audio and video stream file. In this way, the display device 200 separates the accompaniment audio data and the original sound audio data, which is helpful for subsequent audio recording, synthesis and other processing steps, and provides a data basis for subsequent processing.

[0110] In some embodiments, after the user inputs a play instruction for indicating the play of the target song through an interface or voice, the display device 200 can first parse the instruction to determine the target song, and then decode the accompaniment audio data and the original sound audio data of the target song. For example, the decoding process can be performed by a built-in decoder, or the accompaniment audio data and the original sound audio data of the target song can be separated by audio editing software, for example, the accompaniment audio data is usually background music or instrumental accompaniment, and the original sound audio data can be the voice of a singer or speaker. After the accompaniment audio data and the original sound audio data are decoded, a data basis can be provided for the subsequent recording process. After step S2 is executed, the following step S3 may be included.

[0111] Step S3: the display device 200 detects the start time mark contained in the accompaniment audio data, where the start time mark is used to indicate the beginning of the vocal data in the original audio data; and after detecting the start time mark, periodically reads the sound data according to the time corresponding to the start time mark and the acquisition time.

[0112] In certain embodiments, after the display device 200 completes decoding of the accompaniment audio data and the original sound audio data, the start time mark included in the accompaniment audio data can be detected, wherein the start time mark can be a mark or other forms, for indicating the starting moment of the vocal part in the original sound audio data matched with the accompaniment audio data, after detecting the start time mark, the display device 200 can start periodically reading the sound data according to the start time mark and the acquisition time of the sound data stored in the preset position before. The event of periodic reading can be preset, for example, it can be 20 milliseconds, then the display device 200 can read the sound data once every 20 milliseconds after detecting the start time mark. In this way, by detecting the start time mark, the display device 200 can realize the accurate alignment of the human voice and the accompaniment, to enhance the synchronization of audio synthesis. By periodically reading the sound data, the display device 200 can determine whether the sound data can be read, and provide a basis for the subsequent sound data synthesis.

[0113] In some embodiments, when the display device 200 performs periodic reading of sound data, after identifying the start time mark, it can generate a sound reading instruction, and periodically read the sound data in response to the sound reading instruction.

[0114] Exemplary, after the display device 200 monitors and identifies the start time mark, it can generate a sound reading instruction, which is used to trigger the reading operation of subsequent sound row data. In response to the sound reading instruction, the display device 200 can read the sound data according to a preset cycle (such as per second, every 20 milliseconds, etc.), and the reading process will continue. Taking the 5th second as an example, the interval period is 20 milliseconds, then the 5th second 00 starts to read the sound data, and the result of the reading is divided into whether it is read, the 20th second is read again, the 40th second is read again, and so on, until a stop instruction is received or the preset reading duration is reached. In this way, by periodically reading the sound data, the display device 200 can judge the real-time and continuity of the sound data, and provide a basis for the analysis and synthesis of the subsequent sound data.

[0115] In some embodiments, the sound data not only includes the sound of the user singing or speaking, but may also include environmental noise, etc. For this reason, the display device 200 can also clear the sound data with a collection time before the time point corresponding to the start time identifier stored in the preset position at a preset time point before the time point corresponding to the start time identifier before the step of periodically reading the sound data.

[0116] Exemplarily, the display device 200 can determine whether the current time has reached a preset time point before the time point corresponding to the start time identifier. For example, the time point corresponding to the start time identifier is the time from the start of playback to the 5th second, then the preset time point before the time point corresponding to the start time identifier can be set to the 4th second, that is, 1 second before the start time identifier, and the display device 200 can perform a clearing operation at the 4th second. Specifically, the display device 200 needs to clear all sound data stored in the preset position before the time point corresponding to the start time identifier. It can be understood that the collection time of these cleared data is earlier than the time point corresponding to the start time identifier. In this way, the display device 200 can automatically clear outdated or invalid data before formally collecting or processing sound data, avoiding problems such as low processing efficiency or inaccurate results caused by redundant or outdated data, so as to improve the data quality of the sound data. After step S3 is executed, the following step S4 can be included.

[0117] Step S4: when the sound data is read, the target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, the target audio data is generated according to the accompaniment audio data and the replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

[0118] After the display device 200 periodically reads the sound data, there are two reading results, one is that the sound data is read, and the other is that the sound data is not read. According to different reading results, the present application has different sound synthesis methods, which are introduced below.

[0119] In some embodiments, when the display device 200 reads the sound data, it can synthesize the read real-time sound data with the currently played accompaniment audio data to generate target audio data, that is, generate target audio data containing the user's voice and the accompaniment. For example, the display device 200 starts reading at the 5th second of the accompaniment, and reads periodically every 20 milliseconds, and reads the 5th second 00-5th second 20 for the first time. After the sound data is read in this period, the user's voice data of this section and the accompaniment audio data are synthesized into the target audio data.

[0120] In some embodiments, if the display device 200 fails to read the sound data due to some reasons (such as the user pausing the singing, microphone failure, etc.), the display device 200 can use the replacement audio data to represent the user's sound data. For example, the replacement audio data may include the original sound audio data of the time corresponding to the sound data that is not read, and the target audio data is generated according to the accompaniment audio data and the replacement audio data. In this way, even if the user does not occur, the display device 200 can also output the target audio data within the cycle. For example, when the display device 200 reads the sound data for the second time, it is from the 5th second 20-5th second 40. If the sound data is not read in this cycle, the original sound audio data from 5th second 20 to 5th second 40 can be supplemented. It can be understood that 5 seconds 00-5 seconds 20 and 5 seconds 20-5 seconds 40 are the minimum loop units. If the time between the 5th second and the 10th second should be the time for the user to sing karaoke or record a song, the sound data can be periodically read at preset time intervals. Within a small loop unit, the sound data can be synthesized with the accompaniment if it is read. If it is not read, the audio data and the accompaniment are replaced for synthesis. Until the 10th second, the loop of this segment ends.

[0121] In some embodiments, after the step of periodically reading the sound data, the display device 200 can also detect an end time marker corresponding to the start time marker, where the end time marker is used to represent the end of the human voice data in the original audio data, and stop reading the sound data after detecting the end time marker.

[0122] Exemplarily, after detecting the start time marker and periodically reading the sound data, the display device 200 can also detect the end time marker corresponding to the start time marker. The end time marker is used to represent the end moment of the vocal part in the original audio data that matches the accompaniment audio data. After determining that the vocal data has ended, the display device 200 can stop reading the sound data.

[0123] Figure 6 A schematic diagram of a scene in which a display device detects an end time mark corresponding to a start time mark provided in some embodiments of the present application, such as Figure 6 As shown, in some embodiments, the accompaniment audio data can be music, and the original audio data can be the voice of the original singer of the song. The display device 200 can add a start time marker when the human voice begins, such as when the first sentence of lyrics begins, and add an end time marker when the first sentence of lyrics ends. The first sentence of lyrics is a large cycle, and each time the sound data is periodically read at a preset time interval is a small cycle. The entire cycle is completed when the entire accompaniment is played.

[0124] By detecting the end time mark, the display device 200 can avoid unnecessary reading and analysis of sound data, and only read the accompaniment audio data in the interval where the original audio data exists, thereby improving the processing efficiency of the sound data. At the same time, after stopping reading the sound data, the display device 200 can release related resources to improve the sorting efficiency of the display device 200.

[0125] In some embodiments, when the display device 200 stops reading the sound data, it can be implemented through the following process: After identifying the end time mark, the display device 200 can generate a stop reading instruction, and stop reading the sound data in response to the stop reading instruction.

[0126] Exemplarily, after identifying the end time marker, the display device 200 can generate a stop reading instruction. The display device 200 will immediately stop the current sound data reading operation according to the stop reading instruction to ensure that the reading of the sound data can be stopped accurately and timely, thereby improving the processing efficiency of the sound data.

[0127] Figure 7 A schematic diagram of a process for generating replacement audio data corresponding to an unreadable state provided in some embodiments of the present application, such as Figure 7 As shown, in some embodiments, the replacement audio data also includes simulated audio data. Before the step of generating the target audio data according to the accompaniment audio data and the replacement audio data, the display device 200 may also perform the following process to generate the corresponding replacement audio data when the sound data is not read. First, the display device 200 may analyze the voice characteristics of the user according to the sound data, then obtain the target original sound audio data at the time point corresponding to the unread sound data, and generate the simulated audio data of the user at the time point according to the sound characteristics and the target original sound audio data.

[0128] Exemplarily, the display device 200 identifies and extracts the user's voice features by parsing the user's voice data, and these features include but are not limited to pitch, timbre, speech speed, etc. In the process of parsing the voice data, the display device 200 can identify which time points the voice data has not been successfully read or recognized. For these time points where the voice data has not been read, the display device 200 can further obtain the corresponding original sound audio data, and generate the user's simulated audio data at these time points based on the extracted voice features and the original sound audio data at the time points where the voice data has not been read. In this way, the gaps in the voice data can be filled by the generation of simulated audio data. Since the simulated audio data comprehensively considers the user's voice features, combined with the user's voice features and the voice features of the missing parts, the simulated audio is equivalent to the user's actual voice data. In this way, the generation of simulated audio data based on the user's voice features and the voice features of the missing parts can fill in the segments that the user has not sung, ensuring the integrity and continuity of the audio.

[0129] That is to say, in an embodiment of the present application, if the accompaniment audio data requires human voice input, but the sound collector such as a microphone does not collect the human voice, that is, the sound data, the display device 200 will supplement the missing sound data with the original sound or simulated user voice, so that the recorded audio data maintains integrity and continuity, and fills the missing sound data of the user by replacing the audio data, thereby solving the problem of incomplete audio during the audio recording process.

[0130] In order to improve the accuracy of the sound data during the recording process, in some embodiments, the display device 200 can also determine the validity of the sound data when generating the target audio data.

[0131] Figure 8 A schematic diagram of a flow chart of a display device generating target audio data according to some embodiments of the present application, such as Figure 8 As shown, in some embodiments, the display device 200 can calculate the sound energy value corresponding to the sound data and compare the sound energy value with the energy threshold. If the sound energy value is greater than or equal to the energy threshold, the display device 200 can determine that the sound data is valid, and generate target audio data based on the accompaniment audio data and the sound data; if the sound energy value is less than the energy threshold, the display device 200 can determine that the sound data is invalid, and generate target audio data based on the accompaniment audio data and the replacement audio data, and the effect is as follows. Fig. 9 shown.

[0132] Exemplarily, the display device 200 can perform Fourier transform and other algorithm processing on the sound data to calculate the energy value corresponding to the sound data. For example, when the original sound signal is collected by the sound collector, it can be converted into a digital form by analog-to-digital conversion, that is, the continuous analog signal is converted into a discrete digital audio signal. After that, the display device 200 can convert the digital audio signal to the frequency domain through fast Fourier transform. In the frequency domain, the digital audio signal can be represented as a set of different frequency components, each component has its amplitude and phase information, and the energy of each frequency component can be obtained by calculating the square of its amplitude. In the frequency domain, due to the characteristics of Fourier transform, the energy distribution in the frequency domain is equivalent to the energy distribution in the time domain, but distributed on different frequency components. Therefore, by summing the energy of all frequency components, the total energy of the entire signal can be obtained, that is, the sound energy value corresponding to the sound data can be calculated. After the sound energy value is calculated, the display device 200 can set an energy threshold. When the energy of the detected human voice signal exceeds this threshold, the human voice is considered to be valid, otherwise it is invalid.

[0133] In some embodiments, the validity of the sound data can also be verified by other means. For example, the judgment can be based on the signal-to-noise ratio, and the signal-to-noise ratio between the human voice signal and the background noise can be calculated. If the signal-to-noise ratio is higher than a certain preset value, the human voice is considered to be valid, otherwise it is invalid. In actual use scenarios, a specific judgment method can be selected according to actual conditions, and this application does not limit this.

[0134] By calculating the energy value of the sound data and comparing it with a preset energy threshold, the display device 200 can automatically determine the validity of the sound data. If the sound energy value is greater than or equal to the energy threshold, the display device 200 can determine that the sound data is valid, and generate target audio data based on the accompaniment audio data and the sound data; if the sound energy value is less than the energy threshold, the display device 200 can determine that the sound data is invalid, and generate target audio data based on the accompaniment audio data and the replacement audio data. In this way, invalid or low-quality audio data can be avoided from being used to generate target audio data, thereby improving the continuity, integrity and accuracy of audio processing.

[0135] In order to facilitate further understanding of the technical solution of the present application, the above process is summarized and explained below in combination with actual usage scenarios and timing diagrams.

[0136] Fig.10 A timing diagram of a method for synthesizing sound performed by a display device provided in some embodiments of the present application, such as Fig.10 As shown, in order to solve the problem of incomplete audio during the audio recording process, the display device 200 can be provided with an accompaniment function module, a sound collection module and a storage module, wherein the accompaniment function module is used to perform functions such as accompaniment playback and audio synthesis, the sound collection module is used to collect sound data recorded by the user, and the storage module is used to store the sound data recorded by the user. The functions of each functional module will be described in detail below in combination with the actual process.

[0137] Taking the sound collection module as a microphone, the recorded audio as a song, and the display device 200 as a TV as an example, when the user starts recording, if the microphone has been inserted into the TV, the storage module on the TV side will store the sound data collected by the microphone in real time in the preset position of the TV. When the song is played through the accompaniment function module, the beginning of the song is usually accompaniment, without human voice. When the song is played to the time point when the human voice starts, that is, after the start time mark is played, the mechanism for reading the human voice can be triggered, that is, the sound data recorded by the user is read. At this time, the accompaniment function module can periodically read the human voice, that is, the sound data, from the preset position of the storage space, and the storage module can return the sound data to the accompaniment function module. After the accompaniment function module receives the sound data returned by the storage module, it can judge whether the sound data can be read when there should be human voice according to each cycle. When the sound data is read, the target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, the target audio data is generated according to the accompaniment audio data and the replacement audio data, and the replacement audio data includes the original audio data of the time corresponding to the sound data that is not read, or the simulated audio data.

[0138] In some embodiments, if the user's voice data can be read in each cycle, it can be considered that the user sings along throughout the whole process, and all of them are valid voice data. At this time, the voice data and the accompaniment data can be directly synthesized into the target audio through the accompaniment function module. If the sound data is not read in a certain cycle or multiple cycles, if it is not processed, the final synthesized audio will be missing. Therefore, the present application provides two ways to complete the sound data, one is the original sound of the original song, and the data in the original sound audio file can be read according to the playback time point. The other is to generate simulated audio data based on the user's voice characteristics, combined with the sound characteristics of the original sound at the current time point, to simulate the user's voice data, and finally form a complete song.

[0139] It should be noted that the above description is only based on karaoke as an example, but the present application is not limited to karaoke scenarios, and can also be used in other scenarios such as recording songs, and the present application does not make specific limitations on this.

[0140] As can be seen from the above technical solution, the above embodiment provides a display device 200, in response to the access instruction of the sound collector, the display device 200 receives the sound data collected by the sound collector, and stores the sound data and the corresponding collection time in a preset position; according to the input play instruction, decode the accompaniment audio data and the original sound audio data of the target song, wherein the play instruction is used to indicate the play of the target song; detect the start time mark contained in the accompaniment audio data, the start time mark is used to characterize the start of the human voice data in the original sound audio data; and after detecting the start time mark, periodically read the sound data according to the time corresponding to the start time mark and the collection time; when the sound data is read, generate the target audio data according to the accompaniment audio data and the read sound data; when the sound data is not read, generate the target audio data according to the accompaniment audio data and the replacement audio data, wherein the replacement audio data includes the original sound audio data of the time corresponding to the sound data that is not read. The display device 200 can fill the missing sound data of the user by replacing the audio data, ensure the integrity of the target audio, and solve the problem of incomplete audio in the audio recording process.

[0141] Based on the above display device 200, some embodiments of the present application further provide a method for synthesizing sound, which can be applied to the display device 200 in the above embodiment. In some embodiments, the method may include the following contents:

[0142] In response to an access instruction from a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset location;

[0143] Decoding the accompaniment audio data and the original sound audio data of the target song according to the inputted playing instruction, wherein the playing instruction is used to instruct to play the target song;

[0144] Detecting a start time mark included in the accompaniment audio data, wherein the start time mark is used to indicate the start of the vocal data in the original audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the acquisition time;

[0145] When the sound data is read, target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, target audio data is generated according to the accompaniment audio data and replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

[0146] It can be seen from the above technical solution that the above embodiment provides a sound synthesis method, which can fill in the user's missing sound data by replacing the audio data, ensure the integrity of the target audio, and solve the problem of incomplete audio during the audio recording process.

[0147] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0148] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0150] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; The controller is configured as: In response to an access instruction from a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset location; Decoding the accompaniment audio data and the original sound audio data of the target song according to the inputted playing instruction, wherein the playing instruction is used to instruct to play the target song; Detecting a start time mark included in the accompaniment audio data, wherein the start time mark is used to indicate the start of the vocal data in the original audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the acquisition time; When the sound data is read, target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, target audio data is generated according to the accompaniment audio data and replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

2. The display device according to claim 1, characterized in that Before the step of periodically reading the sound data, the controller is further configured to: At a preset time point before the time point corresponding to the start time identifier, the sound data whose collection time is before the time point corresponding to the start time identifier and stored in the preset position is cleared.

3. The display device according to claim 1, characterized in that After the step of periodically reading the sound data, the controller is further configured to: Detecting an end time mark corresponding to the start time mark; the end time mark is used to indicate the end of the human voice data in the original audio data; After the end time mark is detected, the reading of the sound data is stopped.

4. The display device according to claim 3, characterized in that The controller periodically reads the sound data and is specifically configured as follows: After identifying the start time mark, generating a sound reading instruction; In response to the sound reading instruction, the sound data is periodically read.

5. The display device according to claim 3, characterized in that The controller stops reading the sound data and is specifically configured as follows: After identifying the end time mark, generating a stop reading instruction; In response to the stop reading instruction, the reading of the sound data is stopped.

6. The display device according to claim 1, characterized in that The replacement audio data also includes analog audio data. Before the step of generating target audio data according to the accompaniment audio data and the replacement audio data, the controller is further configured as follows: Analyzing the voice characteristics of the user according to the voice data; Acquire the target original audio data at the time point corresponding to the sound data that is not read; The simulated audio data of the user at the time point is generated according to the sound feature and the target original audio data.

7. The display device according to claim 1, characterized in that The controller generates target audio data, and is specifically configured as follows: Calculating a sound energy value corresponding to the sound data; comparing the sound energy value with an energy threshold; If the sound energy value is greater than or equal to the energy threshold, determining that the sound data is valid, and generating target audio data based on the accompaniment audio data and the sound data; If the sound energy value is less than the energy threshold, the sound data is determined to be invalid, and target audio data is generated based on the accompaniment audio data and the replacement audio data.

8. The display device according to claim 1, characterized in that Before the step of decoding the accompaniment audio data and the original sound audio data of the target song, the controller is further configured to: Receive an initial audio and video stream file corresponding to the target song; The initial audio and video stream file is parsed to separate the accompaniment audio data and the original sound audio data in the initial audio and video stream file.

9. The display device according to claim 1, characterized in that: Before the step of receiving the sound data collected by the sound collector, the controller is further configured to: After the sound collector is connected, a start collection instruction is generated; In response to the start collection instruction of the sound collector, the sound collector is controlled to be in a continuous recording state.

10. A method for synthesizing sound, characterized in that: The method comprises: In response to an access instruction from a sound collector, receiving sound data collected by the sound collector, and storing the sound data and the corresponding collection time in a preset location; Decoding the accompaniment audio data and the original sound audio data of the target song according to the inputted playing instruction, wherein the playing instruction is used to instruct to play the target song; Detecting a start time mark included in the accompaniment audio data, wherein the start time mark is used to indicate the start of the vocal data in the original audio data; and after detecting the start time mark, periodically reading the sound data according to the time corresponding to the start time mark and the acquisition time; When the sound data is read, target audio data is generated according to the accompaniment audio data and the read sound data; when the sound data is not read, target audio data is generated according to the accompaniment audio data and replacement audio data, and the replacement audio data includes the original audio data corresponding to the time when the sound data is not read.

Images