A vehicle karaoke audio processing method and system, and storage medium
By identifying the singer's mouth shape and voiceprint parameters, adjusting the singing content and mixing it with the accompaniment music, the problem of singer's low voice, forgetting lyrics or singing in the in-car karaoke system is solved, improving the user experience.
Patent Information
- Application Number
- CN202110153880.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-02-04
AI Technical Summary
The karaoke system in the car is not effective when the singer has a low voice, forgets the lyrics or sings incorrectly.
By obtaining the singer's voiceprint parameters and mouth-shaped continuous frame images, the pre-trained deep learning network model is used to identify the mouth-shaped acoustic parameters, generate singing content parameters, and mix them with the accompaniment music, adjust the lyrics content to improve singing quality.
When the singer has a low voice, forgets the lyrics or sings incorrectly, the singing effect can be achieved in an ideal state and improve the user experience of karaoke in the car.
Smart Images

Figure CN114863898B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio processing, and in particular to a vehicle karaoke audio processing method and system, and a computer-readable storage medium. Background Art
[0002] Currently, in-car karaoke mainly involves mixing the singer's vocal signal with the backing vocals and then playing the resulting audio signal. However, in practice, the singer may sing softly, forget the lyrics, or make mistakes, resulting in a poor user experience for in-car karaoke. Summary of the Invention
[0003] The purpose of the present invention is to propose a vehicle karaoke audio processing method and system, and a computer-readable storage medium, so as to achieve the purpose of enabling the singer to sing well even when the singer sings softly, forgets the lyrics or sings incorrectly, thereby improving the user experience of in-car karaoke.
[0004] A first aspect of the present invention provides a vehicle karaoke audio processing method, comprising:
[0005] Obtain the singer's voiceprint parameters collected by the vehicle-mounted sound collection equipment;
[0006] Acquire continuous frame images of the singer's mouth shape captured by the vehicle-mounted camera device, and use a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain mouth shape acoustic parameters;
[0007] Obtaining corresponding singing content parameters according to the mouth shape acoustic parameters;
[0008] generating a first audio signal according to the voiceprint parameter and the singing content parameter;
[0009] Obtaining a second audio signal corresponding to the accompaniment music of the current song;
[0010] The first audio signal and the second audio signal are mixed to obtain a third audio signal, and the third audio signal is sent to an in-vehicle audio playback device so that the in-vehicle audio playback device plays the third audio signal.
[0011] Optionally, the lip shape acoustic parameters include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics;
[0012] Among them, a lip movement in multiple frames of continuous images corresponds to a lyric content, and the lip shape acoustic parameters of each lip movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics.
[0013] Optionally, obtaining corresponding singing content parameters according to the lip shape acoustic parameters includes:
[0014] Based on the lip shape credibility parameter and the lip shape and current song lyrics consistency parameter, it is determined whether the lyrics content corresponding to each lip shape movement should be retained or modified. The modification includes replacing the lyrics content with the correct lyrics corresponding to the current song, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold.
[0015] Optionally, obtaining corresponding singing content parameters according to the lip shape acoustic parameters includes:
[0016] Based on the lip shape credibility parameter and the lip shape and current song lyrics consistency parameter, it is determined whether to retain or modify a sentence of lyrics corresponding to multiple lip shape movements. The modification includes selecting a correct sentence of lyrics corresponding to the current song to replace the sentence of lyrics, or adjusting the sentence of lyrics so that the similarity between the lyrics content and the correct sentence of lyrics corresponding to the current song is greater than a preset threshold.
[0017] Optionally, the continuous frame images of the mouth shape and the voiceprint parameters are collected at the same time.
[0018] Optionally, the voiceprint parameters include fundamental frequency parameters, formant parameters, harmonic amplitude parameters, and harmonic-to-noise ratio parameters.
[0019] A second aspect of the present invention provides a vehicle karaoke audio processing system, comprising:
[0020] A voiceprint acquisition unit, used to acquire the singer's voiceprint parameters collected by the vehicle-mounted sound collection equipment;
[0021] An acoustic parameter acquisition unit acquires continuous frame images of the singer's mouth shape captured by the vehicle-mounted camera device, and uses a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain the mouth shape acoustic parameters;
[0022] A singing content acquisition unit, configured to acquire corresponding singing content parameters according to the lip shape acoustic parameters;
[0023] a first audio acquisition unit, configured to generate a first audio signal according to the voiceprint parameter and the singing content parameter;
[0024] A first audio acquisition unit is configured to acquire a second audio signal corresponding to the accompaniment music of the current song; and
[0025] The third audio acquisition unit is configured to mix the first audio signal and the second audio signal to obtain a third audio signal, and send the third audio signal to the in-vehicle audio playback device so that the in-vehicle audio playback device plays the third audio signal.
[0026] Optionally, the lip shape acoustic parameters include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics;
[0027] Among them, a lip movement in multiple frames of continuous images corresponds to a lyric content, and the lip shape acoustic parameters of each lip movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics.
[0028] Optionally, the acoustic parameter acquisition unit is specifically configured to:
[0029] Based on the lip shape credibility parameter and the lip shape and current song lyrics consistency parameter, it is determined whether the lyrics content corresponding to each lip shape movement should be retained or modified. The modification includes replacing the lyrics content with the correct lyrics corresponding to the current song, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold.
[0030] Alternatively, based on the lip shape credibility parameter and the lip shape and current song lyrics matching parameter, it is determined whether to retain or modify a sentence of lyrics corresponding to multiple lip shape movements. The modification includes replacing the sentence of lyrics with a correct sentence of lyrics corresponding to the current song, or adjusting the sentence of lyrics so that the similarity between the lyrics and the correct sentence of lyrics corresponding to the current song is greater than a preset threshold.
[0031] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the vehicle karaoke audio processing method described in the first aspect.
[0032] The various aspects of the present invention provide a vehicle karaoke audio processing method and system, and a computer-readable storage medium, which, when implemented, have at least the following beneficial effects:
[0033] An innovative method is proposed to obtain the singing content to be played by intelligently identifying the continuous frame images of the singer's mouth shape. The singing content can be a correction or adjustment of the singer's singing content. Combined with the unique voiceprint characteristics of the singer, the singing content sung by the singer under ideal conditions can be obtained. Finally, it is mixed with the accompaniment, output and played, so as to achieve the purpose of enabling the singer to sing well even if the singer sings softly, forgets the words or sings incorrectly, thereby improving the user experience of in-car karaoke.
[0034] Other features and advantages of the present invention will be set forth in the description that follows. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 The figure is a flow chart of a vehicle karaoke audio processing method according to an embodiment of the present invention.
[0037] Figure 2 Schematic diagram of the framework of a vehicle karaoke audio processing system according to another embodiment of the present invention. DETAILED DESCRIPTION
[0038] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In addition, numerous specific details are provided in the following specific examples to better illustrate the present invention. Those skilled in the art will appreciate that the present invention can be practiced without certain specific details. In some instances, means well known to those skilled in the art are not described in detail in order to highlight the main points of the present invention.
[0039] See Figure 1 An embodiment of the present invention provides a vehicle karaoke audio processing method, comprising the following steps S1 to S6:
[0040] Step S1: obtaining the singer's voiceprint parameters collected by the vehicle-mounted sound collection equipment;
[0041] Specifically, the voiceprint parameters are parameters that characterize the voice characteristics of the singer. In a specific example, the voiceprint parameters include the fundamental frequency parameter, formant parameter, harmonic amplitude parameter, and harmonic-to-noise ratio parameter of the singer's voice.
[0042] Step S2: obtaining continuous frame images of the singer's mouth shape captured by the vehicle-mounted camera device, and using a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain mouth shape acoustic parameters;
[0043] Specifically, the lip shape acoustic parameters are the voice content to be expressed by the singer's lip shape recording;
[0044] In a specific example, the lip shape acoustic parameters include but are not limited to a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics; wherein one lip shape movement in multiple frames of continuous images corresponds to one lyric content, and the lip shape acoustic parameters of each lip shape movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics;
[0045] It is understandable that it takes a certain amount of time for the singer to complete a lip movement. During this time, the car owner's camera will capture multiple frames of continuous images. Therefore, one lip movement corresponds to multiple frames of continuous images. One lip movement actually corresponds to one lyric, such as "I", "you", or "he".
[0046] The mouth shape credibility parameter indicates whether the mouth shape movement is credible. For example, if the mouth shape movement is not obvious, the credibility is relatively low. If the mouth shape movement is obvious, the credibility is relatively high. Specifically, the mouth shape credibility parameter is expressed in a value of 0 to 100%, and the higher the value, the higher the credibility.
[0047] The matching degree parameter of the lip shape and the current song lyrics can be determined by determining the lyrics of the music played according to the timestamp of the image frame corresponding to the lip shape, and then matching the two lyrics to determine the matching degree parameter of the lip shape and the current song lyrics; specifically, the matching degree parameter is represented by a value of 0 to 100%, and the higher the value, the higher the matching degree;
[0048] It should be noted that the deep learning network model is an intelligent tool that can be used for image frame recognition, and it can achieve the recognition purpose through training. It only needs to adjust the input layer and output layer of the existing deep learning network model so that the input layer of the deep learning network model corresponds to the continuous frame image of the mouth shape in this embodiment, and the output layer corresponds to the acoustic parameters of the mouth shape in this embodiment. Given a training sample, the deep learning network model can learn and train on its own to achieve the desired recognition purpose of this embodiment.
[0049] Step S3, obtaining corresponding singing content parameters according to the lip shape acoustic parameters;
[0050] In a specific example, obtaining corresponding singing content parameters according to the lip shape acoustic parameters includes:
[0051] Determining whether to retain or modify the lyrics corresponding to each lip movement based on the lip shape credibility parameter and the consistency parameter between the lip shape and the lyrics of the current song; wherein the modification includes replacing the lyrics with the correct lyrics corresponding to the current song, or adjusting the lyrics so that the similarity between the lyrics and the correct lyrics corresponding to the current song is greater than a preset threshold;
[0052] Specifically, whether to retain or modify the lyrics content corresponding to the lip movement is determined based on a comparison result of the lip shape credibility parameter and the consistency parameter with a preset threshold. For example, if the lip shape credibility parameter corresponding to the lip movement is greater than the credibility threshold and the consistency parameter is greater than the consistency threshold, then the lyric content corresponding to the lip movement is determined to be retained; otherwise, it is modified.
[0053] More specifically, the similarity calculation of lyrics can be performed using a text distance calculation method, so that the distance between two words is less than a preset threshold. The distance can be Euclidean distance, Manhattan distance, etc.
[0054] In another specific example, obtaining corresponding singing content parameters according to the lip shape acoustic parameters includes:
[0055] Determining whether to retain or modify a line of lyrics corresponding to the multiple lip movements based on the lip shape credibility parameter and the consistency parameter between the lip shape and the lyrics of the current song; wherein the modification includes replacing the line of lyrics with a correct line of lyrics corresponding to the current song, or adjusting the line of lyrics so that the similarity between the line of lyrics and the correct line of lyrics corresponding to the current song is greater than a preset threshold;
[0056] Specifically, whether to retain or modify the lyrics corresponding to the lip movement is determined based on a comparison result of the lip shape credibility parameter and the consistency parameter with a preset threshold. For example, if the lip shape credibility parameter corresponding to the lip movement is greater than the credibility threshold and the consistency parameter is greater than the consistency threshold, then the lyrics corresponding to the lip movement is determined to be retained; otherwise, it is modified.
[0057] More specifically, the similarity calculation of lyrics can adopt the text distance calculation method, so that the distance between two sentences is less than a preset threshold. The distance can be Euclidean distance, Manhattan distance, etc.
[0058] Step S4: generating a first audio signal according to the voiceprint parameters and the singing content parameters;
[0059] Specifically, the first audio signal can be understood as the singing content sung by the singer under ideal conditions, thereby improving the user experience of in-car karaoke;
[0060] Step S5: obtaining a second audio signal corresponding to the accompaniment music of the current song;
[0061] Step S6: Mix the first audio signal and the second audio signal to obtain a third audio signal, and send the third audio signal to the in-vehicle audio playback device so that the in-vehicle audio playback device plays the third audio signal.
[0062] Specifically, steps S5 to S6 are conventional karaoke mixing processing. The method of this embodiment mainly improves the acquisition of the singer's vocal audio signal, so that the singer can sing well even if the singer's voice is low, forgets the lyrics, or sings incorrectly, thereby improving the user experience of in-car karaoke.
[0063] In a specific example, the continuous frame images of the mouth shape and the voiceprint parameters are collected at the same time, so that the singer's voiceprint corresponds to the singing content.
[0064] Furthermore, when the singer only moves his mouth and does not make any sound, only the lip shape record is obtained, and the singer's voiceprint parameters cannot be obtained, which indicates that the singer may be singing too softly or may have forgotten the lyrics. At this time, the previously identified voiceprint parameters are used as the current singer's voiceprint parameters for subsequent audio signal processing.
[0065] See Figure 2 Another embodiment of the present invention provides a vehicle karaoke audio processing system, comprising:
[0066] The voiceprint acquisition unit 1 is used to acquire the voiceprint parameters of the singer collected by the vehicle-mounted sound collection equipment;
[0067] Acoustic parameter acquisition unit 2 acquires continuous frame images of the singer's mouth shape captured by the vehicle-mounted camera device, and uses a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain the mouth shape acoustic parameters;
[0068] A singing content acquisition unit 3, configured to obtain corresponding singing content parameters according to the lip shape acoustic parameters;
[0069] A first audio acquisition unit 4 is configured to generate a first audio signal according to the voiceprint parameter and the singing content parameter;
[0070] A first audio acquisition unit 5 is configured to acquire a second audio signal corresponding to the accompaniment music of the current song; and
[0071] The third audio acquisition unit 6 is configured to mix the first audio signal and the second audio signal to obtain a third audio signal, and send the third audio signal to the in-vehicle audio playback device so that the in-vehicle audio playback device plays the third audio signal.
[0072] In a specific example, the lip shape acoustic parameters include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics;
[0073] Among them, a lip movement in multiple frames of continuous images corresponds to a lyric content, and the lip shape acoustic parameters of each lip movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics.
[0074] In a specific example, the acoustic parameter acquisition unit 2 is specifically configured to:
[0075] Based on the lip shape credibility parameter and the lip shape and current song lyrics consistency parameter, it is determined whether the lyrics content corresponding to each lip shape movement should be retained or modified. The modification includes replacing the lyrics content with the correct lyrics corresponding to the current song, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold.
[0076] Alternatively, based on the lip shape credibility parameter and the lip shape and current song lyrics matching parameter, it is determined whether to retain or modify a sentence of lyrics corresponding to multiple lip shape movements. The modification includes replacing the sentence of lyrics with a correct sentence of lyrics corresponding to the current song, or adjusting the sentence of lyrics so that the similarity between the lyrics and the correct sentence of lyrics corresponding to the current song is greater than a preset threshold.
[0077] In a specific example, the continuous frame images of the mouth shape and the voiceprint parameters are collected at the same time.
[0078] In a specific example, the voiceprint parameters include fundamental frequency parameters, formant parameters, harmonic amplitude parameters, and harmonic-to-noise ratio parameters.
[0079] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the objectives of this embodiment based on actual needs.
[0080] It should be noted that the system described in the above embodiment corresponds to the method described in the above embodiment. Therefore, the undescribed part of the system described in the above embodiment can be obtained by referring to the content of the method described in the above embodiment, that is, the specific steps of the method in the above embodiment can be understood as the functions that can be achieved by the system of this embodiment, and will not be repeated here.
[0081] Furthermore, if the vehicle karaoke audio processing system described in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0082] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle karaoke audio processing method described in the above embodiment are implemented.
[0083] Specifically, the computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0084] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A vehicle karaoke audio processing method, characterized in that: include: Obtain the singer's voiceprint parameters collected by the vehicle-mounted sound collection equipment; Acquire continuous frame images of the singer's mouth shape captured by the vehicle-mounted camera device, and use a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain the mouth shape acoustic parameters; Obtaining corresponding singing content parameters according to the lip shape acoustic parameters, including: determining to retain or modify the lyrics content corresponding to each lip shape action according to the lip shape acoustic parameters, wherein the modification includes selecting the correct lyrics corresponding to the current song to replace the lyrics content, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold; or, determining to retain or modify the lyrics content corresponding to a sentence of multiple lip shape actions according to the lip shape acoustic parameters, wherein the modification includes selecting the correct lyrics corresponding to the current song to replace the lyrics content, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold; Generate a first audio signal according to the voiceprint parameter and the singing content parameter; Obtaining a second audio signal corresponding to the accompaniment music of the current song; The first audio signal and the second audio signal are mixed to obtain a third audio signal, and the third audio signal is sent to an in-vehicle audio playback device so that the in-vehicle audio playback device plays the third audio signal.
2. The vehicle karaoke audio processing method according to claim 1, characterized in that: The lip shape acoustic parameters include a lip shape credibility parameter and a lip shape matching parameter with the current song lyrics; Among them, one lip movement in multiple frames of continuous images corresponds to one lyric content, and the lip shape acoustic parameters of each lip movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics.
3. The vehicle karaoke audio processing method according to claim 2, characterized in that: The mouth shape continuous frame images and the voiceprint parameters are collected at the same time.
4. The vehicle karaoke audio processing method according to claim 2, characterized in that: The voiceprint parameters include fundamental frequency parameters, formant parameters, harmonic amplitude parameters, and harmonic-to-noise ratio parameters.
5. A vehicle karaoke audio processing system, characterized in that: include: A voiceprint acquisition unit, used to acquire the voiceprint parameters of the singer collected by the vehicle-mounted sound collection equipment; An acoustic parameter acquisition unit acquires continuous frame images of the singer's mouth shape collected by the vehicle-mounted camera device, and uses a pre-trained deep learning network model to recognize the continuous frame images of the mouth shape to obtain the acoustic parameters of the mouth shape; A singing content acquisition unit, used for obtaining corresponding singing content parameters according to the lip shape acoustic parameters, including: determining to retain or modify the lyrics content corresponding to each lip shape action according to the lip shape acoustic parameters, wherein the modification includes selecting the correct lyrics corresponding to the current song to replace the lyrics content, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold; or, determining to retain or modify the lyrics content corresponding to a plurality of lip shape actions according to the lip shape acoustic parameters, wherein the modification includes selecting the correct lyrics corresponding to the current song to replace the lyrics content, or adjusting the lyrics content so that the similarity between the lyrics content and the correct lyrics corresponding to the current song is greater than a preset threshold; A first audio acquisition unit, configured to generate a first audio signal according to the voiceprint parameter and the singing content parameter; A second audio acquisition unit is used to acquire a second audio signal corresponding to the accompaniment music of the current song; and The third audio acquisition unit is used to obtain a third audio signal after mixing the first audio signal and the second audio signal, and send the third audio signal to the vehicle audio playback device so that the vehicle audio playback device plays the third audio signal.
6. The vehicle karaoke audio processing system according to claim 5, characterized in that: The lip shape acoustic parameters include a lip shape credibility parameter and a lip shape matching parameter with the current song lyrics; Among them, one lip movement in multiple frames of continuous images corresponds to one lyric content, and the lip shape acoustic parameters of each lip movement include a lip shape credibility parameter and a lip shape consistency parameter with the current song lyrics.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the vehicle karaoke audio processing method according to any one of claims 1 to 4 are implemented.
